Zero sample image classification system and method

Abstract

The invention provides a zero sample image classification system and method, and the system comprises a teacher model and a student model which are the same in architecture, and each of the teacher model and the student model comprises a visual encoder, a semantic encoder, a visual-semantic alignment module and a zero sample classification module; a visual encoder of the teacher model is constructed in a manner of cascading a CNN (Convolutional Neural Network) and a Transform; selecting a BERT network as a semantic encoder of the teacher model; the visual-semantic alignment module is used for constructing a teacher model and a student model based on a cross-modal attention mechanism; a full connection layer is adopted as a zero sample classification module of the teacher model and the student model; a visual encoder of the student model is constructed by adopting a lightweight EnviroNet network; and a semantic encoder of the student model is constructed by adopting a Core-BER network. The method has the advantages that under zero sample setting, the high unseen category classification performance can be kept, meanwhile, the model complexity is reduced, and deployment on actual equipment is better facilitated.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a zero-sample image classification system and method. Background Art

[0002] Zero-shot learning transfers the auxiliary information learned from the visible class to the unseen class by learning the visible class images and their auxiliary information, such as attribute information, text description information, etc., and establishes a coupling relationship between the visible class images and the unseen class images, thereby realizing the recognition of unseen categories or concepts under zero-shot conditions. Currently, zero-shot learning has been widely used in terrain scene classification, animal and plant category classification, and can also be used for environmental monitoring, urban planning and natural disaster management. Zero-shot image classification is one of the main application tasks of zero-shot learning in the field of computer vision. Its goal is to classify when the training set and the test set do not contain each other, and is committed to identifying categories that have never been seen in the training stage. It can provide strong support for higher-level applications such as environmental monitoring, urban planning and natural disaster management.

[0003] In actual application scenarios, the dataset used for training cannot contain all categories, so the sample data used to train the classifier is usually a small part of the open set. In the above case, the training process of the classifier will rely heavily on the annotation information of the training set samples, resulting in the classifier only being able to recognize the visible categories that have appeared in the training set, while the classification accuracy of the unseen categories is poor. However, the unseen categories in actual application scenarios are usually more valuable and require high-quality recognition rates. For example, in environmental monitoring, new land cover types may appear due to natural or human changes, and the zero-shot learning classification model can quickly adapt and accurately classify these new categories. Similarly, in disaster management, rapid response and decision-making are crucial, and the model can help identify previously unknown features or objects that are crucial for effective disaster response. Therefore, it is of great significance and value to design a zero-shot image classification method that can identify unseen categories.

[0004] Traditional methods for zero-shot image classification for unseen category recognition directly use pre-trained visual feature extraction networks to extract visual information of training sample data, and design appropriate text encoders to extract corresponding auxiliary information of training sample data, and then directly linearly map the two features to achieve that the two modal features are in the same space, so as to use the nearest neighbor and other metric methods for zero-shot image classification. This method belongs to the embedding model-based method, which constructs the embedding space through the mapping relationship between visual and semantic features, learns the visible class image features and their auxiliary information in the training stage, obtains the relationship between the image features and the corresponding auxiliary information, and then uses this relationship to classify the unseen category images. However, visual and semantic feature extractors are usually complex, which is not conducive to deployment on actual devices. Moreover, only a simple linear embedding mapping is used between the visual space and the semantic space, and there is usually an inconsistency in the class structure between the visual and semantic spaces, that is, the semantic gap problem between modalities. Summary of the invention

[0005] The purpose of this application is to overcome the defects of the prior art that it is not conducive to actual device deployment and has a semantic gap between modalities.

[0006] In order to achieve the above objectives, the present application proposes a zero-sample image classification system, the system comprising a teacher model and a student model;

[0007] The teacher model and the student model have the same architecture, both including: a visual encoder, a semantic encoder, a visual-semantic alignment module and a zero-shot classification module;

[0008] The visual encoder of the teacher model is constructed in a cascaded manner of CNN and Transformer. The BERT network is selected as the semantic encoder of the teacher model. The visual-semantic alignment module of the teacher model is constructed based on the cross-modal attention mechanism. The fully connected layer is used as the zero-shot classification module of the teacher model.

[0009] The visual encoder of the student model uses the ResNet network as the backbone framework, and constructs a residual context Transformer module consisting of two 1×1 convolutions and a Cotlayer; a lightweight EnviroNet network consisting of a mixture of traditional convolution and self-attention mechanisms is established; the semantic encoder of the student model is a lightweight Core-BERT network; the construction of the visual-semantic alignment module and the zero-shot classification module in the student model is the same as that of the teacher model;

[0010] Both the teacher model and the student model are trained in an end-to-end manner.

[0011] As an improvement to the above system, the loss function L of the teacher model and the student model is:

[0012] L=Lcc +λ·L ar

[0013] Among them, L cc is the class semantic vector loss function based on cross entropy loss; L ar is the class regression loss function based on the L2 paradigm, and λ is the weight parameter of the class regression loss function.

[0014] As an improvement of the above system, the class semantic vector loss function L based on cross entropy loss cc for:

[0015]

[0016] Among them, x is the visual feature representation of the image; y is the text information representation of the image; represents the category to which the image belongs; C S is the total number of categories of visible class images in the dataset; Indicates the category to which the image belongs is represented as The text information representation.

[0017] As an improvement of the above system, the proposed method constructs a class regression loss function L based on the L2 paradigm. ar for:

[0018]

[0019] Among them, x is the visual feature representation of the image, y is the text information representation of the image; ||·||2 is the L2 paradigm.

[0020] As an improvement of the above system, the overall loss function L of the system is zsl for:

[0021] L zsl =L stu +λ kd ·L kd

[0022] Among them, L stu is the overall loss function calculated in the student model; L kd is the knowledge distillation loss function based on KL divergence; kd is the weight parameter of the knowledge distillation loss function.

[0023] As an improvement to the above system, the knowledge distillation loss function L kd for:

[0024] L kd = l G (f t ,f s )

[0025] Among them, l G (·) is the knowledge distillation loss function; f t is the predicted log-odds of the teacher model, f s is the predicted log-odds of the Student model.

[0026] As an improvement of the above system, the visual-semantic alignment module of the teacher model and the student model is composed of a cascade of layer normalization layers, cross-modal attention layers, self-attention layers and feedforward layers.

[0027] As an improvement of the above system, the training process of the system includes training of a teacher model, training of a student model, and overall training of the system.

[0028] The present application also provides a zero-sample image classification method, which is implemented based on the above system and includes:

[0029] The unseen category test samples are input into the trained zero-shot image classification system, and the unseen category image classification results are output.

[0030] Compared with the prior art, the advantages of this application are:

[0031] 1. The present invention constructs a lightweight zero-shot image classification network, namely, a student model, which includes an independent visual encoder, an independent semantic encoder, a cross-modal feature fusion module and a classifier. The lightweight feature extraction network, feature alignment network and classifier are integrated into a unified framework, so that the input data can directly obtain the output result through the trained zero-shot image classification network, which is conducive to deployment on actual devices, avoids manual preprocessing and subsequent processing, and improves the zero-shot classification speed of images.

[0032] 2. The visual encoder EnviroNet designed by the present invention, which is based on a mixture of traditional convolution and self-attention mechanisms, fully integrates the advantages of the two network structures compared to traditional visual encoders. It can simultaneously capture local features and contextual information in the image, effectively improving the discriminability of the extracted features.

[0033] 3. The zero-shot image classification framework based on knowledge distillation designed by the present invention transfers the information between visual and semantic features obtained in the complex teacher model to the lightweight student model, so that the overall classification framework can effectively improve the zero-shot classification accuracy with less resource consumption and storage cost.

[0034] In summary, compared with the prior art, the present invention can better utilize the shared information between images and auxiliary information. Under the setting of zero sample, it can not only maintain the overall classification accuracy at a high level, but also effectively reduce the complexity of the model, which is more conducive to deployment on actual equipment and can provide better services for higher-level applications such as ground object monitoring and change detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Shown is a flowchart of a zero-sample image classification method;

[0036] Figure 2 Shown is a block diagram of the zero-shot image classification system model;

[0037] Figure 3 Shown is a zero-sample Mars scene image independently constructed by the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0039] A zero-shot image classification system and method, based on knowledge distillation technology, fully utilizes the visual and semantic interaction information extracted by a complex teacher model, enhances the learning process of the lightweight student model for visual and semantic features, and ensures the zero-shot classification effect of the student model at the lowest possible resource cost.

[0040] In the zero-shot image classification system, both the teacher and student models adopt end-to-end training models, that is, visual semantic feature extraction, alignment and zero-shot classification are combined into a unified process. First, the two models independently use visual and semantic encoders to mine image features and text information. Subsequently, the extracted visual and semantic features are aligned with multimodal information interaction through the cross-modal attention layer. The difference between the two models lies mainly in the structural complexity of the feature extractor. The visual and semantic feature extractors used in the teacher model are more complex (deeper layers and larger parameters), while those of the student model are more lightweight. The training strategy of the zero-shot image classification method based on knowledge distillation is mainly divided into two stages. In the first stage, the best batch prediction results of the teacher model for unseen category images are obtained through end-to-end training, which are usually expressed as soft labels. In the second stage, this set of soft labels is used as the input to guide the student model prediction, and the knowledge distillation loss is constructed to make the visual-semantic features obtained by the student model more discriminative, obtain more accurate feature expression effects, and further improve the classification accuracy of the student model at the lowest possible resource cost.

[0041] The zero-shot image classification system includes a teacher model and a student model. During training, both the teacher and student models are trained end-to-end. Through end-to-end training, the teacher model obtains the best batch prediction results (soft labels) for unseen category images, and the soft labels are used as input to guide the student model prediction to train the student model. During testing, the image classification results can be obtained by simply inputting the image information and semantic information into the student model.

[0042] The architecture of the teacher model and the student model is the same, both including: visual encoder, semantic encoder, visual-semantic alignment module and zero-shot classification module.

[0043] The visual encoder takes in image information and outputs the visual feature vector of the image. The semantic encoder takes in text information corresponding to the image and outputs the semantic feature vector corresponding to the image. The visual-semantic alignment module takes in the visual feature vector of the image and the semantic feature vector of the corresponding image and outputs the aligned visual-semantic features. The zero-shot classification module takes in the aligned visual-semantic features and outputs the classification result of the image.

[0044] The visual encoder of the teacher model is constructed in a cascaded manner of CNN and Transformer. The BERT network is selected as the semantic encoder of the teacher model. The visual-semantic alignment module of the teacher model is constructed based on the cross-modal attention mechanism. The zero-shot classification module of the teacher model is a fully connected layer.

[0045] The construction process of the student model's visual encoder EnviroNet includes:

[0046] Among the many pre-trained models of convolutional neural networks, the ResNet network is selected as the backbone framework of the visual encoder; the context transformer (Cotlayer) is introduced to form a residual context transformer module consisting of two 1×1 convolutions and a Cotlayer; a lightweight visual encoder EnviroNet is established, which is a mixture of traditional convolution and self-attention mechanism; the parameters of the initialization visual encoder include the weight parameter w of EnviroNet v , bias parameter b v , initialization matrix in the query, key and value matrix of the Cotlayer module and the number of heads k of the self-attention layer v And other parameters.

[0047] The construction process of the semantic encoder Core-BERT of the student model includes:

[0048] The BERT pre-trained language model is used to initialize the semantic encoder. The number of layers of the Transformer is a hyperparameter, which is an important parameter of the lightweight semantic encoder Core-BERT. The parameters for initializing the semantic encoder include the number of layers of the Transformer of Core-BERT, as well as the initialization matrices in the query, key, and value matrices. The number of heads k in the self-attention layer s And other parameters.

[0049] The construction process of the visual-semantic alignment module of the student model includes:

[0050] A cross-modal attention layer (visual-semantic alignment module) is constructed based on the cross-modal attention mechanism, where the query, key, and value are set as the semantic vector, visual vector, and category label vector respectively; the parameters of the cross-modal attention layer are initialized including the initialization matrix in the query, key, and value matrices And the number of layers k of the cross-modal feature fusion module c And other parameters.

[0051] The zero-shot classification module of the student model is a fully connected layer.

[0052] The zero-shot image classification system adopts an end-to-end training method and realizes the transfer of teacher model knowledge to the student model through knowledge distillation loss. The training process of this system consists of three parts: end-to-end training of the teacher model, end-to-end training of the student model, and training of the overall zero-shot image classification framework (distillation model).

[0053] The training process of the teacher model and the student model is the same, both of which are performed in an end-to-end manner, including:

[0054] Step 1: Construct the overall loss function L of the network model zsl =L cc +λ·L ar , where L cc is the class semantic vector loss function based on cross entropy loss, L ar is the class regression loss function based on the L2 paradigm, λ is the weight parameter of the class regression loss function;

[0055] Step 2: Initialize the parameters of the teacher model including the weight parameter w in the visual feature extraction module v , bias parameter b v , the initialization matrix in the query, key, and value matrices of the visual transformer Transformer and the number of heads k of the self-attention layer v Parameters such as; initialization matrix in the query, key and value matrix of the BERT network in the semantic feature extraction module and the number of heads k of the self-attention layer s The parameters of the cross-modal attention layer include the initialization matrix in the query, key and value matrices. And the number of layers k of the cross-modal feature fusion module c And other parameters;

[0056] Step 3: Initialize the parameters of the student model including the weight parameter w in the visual encoder v , bias parameter b v , initialization matrix in Cotlayer's query, key and value matrices and the number of heads k of the self-attention layer v Parameters such as the number of layers of the Transformer of the Core-BERT network in the semantic feature extraction module, the initialization matrix in the query, key and value matrices and the number of heads k of the self-attention layer s The parameters of the cross-modal attention layer include the initialization matrix in the query, key and value matrices. And the number of layers k of the cross-modal feature fusion module c And other parameters;

[0057] Step 4: Input the visible category training samples and category semantic information into the teacher / student model, use the gradient descent method to cyclically update the parameters in the teacher / student model to reduce the gradient value of the total loss function until the maximum number of iterations T is reached, and a trained teacher / student model is obtained.

[0058] The training process of the overall zero-shot image classification framework (distillation model) includes:

[0059] Step 1: Construct the overall loss function L of the network model zsl =L stu +λ kd ·L kd , where L stu is the overall loss function calculated in the student model, L kd is the knowledge distillation loss function based on KL divergence, λ kd is the weight parameter of the knowledge distillation loss function;

[0060] Step 2: Initialize the parameters of the zero-shot image classification framework (distillation model) including the weight parameters w in the visual encoder v , bias parameter b v , initialization matrix in Cotlayer's query, key and value matrices and the number of heads k of the self-attention layer v Parameters such as the number of layers of the Transformer of the Core-BERT network in the semantic feature extraction module, the initialization matrix in the query, key and value matrices and the number of heads k of the self-attention layer s The parameters of the cross-modal attention layer include the initialization matrix in the query, key and value matrices. And the number of layers k of the cross-modal feature fusion module c And other parameters;

[0061] Step 3: Input the visible category training samples and category semantic information into the zero-shot image classification framework (distillation model), and use the gradient descent method to cyclically update the parameters in the zero-shot image classification network to reduce the gradient value of the total loss function until the maximum number of iterations T is reached, and the trained zero-shot image classification framework (distillation model) is obtained.

[0062] After training is completed, the unseen category test samples are input into the trained zero-shot image classification network model to output the unseen category image classification results.

[0063] This example uses the zero-sample Mars scene classification standard dataset ZS-Mars as an example to classify it. The images in this dataset and the corresponding true value labels are as follows Figure 3 As shown, it contains 10 types of scenes.

[0064] Reference Figure 1 The specific implementation steps of this example are as follows:

[0065] Step 1: Randomly divide the training and test samples.

[0066] 1.1) For the ZS-Mars dataset, the training (visible categories) and test (unseen categories) samples are divided into 6 / 4, 7 / 3, and 8 / 2 ratios, i.e., 6 visible categories / 4 unseen categories, 7 visible categories / 3 unseen categories, and 8 visible categories / 2 unseen categories;

[0067] 1.2) Encode the label of each training sample into a vector y in one-hot form i , the label of each test sample is encoded in one-hot form as a vector q i .

[0068] Step 2: Select MFINet as the teacher model and build its visual and semantic feature extractors

[0069] 2.1) Build a visual encoder based on CNN+Transformer:

[0070] By cascading CNN and Transformer, a visual encoder of ResNet18 and DeiT is constructed, and the input image size is 224×224×3;

[0071] from Figure 3In the samples shown in the figure, the training set and test set samples and their corresponding labels are obtained according to the division in step 1. The training set is represented by X = {x1,...,x train}, the label is Y = {y1,...,y train}, x i For any training sample, for different visible / invisible category ratios, the subscript train corresponds to the product of the number of visible categories and the number of scenes in each category, x i The corresponding label y i ∈{1,2,...,6}; the test set is denoted as P = {p1,...,p test}, the label is Q = {q1,...,q test}, p i For any test sample, for different visible / invisible category ratios, the subscript test corresponds to the product of the number of invisible categories and the number of scenes in each category, p i The corresponding label q i ∈{1,2,3,4};

[0072] 2.2) Build a semantic encoder based on a pre-trained language model:

[0073] From the pre-trained language model, select the BERT network as the semantic encoder;

[0074] from Figure 3 In the sample labels shown, the corresponding labels of the training set and the test set are obtained according to the division in step 1. Among them, for different visible / invisible category ratios, the number of training samples corresponds to the product of the number of visible categories and the number of scenes in each category, and the test samples correspond to the product of the number of invisible categories and the number of scenes in each category. The remaining sample labels are used as the test set;

[0075] Step 3: Build the feature extractor of the student model:

[0076] 3.1) Construct EnviroNet, a visual encoder based on a mixture of traditional convolution and self-attention mechanisms:

[0077] A lightweight visual encoder EnviroNet is constructed by mixing and cascading traditional convolution and self-attention mechanisms. The input image size is 224×224×3.

[0078] from Figure 3 In the samples shown in the figure, the training set and test set samples and their corresponding labels are obtained according to the division in step 1. The training set is represented by X = {x1,...,x train}, the label is Y = {y1,...,y train}, x iFor any training sample, for different visible / invisible category ratios, the subscript train corresponds to the product of the number of visible categories and the number of scenes in each category, x i The corresponding label y i ∈{1,2,...,6}; the test set is denoted as P = {p1,...,p test}, the label is Q = {q1,...,q test}, p i For any test sample, for different visible / invisible category ratios, the subscript test corresponds to the product of the number of invisible categories and the number of scenes in each category, p i The corresponding label q i ∈{1,2,3,4};

[0079] 3.2) Build a semantic encoder based on the pre-trained language model:

[0080] From the pre-trained language model, the BERT network is selected as the backbone network of the semantic encoder, and the number of series layers of the Transformer in the network is redesigned;

[0081] from Figure 3 In the sample labels shown, the corresponding labels of the training set and the test set are obtained according to the division in step 1. Among them, for different visible / invisible category ratios, the number of training samples corresponds to the product of the number of visible categories and the number of scenes in each category, and the test samples correspond to the product of the number of invisible categories and the number of scenes in each category. The remaining sample labels are used as the test set;

[0082] Step 4: Construct a cross-modal attention based visual-semantic alignment encoder.

[0083] Reference Figure 2 , the specific implementation of this step is as follows:

[0084] 4.1) Establish mapping module φ i :Set up three mapping modules φ 1～3 , are composed of linear matrices, mapping the input sequences q, k, and v to query matrix Q, key matrix K, and value matrix V, respectively. The specific formulas are as follows:

[0085] Q=φ1·q, K=φ2·k, V=φ3·v

[0086] Among them, the input sequences q and v are set as visual features; the input sequence k is set as semantic features.

[0087] 4.2) Set up a cross-modal attention module, which is composed of a cascade of layer normalization layer, cross-modal attention layer, self-attention layer and feedforward layer. The number of input vectors of the layer normalization layer is the feature dimension of the output of the previous module. The size of each vector is (batch size, 1, 224, 224), and the number of output vectors is consistent with the number of input vectors.

[0088] Step 5: Build a zero-shot image classification model based on cross-modal attention.

[0089] 5.1) The above independent visual encoder and independent semantic encoder are cascaded with the visual-semantic alignment encoder to form a visual-semantic feature extraction alignment encoder;

[0090] 5.2) Construct classifier g η . g η Includes 1 fully connected layer, transforms the size of the matrix to extract the above visual semantic features and align the encoder output features to convert them into probability vectors belonging to each category Used to determine the category to which the sample belongs.

[0091] Step 6: Construct the loss function of the zero-shot image classification model based on knowledge distillation.

[0092] 6.1) Construct a loss function for the zero-shot image classification model based on cross-modal attention.

[0093] Construct a class regression loss function L based on the L2 paradigm ar , which is used to constrain the image representation to ensure that it is accurately mapped to the corresponding text embedding. The specific formula is as follows:

[0094]

[0095] Among them, x represents the visual feature representation of the image, and y represents the text information representation of the image;

[0096] Constructing the cross entropy loss function L cc , which is used to force the image to have the highest compatibility score with its corresponding class semantic vector. The specific formula is as follows:

[0097]

[0098] in, Indicates the category to which the image belongs, C S Represents the total number of categories of visible class images in the dataset, Indicates the category to which the image belongs is represented as Text information representation of ;

[0099] 6.2) Constructing the knowledge distillation loss function L based on KL divergence kd, which is used to guide the student model to learn the knowledge transferred from the teacher model and improve the classification accuracy of the student model. The specific formula is as follows:

[0100] L kd = l G (f t ,f s )

[0101] Among them, l G (·) is the knowledge distillation loss function, f t are the predicted logits (log odds) of the teacher model, and f s are the predicted logits of the student model.

[0102] Step 7: Use the training samples and the gradient descent method to train the parameters of the zero-shot image classification model.

[0103] 7.1) Set the initial value of the learning rate α, the maximum number of iterations T, and the number of batches B;

[0104] 7.2) Initialize the parameters of the zero-shot image classification network model including the weight parameter w in the visual feature extraction module v , bias parameter b v , initialization matrix in Cotlayer's query, key and value matrices and the number of heads k of the self-attention layer v Parameters such as; initialization matrix in the query, key and value matrix of the Core-BERT network in the semantic feature extraction module and the number of heads k of the self-attention layer s The parameters of the cross-modal attention layer include the initialization matrix in the query, key and value matrices. And the number of layers k of the cross-modal feature fusion module c Parameters such as , and the parameter η in the classifier;

[0105] 7.3) Divide the training samples into batches according to the batch size, and input them into the zero-shot image classification network in turn for back propagation to update the total loss function The gradient of The network training parameters are updated in the descending direction through forward propagation, and the above calculations are performed for each batch of input networks until the entire training set sample input network is traversed, completing one training;

[0106] 7.4) Repeat the process in 7.3) until the maximum number of iterations T is reached to obtain a trained zero-shot classification model.

[0107] Step 8: Use the trained zero-shot image classification model to classify the test samples.

[0108] Input each test sample into the trained zero-shot image classification model to obtain the probability p(y|x) of the label to which each test sample belongs:

[0109]

[0110] Among them, softmax is the normalization function, L zsl is the total loss function, x is the test sample, y c is the label corresponding to the c-th test sample, z c is the semantic feature corresponding to the c-th test sample, C is the total number of test samples, w v ,b v are the trainable parameters of CNN in the visual feature extraction module, k v is the trainable parameter of Transformer in the visual feature extraction module, is the trainable parameter of the BERT network in the pre-trained language model, and η is the trainable parameter of the classifier.

[0111] The effect of the present invention can be further illustrated by the following simulation results.

[0112] Simulation conditions:

[0113] Test data: ZS-Mars data is used, and word vectors are used for semantic information. Images are divided into three visible / invisible ratios for testing. That is, the visible / invisible ratio is 6 / 4, and 6 categories are selected as visible category images as training sets, and the remaining 4 categories are unseen category images as test sets; the visible / invisible ratio is 7 / 3, and 7 categories are selected as visible category images as training sets, and the remaining 3 categories are unseen category images as test sets; the visible / invisible ratio is 8 / 2, and 8 categories are selected as visible category images as training sets, and the remaining 2 categories are unseen category images as test sets. The specific visible / invisible category division is shown in Table 1.

[0114] Test environment: Linux system, NVIDIA GeForce GTX 2080Ti GPU, Pytorch deep learning framework.

[0115] Table 1 Division of visible / invisible categories

[0116]

[0117] Simulation content:

[0118] Simulation 1. Under the above conditions, the existing zero-shot image scene classification method MFINet based on the embedding model is used to classify the ZS-Mars data. The results are shown in Table 2.

[0119] Simulation 2: Under the above conditions, the ZS-Mars data is classified using the method of the present invention. The results are shown in Table 2.

[0120] By comparing Table 2, it can be found that the method proposed in the present invention achieves higher classification accuracy with lower computational complexity and achieves better classification performance.

[0121] Evaluation indicators:

[0122] The above two methods were tested 10 times using the same random numbers, and the overall classification accuracy OA was calculated. The test results are shown in Table 2.

[0123] The indicator calculation formula is as follows:

[0124]

[0125] Among them, TP is the number of samples correctly classified into this category, FN is the number of samples of this category that are incorrectly classified as other categories, TN is the number of non-class samples classified into other categories, and FP is the number of other samples that are incorrectly classified as samples of this category.

[0126] Table 2 Simulation results of the present invention and the comparative method

[0127]

[0128] It can be seen from Table 2 that, compared with the currently more advanced MFINet method, the present invention can achieve higher classification accuracy with less resource consumption under three visible / invisible ratios.

[0129] The present application may also provide a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together through a bus system. It is understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0130] The user interface may include a display, a keyboard or a pointing device, such as a mouse, a trackball, a touch pad or a touch screen.

[0131] It is understood that the memory in the embodiments disclosed in the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0132] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and applications.

[0133] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application.

[0134] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and is used to:

[0135] Execute the steps of the above method.

[0136] The above method can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The above-disclosed methods, steps and logic block diagrams can be implemented or executed. The general processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the above-disclosed method can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0137] It is understood that the embodiments described in the present application can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0138] For software implementation, the technology of the present application can be implemented by executing the functional modules (such as procedures, functions, etc.) of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0139] The present application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method embodiment can be implemented.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A zero-shot image classification system, comprising a teacher model and a student model, characterized in that: The teacher model and the student model have the same architecture, both including: a visual encoder, a semantic encoder, a visual-semantic alignment module and a zero-shot classification module; The visual encoder of the teacher model is constructed in a cascaded manner of CNN and Transformer. The BERT network is selected as the semantic encoder of the teacher model. The visual-semantic alignment module of the teacher model is constructed based on the cross-modal attention mechanism. The fully connected layer is used as the zero-shot classification module of the teacher model. The visual encoder of the student model uses the ResNet network as the backbone framework, and constructs a residual context Transformer module consisting of two 1×1 convolutions and a Cotlayer; a lightweight EnviroNet network consisting of a mixture of traditional convolution and self-attention mechanisms is established; the semantic encoder of the student model is a lightweight Core-BERT network; the construction of the visual-semantic alignment module and the zero-shot classification module in the student model is the same as that of the teacher model; Both the teacher model and the student model are trained in an end-to-end manner.

2. The zero-shot image classification system according to claim 1, characterized in that: The loss function L of the teacher model and the student model is: L=L cc +λ·L ar Among them, L cc is the class semantic vector loss function based on cross entropy loss; L ar is the class regression loss function based on the L2 paradigm, and λ is the weight parameter of the class regression loss function.

3. The zero-shot image classification system according to claim 2, characterized in that: The class semantic vector loss function L based on cross entropy loss cc for: Among them, x is the visual feature representation of the image; y is the text information representation of the image; represents the category to which the image belongs; C S is the total number of categories of visible class images in the dataset; Indicates the category to which the image belongs is represented as The text information representation.

4. The zero-shot image classification system according to claim 2, characterized in that: The construction of the class regression loss function L based on the L2 paradigm ar for: Among them, x is the visual feature representation of the image, y is the text information representation of the image; ||·||2 is the L2 paradigm.

5. The zero-shot image classification system according to claim 1, characterized in that: The overall loss function of the system is L zsl for: L zsl =L stu +λ kd ·L kd Among them, L stu is the overall loss function calculated in the student model; L kd is the knowledge distillation loss function based on KL divergence; kd is the weight parameter of the knowledge distillation loss function.

6. The zero-shot image classification system according to claim 5, characterized in that: The knowledge distillation loss function L kd for: L kd =l G (in t ,f s ) Among them, l G (·) is the knowledge distillation loss function; f t is the predicted log-odds of the teacher model, f s is the predicted log-odds of the Student model.

7. The zero-shot image classification system according to claim 1, characterized in that: The visual-semantic alignment module of the teacher model and the student model is composed of a cascade of layer normalization layers, cross-modal attention layers, self-attention layers, and feedforward layers.

8. The zero-shot image classification system according to claim 1, characterized in that: The training process of the system includes training of the teacher model, training of the student model, and overall training of the system.

9. A zero-sample image classification method, implemented based on the system of any one of claims 1-8, characterized in that: The method comprises: The unseen category test samples are input into the trained zero-shot image classification system, and the unseen category image classification results are output.

Images