Zero sample image classification system and method

Abstract

The invention provides a zero sample image classification system and method, and the system comprises a visual feature extraction module which is used for obtaining image visual feature data according to an input unseen category image; the semantic information extraction module is used for obtaining image semantic information data according to the input text information corresponding to the unseen category image; the cross-modal feature fusion module is used for obtaining aligned visual-semantic feature data according to the input image visual feature data and image semantic information data; and the classification module is used for obtaining a classification result of the image category according to the input aligned visual-semantic feature data. Compared with the prior art, the method has the advantages that image information, auxiliary information and shared information between the image information and the auxiliary information can be better utilized, and under the setting of zero samples, the classification precision of visible categories can be kept at a high level, and the classification performance of unvisible categories can be effectively improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a zero-sample image classification system and method, which can be used for environmental monitoring, urban planning and natural disaster management. Background Art

[0002] Zero-shot learning transfers the auxiliary information learned from the visible class to the unseen class by learning the visible class images and their auxiliary information, such as attribute information, text description information, etc., and establishes a coupling relationship between the visible class images and the unseen class images, thereby realizing the recognition of unseen categories or concepts under zero-shot conditions. Currently, zero-shot learning has been widely used in terrain scene classification, animal and plant category classification, and can also be used for environmental monitoring, urban planning and natural disaster management. Zero-shot image classification is one of the main application tasks of zero-shot learning in the field of computer vision. Its goal is to classify when the training set and the test set do not contain each other, and is committed to identifying categories that have never been seen in the training stage. It can provide strong support for higher-level applications such as environmental monitoring, urban planning and natural disaster management.

[0003] In actual application scenarios, the dataset used for training cannot contain all categories, so the sample data used to train the classifier is usually a small part of the open set. In the above case, the training process of the classifier will rely heavily on the annotation information of the training set samples, resulting in the classifier only being able to recognize the visible categories that have appeared in the training set, while the classification accuracy of the unseen categories is poor. However, the unseen categories in actual application scenarios are usually more valuable and require high-quality recognition rates. For example, in environmental monitoring, new land cover types may appear due to natural or human changes, and the zero-shot learning classification model can quickly adapt and accurately classify these new categories. Similarly, in disaster management, rapid response and decision-making are crucial, and the model can help identify previously unknown features or objects that are crucial for effective disaster response. Therefore, it is of great significance and value to design a zero-shot image classification method that can identify unseen categories.

[0004] Traditional methods for zero-shot image classification for unseen category recognition directly use pre-trained visual feature extraction networks to extract visual information of training sample data, and design appropriate text encoders to extract corresponding auxiliary information of training sample data for feature extraction, and then directly linearly map the two features to achieve that the two-modal features are in the same space, so as to use nearest neighbor and other metric methods for zero-shot image classification. This method belongs to the embedding model-based method, which constructs the embedding space through the mapping relationship between visual-semantic features, learns visible class image features and their auxiliary information during the training phase, obtains the relationship between image features and corresponding auxiliary information, and then uses this relationship to classify unseen category images. However, there is only a simple linear embedding mapping between the visual space and the semantic space, and there is usually an inconsistency in the class structure between the visual and semantic spaces, that is, the semantic gap problem between modalities.

[0005] Yansheng Li et al. proposed a zero-shot remote sensing scene classification method based on locality-preservation deep cross-modal embedding network (LPDCMENs) in their article "Learning Deep Cross-Modal Embedding Networks for Zero-Shot Remote Sensing Image Scene Classification" (IEEE Transactions on Geoscience & Remote Sensing, 2021, 59: 10590-10603) to overcome the class structure inconsistency problem between visual images and auxiliary semantics in zero-shot image classification. This method maps visual features and semantic features to latent space separately, in which remote sensing image scenes from the same category will be close to each other (locality preservation property) and grouped around the semantic representation of a given category (cross-modal matching property), and a set of interpretable constraints are designed to optimize LPDCMEN. Nevertheless, the method still has the defect of lacking information interaction between visual features and semantic features, failing to fully mine the useful information of visual semantic features, which restricts the improvement of zero-shot image classification performance. Summary of the invention

[0006] The purpose of this application is to overcome the defects of the prior art that lack of information interaction between visual features and semantic features, fail to fully mine the useful information of visual semantic features, and restrict the performance of zero-sample image classification.

[0007] In order to achieve the above objectives, the present application proposes a zero-sample image classification system, the system comprising:

[0008] The visual feature extraction module consists of a convolutional neural network and a cascade of transformers, which is used to obtain image visual feature data based on the input of unseen category images;

[0009] The semantic information extraction module is built based on the pre-trained language model and is used to obtain image semantic information data based on the text information corresponding to the input unseen category image;

[0010] A cross-modal feature fusion module, built based on a cross-modal attention mechanism, for obtaining aligned visual-semantic feature data based on input image visual feature data and image semantic information data; and

[0011] The classification module is built based on the Softmax layer and is used to obtain the classification results of image categories based on the input aligned visual-semantic feature data.

[0012] As an improvement of the above system, the construction process of the visual feature extraction module includes:

[0013] The ResNet18 network is used for local feature extraction of images, and the DeiT model is used for contextual information extraction between local features of images. A visual feature extractor consisting of a cascade of ResNet18 and DeiT structures is established.

[0014] As an improvement of the above system, in the visual feature extraction module, the convolution kernel of the convolutional neural network is (3, 3, 7), (3, 3, 5) or (3, 3, 3);

[0015] The cascade structure is divided into three layers. The first layer is composed of a convolutional neural network, which is used to extract feature maps from the original image; the second layer is a linear mapping layer, which is used to convert the feature map into a sequence; the third layer is a converter network, which is used to generate the final image visual feature data.

[0016] As an improvement to the above system, the pre-trained language model is a Word2Vec or BERT model.

[0017] As an improvement of the above system, the cross-modal feature fusion module includes a cross-modal attention layer, a self-attention layer, a layer normalization layer and a feedforward layer, and its structure is: layer normalization layer->cross-modal attention layer->layer normalization layer->self-attention layer->layer normalization layer->feedforward layer.

[0018] As an improvement of the above system, in the cross-modal attention layer, the query, key and value are set as a semantic vector, a visual vector and a category label vector, respectively.

[0019] As an improvement to the above system, the loss function of the classification model is:

[0020] L zsl =L cc +λ·L ar

[0021] Among them, L cc represents the class semantic vector loss function based on cross entropy loss; L ar Indicates that based on L 2 The class regression loss function of the paradigm; λ represents the weight parameter of the class regression loss function.

[0022] As an improvement of the above system, the L-based 2 The class regression loss function L of the paradigm ar for:

[0023]

[0024] Among them, x represents the visual feature representation of the image, and y represents the text information representation of the image; Indicates L 2 paradigm.

[0025] As an improvement of the above system, the class semantic vector loss function L based on cross entropy loss cc for:

[0026]

[0027] in, Indicates the category to which the image belongs, C S represents the total number of categories of visible class images in the dataset; x represents the visual feature representation of the image, and y represents the text information representation of the image.

[0028] The present application also provides a zero-sample image classification method, which is implemented based on the above system and includes:

[0029] Input the unseen category image into the visual feature extraction module and output the image visual feature data;

[0030] Input text information corresponding to the unseen category image into the semantic information extraction module, and output image semantic information data;

[0031] Input the image visual feature data and image semantic information data into the cross-modal feature fusion module, and output the aligned visual-semantic feature data;

[0032] The aligned visual-semantic feature data is input into the classification module and the classification result of the image category is output.

[0033] Compared with the prior art, the advantages of this application are:

[0034] 1. The present invention constructs a zero-shot image classification network including an independent visual encoder, an independent semantic encoder, a cross-modal feature fusion module and a classifier, and integrates the feature extraction network, the feature alignment network and the classifier into a unified framework, so that the input data can directly obtain the output result through the trained zero-shot image classification network, avoiding manual preprocessing and subsequent processing, and improving the zero-shot classification speed of the image.

[0035] 2. Compared with traditional visual encoders, the visual encoder based on CNN+Transformer cascade designed by the present invention uses convolutional neural networks to extract local feature information of images and uses transformer networks to extract global feature information of images, which fully integrates the advantages of the two network structures and effectively improves the discriminability of the extracted features.

[0036] 3. The feature fusion module designed by the present invention based on cross-modal information interaction can alleviate the problem of inconsistent class structures in visual space and semantic space by learning the shared information between visual features and semantic features. This enables the classifier to have higher accuracy and stronger robustness when performing zero-shot image classification, especially for unseen category data.

[0037] In summary, compared with the prior art, the present invention can better utilize the shared information between images and auxiliary information. Under the setting of zero samples, it can not only maintain the overall classification accuracy at a high level, but also effectively improve the classification performance of unseen categories, and can provide better services for higher-level applications such as ground object monitoring and change detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Shown is a schematic diagram of the system construction and method flow of the present invention;

[0039] Figure 2 Shown is the architecture diagram of the zero-shot image classification system;

[0040] Figure 3 Shown are examples of remote sensing scene images obtained from public websites. DETAILED DESCRIPTION

[0041] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0042] The present invention proposes a zero-shot image classification system and method, which, based on cross-modal information interaction, makes full use of the extracted visual and semantic information, enhances the information interaction between visual and semantic features, and improves the image zero-shot classification effect.

[0043] The technical idea of ​​the present invention is to combine visual semantic feature extraction and zero-shot classification into a unified process, fully mine the input image information and text semantics through their own independent encoders, and interactively learn the visual and semantic features through the cross-modal attention layer, so that the acquired visual-semantic features are more discriminative, and a more accurate feature expression effect is obtained, thereby further improving the classification accuracy.

[0044] Example 1

[0045] like Figure 1 and Figure 2 As shown, according to the above ideas, the zero-shot image classification system of the present invention adopts an end-to-end training method and designs a zero-shot image classification model architecture including a visual feature extraction module based on a cascade of a convolutional neural network (CNN) and a transformer (Transformer), a semantic information extraction module based on a pre-trained language model (PLM), a cross-modal feature fusion module based on cross-modal information interaction, and a classification module, which has a total of four modules;

[0046] 1. Construct a CNN+Transformer cascade visual feature extraction module:

[0047] Among the many pre-trained models of convolutional neural networks, the ResNet18 network is selected as the image local feature extraction module; among the many versions of visual converters, the lightweight DeiT model with model distillation is selected as the context information extraction module between local features; a visual feature extractor consisting of a cascade of ResNet18 and DeiT structures is established; the parameters of the visual feature extraction network are initialized, including the weight parameter w of ResNet18 v , bias parameter b v and the initialization matrices in the query, key, and value matrices of the visual transformer Transformer and the number of heads k of the self-attention layer v And other parameters.

[0048] The convolution kernel of the convolutional neural network can be selected as (3, 3, 7), (3, 3, 5) or (3, 3, 3). The cascade structure can be divided into three layers. The first layer is composed of a convolutional neural network, which is used to extract feature maps from the original image; the second layer is a linear mapping layer, which is responsible for converting the feature map into a sequence; the third layer is a converter network, which is used to generate the final image features.

[0049] 2. Build a semantic information extraction module based on the pre-trained language model:

[0050] Use pre-trained language models such as Word2Vec or BERT model to build a semantic feature extraction network; initialize the parameters of the semantic feature extraction network including the weight parameter w of Word2Vec s or the initialization matrix in the query, key, and value matrices of the BERT model and the number of heads k of the self-attention layer s And other parameters.

[0051] 3. Introduce the cross-modal attention layer to build a cross-modal feature fusion module:

[0052] A cross-modal attention layer is constructed based on the cross-modal attention mechanism, where the query, key, and value are set as the semantic vector, visual vector, and category label vector respectively; the parameters of the cross-modal attention layer are initialized including the initialization matrix in the query, key, and value matrices And the number of layers k of the cross-modal feature fusion module c And other parameters.

[0053] The cross-modal feature fusion module is composed of a cascade of cross-modal attention layer, self-attention layer, layer normalization layer and feed-forward layer, and its structure is: layer normalization layer -> cross-modal attention layer -> layer normalization layer -> self-attention layer -> layer normalization layer -> feed-forward layer.

[0054] 4. Train the zero-shot image classification model:

[0055] Construct the overall loss function L of the network model zsl =L cc +λ·L ar , where L cc is the class semantic vector loss function based on cross entropy loss, L ar Based on L 2 The class regression loss function of the paradigm, λ is the weight parameter of the class regression loss function; initialize the parameters of the zero-shot image classification network model, including the weight parameter w in the visual feature extraction module v , bias parameter b v , the initialization matrix in the query, key, and value matrices of the visual transformer Transformer and the number of heads k of the self-attention layer v Parameters such as; weight parameter w of Word2Vec in the semantic feature extraction module s or the initialization matrix in the query, key, and value matrices of the BERT model and the number of heads k of the self-attention layer s The parameters of the cross-modal attention layer include the initialization matrix in the query, key and value matrices. And the number of layers k of the cross-modal feature fusion module c And other parameters.

[0056] The visible category training samples and category semantic information are input into the zero-shot image classification network model. The gradient descent method is used to cyclically update the parameters in the zero-shot image classification network to reduce the gradient value of the total loss function until the maximum number of iterations T is reached, and a trained zero-shot image classification network model is obtained.

[0057] Example 2

[0058] The present invention also provides a zero-sample image classification method, which is implemented based on the above system and includes:

[0059] Input the unseen category image into the visual feature extraction module and output the image visual feature data;

[0060] Input text information corresponding to the unseen category image into the semantic information extraction module, and output image semantic information data;

[0061] Input the image visual feature data and the image semantic information data into the cross-modal feature fusion module to output aligned visual-semantic feature data;

[0062] Finally, the aligned visual-semantic feature data is input into the classification module to obtain the classification result of the image category.

[0063] Taking the remote sensing scene classification standard dataset RSSDIVCS as an example, the zero-sample image classification system of the present invention is used to classify it. The images in this dataset and the corresponding true value labels are as follows: Figure 3 As shown in the figure, it contains 70 types of scenes, 800 images per type of scene, and a total of 56,000 labeled samples.

[0064] Reference Figure 1 , the specific steps for image classification are as follows:

[0065] Step 1: Randomly divide the training and test samples.

[0066] 1.1) For the RSSDIVCS dataset, the training (visible categories) and test (unseen categories) samples are usually randomly divided in the ratio of 60 / 10, 50 / 20, and 40 / 30, that is, 60 visible categories / 10 unseen categories, 50 visible categories / 20 unseen categories, and 40 visible categories / 30 unseen categories.

[0067] 1.2) Encode the label of each training sample into a vector y in one-hot form i , the label of each test sample is encoded in one-hot form as a vector q i .

[0068] Step 2: Build a visual encoder based on CNN+Tranformer.

[0069] 2.1) Construct a visual encoder with a cascade of ResNet18 and DeiT structures by cascading CNN and Tranformer structures, and the input image size is 224×224×3.

[0070] 2.2) From Figure 3 In the samples shown in the figure, the training set and test set samples and their corresponding labels are obtained according to the division in step 1. The training set is represented by X = {x 1 ,...,x train}, the label is Y = {y 1 ,...,y train}, x i For any training sample, for different visible / invisible category ratios, the subscript train corresponds to the product of the number of visible categories and the number of scenes in each category. For example, for a 60 / 10 ratio, train is 60*800=48000, x i The corresponding label y i ∈{1,2,...,60}; the test set is denoted as P={p 1 ,...,p test}, the label is Q = {q 1 ,...,q test}, p i For any test sample, for different visible / invisible category ratios, the subscript test corresponds to the product of the number of invisible categories and the number of scenes in each category. For example, for a 60 / 10 ratio, test is 10*800=8000, p i The corresponding label q i ∈{1,2,...,10}.

[0071] Step 3: Build a semantic encoder based on the pre-trained language model.

[0072] 3.1) From the pre-trained language models, select a suitable model as the semantic encoder, such as the Word2Vec model that takes label words as input, or the BERT model that takes label sentence descriptions as input.

[0073] 3.2) From Figure 3 In the sample labels shown, the corresponding labels of the training set and the test set are obtained according to the division in step 1. Among them, for different visible / invisible category ratios, the number of training samples corresponds to the product of the number of visible categories and the number of scenes in each category, and the test samples correspond to the product of the number of invisible categories and the number of scenes in each category. For example, under the 60 / 10 ratio, the training sample labels total 60*800=48000, and the remaining sample labels are used as the test set, totaling 10*800=8000.

[0074] Step 4: Construct a cross-modal attention based visual-semantic alignment encoder.

[0075] Reference Figure 2 , the specific implementation of this step is as follows:

[0076] 4.1) Establish mapping module φ i :Set up three mapping modules φ 1～3 , are composed of linear matrices, mapping the input sequences q, k, and v to query matrix Q, key matrix K, and value matrix V, respectively. The specific formulas are as follows:

[0077] Q=φ 1 q,K=φ 2 k,V=φ 3 ·v

[0078] Among them, the input sequences q and v are set as visual features; the input sequence k is set as semantic features.

[0079] 4.2) Set up a cross-modal attention module, which is composed of a cascade of layer normalization layer, cross-modal attention layer, self-attention layer and feedforward layer. The number of input vectors of the layer normalization layer is the feature dimension of the output of the previous module. The size of each vector is (batch size, 1, 224, 224), and the number of output vectors is consistent with the number of input vectors.

[0080] Step 5: Build a zero-shot image classification model based on cross-modal attention.

[0081] 5.1) The above-mentioned independent visual encoder, independent semantic encoder, and visual semantic alignment encoder are cascaded in sequence to form a visual semantic feature extraction alignment encoder.

[0082] 5.2) Construct a classification module, which includes a classifier g η . g η It includes 1 fully connected layer and 1 Softmax layer, and transforms the size of the matrix to extract the above visual semantic features and align the encoder output features to convert them into probability vectors belonging to each category. Used to determine the category to which the sample belongs.

[0083] Step 6: Construct the loss function of the cross-modal attention based zero-shot image classification model.

[0084] 6.1) Build based on L 2 The class regression loss function L of the paradigm ar , which is used to constrain the image representation to ensure that it is accurately mapped to the corresponding text embedding. The specific formula is as follows:

[0085]

[0086] Among them, x represents the visual feature representation of the image, and y represents the text information representation of the image; Indicates L 2 paradigm.

[0087] 6.2) Constructing the cross entropy loss function L cc , which is used to force the image to have the highest compatibility score with its corresponding class semantic vector. The specific formula is as follows:

[0088]

[0089] in, Indicates the category to which the image belongs, C S Represents the total number of classes of images seen in the dataset.

[0090] Step 7: Use the training samples and the gradient descent method to train the parameters of the zero-shot image classification model.

[0091] 7.1) Set the initial value of the learning rate α, the maximum number of iterations T, and the number of batches B.

[0092] 7.2) Initialize the parameters of the zero-shot image classification network model including the weight parameter w in the visual feature extraction module v , bias parameter b v , the initialization matrix in the query, key, and value matrices of the visual transformer Transformer and the number of heads k of the self-attention layer v Parameters such as; weight parameter w of Word2Vec in the semantic feature extraction module s or the initialization matrix in the query, key, and value matrices of the BERT model and the number of heads k of the self-attention layer s The parameters of the cross-modal attention layer include the initialization matrix in the query, key and value matrices. And the number of layers k of the cross-modal feature fusion module c , and other parameters, as well as the parameter η in the classifier.

[0093] 7.3) Divide the training samples into batches according to the batch size, and input them into the zero-shot image classification network in turn for back propagation to update the total loss function The gradient of The network training parameters are updated in the descending direction through forward propagation, and the above calculations are performed for each batch of input networks until the entire training set sample input network is traversed, completing one training session.

[0094] 7.4) Repeat the process in 7.3) until the maximum number of iterations T is reached to obtain a trained zero-shot classification model.

[0095] Step 8: Use the trained zero-shot image classification model to classify the test samples.

[0096] Input each test sample into the trained zero-shot image classification model to obtain the probability p(y|x) of the label to which each test sample belongs:

[0097]

[0098] Among them, softmax is the normalization function, L zsl is the total loss function, x is the test sample, y c is the label corresponding to the c-th test sample, z c is the semantic feature corresponding to the c-th test sample, C is the total number of test samples, w v ,b v are the trainable parameters of CNN in the visual feature extraction module, is the trainable parameter of Transformer in the visual feature extraction module, is the trainable parameter of the BERT model / word2vec model in the pre-trained language model, and η is the trainable parameter of the classifier.

[0099] The effect of the present invention can be further illustrated by the following simulation results.

[0100] 1. Simulation conditions

[0101] Test data: RSSDIVCS data was used, and the semantic information used was Word2Vec semantic vectors. Images were tested according to three visible / invisible ratios. That is, the visible / invisible ratio was 60 / 10, 60 categories of visible category images were randomly selected as training sets, and the remaining 10 categories were unseen category images as test sets; the visible / invisible ratio was 50 / 20, 50 categories of visible category images were randomly selected as training sets, and the remaining 20 categories were unseen category images as test sets; the visible / invisible ratio was 40 / 30, 40 categories of visible category images were randomly selected as training sets, and the remaining 30 categories were unseen category images as test sets.

[0102] Test environment: Linux system, NVIDIA GeForce GTX 2080Ti GPU, Pytorch deep learning framework.

[0103] 2. Simulation content

[0104] Simulation 1), under the above conditions, the RSSDIVCS data is classified using the existing zero-shot image scene classification method LPDCMENs based on the embedding model, and the results are shown in Table 1.

[0105] Simulation 2), under the above conditions, the RSSDIVCS data are classified using the method of the present invention, and the results are shown in Table 1.

[0106] By comparing Table 1, it can be found that the classification result obtained by the method proposed in the present invention is more accurate and achieves better classification performance.

[0107] The above two methods were tested 10 times using the same random numbers, and the overall classification accuracy (OA) was calculated. The test results are shown in Table 1.

[0108] The indicator calculation formula is as follows:

[0109]

[0110] Among them, TP is the number of samples correctly classified into this category, FN is the number of samples of this category that are incorrectly classified as other categories, TN is the number of non-class samples classified into other categories, and FP is the number of other samples that are incorrectly classified as samples of this category.

[0111] Table 1 Simulation experimental results of the present invention and the comparative method (OA)

[0112]

[0113] It can be seen from Table 1 that, compared with the current more advanced LPDCMENs method, the classification accuracy of the present invention under the three visible / invisible ratios is improved.

[0114] The present application may also provide a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together through a bus system. It is understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0115] The user interface may include a display, a keyboard or a pointing device, such as a mouse, a trackball, a touch pad or a touch screen.

[0116] It is understood that the memory in the embodiments disclosed in the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0117] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and applications.

[0118] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application.

[0119] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and is used to:

[0120] Execute the steps of the above method.

[0121] The above method can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The above-disclosed methods, steps and logic block diagrams can be implemented or executed. The general processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the above-disclosed method can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0122] It is understood that the embodiments described in the present application can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0123] For software implementation, the technology of the present application can be implemented by executing the functional modules (such as procedures, functions, etc.) of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0124] The present application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method embodiment can be implemented.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A zero-shot image classification system, characterized in that: The system comprises: The visual feature extraction module consists of a convolutional neural network and a cascade of transformers, which is used to obtain image visual feature data based on the input of unseen category images; The semantic information extraction module is built based on the pre-trained language model and is used to obtain image semantic information data based on the text information corresponding to the input unseen category image; A cross-modal feature fusion module, built based on a cross-modal attention mechanism, for obtaining aligned visual-semantic feature data based on input image visual feature data and image semantic information data; and The classification module is built based on the Softmax layer and is used to obtain the classification results of image categories based on the input aligned visual-semantic feature data.

2. The zero-shot image classification system according to claim 1, characterized in that: The construction process of the visual feature extraction module includes: The ResNet18 network is used for local feature extraction of images, and the DeiT model is used for contextual information extraction between local features of images. A visual feature extractor consisting of a cascade of ResNet18 and DeiT structures is established.

3. The zero-shot image classification system according to claim 2, characterized in that: In the visual feature extraction module, the convolution kernel of the convolutional neural network is (3, 3, 7), (3, 3, 5) or (3, 3, 3); The cascade structure is divided into three layers. The first layer consists of a convolutional neural network, which is used to extract feature maps from the original image; The second layer is a linear mapping layer, which is used to convert the feature map into a sequence; the third layer is a converter network, which is used to generate the final image visual feature data.

4. The zero-shot image classification system according to claim 1, characterized in that: The pre-trained language model is a Word2Vec or BERT model.

5. The zero-shot image classification system according to claim 1, characterized in that: The cross-modal feature fusion module includes a cross-modal attention layer, a self-attention layer, a layer normalization layer and a feedforward layer, and its structure is: layer normalization layer->cross-modal attention layer->layer normalization layer->self-attention layer->layer normalization layer->feedforward layer.

6. The zero-shot image classification system according to claim 5, characterized in that: In the cross-modal attention layer, the query, key, and value are set as the semantic vector, visual vector, and category label vector, respectively.

7. The zero-shot image classification system according to claim 1, characterized in that: The loss function of the classification model is: L zsl =L cc +λ·L ar Among them, L cc represents the class semantic vector loss function based on cross entropy loss; L ar represents the class regression loss function based on the L2 paradigm; λ represents the weight parameter of the class regression loss function.

8. The zero-shot image classification system according to claim 7, characterized in that: The class regression loss function L based on the L2 paradigm ar for: Among them, x represents the visual feature representation of the image, and y represents the text information representation of the image; Denotes L2 norm.

9. The zero-shot image classification system according to claim 7, characterized in that: The class semantic vector loss function L based on cross entropy loss cc for: in, Indicates the category to which the image belongs, C S represents the total number of categories of visible class images in the dataset; x represents the visual feature representation of the image, and y represents the text information representation of the image.

10. A zero-sample image classification method, implemented based on the system according to any one of claims 1 to 9, the method comprising: Input the unseen category image into the visual feature extraction module and output the image visual feature data; Input text information corresponding to the unseen category image into the semantic information extraction module, and output image semantic information data; Input the image visual feature data and image semantic information data into the cross-modal feature fusion module, and output the aligned visual-semantic feature data; The aligned visual-semantic feature data is input into the classification module and the classification result of the image category is output.

Images