A zero-sample image classification method, device, terminal and medium

Abstract

The present invention discloses a zero-shot image classification method, device, terminal, and medium, comprising: projecting visual features and semantic features into a common space to learn latent low-dimensional features according to a conditional primary label space conversion algorithm; performing supervised learning based on the latent low-dimensional features to train an encoder that projects visual features into low-dimensional features, and training a decoder that projects low-dimensional features into visual features; performing supervised learning based on the latent low-dimensional features to train an encoder that projects semantic features into low-dimensional features, and training a decoder that projects low-dimensional features into semantic features; and performing category determination in the visual feature space, the semantic feature space, and the latent common low-dimensional feature space according to a multi-level semantic coding image classification model, and outputting an image classification result. The present invention improves the accuracy and efficiency of zero-shot image classification tasks for multi-level semantic information.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a zero-sample image classification method, device, terminal and medium. Background Art

[0002] Zero-shot image classification tasks in emerging intelligent scenarios such as autonomous driving, extended reality (XR), and smart cities have brought new challenges to wireless communications. Designing a lightweight and multi-level semantic information encoding and decoding method based on the local semantic knowledge base of the intelligent agent to efficiently complete zero-shot image classification tasks and thus support intelligent and simplified communications is an urgent issue.

[0003] Zero-shot image classification is a critical requirement for emerging intelligent application scenarios such as autonomous driving, XR, and smart cities. For example, in autonomous driving, the high speed of vehicles creates a constantly changing traffic landscape. Consequently, the distribution of the datasets perceived by the vehicle is also constantly changing, making zero-shot classification a common problem. Furthermore, traditional deep learning-based image classification methods often require the collection and annotation of large-scale image datasets, consuming significant manpower, material resources, and time. This is inconsistent with the low-latency, lightweight, and efficient requirements of future intelligent communications. Therefore, achieving efficient zero-shot image classification is crucial for intelligent communications.

[0004] To efficiently solve the zero-shot image classification problem and improve the generalization capabilities of classification methods, a proposal has been made to design semantic autoencoders supervised by semantic information. This approach directly projects the visual feature information in the training samples into the semantic feature space, enabling category judgment in a shared semantic space across all categories. However, due to the inconsistent data structures of image features and semantic features, the direct projection of visual features onto semantic features fails to bridge the semantic gap between visual and semantic features. Furthermore, the dimensionality of the latent semantic space obtained using this approach depends on manual semantic attribute descriptions or Word2Net, making it difficult to flexibly adjust. High dimensionality is therefore detrimental to the implementation of intelligent communication.

[0005] Another proposal proposes a zero-shot image classification method based on a dual autoencoder, which learns latent semantic features by projecting visual and semantic features into a common space. However, this method makes category decisions by projecting category semantic features onto latent semantic features and then performing a distance judgment with the latent semantic features of the test sample. It does not consider the ability to efficiently decode latent features into the semantic and visual feature spaces, resulting in a single decision hierarchy. The performance of category decisions at the visual or semantic level remains to be verified. Furthermore, this method still requires iterative operations to update the training model, which is not conducive to the implementation of intelligent communication.

[0006] In the above-mentioned zero-shot image classification methods, it is still impossible to complete the zero-shot image classification task accurately and efficiently.

[0007] Therefore, the existing technology needs to be improved. Summary of the Invention

[0008] The technical problem to be solved by the present invention is that, in response to the defects of the existing technology, the present invention provides a zero-sample image classification method, device, terminal and medium to solve the technical problems of low accuracy and efficiency of the existing zero-sample image classification method for multi-level semantic information classification tasks.

[0009] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0010] In a first aspect, the present invention provides a zero-shot image classification method, comprising:

[0011] The conditional main label space conversion algorithm is used to project the visual features and semantic features into a common space to learn the latent low-dimensional features.

[0012] Performing supervised learning based on the potential low-dimensional features to train an encoder that projects the visual features to low-dimensional features, and training a decoder that projects the low-dimensional features to the visual features;

[0013] Performing supervised learning based on the potential low-dimensional features to train an encoder that projects the semantic features to the low-dimensional features, and training a decoder that projects the low-dimensional features to the semantic features;

[0014] According to the multi-level semantic coding image classification model, category judgment is performed in the visual feature space, semantic feature space and potential common low-dimensional feature space, and the image classification results are output.

[0015] In one implementation, the conditional main label space conversion algorithm projects visual features and semantic features into a common space to learn latent low-dimensional features, which includes:

[0016] Obtaining an image sample dataset, and using a deep residual network to obtain a set of visual feature vectors corresponding to the sample image dataset;

[0017] The categories of each sample image in the sample image dataset are labeled, and a semantic knowledge base of each category is constructed.

[0018] In one implementation, projecting visual features and semantic features into a common space to learn latent low-dimensional features according to a conditional primary label space conversion algorithm includes:

[0019] Obtaining a semantic feature vector of each sample image according to the semantic knowledge base;

[0020] According to the semantic features and visual features of each sample image, a low-dimensional feature extractor is constructed to project the visual features and semantic features of each sample image into a common space to learn the potential low-dimensional features.

[0021] In one implementation, constructing a low-dimensional feature extractor based on the semantic features and visual features of each sample image, and projecting the visual features and semantic features of each sample image into a common space to learn potential low-dimensional features includes:

[0022] The problem modeling of obtaining low-dimensional features based on the conditional main label algorithm is to optimize the visual mapping matrix and the semantic mapping matrix with the joint goal of minimizing the prediction error and encoding error:

[0023]

[0024]

[0025] in, represents the visual feature vector matrix;

[0026] Represents the semantic feature vector matrix;

[0027] express dimensional cell matrix;

[0028] Based on the semantic mapping matrix, the semantic feature vector of each sample image is mapped to a low-dimensional feature space.

[0029] In one implementation, the performing supervised learning based on the potential low-dimensional features to train an encoder that projects the visual features to low-dimensional features, and training a decoder that projects the low-dimensional features to the visual features, includes:

[0030] Using the extracted low-dimensional latent feature vector as supervision, we construct a visual autoencoder and a visual autodecoder.

[0031] Projecting the visual feature vector into a low-dimensional latent feature space under the visual autoencoder mapping;

[0032] Under the visual self-decoder mapping, the low-dimensional latent feature space is restored to the visual feature vector.

[0033] In one implementation, the construction of a visual autoencoder and a visual autodecoder using the extracted low-dimensional latent feature vector as supervision includes:

[0034] Taking the low-dimensional feature space For supervision, optimize the following visual autoencoder problem:

[0035]

[0036]

[0037] in, Indicates that the visual features Mapping to low-dimensional latent feature space The projection matrix is ​​denoted as the visual autoencoder;

[0038] represents the visual self-decoder;

[0039] The constraints of the visual autoencoder problem are scaled into a first objective function, and the first-order derivative of the first objective function is set to 0, and the optimal solution of the visual autoencoder is obtained by the Bartels-Stewart algorithm.

[0040] In one implementation, the performing supervised learning based on the potential low-dimensional features to train an encoder that projects the semantic features to the low-dimensional features, and the training obtaining a decoder that projects the low-dimensional features to the semantic features, includes:

[0041] Using the extracted low-dimensional latent feature vector as supervision, a semantic autoencoder and semantic self-decoder are constructed;

[0042] Under the semantic autoencoder mapping, the semantic feature vector is projected into a low-dimensional latent feature space;

[0043] Under the semantic self-decoder mapping, the low-dimensional latent feature space is restored to the semantic feature space.

[0044] In one implementation, the construction of a semantic autoencoder and a semantic decoder using the extracted low-dimensional latent feature vector as supervision includes:

[0045] Using the low-dimensional feature space as supervision, optimize the following semantic autoencoder problem:

[0046]

[0047]

[0048] in, Indicates that the semantic features Mapping to low-dimensional latent feature space The projection matrix is ​​denoted as semantic autoencoder;

[0049] Represents a semantic self-decoder;

[0050] The constraints of the semantic autoencoder problem are scaled into a second objective function, and the first-order derivative of the second objective function is set to 0, and the optimal solution of the semantic autoencoder is obtained by the Bartels-Stewart algorithm.

[0051] In one implementation, the image classification model based on multi-level semantic coding performs category judgment in the visual feature space, the semantic feature space, and the potential common low-dimensional feature space, and outputs the image classification result, including:

[0052] Based on the visual autoencoder and semantic autoencoder, a multi-level semantic encoder is constructed to characterize the mapping relationship between the dimension size, classification accuracy, and semantic loss of different levels, and an image classification model of the multi-level semantic encoding is obtained;

[0053] According to the multi-level semantic coding image classification model, category judgment is performed in the visual feature space, the semantic feature space and the potential common low-dimensional feature space respectively, and the image classification result is output.

[0054] In one implementation, a multi-level semantic encoder is constructed based on the visual autoencoder and the semantic autoencoder to characterize the mapping relationship between the dimension size, classification accuracy, and semantic loss amount of different levels, including:

[0055] First-level category judgment: project the semantic representation of each category into the visual space and make category judgments on the test visual sample vectors;

[0056] Second-level category decision: Projecting the test visual sample vector into a low-dimensional space, and projecting the semantic representation of each category into the low-dimensional space to make a category decision;

[0057] The third level category decision: the test visual sample vector is projected into the semantic space to make a category decision.

[0058] In a second aspect, the present invention provides a zero-shot image classification device, comprising:

[0059] Latent low-dimensional feature module, which is used to project visual features and semantic features into a common space based on the conditional main label space conversion algorithm to learn latent low-dimensional features;

[0060] a visual feature encoding and decoding module, configured to perform supervised learning based on the potential low-dimensional features, train an encoder for projecting the visual features into low-dimensional features, and train a decoder for projecting the low-dimensional features into the visual features;

[0061] A semantic feature encoding and decoding module, configured to perform supervised learning based on the potential low-dimensional features, train an encoder that projects the semantic features to the low-dimensional features, and train a decoder that projects the low-dimensional features to the semantic features;

[0062] The category judgment module is used to make category judgments in the visual feature space, semantic feature space and potential common low-dimensional feature space based on the multi-level semantic coding image classification model, and output the image classification results.

[0063] In a third aspect, the present invention provides a terminal comprising: a processor and a memory, wherein the memory stores a zero-sample image classification program, and when the zero-sample image classification program is executed by the processor, it is used to implement the operation of the zero-sample image classification method as described in the first aspect.

[0064] In a fourth aspect, the present invention further provides a medium, which is a computer-readable storage medium, storing a zero-sample image classification program, and when the zero-sample image classification program is executed by a processor, it is used to implement the operation of the zero-sample image classification method as described in the first aspect.

[0065] The present invention adopts the above technical solution to achieve the following effects:

[0066] The present invention is based on the conditional main label space conversion method, which learns potential low-dimensional features by projecting visual and semantic features into a common space. Under the supervision of the extracted potential low-dimensional features, a visual autoencoder is designed, and an encoder that projects visual features into low-dimensional features and a decoder that projects low-dimensional features into visual features are trained. Under the supervision of the extracted potential low-dimensional features, a semantic autoencoder is designed, and an encoder that projects semantic features into low-dimensional features and a decoder that projects low-dimensional features into semantic features are trained. Finally, an image classification method based on multi-level semantic coding is designed, and category judgment is performed in the visual feature space, the semantic feature space, and the potential common low-dimensional feature space respectively. The present invention can realize multi-level semantic category judgment, improve classification accuracy, reduce computational complexity and required processing delay, and realize more efficient intelligent communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0068] Figure 1 It is a flowchart of a zero-sample image classification method in one implementation of the present invention.

[0069] Figure 2 It is a schematic diagram of zero-shot image classification based on a multi-level semantic encoder in one implementation of the present invention.

[0070] Figure 3 It is a functional principle diagram of a terminal in one implementation of the present invention.

[0071] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0073] Exemplary Methods

[0074] To efficiently solve the zero-shot image classification problem and improve the generalization capabilities of classification methods, a proposal has been made to design semantic autoencoders supervised by semantic information. This approach directly projects the visual feature information in the training samples into the semantic feature space, enabling category judgment in a shared semantic space across all categories. However, due to the inconsistent data structures of image features and semantic features, the direct projection of visual features onto semantic features fails to bridge the semantic gap between visual and semantic features. Furthermore, the dimensionality of the latent semantic space obtained using this approach depends on manual semantic attribute descriptions or Word2Net, making it difficult to flexibly adjust. High dimensionality is therefore detrimental to the implementation of intelligent communication.

[0075] Another proposal proposes a zero-shot image classification method based on a dual autoencoder, which learns latent semantic features by projecting visual and semantic features into a common space. However, the category judgment based on this method is obtained by projecting the category semantic features into the latent semantic features and then performing a distance judgment with the latent semantic features of the test sample. The ability to efficiently decode the latent features into the semantic and visual feature spaces has not been considered, resulting in a single judgment level. The performance of category judgment at the visual level or the semantic level remains to be verified. On the other hand, the training model of this method still requires iterative operations when updating, which is not conducive to the realization of intelligent communication. Among the above-mentioned zero-shot image classification methods, it is still impossible to accurately and efficiently complete the zero-shot image classification task.

[0076] To address the above technical issues, this embodiment provides a zero-shot image classification method. This method, based on multi-level semantic coding, performs category determination in the visual feature space, the semantic feature space, and the underlying common low-dimensional feature space. This embodiment implements multi-level semantic category determination, improving classification accuracy while reducing computational complexity and processing latency, enabling more efficient intelligent communication.

[0077] like Figure 1 As shown, an embodiment of the present invention provides a zero-sample image classification method, comprising the following steps:

[0078] Step S100 , projecting visual features and semantic features into a common space to learn potential low-dimensional features according to a conditional main label space conversion algorithm.

[0079] In this embodiment, the zero-sample image classification method is applied to a terminal, which includes but is not limited to: a computer and other devices.

[0080] This embodiment provides a zero-shot image classification method based on a multi-level semantic codec. First, a conditional principal label space conversion method is used to project visual and semantic features into a common space to learn latent low-dimensional features. This conditional principal label space conversion method allows the optimal projection matrix to be directly obtained in one step using singular value decomposition.

[0081] Specifically, in one implementation of this embodiment, the following steps are included before step S100:

[0082] Step S001: obtaining an image sample dataset, and using a deep residual network to obtain a set of visual feature vectors corresponding to the sample image dataset;

[0083] Step S002: annotate the category of each sample image in the sample image dataset and construct a semantic knowledge base of each category.

[0084] In this embodiment, before projecting the visual features and semantic features into a common space to learn the potential low-dimensional features, the image sample data needs to be preprocessed. The preprocessing process includes: first, obtaining an image sample dataset, and using a deep residual network (e.g., ResNet network, VGG network, etc.) to obtain a set of visual feature vectors of the sample image. ; Secondly, mark the category to which the sample image belongs , using Word2Net network or manual annotation method to build the semantic knowledge base of the category .

[0085] Specifically, in one implementation of this embodiment, step S100 includes the following steps:

[0086] Step S101, obtaining a semantic feature vector of each sample image according to the semantic knowledge base;

[0087] Step S102: construct a low-dimensional feature extractor based on the semantic features and visual features of each sample image, and project the visual features and semantic features of each sample image into a common space to learn potential low-dimensional features.

[0088] In this embodiment, after preprocessing the image sample data, the semantic feature vector of each sample image is obtained based on the semantic knowledge base. In this embodiment, a low-dimensional latent feature extractor is designed based on visual and semantic features to learn latent features. This method not only helps obtain a controllable low-dimensional latent vector, but also considers both visual and semantic features, thus bridging the gap between their statistical properties.

[0089] In one implementation of this embodiment, step S102 includes the following steps:

[0090] Step S102a, modeling the problem of obtaining low-dimensional features based on the conditional primary label algorithm, and optimizing the visual mapping matrix and the semantic mapping matrix with the joint goal of minimizing the prediction error and the encoding error;

[0091] Step S102b: Mapping the semantic feature vector of each sample image to a low-dimensional feature space based on the semantic mapping matrix.

[0092] In this embodiment, the visual mapping matrix is ​​optimized with the joint goal of minimizing the prediction error and the coding error. and semantic mapping matrix The relevant optimization problem is modeled as:

[0093]

[0094]

[0095] in, represents the visual feature vector matrix;

[0096] Represents the semantic feature vector matrix;

[0097] express dimensional cell matrix.

[0098] In the above modeling problem, the vectors in the visual feature vector matrix are: feature vectors extracted by large-scale pre-trained models, such as ResNet networks and VGG networks; while the vectors in the semantic feature vector matrix are: semantic descriptions of the categories to which the corresponding samples belong, such as attribute descriptions, semantic vectors obtained based on the Word2Net network, etc.

[0099] In order to solve the above problem, first, given , we can directly get the corresponding optimal The closed-form solution of ,in express Then, Replace the original question , the original problem can be equivalently transformed into:

[0100]

[0101]

[0102] Based on the Eckart-Young theorem, we can solve the matrix The eigenvector of .

[0103] Get the best Then, based on the semantic mapping matrix , the semantic feature vector Mapping to low-dimensional feature space ,Right now .

[0104] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the zero-sample image classification method further includes the following steps:

[0105] Step S200 , performing supervised learning based on the potential low-dimensional features, training an encoder that projects the visual features to low-dimensional features, and training a decoder that projects the low-dimensional features to the visual features.

[0106] In this example, a visual autoencoder is designed under the supervision of the extracted latent low-dimensional features. The encoder is trained to project visual features onto low-dimensional features, and the decoder is trained to project low-dimensional features onto visual features. This problem is essentially equivalent to solving the Sylvester equation, and the optimal solution can be obtained in a single step using Matlab code.

[0107] Specifically, in one implementation of this embodiment, step S200 includes the following steps:

[0108] Step S201, constructing a visual autoencoder and a visual autodecoder using the extracted low-dimensional latent feature vector as supervision;

[0109] Step S202, projecting the visual feature vector into a low-dimensional latent feature space under the mapping of the visual autoencoder;

[0110] Step S203 , restoring the low-dimensional latent feature space to the visual feature vector under the visual self-decoder mapping.

[0111] In this embodiment, a visual autoencoder and a visual autodecoder are designed with the extracted low-dimensional latent feature vector as supervision, wherein under the mapping of the visual autoencoder, the visual feature vector is projected into the low-dimensional latent feature space; under the mapping of the visual autodecoder, the low-dimensional latent feature space is restored to the original visual feature vector.

[0112] In one implementation of this embodiment, step S201 includes the following steps:

[0113] Step S201a, using the low-dimensional feature space For supervision, optimize the following visual autoencoder problem:

[0114]

[0115]

[0116] in, Indicates that the visual features Mapping to low-dimensional latent feature space The projection matrix is ​​denoted as the visual autoencoder;

[0117] represents the visual self-decoder.

[0118] Step S201b: scaling the constraints of the visual autoencoder problem into a first objective function, setting the first-order derivative of the first objective function to 0, and obtaining the optimal solution of the visual autoencoder using the Bartels-Stewart algorithm.

[0119] In this embodiment, in order to solve the above problem, the above equality constraint is first scaled into the objective function (i.e., the first objective function):

[0120]

[0121] Then, the first derivative of the function is set to 0, that is:

[0122]

[0123] Finally, since the above equation belongs to the Sylvester equation, the optimal solution can be directly obtained by the Bartels-Stewart algorithm, which can be directly implemented using a single line of Matlab code.

[0124] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the zero-sample image classification method further includes the following steps:

[0125] Step S300 , performing supervised learning based on the potential low-dimensional features, training an encoder that projects the semantic features to the low-dimensional features, and training a decoder that projects the low-dimensional features to the semantic features.

[0126] In this example, a semantic autoencoder is designed under the supervision of the extracted latent low-dimensional features. The encoder is trained to project semantic features onto low-dimensional features, and the decoder is trained to project low-dimensional features onto semantic features. This problem is essentially equivalent to solving the Sylvester equation, which can be solved in one step using Matlab code.

[0127] Specifically, in one implementation of this embodiment, step S300 includes the following steps:

[0128] Step S301, constructing a semantic autoencoder and a semantic autodecoder using the extracted low-dimensional latent feature vector as supervision;

[0129] Step S302, projecting the semantic feature vector into a low-dimensional latent feature space under the semantic autoencoder mapping;

[0130] Step S303 : Restoring the low-dimensional latent feature space to the semantic feature space under the semantic self-decoder mapping.

[0131] In this embodiment, a semantic autoencoder and a semantic self-decoder are designed with the extracted low-dimensional latent feature vector as supervision, wherein under the mapping of the semantic autoencoder, the semantic feature vector is projected to the low-dimensional latent feature space; under the mapping of the semantic self-decoder, the low-dimensional latent feature space is restored to the semantic feature space.

[0132] In one implementation of this embodiment, step S301 includes the following steps:

[0133] Step S301a, using the low-dimensional feature space as supervision, optimize the following semantic autoencoder problem:

[0134]

[0135]

[0136] in, Indicates that the semantic features Mapping to low-dimensional latent feature space The projection matrix is ​​denoted as semantic autoencoder;

[0137] Represents a semantic self-decoder.

[0138] Step S301b: scaling the constraints of the semantic autoencoder problem into a second objective function, setting the first-order derivative of the second objective function to 0, and obtaining the optimal solution of the semantic autoencoder using the Bartels-Stewart algorithm.

[0139] In this embodiment, in order to solve the above problem, the above equality constraint is first scaled into the objective function (i.e., the second objective function):

[0140]

[0141] Then, the first derivative of the objective function is set to 0, that is:

[0142]

[0143] Finally, since the above equation belongs to the Sylvester equation, the optimal solution can be directly obtained by the Bartels-Stewart algorithm, which can be directly implemented using a single line of Matlab code.

[0144] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the zero-sample image classification method further includes the following steps:

[0145] Step S400 , based on the multi-level semantic coding image classification model, class decisions are made in the visual feature space, the semantic feature space, and the potential common low-dimensional feature space, and the image classification result is output.

[0146] In this embodiment, a multi-level semantic coding-based image classification method is designed to perform category determination in the visual feature space, semantic feature space, and potential common low-dimensional feature space. The zero-shot image classification method proposed in this embodiment can achieve multi-level semantic category determination.

[0147] Specifically, in one implementation of this embodiment, step S400 includes the following steps:

[0148] Step S401: Based on the visual autoencoder and the semantic autoencoder, a multi-level semantic encoder is constructed to characterize the mapping relationship between the dimensionality, classification accuracy, and semantic loss of different levels, thereby obtaining an image classification model of the multi-level semantic encoding;

[0149] Step S402 , performing category determination in the visual feature space, the semantic feature space, and the potential common low-dimensional feature space according to the multi-level semantic coding image classification model, and outputting the image classification result.

[0150] In this embodiment, a multi-level semantic encoder is constructed based on the visual autoencoder and the semantic autoencoder to characterize the mapping relationship between the dimensional size, classification accuracy, and semantic loss of different levels, and a multi-level semantic coding image classification model is obtained. Through this model, multi-level semantic category judgment can be realized, thereby improving classification accuracy.

[0151] In one implementation of this embodiment, step S401 includes the following steps:

[0152] Step S401a, first-level category decision: project the semantic representation of each category into the visual space and make category decisions on the test visual sample vector;

[0153] Step S401b, second-level category decision: projecting the test visual sample vector into a low-dimensional space, and projecting the semantic representation of each category into the low-dimensional space, and performing category decision;

[0154] Step S401c, third-level category decision: projecting the test visual sample vector into the semantic space and performing category decision.

[0155] like Figure 2 As shown, in this embodiment, multi-level semantic category determination includes but is not limited to:

[0156] First-level category judgment: First, the semantic representation of each category is projected into the visual space, that is, ; Then test the visual sample vector , make category judgment:

[0157]

[0158] Second level category judgment: First, the test visual sample vector Projected into low-dimensional space, that is , and project the semantic representation of each category into a low-dimensional space, that is, , and then make a category decision:

[0159]

[0160] The third level category judgment: First, the test visual sample vector Projected into semantic space, i.e. , and then make a category decision:

[0161]

[0162] The above image classification decision method based on multi-level semantic encoder is as follows Figure 2 shown.

[0163] In this embodiment, visual and semantic feature information are jointly utilized to design a latent common low-dimensional feature encoder based on the conditional main label space conversion method; under the supervision of the extracted latent low-dimensional feature vector, a visual autoencoder and a semantic autoencoder are designed respectively; a classification method based on a multi-level semantic encoder is designed, and the final category decision is made according to the visual level, the low-dimensional latent feature level, and the semantic level vector representation.

[0164] This embodiment achieves the following technical effects through the above technical solution:

[0165] This embodiment is based on the conditional main label space conversion method, which learns potential low-dimensional features by projecting visual and semantic features into a common space. Under the supervision of the extracted potential low-dimensional features, a visual autoencoder is designed, and the encoder that projects visual features to low-dimensional features and the decoder that projects low-dimensional features to visual features are trained. Under the supervision of the extracted potential low-dimensional features, a semantic autoencoder is designed, and the encoder that projects semantic features to low-dimensional features and the decoder that projects low-dimensional features to semantic features are trained. Finally, an image classification method based on multi-level semantic coding is designed, and category decisions are made in the visual feature space, semantic feature space, and potential common low-dimensional feature space respectively. This embodiment can realize multi-level semantic category decisions, improve classification accuracy, reduce computational complexity and required processing delay, and achieve more efficient intelligent communication.

[0166] Exemplary devices

[0167] Based on the above embodiment, the present invention further provides a zero-sample image classification device, comprising:

[0168] Latent low-dimensional feature module, which is used to project visual features and semantic features into a common space based on the conditional main label space conversion algorithm to learn latent low-dimensional features;

[0169] a visual feature encoding and decoding module, configured to perform supervised learning based on the potential low-dimensional features, train an encoder for projecting the visual features into low-dimensional features, and train a decoder for projecting the low-dimensional features into the visual features;

[0170] A semantic feature encoding and decoding module, configured to perform supervised learning based on the potential low-dimensional features, train an encoder that projects the semantic features to the low-dimensional features, and train a decoder that projects the low-dimensional features to the semantic features;

[0171] The category judgment module is used to make category judgments in the visual feature space, semantic feature space and potential common low-dimensional feature space based on the multi-level semantic coding image classification model, and output the image classification results.

[0172] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 3 shown.

[0173] The terminal includes: a processor, memory, interface, display screen and communication module connected via a system bus; wherein the processor of the terminal is used to provide computing and control capabilities; the memory of the terminal includes a storage medium and an internal memory; the storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used to connect to external devices, such as mobile terminals and computers; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0174] When the computer program is executed by a processor, it is used to implement the operation of the zero-shot image classification method.

[0175] It will be understood by those skilled in the art that Figure 3 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0176] In one embodiment, a terminal is provided, comprising: a processor and a memory, wherein the memory stores a zero-sample image classification program, and when the zero-sample image classification program is executed by the processor, it is used to implement the operations of the above zero-sample image classification method.

[0177] In one embodiment, a storage medium is provided, wherein the storage medium stores a zero-shot image classification program, and the zero-shot image classification program is used to implement the operations of the above zero-shot image classification method when executed by a processor.

[0178] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile storage medium. When executed, the computer program can include the processes in the above-described method embodiments. Any reference to memory, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory.

[0179] In summary, the present invention provides a zero-shot image classification method, device, terminal, and medium. The method includes: projecting visual features and semantic features into a common space to learn latent low-dimensional features according to a conditional main label space conversion algorithm; performing supervised learning based on the latent low-dimensional features to train an encoder that projects visual features into low-dimensional features, and training a decoder that projects low-dimensional features into visual features; performing supervised learning based on the latent low-dimensional features to train an encoder that projects semantic features into low-dimensional features, and training a decoder that projects low-dimensional features into semantic features; and performing category judgment in the visual feature space, the semantic feature space, and the latent common low-dimensional feature space according to an image classification model based on multi-level semantic coding, and outputting an image classification result. The present invention improves the accuracy and efficiency of zero-shot image classification tasks for multi-level semantic information.

[0180] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A zero-shot image classification method, characterized in that: include: The conditional main label space conversion algorithm is used to project the visual features and semantic features into a common space to learn the latent low-dimensional features. Performing supervised learning based on the potential low-dimensional features to train an encoder that projects the visual features to low-dimensional features, and training a decoder that projects the low-dimensional features to the visual features; Performing supervised learning based on the potential low-dimensional features to train an encoder that projects the semantic features to the low-dimensional features, and training a decoder that projects the low-dimensional features to the semantic features; Based on the multi-level semantic coding image classification model, category judgment is performed in the visual feature space, semantic feature space and potential common low-dimensional feature space, and the image classification result is output; The conditional main label space conversion algorithm projects visual features and semantic features into a common space to learn potential low-dimensional features, including: According to the semantic knowledge base, the semantic feature vector of each sample image is obtained; Based on the semantic features and visual features of each sample image, a low-dimensional feature extractor is constructed to project the visual features and semantic features of each sample image into a common space to learn the potential low-dimensional features; The method of constructing a low-dimensional feature extractor based on the semantic features and visual features of each sample image and projecting the visual features and semantic features of each sample image into a common space to learn potential low-dimensional features includes: The problem modeling of obtaining low-dimensional features based on the conditional main label algorithm is to optimize the visual mapping matrix and the semantic mapping matrix with the joint goal of minimizing the prediction error and encoding error: in, represents the visual mapping matrix; represents the semantic mapping matrix; represents the visual feature vector matrix; Represents the semantic feature vector matrix; express dimensional cell matrix; Based on the semantic mapping matrix, mapping the semantic feature vector of each sample image to a low-dimensional feature space; The image classification model based on multi-level semantic coding performs category judgment in the visual feature space, semantic feature space and potential common low-dimensional feature space respectively, and outputs the image classification results, including: Based on the visual autoencoder and semantic autoencoder, a multi-level semantic encoder is constructed to characterize the mapping relationship between the dimension size, classification accuracy, and semantic loss of different levels, and an image classification model of the multi-level semantic encoding is obtained; According to the multi-level semantic coding image classification model, category judgment is performed in the visual feature space, the semantic feature space and the potential common low-dimensional feature space respectively, and the image classification result is output.

2. The zero-shot image classification method according to claim 1, wherein: The conditional main label space conversion algorithm projects visual features and semantic features into a common space to learn potential low-dimensional features, which previously included: Obtaining an image sample dataset, and using a deep residual network to obtain a set of visual feature vectors corresponding to the sample image dataset; The categories of each sample image in the sample image dataset are labeled, and a semantic knowledge base of each category is constructed.

3. The zero-shot image classification method according to claim 1, wherein: The supervised learning is performed based on the potential low-dimensional features to train an encoder that projects the visual features to low-dimensional features, and the decoder that projects the low-dimensional features to the visual features is trained, comprising: Using the extracted low-dimensional latent feature vector as supervision, we construct a visual autoencoder and a visual autodecoder. Projecting the visual feature vector into a low-dimensional latent feature space under the visual autoencoder mapping; Under the visual self-decoder mapping, the low-dimensional latent feature space is restored to the visual feature vector.

4. The zero-shot image classification method according to claim 3, wherein: The method of constructing a visual autoencoder and a visual autodecoder using the extracted low-dimensional latent feature vector as supervision includes: Taking the low-dimensional feature space For supervision, optimize the following visual autoencoder problem: in, Indicates that the visual features Mapping to low-dimensional latent feature space The projection matrix is ​​denoted as the visual autoencoder; represents the visual self-decoder; The constraints of the visual autoencoder problem are scaled into a first objective function, and the first-order derivative of the first objective function is set to 0, and the optimal solution of the visual autoencoder is obtained by the Bartels-Stewart algorithm.

5. The zero-shot image classification method according to claim 1, wherein: The supervised learning is performed based on the potential low-dimensional features to train an encoder that projects the semantic features to the low-dimensional features, and the decoder that projects the low-dimensional features to the semantic features is trained, including: Using the extracted low-dimensional latent feature vector as supervision, a semantic autoencoder and semantic self-decoder are constructed; Under the semantic autoencoder mapping, the semantic feature vector is projected into a low-dimensional latent feature space; Under the semantic self-decoder mapping, the low-dimensional latent feature space is restored to the semantic feature space.

6. The zero-shot image classification method according to claim 5, wherein: The method uses the extracted low-dimensional latent feature vector as supervision to construct a semantic autoencoder and a semantic decoder, including: Using the low-dimensional feature space as supervision, optimize the following semantic autoencoder problem: in, Indicates that the semantic features Mapping to low-dimensional latent feature space The projection matrix is ​​denoted as semantic autoencoder; Represents a semantic self-decoder; The constraints of the semantic autoencoder problem are scaled into a second objective function, and the first-order derivative of the second objective function is set to 0, and the optimal solution of the semantic autoencoder is obtained by the Bartels-Stewart algorithm.

7. The zero-shot image classification method according to claim 1, wherein: Based on the visual autoencoder and semantic autoencoder, a multi-level semantic encoder is constructed to characterize the mapping relationship between the dimension size, classification accuracy, and semantic loss amount of different levels, including: First-level category judgment: project the semantic representation of each category into the visual space and make category judgments on the test visual sample vectors; Second-level category decision: Projecting the test visual sample vector into a low-dimensional space, and projecting the semantic representation of each category into the low-dimensional space to make a category decision; The third level category decision: the test visual sample vector is projected into the semantic space to make a category decision.

8. A zero-shot image classification device, configured to implement the zero-shot image classification method according to any one of claims 1 to 7, characterized in that: include: Latent low-dimensional feature module, which is used to project visual features and semantic features into a common space based on the conditional main label space conversion algorithm to learn latent low-dimensional features; a visual feature encoding and decoding module, configured to perform supervised learning based on the potential low-dimensional features, train an encoder for projecting the visual features into low-dimensional features, and train a decoder for projecting the low-dimensional features into the visual features; A semantic feature encoding and decoding module, configured to perform supervised learning based on the potential low-dimensional features, train an encoder that projects the semantic features to the low-dimensional features, and train a decoder that projects the low-dimensional features to the semantic features; The category judgment module is used to make category judgments in the visual feature space, semantic feature space and potential common low-dimensional feature space based on the multi-level semantic coding image classification model, and output the image classification results.

9. A terminal, characterized in that: include: A processor and a memory, wherein the memory stores a zero-shot image classification program, and when the zero-shot image classification program is executed by the processor, it is used to implement the operation of the zero-shot image classification method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and the storage medium stores a zero-sample image classification program. When the zero-sample image classification program is executed by a processor, it is used to implement the operation of the zero-sample image classification method according to any one of claims 1 to 7.

Images