Image classification method, device, terminal and computer readable storage medium

Abstract

The present invention provides an image classification method, device, terminal and computer-readable storage medium. The image classification method includes: obtaining a plurality of image data corresponding to a basic classification model and a newly added category; the basic classification model supports classification of each preset category in a preset category library; the newly added category is different from each preset category; extracting feature information of each image data based on the basic classification model, determining a feature vector of the newly added category; constructing an incremental classification model based on the feature vector of the newly added category and the preset category library in the basic classification model; classifying the image to be classified through the incremental classification model to obtain a classification result of the image to be classified, and the classification result of the image to be classified is any preset category or any newly added category. Compared with the image classification model in the prior art, the incremental classification model in the present application does not require sample images for training, and can realize rapid support for new categories.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to an image classification method, device, terminal and computer-readable storage medium. Background Art

[0002] Incremental learning refers to the ability of a system to gradually update and optimize a model by accumulating new data without retraining the entire model. This learning method is important for coping with ever-changing data streams and dynamic environments, avoiding the tedious process of training from scratch every time new data appears.

[0003] In machine learning, a special case of incremental learning is class incremental learning. Class incremental learning focuses on the ability of the model to effectively integrate new knowledge without forgetting existing category knowledge when faced with data of new categories. Unlike traditional batch learning, class incremental learning is more in line with the situation where new categories are constantly emerging in the real world. For example, some new object categories may be added in image classification tasks, and new pronunciation patterns may appear in speech recognition tasks. However, existing incremental learning requires the use of new knowledge for training, and it is difficult to achieve rapid support for new categories. Summary of the invention

[0004] The main technical problem solved by the present invention is to provide an image classification method, device, terminal and computer-readable storage medium to solve the problem that the prior art cannot quickly support new categories.

[0005] In order to solve the above technical problems, the first technical solution adopted by the present invention is: to provide an image classification method, the image classification method comprising:

[0006] Obtain multiple image data corresponding to the basic classification model and the newly added categories; the basic classification model supports the classification of each preset category in the preset category library; the newly added categories are different from each preset category;

[0007] Extract feature information of each image data based on the basic classification model and determine feature vectors of newly added categories;

[0008] An incremental classification model is constructed based on the feature vectors of the newly added categories and the preset category library in the basic classification model;

[0009] The image to be classified is classified by using the incremental classification model to obtain a classification result of the image to be classified, and the classification result of the image to be classified is any preset category or any newly added category.

[0010] Among them, the basic classification model includes a feature extraction layer;

[0011] Based on the basic classification model, feature information of each image data is extracted to determine the feature vector of the newly added category, including:

[0012] Based on the feature extraction layer, feature extraction is performed on each image data to obtain feature information of the image data;

[0013] The feature information of all image data corresponding to the newly added category is averaged to obtain the feature vector of the newly added category.

[0014] Among them, the basic classification model also includes a classification layer; the incremental classification model includes a feature extraction layer and an incremental classification layer;

[0015] Based on the feature vectors of the newly added categories and the preset category library in the basic classification model, an incremental classification model is constructed, including:

[0016] The feature extraction layer of the basic classification model is used as the feature extraction layer of the incremental classification model;

[0017] The feature vector of the newly added category is added to the classification layer of the basic classification model to generate an incremental classification layer of the incremental classification model; the output dimension of the incremental classification layer is the sum of the total number of preset categories in the preset category library and the number of newly added categories.

[0018] Among them, the incremental classification model includes a feature extraction layer, an incremental classification layer, and a category embedding layer;

[0019] The image to be classified is classified by the incremental classification model to obtain the classification result of the image to be classified, including:

[0020] The feature extraction layer is used to extract features of the image to be classified to obtain feature information of the image to be classified;

[0021] A category embedding layer is used to encode each preset category and each newly added category, and a feature vector corresponding to each preset category and each newly added category is obtained;

[0022] The incremental classification layer is used to compare the feature information of the image to be classified with the feature vectors corresponding to each preset category and each newly added category to determine the classification result of the image to be classified.

[0023] Among them, the training method of the basic classification model includes:

[0024] A training image set is obtained, wherein the training image set includes a plurality of sample images; each sample image has a corresponding category label; the category labels included in the training image set are part of the preset categories in the preset category library; each category label and the preset category respectively has a corresponding feature vector;

[0025] Extract and identify features of the sample image through the initial classification model to obtain feature information of the sample image and the predicted probability that the sample image belongs to each preset category;

[0026] Determine a final error value of the training image set based on feature information corresponding to each sample image, a predicted probability that each sample image belongs to each preset category, and feature vectors corresponding to the category label and the preset category respectively;

[0027] The initial classification model is trained based on the final error value of the training image set to obtain a basic classification model.

[0028] Wherein, based on the feature information corresponding to each sample image, the predicted probability that each sample image belongs to each preset category, and the feature vectors corresponding to the category label and the preset category, the final error value of the training image set is determined, including:

[0029] Determine the local loss value of the training image set based on the similarity between the feature information corresponding to each sample image and the feature vector of each category label in the training image set, and the category label of each sample image;

[0030] Determine a global loss value of the training image set based on the similarity between the feature information corresponding to each sample image and the feature vectors of each preset category in the preset category library and the category label of each sample image;

[0031] Determine the divergence loss value of the training image set based on the similarity between the feature information of each sample image and the feature vectors of each preset category in the preset category library and the predicted probability that each sample image belongs to each preset category;

[0032] Based on the local loss value, global loss value and divergence loss value corresponding to the training image set, the final error value of the training image set is determined.

[0033] Wherein, based on the similarity between the feature information corresponding to each sample image and the feature vector of each category label in the training image set, and the category label of each sample image, the local loss value of the training image set is determined, including:

[0034] Determine a first error value of the sample image based on the similarity between the feature information of the sample image and the feature vectors of each category label in the training image set and the difference between the category labels of the corresponding sample image;

[0035] Adding and averaging the first error values ​​of the sample images in the training image set to obtain a first mean error of the training image set;

[0036] Determine a second error value of the category label based on the similarity between the feature vector of the category label and the feature information of each sample image in the training image set and the difference between the sample images marked with the category label in the training image set;

[0037] The second error values ​​of the category labels corresponding to the training image set are added and averaged to obtain a second mean error of the training image set;

[0038] Based on the first mean error and the second mean error corresponding to the training image set, a local loss value of the training image set is determined.

[0039] In order to solve the above technical problems, the second technical solution adopted by the present invention is: to provide an image classification device, the image classification device comprising:

[0040] The acquisition module is used to obtain multiple image data corresponding to the basic classification model and the newly added categories; the basic classification model supports the classification of each preset category in the preset category library; the newly added category is different from each preset category;

[0041] A feature extraction module is used to extract feature information of each image data based on the basic classification model and determine the feature vector of the newly added category;

[0042] A construction module, used to construct an incremental classification model based on feature vectors of newly added categories and a preset category library in a basic classification model;

[0043] The classification module is used to classify the image to be classified through the incremental classification model to obtain the classification result of the image to be classified, and the classification result of the image to be classified is any preset category or any newly added category.

[0044] To solve the above technical problems, the third technical solution adopted by the present invention is: to provide a terminal, the terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, the processor is used to execute program data to implement the steps in the above-mentioned image classification method.

[0045] In order to solve the above technical problems, the fourth technical solution adopted by the present invention is: providing a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned image classification method are implemented.

[0046] The beneficial effects of the present invention are as follows: different from the prior art, an image classification method, device, terminal and computer-readable storage medium are provided, and the image classification method includes: obtaining a basic classification model and multiple image data corresponding to the newly added category; the basic classification model supports the classification of each preset category in the preset category library; the newly added category is different from each preset category; extracting feature information of each image data based on the basic classification model, and determining the feature vector of the newly added category; constructing an incremental classification model based on the feature vector of the newly added category and the preset category library in the basic classification model; performing image classification on the image to be classified through the incremental classification model to obtain the classification result of the image to be classified, and the classification result of the image to be classified is any preset category or any newly added category. The present application extracts the features of the image data corresponding to the newly added category through the basic classification model, and then obtains the feature vector of the newly added category; the feature vector corresponding to the newly added category and the feature vector of the preset category in the basic classification model are saved, so that the incremental classification model can identify the preset category and the newly added category. Compared with the image classification model in the prior art, the incremental classification model in the present application does not require sample images for training, and can achieve rapid support for new categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is a flowchart of the image classification method provided by the present invention;

[0049] Figure 2 It is a flowchart of a specific embodiment of the basic classification model training method provided by the present invention;

[0050] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step S12 in the provided basic classification model training method;

[0051] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step S123 in the basic classification model training method provided;

[0052] Figure 5 It is a schematic diagram of the framework of an embodiment of an image classification device provided by the present invention;

[0053] Figure 6 It is a schematic diagram of the framework of a terminal embodiment provided by the present invention;

[0054] Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0055] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0056] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0057] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0058] In order to enable those skilled in the art to better understand the technical solution of the present invention, an image classification method provided by the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0061] Existing neural network models often encounter "catastrophic forgetting" when incrementally learning new knowledge. After learning a new task, the model's performance on the old task degrades catastrophically.

[0062] The image classification method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. In some embodiments, the terminal or server can implement the image classification method provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports virtual scenes, such as a game APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0063] The image classification method provided in the embodiment of the present application is explained below using a server implementation as an example.

[0064] See also Figure 1 , Figure 1 It is a flowchart of the image classification method provided by the present invention.

[0065] This embodiment provides an image classification method, which includes the following steps.

[0066] S1: Acquire multiple image data corresponding to the basic classification model and the newly added category; the basic classification model supports the classification of each preset category in the preset category library; the newly added category is different from each preset category.

[0067] S2: Extract feature information of each image data based on the basic classification model and determine the feature vector of the newly added category.

[0068] S3: Build an incremental classification model based on the feature vectors of the newly added categories and the preset category library in the basic classification model.

[0069] S4: performing image classification on the image to be classified by using the incremental classification model to obtain a classification result of the image to be classified, where the classification result of the image to be classified is any preset category or any newly added category.

[0070] In one embodiment, the training method of the basic classification model includes the following embodiments.

[0071] See also Figure 2 , Figure 2 It is a flowchart of a specific embodiment of the basic classification model training method provided by the present invention.

[0072] S11: Build an initial classification model.

[0073] In one embodiment, the initial classification model includes a feature extraction layer and a classification layer cascaded in sequence, wherein the feature extraction layer may be a convolutional neural network (CNN) or a visual transformer (VIT) model.

[0074] In one embodiment, an initial feature extraction layer is obtained, where the initial feature extraction layer has an initialized model parameter.

[0075] When the initial feature extraction layer is CNN, CNN is pre-trained using contrast learning methods such as Momentum Contrast (MoCo) and SimCLR (Simplified Contrastive Learning with Large Datasets). When the initial feature extraction layer is VIT, VIT is pre-trained using self-supervisory methods such as MAE. The initial feature extraction layer is pre-trained using the above-mentioned self-supervisory method to obtain a pre-trained feature extraction layer, and the pre-trained feature extraction layer has a first weight parameter. The purpose of pre-training the feature extraction layer is to use a large amount of unlabeled image data to initialize the model parameters of the initial feature extraction layer, thereby improving the generalization ability of the feature extraction layer. In one embodiment, the weight of the feature extraction layer obtained from the last round of training is used as the first weight parameter.

[0076] MoCo is a contrastive learning method for self-supervised learning, which improves the learning efficiency of the model and the quality of features by maintaining a dynamically updated queue to provide a large number of negative samples.

[0077] Masked Autoencoder (MAE), a technique for self-supervised learning, is particularly used in the field of computer vision to learn image representations. MAE is a method based on the Transformer architecture that learns image features by masking parts of the input image region and trying to reconstruct these masked parts.

[0078] The pre-trained feature extraction layer is connected to the classification layer to obtain an initial classification model. In one embodiment, the initial classification model also includes a pooling layer, and the feature extraction layer and the classification layer are connected through the pooling layer. The feature information output by the feature extraction layer is subjected to dimensionality reduction processing by the pooling layer and then input into the classification layer. In one embodiment, the pooling layer can also perform dimensionality reduction processing on the features of the preset category. The weight of the feature extraction layer in the initial classification model is the first weight parameter of the pre-trained feature extraction layer.

[0079] In one embodiment, the initial classification model further includes a category embedding layer, which is used to map discrete input items into learnable vector representations.

[0080] S12: Train the initial classification model to obtain a basic classification model.

[0081] Specifically, the initial classification model is trained through the following steps, and the training enables the initial classification model to support the recognition of preset categories in the preset category library.

[0082] See also Figure 3 , Figure 3 yes Figure 2 A flowchart of a specific embodiment of step S12 in the basic classification model training method is provided.

[0083] S121: Obtain a training image set, the training image set includes multiple sample images; each sample image has a corresponding category label; the category labels included in the training image set are part of the preset categories in the preset category library; each category label and preset category has a corresponding feature vector.

[0084] In one embodiment, a plurality of training image sets are obtained, and the training image sets include a plurality of sample images. Sample images refer to images for which classification is required, for example, animal images for which the classification of animals in the images needs to be determined; the sample images include target objects, and the target objects refer to the subjects in the sample images, such as the subject of a tree in a tree image as the target object. The label category corresponding to the sample image may be the category of the target object. The category labels included in each training image set are part of the preset categories in the preset category library. The category labels included in the plurality of training image sets for training the initial classification model cover all preset categories in the preset category library.

[0085] In one embodiment, the category label and the preset category are encoded by the category embedding layer to obtain feature vectors corresponding to the category label and the preset category. The feature vectors of all category labels contained in the training image set can be The dimension of the feature vector of all category labels contained in the training image set is (b, dim), where dim is the dimension of the feature vector and b is the number of sample images contained in the training image set. The feature vectors of all preset categories in the preset category library can be . The dimension of the feature vector of the preset category in the preset category library is (num_classes, dim), dim is the dimension of the feature vector, and num_classes is the number of preset categories in the preset category library.

[0086] S122: Extracting and identifying features of the sample image through the initial classification model to obtain feature information of the sample image and the predicted probability that the sample image belongs to each preset category.

[0087] In one embodiment, the initial classification model includes a feature extraction layer, a classification layer, and a category embedding layer.

[0088] In a specific embodiment, a feature extraction layer is used to extract features of a sample image to obtain feature information of the sample image; a category embedding layer is used to encode each preset category separately to obtain a feature vector of each preset category; a classification layer is used to compare the feature information of the sample image with the feature vector of each preset category to determine the predicted probability that the sample image belongs to each preset category, and the dimension of the predicted probability that the sample image belongs to each preset category is (b, num_classes).

[0089] Through the above steps, the predicted probability that each sample image in the training image set belongs to each preset category can be obtained.

[0090] S123: Determine a final error value of the training image set based on feature information corresponding to each sample image, the predicted probability that each sample image belongs to each preset category, and the category label and the feature vectors corresponding to the preset category.

[0091] See also Figure 4 , Figure 4 yes Figure 3 A flowchart of a specific embodiment of step S123 in the basic classification model training method is provided.

[0092] S1231: Determine the local loss value of the training image set based on the similarity between the feature information corresponding to each sample image and the feature vectors of each category label in the training image set, and the category label of each sample image.

[0093] In one embodiment, the first error value of the sample image is determined based on the similarity between the feature information of the sample image and the feature vectors of each category label in the training image set and the difference between the category labels of the corresponding sample image.

[0094] In a specific embodiment, the similarity between the feature information of the sample image and the feature vectors of each category label in the training image set is calculated based on the following formula.

[0095] (Formula 1)

[0096] Where: Represents the similarity between the sample image and each category label in the training image set; Represents the characteristic information of the sample image; A feature vector representing the class labels in the training image set.

[0097] The similarities between the sample image and each category label in the training image set and the category label of the sample image are compared to determine a first error value of the sample image.

[0098] The first error values ​​of the sample images in the training image set are added and averaged to obtain a first mean error of the training image set.

[0099] In one embodiment, the second error value of the category label is determined based on the similarity between the feature vector of the category label and the feature information of each sample image in the training image set and the difference between the sample images marked with the category label in the training image set.

[0100] In a specific embodiment, the similarity between the feature vector of the category label and the feature information of each sample image in the training image set is calculated based on the following formula.

[0101] (Formula 2)

[0102] Where: Represents the similarity between the category label and each sample image in the training image set.

[0103] The similarity between the category label and each sample image in the training image set and the category label of the sample image are compared to determine a second error value of the sample image.

[0104] The second error values ​​of the category labels corresponding to the training image set are added and averaged to obtain the second mean error of the training image set.

[0105] Based on the first mean error and the second mean error corresponding to the training image set, a local loss value of the training image set is determined.

[0106] In a specific embodiment, the local loss value of the training image set is determined by adding and averaging the first mean error and the second mean error corresponding to the training image set.

[0107] S1232: Determine a global loss value of the training image set based on the similarity between the feature information corresponding to each sample image and the feature vectors of each preset category in the preset category library, and the category label of each sample image.

[0108] In one embodiment, the similarity between the feature information corresponding to the sample image and the feature vectors of each preset category in the preset category library is calculated based on the following formula.

[0109] (Formula 3)

[0110] Where: Indicates the similarity between the sample image and each preset category in the preset category library; Represents the characteristic information of the sample image; The feature vector representing each preset category in the preset category library.

[0111] In one embodiment, the global loss values ​​corresponding to all sample images in the training image set are added and averaged to obtain the global loss value corresponding to the training image set.

[0112] S1233: Determine the divergence loss value of the training image set based on the similarity between the feature information of each sample image and the feature vectors of each preset category in the preset category library and the predicted probability that each sample image belongs to each preset category.

[0113] In one embodiment, the divergence loss value of the sample image is determined by the similarity between the sample image and each preset category in the preset category library and the difference between the predicted probabilities that the sample image belongs to each preset category.

[0114] In one embodiment, the divergence loss values ​​corresponding to all sample images in the training image set are added and averaged to obtain the divergence loss value corresponding to the training image set.

[0115] S1234: Determine a final error value of the training image set based on the local loss value, global loss value, and divergence loss value corresponding to the training image set.

[0116] In one embodiment, a weighted sum is performed on the local loss value, the global loss value, and the divergence loss value corresponding to the training image set to obtain a final error value of the training image set.

[0117] S124: Training the initial classification model based on the final error value of the training image set to obtain a basic classification model.

[0118] In one embodiment, the weights in the feature extraction layer, classification layer, and category embedding layer in the initial classification model are trained by the final error value of the training image set. When the number of training times reaches the preset number, the weights of the feature extraction layer, classification layer, and category embedding layer after the last training are retained, and the trained initial classification model is used as the basic classification model. The weight of the feature extraction layer in the basic classification model is the second weight parameter. The basic classification model supports the classification of all preset categories in the preset database.

[0119] Specifically, the specific implementation method of obtaining the basic classification model and multiple image data corresponding to the newly added categories in step S1 is as follows.

[0120] In one embodiment, the basic classification model supports classification of each preset category in the preset category library, and the newly added category is different from all the preset categories in the preset category library.

[0121] The label categories of multiple images corresponding to the newly added category are all newly added categories. There can be one or more newly added categories. Each newly added category corresponds to an image set, and the image set contains multiple image data. The category of the image data in the same image set is the newly added category.

[0122] Specifically, in step S2, the feature information of each image data is extracted based on the basic classification model, and a specific implementation method for determining the feature vector of the newly added category is as follows.

[0123] In one embodiment, feature extraction is performed on each image data based on the feature extraction layer in the basic classification model to obtain feature information of the image data; the feature information of all image data corresponding to the newly added category is averaged to obtain a feature vector of the newly added category.

[0124] In one embodiment, feature extraction is performed on each image data based on the feature extraction layer in the basic classification model to obtain feature information of the image data; and the most representative feature information is selected from the feature information of all image data corresponding to the newly added category as the feature vector of the newly added category.

[0125] Specifically, in step S3, a specific implementation method of constructing an incremental classification model based on the feature vector of the newly added category and the preset category library in the basic classification model is as follows.

[0126] In one embodiment, the feature extraction layer of the basic classification model is used as the feature extraction layer of the incremental classification model; the feature vector of the newly added category is added to the classification layer of the basic classification model to generate the incremental classification layer of the incremental classification model. The output dimension of the incremental classification layer is the sum of the total number of preset categories in the preset category library and the number of newly added categories.

[0127] In one embodiment, the feature extraction layer of the basic classification model is used as the feature extraction layer of the incremental classification model. The feature vectors of the preset categories in the preset category library corresponding to the basic classification model are copied to the incremental classification layer, and the feature vectors of the newly added categories are added to the incremental classification layer, so that the incremental classification layer can recognize the newly added categories and the preset categories in the preset category library.

[0128] Specifically, in step S4, the image to be classified is classified by using the incremental classification model to obtain the classification result of the image to be classified. The specific implementation method is as follows.

[0129] In one embodiment, an image to be classified is obtained, and the image to be classified is an image with classification requirements. For example, the image to be classified may be an image containing animals or an image containing plants. The image to be classified contains a target object, and the target object refers to the subject in the image to be classified, such as the tree subject in a tree image is the target object.

[0130] The image to be classified is classified by the incremental classification model to obtain the classification result of the image to be classified. The incremental classification model includes a feature extraction layer, an incremental classification layer, and a category embedding layer. In this embodiment, the category embedding layer in the incremental classification model is the category embedding layer in the basic classification model.

[0131] Specifically, the weight of the feature extraction layer in the incremental classification model is the second weight parameter. The feature extraction layer corresponding to the second weight parameter is used to extract features of the image to be classified to obtain feature information of the image to be classified. Specifically, the feature information of the image to be classified is compared with the feature vectors corresponding to the newly added category and the preset category respectively through the incremental classification layer to determine the classification result of the image to be classified. The classification result of the image to be classified is any preset category or any newly added category.

[0132] In one embodiment, a feature extraction layer is used to extract features of an image to be classified to obtain feature information of the image to be classified; a category embedding layer is used to encode each preset category and each newly added category to obtain feature vectors corresponding to each preset category and each newly added category; an incremental classification layer is used to compare the feature information of the image to be classified with the feature vectors corresponding to each preset category and each newly added category to determine the classification result of the image to be classified.

[0133] Specifically, the feature similarity between the feature information of the image to be classified and the feature vectors corresponding to each preset category and each newly added category is calculated.

[0134] The preset category or newly added category corresponding to the feature similarity with the maximum value is selected as the classification category of the image to be classified.

[0135] In another embodiment, in response to the feature similarity being greater than the similarity threshold, the preset categories and newly added categories corresponding to the feature similarities greater than the similarity threshold are displayed to the user for the user to select.

[0136] The image classification method provided in this embodiment extracts the features of the image data corresponding to the newly added category through the basic classification model, and then obtains the feature vector of the newly added category; the feature vector corresponding to the newly added category and the feature vector of the preset category in the basic classification model are saved, so that the incremental classification model can recognize the preset category and the newly added category. Compared with the image classification model in the prior art, the incremental classification model in the present application does not require additional training of sample images, and can achieve rapid support for new categories.

[0137] See also Figure 5 , Figure 5 It is a schematic diagram of the framework of an embodiment of the image classification device provided by the present invention.

[0138] This embodiment provides an image classification device 60 , which includes an acquisition module 61 , a feature extraction module 62 , a construction module 63 and a classification module 64 .

[0139] The acquisition module 61 is used to acquire multiple image data corresponding to the basic classification model and the newly added category; the basic classification model supports the classification of each preset category in the preset category library; the newly added category is different from each preset category.

[0140] The feature extraction module 62 is used to extract feature information of each image data based on the basic classification model and determine the feature vector of the newly added category.

[0141] The construction module 63 is used to construct an incremental classification model based on the feature vector of the newly added category and the preset category library in the basic classification model.

[0142] The classification module 64 is used to classify the image to be classified by using an incremental classification model to obtain a classification result of the image to be classified. The classification result of the image to be classified is any preset category or any newly added category.

[0143] The image classification device provided in this embodiment extracts the features of the image data corresponding to the newly added category through the basic classification model, and then obtains the feature vector of the newly added category; the feature vector corresponding to the newly added category and the feature vector of the preset category in the basic classification model are saved, so that the incremental classification model can identify the preset category and the newly added category. Compared with the image classification model in the prior art, the incremental classification model in the present application does not require sample images for training, and can achieve rapid support for new categories.

[0144] See also Figure 6 , Figure 6 It is a schematic diagram of the framework of a terminal embodiment provided by the present invention.

[0145] The terminal 80 includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is used to execute the program instructions stored in the memory 81 to implement the steps of any of the above-mentioned image classification method embodiments. In a specific implementation scenario, the terminal 80 may include but is not limited to: a microcomputer, a server, and in addition, the terminal 80 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0146] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps of any of the above-mentioned image classification method embodiments. The processor 82 can also be called a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 82 can be implemented by an integrated circuit chip.

[0147] The above scheme is an image classification method, and the image classification method includes:

[0148] Obtain multiple image data corresponding to the basic classification model and the newly added categories; the basic classification model supports the classification of each preset category in the preset category library; the newly added categories are different from each preset category;

[0149] Extract feature information of each image data based on the basic classification model and determine feature vectors of newly added categories;

[0150] An incremental classification model is constructed based on the feature vectors of the newly added categories and the preset category library in the basic classification model;

[0151] The image to be classified is classified by using the incremental classification model to obtain a classification result of the image to be classified, and the classification result of the image to be classified is any preset category or any newly added category.

[0152] See also Figure 7 , Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present invention.

[0153] The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor, and the program instructions 901 are used to implement the steps of any of the above-mentioned image classification method embodiments.

[0154] The above scheme is an image classification method, and the image classification method includes:

[0155] Obtain multiple image data corresponding to the basic classification model and the newly added categories; the basic classification model supports the classification of each preset category in the preset category library; the newly added categories are different from each preset category;

[0156] Extract feature information of each image data based on the basic classification model and determine feature vectors of newly added categories;

[0157] An incremental classification model is constructed based on the feature vectors of the newly added categories and the preset category library in the basic classification model;

[0158] The image to be classified is classified by using the incremental classification model to obtain a classification result of the image to be classified, and the classification result of the image to be classified is any preset category or any newly added category.

[0159] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0160] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0161] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0164] The above are only implementation modes of the present invention, and are not intended to limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image classification method, characterized in that: The method comprises: Acquire multiple image data corresponding to a basic classification model and a newly added category; the basic classification model supports classification of each preset category in a preset category library; the newly added category is different from each preset category; Extracting feature information of each of the image data based on the basic classification model, and determining a feature vector of the newly added category; Building an incremental classification model based on the feature vector of the newly added category and the preset category library in the basic classification model; Performing image classification on the image to be classified by using the incremental classification model to obtain a classification result of the image to be classified, wherein the classification result of the image to be classified is any of the preset categories or any of the newly added categories; Wherein, the basic classification model includes a feature extraction layer; The extracting feature information of each of the image data based on the basic classification model to determine the feature vector of the newly added category includes: Performing feature extraction on each of the image data based on the feature extraction layer to obtain feature information of the image data; The feature information of all the image data corresponding to the newly added category is averaged to obtain a feature vector of the newly added category.

2. The image classification method according to claim 1, characterized in that: The basic classification model also includes a classification layer; the incremental classification model includes a feature extraction layer and an incremental classification layer; The step of constructing an incremental classification model based on the feature vector of the newly added category and the preset category library in the basic classification model includes: Using the feature extraction layer of the basic classification model as the feature extraction layer of the incremental classification model; The feature vector of the newly added category is added to the classification layer of the basic classification model to generate an incremental classification layer of the incremental classification model; the output dimension of the incremental classification layer is the sum of the total number of preset categories in the preset category library and the number of the newly added categories.

3. The image classification method according to claim 1, characterized in that: The incremental classification model includes a feature extraction layer, an incremental classification layer, and a category embedding layer; The step of performing image classification on the image to be classified by using the incremental classification model to obtain a classification result of the image to be classified includes: Using the feature extraction layer to extract features from the image to be classified to obtain feature information of the image to be classified; Using the category embedding layer to encode each of the preset categories and the newly added categories respectively, to obtain feature vectors corresponding to each of the preset categories and each of the newly added categories respectively; The incremental classification layer is used to compare the feature information of the image to be classified with the feature vectors corresponding to each of the preset categories and each of the newly added categories, so as to determine the classification result of the image to be classified.

4. The image classification method according to claim 1, characterized in that: The training method of the basic classification model includes: Acquire a training image set, the training image set comprising a plurality of sample images; each of the sample images has a corresponding category label; the category labels contained in the training image set are part of the preset categories in the preset category library; each of the category labels and the preset categories respectively has a corresponding feature vector; Extracting and identifying features of the sample image through an initial classification model to obtain feature information of the sample image and a predicted probability that the sample image belongs to each of the preset categories; Determining a final error value of the training image set based on the feature information corresponding to each of the sample images, the predicted probability that each of the sample images belongs to each of the preset categories, and the category labels and the feature vectors corresponding to the preset categories respectively; The initial classification model is trained based on the final error value of the training image set to obtain the basic classification model.

5. The image classification method according to claim 4, characterized in that: The determining of the final error value of the training image set based on the feature information corresponding to each of the sample images, the predicted probability that each of the sample images belongs to each of the preset categories, and the category labels and the feature vectors corresponding to the preset categories respectively, comprises: Determine a local loss value of the training image set based on the similarity between the feature information corresponding to each of the sample images and the feature vectors of each of the category labels in the training image set, and the category label of each of the sample images; Determine the global loss value of the training image set based on the similarity between the feature information corresponding to each of the sample images and the feature vectors of each of the preset categories in the preset category library, and the category label of each of the sample images; Determining the divergence loss value of the training image set based on the similarity between the feature information of each of the sample images and the feature vectors of each of the preset categories in the preset category library and the predicted probability that each of the sample images belongs to each of the preset categories; Based on the local loss value, the global loss value, and the divergence loss value corresponding to the training image set, a final error value of the training image set is determined.

6. The image classification method according to claim 5, characterized in that: The determining the local loss value of the training image set based on the similarity between the feature information corresponding to each of the sample images and the feature vectors of each of the category labels in the training image set, and the category label of each of the sample images, comprises: Determine a first error value of the sample image based on the similarity between the feature information of the sample image and the feature vector of each of the category labels in the training image set and the difference between the corresponding category labels of the sample image; Adding and averaging the first error values ​​of the sample images in the training image set to obtain a first mean error of the training image set; Determine a second error value of the category label based on the similarity between the feature vector of the category label and the feature information of each of the sample images in the training image set and the difference between the sample images marked with the category label in the training image set; Adding and averaging the second error values ​​of the category labels corresponding to the training image set to obtain a second mean error of the training image set; Based on the first mean error and the second mean error corresponding to the training image set, a local loss value of the training image set is determined.

7. An image classification device, characterized in that: The image classification device comprises: An acquisition module, used to acquire multiple image data corresponding to a basic classification model and a newly added category; the basic classification model supports classification of each preset category in a preset category library; the newly added category is different from each preset category; wherein the basic classification model includes a feature extraction layer; a feature extraction module, configured to extract feature information of each of the image data based on the basic classification model, and determine a feature vector of the newly added category; further configured to extract features of each of the image data based on the feature extraction layer, and obtain feature information of the image data; and average the feature information of all the image data corresponding to the newly added category, and obtain a feature vector of the newly added category; A construction module, used to construct an incremental classification model based on the feature vector of the newly added category and the preset category library in the basic classification model; The classification module is used to classify the image to be classified by using the incremental classification model to obtain a classification result of the image to be classified, where the classification result of the image to be classified is any of the preset categories or any of the newly added categories.

8. A terminal, characterized in that: The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor is used to execute program data to implement the steps in the image classification method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the image classification method according to any one of claims 1 to 6 are implemented.

Images