Image classification method, image classification model training method and device

Abstract

The present application relates to image recognition technology in the field of artificial intelligence, and provides an image classification method, a training method for an image classification model, and a device thereof. It relates to the field of artificial intelligence, and specifically to the field of computer vision. The method comprises: obtaining an image to be processed; classifying the image to be processed according to a preset global category feature, and obtaining a classification result of the image to be processed, wherein the global category feature comprises a plurality of category features obtained by training based on a plurality of training images in a training set, and the plurality of category features in the global category feature are used to indicate the visual features of all categories in the training set, and all categories in the training set are categories to which all training images in the training set belong, and the training set comprises images in a base class and images in a new class. The image classification method in the embodiment of the present application can better classify images.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to an image classification method, a training method for an image classification model, and a device thereof. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a science about how to use cameras / camcorders and computers to obtain the data and information of the objects we need. Figuratively speaking, it is to install eyes (cameras / camcorders) and brains (algorithms) on computers to replace human eyes to identify, track and measure targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0003] Image (or picture) classification is the basis of various image processing applications. Computer vision often involves the problem of how to classify acquired images. With the rapid development of machine learning (or deep learning), machine learning algorithms have been more and more widely used in image classification processing, and have achieved very good results. However, machine learning algorithms are highly dependent on a large amount of labeled training data, and in many cases it is very difficult to obtain data.

[0004] Therefore, how to better classify images when training data is insufficient is an urgent problem to be solved. Summary of the invention

[0005] The present application provides an image classification method, an image classification model training method and a device thereof, which can better classify images.

[0006] In a first aspect, an image classification method is provided, the method comprising: acquiring an image to be processed; classifying the image to be processed according to a preset global category feature to obtain a classification result of the image to be processed, wherein the global category feature comprises a plurality of category features trained according to a plurality of training images in a training set, the plurality of category features in the global category feature are used to indicate visual features of all categories in the training set, all categories in the training set are categories to which all training images in the training set belong, and the training set comprises images in a base class and images in a new class.

[0007] The above base class can be understood as a large-scale training image set. The above base class usually includes a large number of annotated images for model training. The images in the base class can be annotated images. The annotated images here can refer to images that have been annotated with the category to which the image belongs.

[0008] Correspondingly, compared with the above base class, the above new class usually includes a small number of annotated samples, and the images in the new class can also be annotated images. That is, in the embodiment of the present application, the new class is a small sample, that is, the new class includes a small number of images that have been annotated to the category to which it belongs.

[0009] For example, the base class includes 100 categories, each category includes 1000 training images, and the new class includes 5 categories, each category includes 5 training images.

[0010] In the present application, global category features are obtained by training multiple training images in a training set. The global category features include multiple category features that can indicate visual features corresponding to all categories in the training set. At the same time, since the training set used in the global category training process includes images in the base class and images in the new class, the global category features can be avoided from overfitting to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0011] Optionally, the image classification model in the present application can be trained based on a multi-stage training strategy. For example, the model training process can be divided into multiple training episodes, and in each training episode, several categories in the training set can be randomly selected to train the model. Finally, after multiple training episodes, the model training is completed.

[0012] Specifically, the global category features can be updated in multiple training stages during the model training process, so that the global category features obtained through training can have better consistency. At the same time, the training set includes images in the base class and images in the new class. The effect of the new class (training images in the new class) during the training process can be continuously accumulated with the global category features. Therefore, it is possible to avoid the trained image classification model from overfitting to the base class. By classifying the images to be processed according to the image classification model, better image classification results can be obtained.

[0013] In combination with the first aspect, in certain implementations of the first aspect, the image to be processed is classified according to a preset global category feature to obtain a classification result of the image to be processed, including: extracting a feature vector of the image to be processed; determining, based on the feature vector of the image to be processed, a confidence level that the image to be processed belongs to a candidate category, wherein the candidate category is one or more of a plurality of categories indicated by the global category feature; and determining, based on the confidence level, a classification result of the image to be processed from the candidate categories.

[0014] In combination with the first aspect, in certain implementations of the first aspect, before determining the confidence that the image to be processed belongs to a candidate category based on the feature vector of the image to be processed, the method also includes: determining the local category features of the image to be processed based on the support set of the image to be processed; determining the candidate category based on the local category features of the image to be processed and the global category features; wherein the support set of the image to be processed includes multiple images, and the categories to which the multiple images belong are one or more of the multiple categories indicated by the global category features.

[0015] In combination with the first aspect, in certain implementations of the first aspect, determining the confidence that the image to be processed belongs to a candidate category based on the feature vector of the image to be processed includes: determining, based on the feature vector of the image to be processed, the distance between the feature vector of the image to be processed and the feature vector corresponding to each category in the candidate categories; and determining, based on the distance, the confidence that the image to be processed belongs to the candidate category.

[0016] In combination with the first aspect, in certain implementations of the first aspect, determining the classification result of the image to be processed from the candidate categories based on the confidence level includes: determining the category with the highest confidence level among the candidate categories as the classification result of the image to be processed.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the global category feature is obtained by training based on classification error, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated with the training images in the query set, the labels are used to indicate the category to which the training images belong, and the query set includes some training images in some categories in the training set.

[0018] In the present application, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated by the training images. The training images in the query set include images in the base class and images in the new class. Updating the global category feature based on the classification error can avoid overfitting of the global category feature to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0019] Furthermore, in multiple training stages during the model training process, the classification error of the current training stage can be used to update the global category features. At the same time, the training set includes images in the base class and images in the new class. The effect of the new class in the training process can be accumulated continuously with the global category features. Therefore, it is possible to avoid the trained image classification model from overfitting to the base class. By classifying the image to be processed according to the image classification model, better image classification results can be obtained.

[0020] In combination with the first aspect, in certain implementations of the first aspect, the global category feature is obtained by training based on the classification error and the registration error, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated with the training images in the query set, the labels are used to indicate the category to which the training image belongs, the query set includes some training images in some categories in the training set, the registration error is determined based on the local category features of the training images and multiple category features in the global category features, the local category features of the training images include multiple category features determined based on multiple training images in a support set, multiple category features in the local category features of the training images are used to indicate the visual features of all categories in the support set, and the support set includes some training images in some categories in the training set.

[0021] In the present application, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated by the training images, the training images in the query set include images in the base class and images in the new class, the registration error is determined based on the local category features of the training images and multiple category features in the global category features, the local category features of the training images include multiple category features determined based on multiple training images in the support set, and the global category features are updated based on the classification error and the registration error, which can avoid overfitting of the global category features to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0022] Furthermore, in multiple training stages during the model training process, the classification error and registration error of the current training stage can be used to update the global category features. At the same time, the training set includes images in the base class and images in the new class. The effect of the new class in the training process can be accumulated continuously with the global category features. Therefore, it is possible to avoid the trained image classification model from overfitting to the base class. By classifying the image to be processed according to the image classification model, better image classification results can be obtained.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the local category features of the training image are determined by multiple training images in the support set that have been expanded, and the expansion processing includes cropping, flipping and / or data illusion processing of the image.

[0024] Many of the training images in the support set above are images of the new class.

[0025] In the present application, multiple training images in the new class are expanded through cropping, flipping and / or data illusion processing to increase the number of training images in the new class. Therefore, it is possible to avoid overfitting of the trained image classification model to the base class. By classifying the processed images according to the image classification model, better image classification results can be obtained.

[0026] In a second aspect, a training method for an image classification model is provided, the method comprising: acquiring multiple training images in a training set, wherein the training set comprises a support set and a training set, and the multiple training images comprise multiple training images in the support set and multiple training images in the query set; extracting feature vectors of the multiple training images in the query set according to a preset first neural network, the query set comprising some images in some categories in the training set; processing the feature vectors of the multiple training images in the query set according to a preset second neural network and a preset global category feature to obtain classification results of the multiple training images in the query set, wherein the global category feature comprises multiple category features, and multiple category features in the global category feature are used to indicate visual features of all categories in the training set, all categories in the training set are categories to which all training images in the training set belong, and the training set comprises images in a base class and images in a new class; updating the global category feature according to the classification results of the multiple training images in the query set.

[0027] The above base class can be understood as a large-scale training image set. The above base class usually includes a large number of annotated images for model training. The images in the base class can be annotated images. The annotated images here can refer to images that have been annotated with the category to which the image belongs.

[0028] Correspondingly, compared with the above base class, the above new class usually includes a small number of annotated samples, and the images in the new class can also be annotated images. That is, in the embodiment of the present application, the new class is a small sample, that is, the new class includes a small number of images that have been annotated to the category to which it belongs.

[0029] For example, the base class includes 100 categories, each category includes 1000 training images, and the new class includes 5 categories, each category includes 5 training images.

[0030] In the present application, global category features are obtained by training the classification results of training images in a training set. The global category features include multiple category features that can indicate visual features corresponding to all categories in the training set. At the same time, since the training set used in the global category training process includes images in the base class and images in the new class, the global category features can be avoided from overfitting to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0031] Optionally, the training method of the image classification model in the present application can train the model based on a multi-stage training (episodic training) strategy. For example, the model training process can be divided into multiple training episodes, and in each training episode, several categories in the training set can be randomly selected to train the model. Finally, after multiple training episodes, the model training is completed.

[0032] Specifically, the global category features can be updated in multiple training stages during the model training process, so that the global category features obtained through training can have better consistency. At the same time, the training set includes images in the base class and images in the new class. The effect of the new class (training images in the new class) during the training process can be continuously accumulated with the global category features. Therefore, it is possible to avoid the trained image classification model from overfitting to the base class. By classifying the images to be processed according to the image classification model, better image classification results can be obtained.

[0033] Optionally, feature vectors of a plurality of training images in the support set and feature vectors of a plurality of training images in the query set may be extracted.

[0034] In combination with the second aspect, in certain implementations of the second aspect, the feature vectors of multiple training images in the query set are processed according to a preset second neural network and a preset global category feature to obtain classification results of the multiple training images in the query set, including: extracting feature vectors of multiple training images in the support set, the support set including some training images in some categories in the training set; determining local category features of the training images according to the feature vectors of the multiple training images in the support set, multiple category features in the local category features of the training images are used to indicate visual features of all categories in the support set, the support set including some training images in some categories in the training set; determining classification results of the multiple training images in the query set according to the second neural network, the local category features of the training images and the global category features.

[0035] In combination with the second aspect, in some implementations of the second aspect, updating the global category feature based on the classification results of the multiple training images in the query set includes: updating the global category feature, the first neural network, and the second neural network based on the classification results of the multiple training images in the query set.

[0036] In combination with the second aspect, in certain implementations of the second aspect, updating the global category feature based on the classification results of the multiple training images in the query set includes: updating the global category feature based on the classification error, wherein the classification error is determined based on the classification results of the multiple training images in the query set and pre-annotated labels of the multiple training images in the query set, and the label is used to indicate the category to which the training image belongs.

[0037] In the present application, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated by the training images. The training images in the query set include images in the base class and images in the new class. Updating the global category feature based on the classification error can avoid overfitting of the global category feature to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0038] Furthermore, in multiple training stages during the model training process, the classification error of the current training stage can be used to update the global category features. At the same time, the training set includes images in the base class and images in the new class. The effect of the new class in the training process can be accumulated continuously with the global category features. Therefore, it is possible to avoid the trained image classification model from overfitting to the base class. By classifying the image to be processed according to the image classification model, better image classification results can be obtained.

[0039] In combination with the second aspect, in certain implementations of the second aspect, updating the global category feature based on the classification results of the multiple training images includes: updating the global category feature based on the classification error and the registration error, wherein the registration error is determined based on the local category features of the training image and multiple category features in the global category feature.

[0040] In the present application, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated by the training images, the training images in the query set include images in the base class and images in the new class, the registration error is determined based on the local category features of the training images and multiple category features in the global category features, the local category features of the training images include multiple category features determined based on multiple training images in the support set, and the global category features are updated based on the classification error and the registration error, which can avoid overfitting of the global category features to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0041] Furthermore, in multiple training stages during the model training process, the classification error and registration error of the current training stage can be used to update the global category features. At the same time, the training set includes images in the base class and images in the new class. The effect of the new class in the training process can be accumulated continuously with the global category features. Therefore, it is possible to avoid the trained image classification model from overfitting to the base class. By classifying the image to be processed according to the image classification model, better image classification results can be obtained.

[0042] In combination with the second aspect, in certain implementations of the second aspect, the local category features of the training image are determined by multiple training images in the support set that have been expanded, and the expansion processing includes cropping, flipping and / or data illusion processing of the image.

[0043] Many of the training images in the support set above are images of the new class.

[0044] In the present application, multiple training images in the new class are expanded through cropping, flipping and / or data illusion processing to increase the number of training images in the new class. Therefore, it is possible to avoid overfitting of the trained image classification model to the base class. By classifying the processed images according to the image classification model, better image classification results can be obtained.

[0045] In a third aspect, an image classification device is provided, comprising: an acquisition module, used to acquire an image to be processed; and a classification module, used to classify the image to be processed according to a preset global category feature to obtain a classification result of the image to be processed, wherein the global category feature includes a plurality of category features trained according to a plurality of training images in a training set, and the plurality of category features in the global category feature are used to indicate visual features of all categories in the training set, and all categories in the training set are categories to which all training images in the training set belong, and the training set includes images in a base class and images in a new class.

[0046] In an embodiment of the present application, the global category feature in the image classification device is obtained by training with multiple training images in a training set, and the global category feature includes multiple category features that can indicate visual features corresponding to all categories in the training set. At the same time, since the training set used in the global category training process includes images in the base class and images in the new class, the global category feature can be avoided from overfitting to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0047] Specifically, in multiple training stages during the model training process, the global category features in the image classification device can be updated, so that the global category features obtained through training can have better consistency. At the same time, the training set includes images in the base class and images in the new class. The effect produced by the new class (training images in the new class) during the training process can be continuously accumulated with the global category features. Therefore, it is possible to avoid overfitting of the trained image classification device to the base class. By classifying the image to be processed according to the image classification device, better image classification results can be obtained.

[0048] In combination with the third aspect, in certain implementations of the third aspect, the classification module is specifically used to: extract a feature vector of the image to be processed; determine, based on the feature vector of the image to be processed, a confidence level that the image to be processed belongs to a candidate category, wherein the candidate category is one or more of a plurality of categories indicated by the global category feature; and determine, based on the confidence level, a classification result of the image to be processed from the candidate categories.

[0049] In combination with the third aspect, in certain implementations of the third aspect, the device also includes a determination module, which is used to: determine the local category features of the image to be processed based on the support set of the image to be processed; determine the candidate category based on the local category features of the image to be processed and the global category features; wherein the support set of the image to be processed includes multiple images, and the categories to which the multiple images belong are one or more of the multiple categories indicated by the global category features.

[0050] In combination with the third aspect, in certain implementations of the third aspect, the classification module is specifically used to: determine, based on the feature vector of the image to be processed, the distance between the feature vector of the image to be processed and the feature vector corresponding to each category in the candidate categories; and determine, based on the distance, the confidence that the image to be processed belongs to the candidate category.

[0051] In combination with the third aspect, in certain implementations of the third aspect, the classification module is specifically used to: determine the category with the highest confidence among the candidate categories as the classification result of the image to be processed.

[0052] In combination with the third aspect, in certain implementations of the third aspect, the global category feature is obtained by training based on classification error, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated with the training images in the query set, the labels are used to indicate the category to which the training image belongs, and the query set includes some training images in some categories in the training set.

[0053] In the present application, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated by the training images. The training images in the query set include images in the base class and images in the new class. Updating the global category feature based on the classification error can avoid overfitting of the global category feature to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0054] In combination with the third aspect, in certain implementations of the third aspect, the global category feature is obtained by training based on the classification error and the registration error, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated with the training images in the query set, the labels are used to indicate the category to which the training image belongs, the query set includes some training images in some categories in the training set, the registration error is determined based on the local category features of the training images and multiple category features in the global category features, the local category features of the training images include multiple category features determined based on multiple training images in the support set, multiple category features in the local category features of the training images are used to indicate the visual features of all categories in the support set, and the support set includes some training images in some categories in the training set.

[0055] In the present application, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated by the training images, the training images in the query set include images in the base class and images in the new class, the registration error is determined based on the local category features of the training images and multiple category features in the global category features, the local category features of the training images include multiple category features determined based on multiple training images in the support set, and the global category features are updated based on the classification error and the registration error, which can avoid overfitting of the global category features to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0056] In combination with the third aspect, in certain implementations of the third aspect, the local category features of the training image are determined by multiple training images in the support set that have been expanded, and the expansion processing includes cropping, flipping and / or data illusion processing of the image.

[0057] Many of the training images in the support set above are images of the new class.

[0058] In the present application, multiple training images in the new class are expanded through cropping, flipping and / or data illusion processing to increase the number of training images in the new class. Therefore, it is possible to avoid overfitting of the trained image classification device to the base class. By classifying the images to be processed according to the image classification device, better image classification results can be obtained.

[0059] In a fourth aspect, a training device for an image classification model is provided, comprising: an acquisition module, used to acquire multiple training images in a training set, wherein the training set comprises a support set and a training set, and the multiple training images comprise multiple training images in the support set and multiple training images in the query set; a feature extraction module, used to extract feature vectors of the multiple training images in the query set according to a preset first neural network, and the query set comprises some images in some categories in the training set; a classification module, used to process the feature vectors of the multiple training images in the query set according to a preset second neural network and a preset global category feature to obtain classification results of the multiple training images in the query set, wherein the global category feature comprises multiple category features, and multiple category features in the global category feature are used to indicate visual features of all categories in the training set, all categories in the training set are categories to which all training images in the training set belong, and the training set comprises images in a base class and images in a new class; an update module, used to update the global category feature according to the classification results of the multiple training images in the query set.

[0060] In the present application, global category features are obtained by training based on the classification results of multiple training images in a training set. The global category features include multiple category features that can indicate visual features corresponding to all categories in the training set. At the same time, since the training set used in the global category training process includes images in the base class and images in the new class, the global category features can be prevented from overfitting to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0061] Specifically, the global category features can be updated in multiple training stages during the model training process, so that the global category features obtained through training can have better consistency. At the same time, the training set includes images in the base class and images in the new class. The effect of the new class (training images in the new class) during the training process can be continuously accumulated with the global category features. Therefore, it is possible to avoid the trained image classification model from overfitting to the base class. By classifying the images to be processed according to the image classification model, better image classification results can be obtained.

[0062] In combination with the fourth aspect, in certain implementations of the fourth aspect, the classification module is specifically used to: extract feature vectors of multiple training images in the support set, the support set including some training images in some categories in the training set; determine local category features of the training images based on the feature vectors of the multiple training images in the support set, multiple category features among the local category features of the training images are used to indicate visual features of all categories in the support set, and the support set includes some training images in some categories in the training set; determine classification results of the multiple training images in the query set based on the second neural network, the local category features of the training images and the global category features.

[0063] In combination with the fourth aspect, in certain implementations of the fourth aspect, the updating module is specifically used to: update the global category feature, the first neural network, and the second neural network according to classification results of multiple training images in the query set.

[0064] In combination with the fourth aspect, in certain implementations of the fourth aspect, the update module is specifically used to: update the global category feature based on the classification error, wherein the classification error is determined based on the classification results of multiple training images in the query set and pre-annotated labels of the multiple training images in the query set, and the label is used to indicate the category to which the training image belongs.

[0065] In the present application, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated by the training images. The training images in the query set include images in the base class and images in the new class. Updating the global category feature based on the classification error can avoid overfitting of the global category feature to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0066] In combination with the fourth aspect, in certain implementations of the fourth aspect, the update module is specifically used to: update the global category feature based on the classification error and the registration error, wherein the registration error is determined based on the local category features of the training image and multiple category features in the global category feature.

[0067] In the present application, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated by the training images, the training images in the query set include images in the base class and images in the new class, the registration error is determined based on the local category features of the training images and multiple category features in the global category features, the local category features of the training images include multiple category features determined based on multiple training images in the support set, and the global category features are updated based on the classification error and the registration error, which can avoid overfitting of the global category features to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0068] In combination with the fourth aspect, in certain implementations of the fourth aspect, the local category features of the training image are determined by multiple training images in the support set that have been expanded, and the expansion processing includes cropping, flipping and / or data illusion processing of the image.

[0069] Many of the training images in the support set above are images of the new class.

[0070] In the present application, multiple training images in the new class are expanded through cropping, flipping and / or data illusion processing to increase the number of training images in the new class. Therefore, it is possible to avoid overfitting of the trained image classification model to the base class. By classifying the processed images according to the image classification model, better image classification results can be obtained.

[0071] In a fifth aspect, an image classification device is provided, comprising: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in any one of the implementation modes of the above-mentioned first aspect.

[0072] In a sixth aspect, a training device for an image classification model is provided, the device comprising: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in any one of the implementation modes of the above-mentioned second aspect.

[0073] The processor in the fifth and sixth aspects above can be a central processing unit (CPU) or a combination of a CPU and a neural network processor, where the neural network processor can include a graphics processing unit (GPU), a neural network processing unit (NPU) and a tensor processing unit (TPU), etc. Among them, TPU is a dedicated integrated circuit for artificial intelligence accelerator fully customized by Google for machine learning.

[0074] In a seventh aspect, a computer-readable medium is provided, which stores a program code for execution by a device, wherein the program code includes a method for executing any one of the implementations of the first aspect or the second aspect.

[0075] In an eighth aspect, a computer program product comprising instructions is provided, and when the computer program product is run on a computer, the computer executes a method in any one of the implementations of the first aspect or the second aspect.

[0076] In a ninth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute a method in any one of the implementations of the first or second aspect.

[0077] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in any one of the implementation methods of the first aspect or the second aspect.

[0078] The chip may specifically be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0079] In a tenth aspect, an electronic device is provided, which includes the image classification device of any one of the third aspects above, or the electronic device includes the training device of the image classification model of any one of the fourth aspects above.

[0080] When the electronic device includes the image classification device in any one of the third aspects, the electronic device may specifically be a terminal device.

[0081] When the above-mentioned electronic device includes the training device for the image classification model in any one of the above-mentioned fourth aspects, the electronic device may specifically be a server.

[0082] In the present application, the global category features can be updated in multiple training stages during the model training process, so that the global category features obtained through training can have better consistency. At the same time, the training set includes images in the base class and images in the new class. The effect produced by the new class (training images in the new class) during the training process can be continuously accumulated with the global category features. Therefore, it is possible to avoid the trained image classification model from overfitting to the base class. By classifying the images to be processed according to the image classification model, it is possible to more accurately identify images in the new class. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a structural diagram of the system architecture provided in the embodiment of the present application.

[0084] Figure 2 It is a schematic block diagram of the convolutional neural network model provided in an embodiment of the present application.

[0085] Figure 3 It is a schematic diagram of a chip hardware structure provided in an embodiment of the present application.

[0086] Figure 4 It is a schematic diagram of an application scenario provided in an embodiment of the present application.

[0087] Figure 5 It is a schematic flowchart of a training method for an image classification model provided in one embodiment of the present application.

[0088] Figure 6 It is a schematic block diagram of a training method for an image classification model provided in one embodiment of the present application.

[0089] Figure 7 It is a schematic flowchart of a training method for an image classification model provided in another embodiment of the present application.

[0090] Figure 8 It is a schematic flowchart of an image classification method provided by an embodiment of the present application.

[0091] Fig. 9 It is a schematic diagram of the hardware structure of the image classification device according to an embodiment of the present application.

[0092] Fig.10 It is a schematic diagram of the hardware structure of the training device of the image classification model of the embodiment of the present application. DETAILED DESCRIPTION

[0093] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0094] The image classification method provided in the embodiment of the present application can be applied to image retrieval, album management, safe city, human-computer interaction and other scenes that require image classification or image recognition. It should be understood that the image in the embodiment of the present application can be a static image (or a static picture) or a dynamic image (or a dynamic picture). For example, the image in the present application can be a video or a dynamic picture, or the image in the present application can also be a static picture or a photo. For ease of description, the present application will uniformly refer to static images or dynamic images as images in the following embodiments.

[0095] Specifically, the image classification method of the embodiment of the present application can be specifically applied to album classification and photo recognition scenarios. These two scenarios are introduced in detail below.

[0096] Album Category:

[0097] Users store a large number of pictures on their mobile phones and cloud disks, and classifying and managing albums by category can improve the user experience. By using the image classification method of the embodiment of the present application to classify the pictures in the album, an album arranged or stored by category can be obtained. The image classification method of the embodiment of the present application can facilitate users to classify and manage different object categories, thereby facilitating users' searches, saving users' management time, and improving the efficiency of album management.

[0098] Specifically, when the image classification method of the embodiment of the present application is used to classify the album, the image features of the pictures in the album can be extracted first, and then the pictures in the album can be classified according to the extracted image features to obtain the classification results of the pictures. Next, the pictures in the album can be classified according to the classification results of the pictures to obtain an album arranged according to the picture categories. Among them, when arranging the pictures in the album according to the picture categories, pictures belonging to the same category can be arranged in one row or one row. For example, in the final album, the pictures in the first row are all airplanes, and the pictures in the second row are all cars.

[0099] Photographing and identifying objects:

[0100] When taking photos, users can use the image classification method of the embodiment of the present application to process the photos taken, and can automatically identify the category of the photographed object. For example, it can automatically identify whether the photographed object is a flower, an animal, etc. Further, the image classification method of the embodiment of the present application can be used to identify the object obtained by taking photos and identify the category to which the object belongs. For example, if the photos taken by the user include shared bicycles, the image classification method of the embodiment of the present application can be used to identify the shared bicycles and identify that the object belongs to a bicycle. Furthermore, relevant information about the bicycle can also be displayed.

[0101] Since the present invention has excellent generalized small sample learning effect, no matter the photographed object is from a base class or a new class, it can be well identified.

[0102] In the prior art, for image categories with few training samples, it is often impossible to effectively classify or recognize images belonging to the category. Using the image classification method of the embodiment of the present application, image classification or image recognition can also be well achieved for new categories (small samples) with few training samples.

[0103] It should be understood that the album classification and photo object recognition introduced above are only two specific scenarios in which the image classification method of the embodiment of the present application is applied. The image classification method of the embodiment of the present application is not limited to the above two scenarios when applied. The image classification method of the embodiment of the present application can be applied to any scenario that requires image classification or image recognition.

[0104] The embodiments of the present application involve a large number of neural network-related applications. In order to better understand the solutions of the embodiments of the present application, the relevant terms and other related concepts of the neural network that may be involved in the embodiments of the present application are first introduced below.

[0105] (1) Neural Network

[0106] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be shown as formula (1-1):

[0107]

[0108] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0109] (2) Deep Neural Networks

[0110] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0111] Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0112] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0113] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0114] (3) Convolutional Neural Network

[0115] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0116] (4) Loss Function

[0117] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0118] (5) Back propagation algorithm

[0119] Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0120] (6) Pixel value

[0121] The pixel value of an image can be a red, green, and blue (RGB) color value, and the pixel value can be a long integer representing the color. For example, the pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness. For a grayscale image, the pixel value can be a grayscale value.

[0122] (7) Base class

[0123] In the prior art, a base class includes a large number of labeled samples for training a model, and the number of these labeled samples is sufficient to meet the needs of model training. For example, a base class may include multiple images that have been labeled with their categories, and the multiple images may belong to one category, or the multiple images may belong to multiple different categories. The base class can be used to train the image classification model in the embodiment of the present application.

[0124] (8) New Class

[0125] In the prior art, a novel class is a concept relative to a base class. For example, if a model is trained using multiple labeled samples, then for the (trained) model, the multiple labeled samples used in training are the base class, and the categories not included in the base class are the novel class. For example, if we have trained a model with a large number of images of animals (except dogs), and at this time, we want the model to recognize dogs, then the large number of animal images are the base class, and the dog images are the novel class.

[0126] Typically, each category in a new class includes only a small number of labeled samples. In this application, a new class may refer to a few-shot, i.e., a new class includes a small number of images that have been labeled to belong to the same category, or the images may be divided into multiple different categories.

[0127] (9) Small Sample Learning

[0128] Few-shot learning (FSL) refers to using a large-scale training set (including one or more base classes) to train an image classification model, and then accurately identifying the test samples of a new class (the category to which it belongs) that has never been seen before (the new class does not overlap with the base class) with the help of a few training samples included in each new class.

[0129] Furthermore, small sample learning can include standard small sample learning and generalized small sample learning. For example, if the test sample in small sample learning only includes new classes, this type of problem can be called standard small sample learning; if the test sample includes not only new classes but also base classes, this type of problem can be called generalized small sample learning.

[0130] The image classification method in the embodiment of the present application can be applied to both standard small sample learning and generalized small sample learning. Figure 1 The system architecture applicable to the embodiments of the present application is introduced.

[0131] like Figure 1 As shown, the embodiment of the present application provides a system architecture 100. Figure 1In the embodiment of the present invention, the data acquisition device 160 is used to acquire training data. For the image classification method of the present invention, the training data may include training images and classification results corresponding to the training images, wherein the classification results of the training images may be manually pre-labeled results.

[0132] After collecting the training data, the data collection device 160 stores the training data in the database 130 , and the training device 120 obtains the target model / rule 101 through training based on the training data maintained in the database 130 .

[0133] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input original image and compares the output image with the original image until the difference between the image output by the training device 120 and the original image is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0134] The above-mentioned target model / rule 101 can be used to implement the image classification method of the embodiment of the present application, that is, the image to be processed is input into the target model / rule 101 after relevant pre-processing, and the classification result of the image can be obtained. The target model / rule 101 in the embodiment of the present application can specifically be an image classification model in the embodiment of the present application. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 completely based on the training data maintained by the database 130, and it is also possible to obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0135] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud device, etc. Figure 1 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include an image to be processed input by the client device in the embodiment of the present application.

[0136] Preprocessing module 113 and preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be processed) received by I / O interface 112. In the embodiment of the present application, preprocessing module 113 and preprocessing module 114 may be omitted (or only one of the preprocessing modules may be present), and the computing module 111 may be used directly to process the input data.

[0137] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0138] Finally, the I / O interface 112 returns the processing result, such as the classification result of the image to be processed obtained above, to the client device 140 , thereby providing it to the user.

[0139] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0140] exist Figure 1 In the case shown, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0141] It is worth noting that Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 1In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0142] like Figure 1 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 in the embodiment of the present application may be the image classification model in the present application. Specifically, the image classification model provided in the embodiment of the present application may include one or more neural networks, and the one or more neural networks may include CNN, deep convolutional neural networks (DCNN) and / or recurrent neural networks (RNNS), etc.

[0143] Since CNN is a very common neural network, Figure 2 The focus is on a detailed introduction to the structure of CNN. As mentioned in the basic concept introduction above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.

[0144] like Figure 2 As shown, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (wherein the pooling layer is optional), and a neural network layer 230. The relevant contents of these layers are described in detail below.

[0145] Convolutional layer / pooling layer 220:

[0146] Convolutional Layer:

[0147] like Figure 2 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0148] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0149] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unwanted noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0150] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0151] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0152] Pooling layer / Pooling layer 220:

[0153] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 2 Each layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0154] Neural Network Layer 230:

[0155] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 2 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0156] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 2 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0157] It should be noted that if Figure 2 The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0158] In this application, the image classification model may include Figure 2 The convolutional neural network 200 shown in the figure can process the image to be processed and obtain the classification result of the image to be processed.

[0159] Figure 3 A chip hardware structure provided in an embodiment of the present application includes a neural network processor 30. The chip can be arranged as follows: Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 2 The algorithms of each layer in the convolutional neural network shown can be found in Figure 3 The convolutional neural network can be implemented in the chip shown, and optionally, the convolutional neural network can be one of the (one or more) neural networks included in the above-mentioned image classification model.

[0160] The neural network processor NPU 30 is mounted on the host CPU as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 303, and the controller 304 controls the operation circuit 303 to extract data from the memory (weight memory or input memory) and perform operations.

[0161] In some implementations, the operation circuit 303 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 303 is a two-dimensional systolic array. The operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0162] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 303 takes the corresponding data of the matrix B from the weight memory 302 and caches it on each PE in the operation circuit 303. The operation circuit 303 takes the matrix A data from the input memory 301 and performs a matrix operation with the matrix B, and the partial result or the final result of the matrix is ​​stored in the accumulator 308.

[0163] The vector calculation unit 307 can further process the output of the operation circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0164] In some implementations, the vector calculation unit 307 can store the processed output vector to the unified buffer 306. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 303, for example, for use in a subsequent layer in a neural network.

[0165] The unified memory 306 is used to store input data and output data.

[0166] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 306 through the direct memory access controller 305 (DMAC), the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0167] The bus interface unit (BIU) 310 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 309 through the bus.

[0168] An instruction fetch buffer 309 connected to the controller 304 and used to store instructions used by the controller 304;

[0169] The controller 304 is used to call the instructions cached in the memory 309 to control the working process of the computing accelerator.

[0170] Generally, the unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM for short), a high bandwidth memory (HBM) or other readable and writable memory.

[0171] in, Figure 2 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 303 or the vector calculation unit 307.

[0172] Introduced above Figure 1 The execution device 110 in the embodiment can execute each step of the image classification method of the present application. Optionally, Figure 1 The execution device 110 may include Figure 2 The CNN model shown and Figure 3 The chip shown. The image classification method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0173] The image classification method provided in the embodiment of the present application can be executed on a server, on a cloud, or on a terminal device. Figure 4 As shown, the technical solution of the embodiment of the present invention can be applied to a terminal device, and the image classification method in the embodiment of the present application can classify the input image to obtain the classification result of the input image. The terminal device can be mobile or fixed, for example, the terminal device can be a mobile phone with image processing function, a tablet personal computer (tablet personal computer, TPC), a media player, a smart TV, a laptop computer (laptop computer, LC), a personal digital assistant (personal digital assistant, PDA), a personal computer (personal computer, PC), a camera, a video camera, a smart watch, a wearable device (wearable device, WD) or an autonomous driving vehicle, etc., and the embodiment of the present invention is not limited to this.

[0174] Image (or picture) classification is the basis of various image processing applications. Computer vision often involves the problem of how to classify acquired images. However, the training of image classification models requires a large amount of well-labeled training data. For image categories with few training samples (such as new categories), it is often impossible to effectively classify or recognize images belonging to this category. In many cases, it is very difficult to obtain effective data, such as in the medical field and security field.

[0175] Based on the above problems, the embodiments of the present application propose an image classification method and a training method for an image classification model, which can well implement image classification or image recognition for new classes (small samples) with very few training samples.

[0176] Figure 5 It is a schematic flowchart of a training method 500 for an image classification model according to an embodiment of the present application. Figure 5 The method shown can be executed by a computer device, a server device, or a computing device with strong computing power. For example, the method can be executed by Figure 4 Executed by the terminal device in. Figure 5 The method shown includes steps 510, 520, 530 and 540, which are described in detail below.

[0177] S510, obtaining a plurality of training images in a training set.

[0178] The training set may be a set of all training images used during training.

[0179] Optionally, the training set includes images in a base class and images in a new class, and each category in the base class includes far more training images than each category in the new class.

[0180] Here, “far more than” can be understood as the number of training images included in each category in the base class is at least one order of magnitude higher than the number of training images included in each category in the new class. That is to say, the number of training images included in each category in the base class is at least ten times the number of training images included in each category in the new class.

[0181] For example, the base class includes 100 categories, each category includes 1000 training images, and the new class includes 5 categories, each category includes 5 training images.

[0182] Optionally, the image classification model in the present application can be trained based on a multi-stage training strategy. For example, the model training process can be divided into multiple training episodes. In each training episode, several categories in the training set can be randomly selected to train the model. Finally, after multiple training episodes, the training of the image classification model is completed. The description of multi-stage training can be specifically referred to the prior art, which will not be repeated here.

[0183] Specifically, the global category features can be updated in multiple training stages during the model training process, so that the global category features obtained through training can have better consistency. At the same time, the training set includes images in the base class and images in the new class. The effect of the new class (training images in the new class) during the training process can be continuously accumulated with the global category features. Therefore, it is possible to avoid the trained image classification model from overfitting to the base class. By classifying the images to be processed according to the image classification model, better image classification results can be obtained.

[0184] The training method of the image classification model in the embodiment of the present application can better balance the difference in the number of samples in the base class and the number of samples in the new class, and the impact of the difference in the model training process. The detailed description of the training method of the image classification model involved in the present application can be as follows: Figure 7 As described in method 700.

[0185] Based on the above-mentioned multi-stage training strategy, the method 500 in the present application can be divided into multiple training stages. Accordingly, the above-mentioned S510 can refer to, in a training stage, randomly extracting multiple training images from a training set, and dividing the extracted multiple training images in the training set into a support set and a query set.

[0186] Therefore, it can also be said that the training set includes a support set and a query set, and the multiple training images in the training set include: multiple training images in the support set and multiple training images in the query set.

[0187] S520: Extract feature vectors of a plurality of training images in the query set according to a preset first neural network.

[0188] The feature vector of the training image may be used to indicate the visual features (or image features) of the training image, and the query set includes some images in some categories in the training set.

[0189] Optionally, when extracting the feature vectors of the multiple training images in the query set, the feature vectors of the multiple training images in the support set may also be extracted.

[0190] Optionally, the first neural network may be composed of Figure 1 The system 100 shown is trained.

[0191] Optionally, the first neural network may be Figure 2 The convolutional neural network shown.

[0192] S530, processing feature vectors of the plurality of training images in the query set according to a preset second neural network and a preset global category feature to obtain classification results of the plurality of training images in the query set.

[0193] It should be understood that the classification result here may refer to the label of the category to which the image belongs.

[0194] Optionally, the second neural network may be composed of Figure 1 The system 100 shown is trained.

[0195] Optionally, the second neural network may be Figure 2 The convolutional neural network shown.

[0196] The global category feature includes multiple category features, and the multiple category features in the global category feature are used to indicate visual features of all categories in the training set, and all categories in the training set are categories to which all training images in the training set belong.

[0197] In the present application, the feature vectors of the multiple training images in the query set are processed according to the preset second neural network and the preset global category features to obtain the classification results of the multiple training images in the query set, which may include: extracting the feature vectors of the multiple training images in the support set; determining the local category features of the training images according to the feature vectors of the multiple training images in the support set; and determining the classification results of the multiple training images in the query set according to the second neural network, the local category features of the training images and the global category features.

[0198] Among them, multiple category features in the local category features of the training image are used to indicate the visual features of all categories in the support set, the support set includes some training images in some categories in the training set, and the query set includes some training images in some categories in the training set.

[0199] Optionally, the categories in the support set and the categories in the query set may be the same.

[0200] Combine the following Figure 6 The step of determining the classification result in the above S530 is described in detail.

[0201] like Figure 6 As shown, the multiple training images in the training set are divided into a support set and a query set, and the multiple training images in the support set may include images in the base class and images in the new class.

[0202] It should be noted that, in a training stage, the support set and the query set are determined by multiple training images randomly selected from the training set, and the training set includes images in the base class and images in the new class. Therefore, the multiple training images in the support set may also include only images in the base class, or, the multiple training images in the support set may also include only images in the new class.

[0203] like Figure 6 As shown, the category features of multiple training images in the base class in the support set and the category features of multiple training images in the new class can be determined. For example, the feature vector of each training image in the base class can be determined, and the feature vectors of multiple training images belonging to the same class can be averaged, and the average can be used as the category feature of the class. The method of determining the category features of multiple training images in the new class will not be repeated here.

[0204] In particular, before determining the category features of multiple training images in the new class, the multiple training images in the new class may be subjected to image expansion processing such as cropping, flipping and / or hallucinator processing to obtain more new class images.

[0205] In the present application, the above-mentioned (obtained) category features (including category features of multiple training images in the base class and / or category features of multiple training images in the new class) can be called local category features (local class representations) of the training images.

[0206] It can be seen that the local category features of the training images are determined by a plurality of training images randomly selected (in the support set) in a training phase, that is, these local category features only act in a training phase in the model training process, therefore, the local category features can also be called episodic class representations. Alternatively, the local category features can also be other names, which are not limited in this application.

[0207] In contrast, global class representations can be considered as parameters of the image classification model, sharing the class features of all categories in the training set. Therefore, global class representations can be used in multiple training stages during the model training process.

[0208] Optionally, after obtaining the local category features of the training image, the local category features of the training image may be registered to the global category features, thereby obtaining a registration result, ie, a registered global category feature.

[0209] The registration here can also be understood as finding the category feature in the global category feature corresponding to each category feature in the local category feature. For example, finding a category feature in the global category feature that has the highest similarity with each category feature in the local category feature.

[0210] Furthermore, the registration error may be determined according to the similarity between each category feature in the local category features and its corresponding category feature in the global category features.

[0211] Specifically, the registration error can be determined based on the similarity between each category feature in the local category features and the category features in the corresponding global category features. In other words, the registration error in the current training stage is determined based on the local category features of the training image and multiple category features in the global category features.

[0212] Optionally, the second neural network may be used to perform dimensionality reduction processing on the registration result, so as to process feature vectors of multiple training images in the query set in a low-dimensional vector space.

[0213] In the present application, the registration result may be used to predict each training image in the query set to obtain a classification result for each training image in the query set.

[0214] Specifically, the nearest neighbor method can be used to predict the classification results of the training images.

[0215] For example, the distance between the category feature in the registration result and the feature vector of each training image in the query set can be calculated, and each distance can be normalized to obtain the probability that each training image in the training image belongs to the category indicated by the category feature in the registration result, that is, the classification result of each training image in the query set.

[0216] Furthermore, the classification error can be obtained by comparing the classification result of each training image with the pre-labeled label of the training image, wherein the pre-labeled label is used to indicate the true category to which the training image belongs. In other words, the classification error of the current training stage is determined based on the classification results of the multiple training images and the pre-labeled labels of the multiple training images.

[0217] S540: Update the global category feature according to the classification results of the multiple training images in the query set.

[0218] In the present application, the global category feature, the first neural network and the second neural network can be updated according to the classification results of the multiple training images in the query set.

[0219] Optionally, the global category feature may be updated according to the classification error. Optionally, the classification error may be determined according to the classification results of the plurality of training images by the method in S530 above.

[0220] Furthermore, the global category feature, the first neural network and the second neural network may be updated according to the classification error.

[0221] Optionally, the global category feature may be updated according to the classification error and the registration error. Optionally, the classification error may be determined according to the local category feature of the training image and multiple category features in the global category feature by the method in S530 above.

[0222] Furthermore, the global category feature, the first neural network and the second neural network can be updated according to the classification error and the registration error.

[0223] In the present application, the global category features can be updated in multiple training stages during the model training process, so that the global category features obtained through training can have better consistency. At the same time, the training set includes images in the base class and images in the new class. The effect produced by the new class (training images in the new class) during the training process can be continuously accumulated with the global category features. Therefore, it is possible to avoid the trained image classification model from overfitting to the base class. By classifying the images to be processed according to the image classification model, it is possible to more accurately identify images in the new class.

[0224] Figure 7 It is a schematic flowchart of a training method 700 for an image classification model according to another embodiment of the present application. Figure 7 The method shown can be executed by a computer device, a server device, or a computing device with strong computing power. For example, the method can be executed by Figure 4 The following describes each step in method 700 in detail.

[0225] S710, feature extraction (module).

[0226] Feature vectors of a plurality of training images in a support set are extracted, the plurality of training images comprising images in a base class and images in a new class.

[0227] Optionally, the first neural network mentioned above can be used to extract feature vectors of multiple training images in the support set.

[0228] For example, suppose there is a set (multiple) of training images that can be used to train the above image classification model, including C total ={c1,...,c N} categories, where N represents the total number of categories, and the cth j The categories include k j training images, j is an integer. Based on the multiple training images, a training set D can be obtained train , training set D train Includes base class images and new class images.

[0229] In the current training stage, from the training set D train Randomly select C train categories, and n samples are extracted from each category s training images to form a support set S = {(x i ,y i ),i=1,...,n s ×C train},y i is the i-th training image x i Label, C train is the number of categories extracted; from the training set D train C total From the categories, we extract C train categories, each category includes n q training images, forming a query set Q = {(x q ,y q ),q=1,...,n q ×C train}, where n s 、n q Is a positive integer.

[0230] Alternatively, F(·) can be used to represent feature extraction, then for the training set D train The i-th training image x in i , the eigenvector is f i =F(x i ), where i is an integer.

[0231] S701, image expansion.

[0232] Optionally, for new classes in the support set, image expansion processing such as cropping, flipping and / or hallucinator processing may be performed on the images in the new class to obtain more images of the new class.

[0233] It should be noted that in the training set Dtrain In the example, each new class has only n few training images, and n few Often less than n s +n q .

[0234] Therefore, in S701, it is necessary to change n in the new class few The training images are expanded to n s +n q training images, and then n of them s training images into the support set, and n q The training images are put into the query set.

[0235] S720, local category feature calculation.

[0236] Optionally, for multiple training images in the support set, the feature vectors of multiple training images belonging to the same category are calculated to find the average value, and the average value can be used as the category feature of the category.

[0237] Calculate the category features of the categories to which all training images in the support set belong, so that the local category features of the training images can be obtained, denoted as in, To support the concentration of j The category characteristics of the categories.

[0238] S730, category feature registration (module).

[0239] Optionally, the local category features of the training image are registered to the global category features to obtain a registration result.

[0240] In other words, find the category feature in the global category feature corresponding to each category feature in the local category feature.

[0241] For example, the global category feature is recorded as in, is the cth j The category characteristics of the categories.

[0242] For the j-th category feature f in the local category feature i and the jth category feature in the global category feature The category feature f in the local category feature can be calculated in the low-dimensional embedding space i and the category features in the global category features The similarity of Among them, the jth element represents f i and The similarity.

[0243] For example, the similarity may be calculated using the following formula

[0244]

[0245] Among them, θ(·) is the embedding function of the visual features of the training image, and φ(·) is the embedding function of the global category features.

[0246] At this time, the global category feature is consistent with the category feature f i The most similar category feature is the category feature f i The corresponding global category features.

[0247] Furthermore, the registration error may be determined according to the similarity between each category feature in the local category features and its corresponding category feature in the global category features.

[0248] For example, in this application, the training image x i The loss function L reg As the registration error, the training image is closest to its global category characteristics in the low-dimensional embedding space. Calculate the loss function L reg The formula is as follows.

[0249] L reg =CE(y i ,V i )

[0250] Among them, CE(·) is the cross entropy loss, y i For the training image x i .

[0251] Therefore, the c j Local category features of categories The registration error can be calculated by the following formula.

[0252]

[0253]

[0254] Next, we can use the softmax function to Normalization is performed to obtain the probability distribution of each category The probability distribution P i As the weight, estimate the weighted sum of the category features of all categories (in the global category features), and use this weighted sum as the C of the current training stage. trainThe category representation of the i-th category among the categories is denoted by ξ i , that is, i =P i G. ξ i This is the registration result in the above embodiment.

[0255] S740, image classification.

[0256] Optionally, the registration result may be used to predict each training image in the query set to obtain a classification result for each training image in the query set.

[0257] Furthermore, by comparing the classification result of each training image with the pre-labeled label (true label) of the training image, the classification error can be obtained.

[0258] For example, given a category feature ξ i , the loss function L fsl As the training image in the query set (x k ,y k ) classification error, loss function L fsl As shown in the following formula.

[0259] L fsl =CE(y k ,W k )

[0260]

[0261] in, Represents the training image x in the query set k and the estimated global category feature ξ i The similarity.

[0262] At this time, the one with the highest similarity can be The corresponding category is used as the training image x k The category of the training image x k The classification results.

[0263] S750, image classification model update.

[0264] Optionally, the registration error and / or classification error may be used to update the image classification model.

[0265] For example, in this application, the registration error (L reg ) and classification error (L fsl ) combined, the total loss function L for multiple training stages total (·) can be calculated by the following formula.

[0266]

[0267] Optionally, S750 may include at least one of steps S751, S752 and S753.

[0268] S751, update feature extraction (module).

[0269] Alternatively, the registration error (L reg ), classification error (L fsl ) and / or the total loss function (L total ), update feature extraction (module).

[0270] S752, update category feature registration (module).

[0271] Alternatively, the registration error (L reg ), classification error (L fsl ) and / or the total loss function (L total ), update category feature registration (module).

[0272] S753, update the global category features.

[0273] Alternatively, the registration error (L reg ), classification error (L fsl ) and / or the total loss function (L total ), update the global category features.

[0274] Figure 8 800 is a schematic flow chart of an image classification method 800 provided in an embodiment of the present application. Figure 8 The method shown can be executed by a computer device, a server device, or a computing device with strong computing power. For example, the method can be executed by Figure 4 Executed by the terminal device in. Figure 8 The method shown includes steps 810 and 820, which are described in detail below.

[0275] S810, obtaining an image to be processed.

[0276] Optionally, when Figure 8 The method 800 shown consists of Figure 4 When the terminal device in is executed, the image to be processed may be an image captured by the terminal device through a camera; or, the image to be processed may also be an image obtained from inside the terminal device, for example, an image stored in an album of the terminal device, or an image obtained by the terminal device from the cloud.

[0277] S820: Classify the image to be processed according to a preset global category feature to obtain a classification result of the image to be processed.

[0278] Among them, the global category feature includes multiple category features trained according to multiple training images in a training set, and the multiple category features in the global category feature are used to indicate the visual features of all categories in the training set. All categories in the training set are categories to which all training images in the training set belong, and the training set includes images in a base class and images in a new class.

[0279] Optionally, classifying the image to be processed according to a preset global category feature to obtain a classification result of the image to be processed may include: extracting a feature vector of the image to be processed; determining, based on the feature vector of the image to be processed, a confidence level that the image to be processed belongs to a candidate category, wherein the candidate category is one or more of a plurality of categories indicated by the global category feature; and determining, based on the confidence level, a classification result of the image to be processed from the candidate categories.

[0280] Optionally, before determining the confidence that the image to be processed belongs to a candidate category based on the feature vector of the image to be processed, the method may also include: determining local category features of the image to be processed based on the support set of the image to be processed; and determining the candidate category based on the local category features of the image to be processed and the global category features.

[0281] The support set of the image to be processed includes a plurality of images, and the categories to which the plurality of images belong are one or more of the plurality of categories indicated by the global category feature.

[0282] Optionally, the candidate categories may include categories to which all training images in the support set belong.

[0283] Optionally, determining the confidence that the image to be processed belongs to a candidate category based on the feature vector of the image to be processed may include: determining, based on the feature vector of the image to be processed, the distance between the feature vector of the image to be processed and the feature vector corresponding to each category in the candidate categories; and determining, based on the distance, the confidence that the image to be processed belongs to the candidate category.

[0284] Optionally, determining the classification result of the image to be processed from the candidate categories according to the confidence level may include: determining the category with the highest confidence level among the candidate categories as the classification result of the image to be processed.

[0285] It should be understood that the confidence here may be the probability that the image to be processed belongs to the candidate category. Therefore, the maximum confidence can also be said to be the maximum probability that the image to be processed belongs to the candidate category.

[0286] Optionally, the global category feature is obtained by training based on classification error, the classification error is determined based on the classification results of the training images in the query set and the labels pre-labeled for the training images in the query set, the labels are used to indicate the category to which the training images belong, and the query set includes some training images in some categories in the training set. The specific process of determining the classification error can refer to Figure 5 Method 500 in .

[0287] Optionally, the global category feature is obtained by training based on the registration error, the registration error is determined based on the local category feature of the training image and multiple category features in the global category feature, the local category feature of the training image includes multiple category features determined based on multiple training images in the support set, multiple category features in the local category feature of the training image are used to indicate the visual features of all categories in the support set, and the support set includes some training images in some categories in the training set. The specific process of determining the registration error can refer to Figure 5 Method 500 in .

[0288] Optionally, the global category feature is obtained by training based on classification error and registration error.

[0289] Optionally, the local category features of the training image are determined by a plurality of training images in the support set that have been subjected to an expansion process, wherein the expansion process includes cropping, flipping and / or data hallucination processing of the image.

[0290] In the present application, global category features are obtained by training based on the classification results of multiple training images in a training set. The global category features include multiple category features that can indicate visual features corresponding to all categories in the training set. At the same time, since the training set used in the global category training process includes images in the base class and images in the new class, the global category features can be prevented from overfitting to the images in the base class, thereby enabling more accurate identification of images in the new class.

[0291] Fig. 9 It is a schematic diagram of the hardware structure of the image classification device according to an embodiment of the present application. Fig. 9 The image classification device 4000 shown includes a memory 4001, a processor 4002, a communication interface 4003 and a bus 4004. The memory 4001, the processor 4002 and the communication interface 4003 are connected to each other through the bus 4004.

[0292] The memory 4001 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 4001 may store a program. When the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute each step of the image classification method of the embodiment of the present application.

[0293] Processor 4002 can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the image classification device of the embodiment of the present application, or to execute the image classification method of the method embodiment of the present application.

[0294] The processor 4002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image classification method of the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 4002 or by instructions in the form of software.

[0295] The processor 4002 may also be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 4001, and the processor 4002 reads the information in the memory 4001, and combines its hardware to complete the functions required to be performed by the units included in the image classification device of the embodiment of the present application, or executes the image classification method of the method embodiment of the present application.

[0296] The communication interface 4003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 4000 and other devices or a communication network. For example, the image to be processed can be obtained through the communication interface 4003.

[0297] The bus 4004 may include a path for transmitting information between various components of the device 4000 (eg, the memory 4001 , the processor 4002 , and the communication interface 4003 ).

[0298] Fig.10 Schematic diagram of the hardware structure of the training device 5000 for the image classification model of the embodiment of the present application. Similar to the above-mentioned device 4000, Fig.10 The training device 5000 of the image classification model shown includes a memory 5001, a processor 5002, a communication interface 5003 and a bus 5004. The memory 5001, the processor 5002 and the communication interface 5003 are connected to each other through the bus 5004.

[0299] The memory 5001 can store programs. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 is used to execute each step of the neural network training method of the embodiment of the present application.

[0300] The processor 5002 can adopt a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits to execute relevant programs to implement the training method of the image classification model of the embodiment of the present application.

[0301] The processor 5002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the training method of the image classification model of the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 5002 or by instructions in the form of software.

[0302] It should be understood that through Fig.10 The training device 5000 of the image classification model shown in the figure trains the image classification model, and the trained image classification model can be used to execute the image classification method of the embodiment of the present application. Specifically, the image classification model can be trained by the device 5000 to obtain Figure 5 as well as Figure 8 Image classification model in the method shown.

[0303] Specifically, Fig.10 The device shown can obtain training data and the image classification model to be trained from the outside world through the communication interface 5003, and then the processor trains the image classification model to be trained according to the training data.

[0304] It should be noted that although the above-mentioned apparatus 4000 and apparatus 5000 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the apparatus 4000 and apparatus 5000 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the apparatus 4000 and apparatus 5000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the apparatus 4000 and apparatus 5000 may also only include the devices necessary for implementing the embodiments of the present application, and do not necessarily include Fig. 9 and Fig.10 All devices shown in .

[0305] It should be understood that the processor in the embodiment of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0306] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0307] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0308] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0309] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0310] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0311] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0312] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0313] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0314] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0315] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0316] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0317] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image classification method, characterized in that: include: Get the image to be processed; According to a preset global category feature, the image to be processed is classified to obtain a classification result of the image to be processed, wherein the global category feature includes a plurality of category features trained according to a plurality of training images in a training set, and the plurality of category features in the global category feature are used to indicate visual features of all categories in the training set, and all categories in the training set are categories to which all training images in the training set belong, and the training set includes images in a base class and images in a new class; The global category feature is obtained by training based on the classification error and / or the registration error, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated on the training images in the query set, the labels are used to indicate the category to which the training images belong, the query set includes some training images in some categories in the training set, the registration error is determined based on the local category features of the training images and multiple category features in the global category features, the local category features of the training images include multiple category features determined based on multiple training images in the support set, multiple category features in the local category features of the training images are used to indicate the visual features of all categories in the support set, and the support set includes some training images in some categories in the training set.

2. The method according to claim 1, characterized in that: The step of classifying the image to be processed according to the preset global category features to obtain the classification result of the image to be processed includes: Extracting a feature vector of the image to be processed; Determining, according to the feature vector of the image to be processed, the confidence that the image to be processed belongs to a candidate category, where the candidate category is one or more of the multiple categories indicated by the global category feature; According to the confidence level, a classification result of the image to be processed is determined from the candidate categories.

3. The method according to claim 2, characterized in that Before determining the confidence that the image to be processed belongs to a candidate category according to the feature vector of the image to be processed, the method further includes: Determining local category features of the image to be processed according to the support set of the image to be processed; Determining the candidate category according to the local category features of the image to be processed and the global category features; The support set of the image to be processed includes a plurality of images, and the categories to which the plurality of images belong are one or more of the plurality of categories indicated by the global category feature.

4. The method according to claim 2, characterized in that: Determining the confidence that the image to be processed belongs to a candidate category according to the feature vector of the image to be processed includes: Determining, according to the feature vector of the image to be processed, the distance between the feature vector of the image to be processed and the feature vector corresponding to each category in the candidate categories; The confidence level that the image to be processed belongs to the candidate category is determined according to the distance.

5. The method according to claim 2, characterized in that: Determining the classification result of the image to be processed from the candidate categories according to the confidence level includes: The category with the highest confidence among the candidate categories is determined as the classification result of the image to be processed.

6. The method according to any one of claims 3 to 5, characterized in that: The local category features of the training images are determined by a plurality of training images in the support set that have been expanded, wherein the expansion process includes cropping, flipping and / or data hallucination processing of the images.

7. A method for training an image classification model, characterized in that: include: Acquire a plurality of training images in a training set, wherein the training set includes a support set and a query set, and the plurality of training images include a plurality of training images in the support set and a plurality of training images in the query set; Extracting feature vectors of a plurality of training images in the query set according to a preset first neural network, wherein the query set includes some images in some categories in the training set; According to a preset second neural network and a preset global category feature, feature vectors of a plurality of training images in the query set are processed to obtain classification results of the plurality of training images in the query set, wherein the global category feature includes a plurality of category features, and the plurality of category features in the global category feature are used to indicate visual features of all categories in the training set, and all categories in the training set are categories to which all training images in the training set belong, and the training set includes images in a base class and images in a new class; updating the global category feature according to the classification results of the plurality of training images in the query set; Wherein, updating the global category feature according to the classification results of the plurality of training images in the query set includes: According to the classification error, updating the global category feature; or The global category feature is updated according to the classification error and the registration error, wherein the registration error is determined according to the local category feature of the training image and multiple category features in the global category feature, and the classification error is determined according to the classification results of multiple training images in the query set and labels pre-annotated for the multiple training images in the query set, and the label is used to indicate the category to which the training image belongs.

8. The method according to claim 7, characterized in that The step of processing the feature vectors of the plurality of training images in the query set according to the preset second neural network and the preset global category features to obtain the classification results of the plurality of training images in the query set includes: Extracting feature vectors of a plurality of training images in the support set, wherein the support set includes some training images in some categories in the training set; Determine local category features of the training images according to feature vectors of the plurality of training images in the support set, wherein a plurality of category features in the local category features of the training images are used to indicate visual features of all categories in the support set; The classification results of the plurality of training images in the query set are determined according to the second neural network, the local category features of the training images, and the global category features.

9. The method according to claim 8, characterized in that The updating of the global category feature according to the classification results of the plurality of training images in the query set comprises: The first neural network and the second neural network are updated according to the classification results of the plurality of training images in the query set.

10. The method according to claim 9, characterized in that The local category features of the training images are determined by a plurality of training images in the support set that have been expanded, wherein the expansion process includes cropping, flipping and / or data hallucination processing of the images.

11. An image classification device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A classification module, used for classifying the image to be processed according to a preset global category feature to obtain a classification result of the image to be processed, wherein the global category feature includes a plurality of category features obtained by training according to a plurality of training images in a training set, and the plurality of category features in the global category feature are used to indicate visual features of all categories in the training set, and all categories in the training set are categories to which all training images in the training set belong, and the training set includes images in a base class and images in a new class; The global category feature is obtained by training based on the classification error and / or the registration error, the classification error is determined based on the classification results of the training images in the query set and the labels pre-annotated on the training images in the query set, the labels are used to indicate the category to which the training images belong, the query set includes some training images in some categories in the training set, the registration error is determined based on the local category features of the training images and multiple category features in the global category features, the local category features of the training images include multiple category features determined based on multiple training images in the support set, multiple category features in the local category features of the training images are used to indicate the visual features of all categories in the support set, and the support set includes some training images in some categories in the training set.

12. The device according to claim 11, characterized in that The classification module is specifically used for: Extracting a feature vector of the image to be processed; Determining, according to the feature vector of the image to be processed, the confidence that the image to be processed belongs to a candidate category, where the candidate category is one or more of the multiple categories indicated by the global category feature; According to the confidence level, a classification result of the image to be processed is determined from the candidate categories.

13. The device according to claim 12, characterized in that The device also includes a determining module, configured to: Determining local category features of the image to be processed according to the support set of the image to be processed; Determining the candidate category according to the local category features of the image to be processed and the global category features; The support set of the image to be processed includes a plurality of images, and the categories to which the plurality of images belong are one or more of the plurality of categories indicated by the global category feature.

14. The device according to claim 12, characterized in that The classification module is specifically used for: Determining, according to the feature vector of the image to be processed, the distance between the feature vector of the image to be processed and the feature vector corresponding to each category in the candidate categories; The confidence level that the image to be processed belongs to the candidate category is determined according to the distance.

15. The device according to claim 12, characterized in that The classification module is specifically used for: The category with the highest confidence among the candidate categories is determined as the classification result of the image to be processed.

16. The device according to any one of claims 13 to 15, characterized in that The local category features of the training images are determined by a plurality of training images in the support set that have been expanded, wherein the expansion process includes cropping, flipping and / or data hallucination processing of the images.

17. A training device for an image classification model, characterized in that: include: An acquisition module, configured to acquire a plurality of training images in a training set, wherein the training set includes a support set and a query set, and the plurality of training images include a plurality of training images in the support set and a plurality of training images in the query set; A feature extraction module, used for extracting feature vectors of a plurality of training images in the query set according to a preset first neural network, wherein the query set includes some images in some categories in the training set; a classification module, used for processing feature vectors of a plurality of training images in the query set according to a preset second neural network and a preset global category feature to obtain classification results of the plurality of training images in the query set, wherein the global category feature includes a plurality of category features, and the plurality of category features in the global category feature are used to indicate visual features of all categories in the training set, and all categories in the training set are categories to which all training images in the training set belong, and the training set includes images in a base class and images in a new class; An updating module, configured to update the global category feature according to the classification results of the plurality of training images in the query set; The update module is specifically used for: According to the classification error, updating the global category feature; or The global category feature is updated according to the classification error and the registration error, wherein the registration error is determined according to the local category feature of the training image and multiple category features in the global category feature, and the classification error is determined according to the classification results of multiple training images in the query set and labels pre-annotated for the multiple training images in the query set, and the label is used to indicate the category to which the training image belongs.

18. The device according to claim 17, characterized in that The feature extraction module is also used for: Extracting feature vectors of a plurality of training images in the support set, wherein the support set includes some training images in some categories in the training set; The classification module is specifically used for: Determine local category features of the training images according to feature vectors of the plurality of training images in the support set, wherein a plurality of category features in the local category features of the training images are used to indicate visual features of all categories in the support set, and the support set includes some training images in some categories in the training set; The classification results of the plurality of training images in the query set are determined according to the second neural network, the local category features of the training images, and the global category features.

19. The device according to claim 17, characterized in that The update module is specifically used for: The first neural network and the second neural network are updated according to the classification results of the plurality of training images in the query set.

20. The device according to claim 18 or 19, characterized in that The local category features of the training images are determined by a plurality of training images in the support set that have been expanded, wherein the expansion process includes cropping, flipping and / or data hallucination processing of the images.

21. An image classification device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 1 to 6.

22. A training device for an image classification model, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 7 to 10.

23. A computer-readable storage medium, characterized in that: The computer-readable medium stores a program code for execution by a device, the program code including a program for executing the method according to any one of claims 1 to 6 or 7 to 10.

24. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method as claimed in any one of claims 1 to 6 or 7 to 10.

Images