Continual small sample image classification method and system based on dynamic neural network expansion

Abstract

The application belongs to the field of computer vision, and particularly relates to a kind of based on dynamic neural network extension's continuous small sample image classification method and system, aims at solving the problem that model overall is difficult to adjust when training new class in prior art.The application includes: training basic feature extraction network with sample with known class data label, feature extraction is carried out to sample with known class data label by feature extractor, then projection layer is projected into expandable feature space and unused area is kept;Sample with new class data label and sample with known class data label are combined for training;New class visual feature is projected into unused area by projection layer of extended part parameter and aligned with known class, and classification is carried out by incremental class prototype classifier.The application is simple and flexible, can significantly improve the classification performance of new class object, and can effectively improve the classification performance of historical class object.

Description

TECHNICAL FIELD

[0001] The present specification belongs to the field of computer vision, and particularly relates to a continuous few-shot image classification method and system based on dynamic neural network expansion. BACKGROUND

[0002] The core task of few-shot learning is image classification, which requires the target model to be able to expand to the specified class and make correct judgments. When using a deep model to solve the classic image classification problem, the training set contains a large number of labeled samples of target classes, so the model can obtain a large amount of related knowledge of the target task in the training stage, thereby having good performance on the test set. Unlike previous use of deep models to solve image classification problems, the target classes in the few-shot image classification problem contain only a small number of labeled samples, or even only one labeled sample, so the traditional deep model training method is not suitable for few-shot image classification problems. Since the fitting of a deep model requires the support of a large number of training samples, and the number of labeled samples of a few-shot task is difficult to meet this requirement, the model needs to be pre-trained on a large existing dataset. After pre-training, the model has certain prior knowledge and feature extraction capability, and we migrate it to the downstream few-shot task to combine the knowledge learned by the model with the small amount of labeled samples of the target task to solve the classification problem.

[0003] Continuous few-shot learning aims to design a machine learning algorithm that enables the model to continuously learn the knowledge of new classes through a small number of labeled samples, while not forgetting the knowledge of old classes. The difficulty of this problem lies in the fact that limited data from new classes not only leads to serious overfitting, but also exacerbates catastrophic forgetting of old class knowledge. In addition, since the training data is input sequentially, the classifier learned in the current stage can only provide discrimination information for specific classes, while continuous few-shot learning requires all classes to participate in testing. Considering that there are a large number of open problems with continuous growth in real-world scenarios, continuous few-shot learning is a valuable research direction.

[0004] To solve this problem, a simple approach is to retain part of the labeled samples of the appeared classes, and when new classes are added, input the labeled samples of the old classes and the labeled samples of the new classes into the neural network at the same time, and fine-tune the model as a whole, which ensures that the model does not forget the knowledge of old classes while increasing the recognition ability of the model for new classes. The disadvantage of this method is that there are only a small number of labeled samples for new classes, which makes it difficult to adjust the model as a whole, and at the same time, it will destroy the efficient feature extraction network obtained by the model under a large amount of training data. Therefore, how to use only a small number of labeled samples to help the model incrementally learn the feature extraction ability of new classes is the research difficulty of current continuous few-shot learning. SUMMARY

[0005] To solve the above problems in the prior art, that is, when a new class is added, the labeled samples of the old class and the labeled samples of the new class are input into the neural network at the same time, the model is overall fine-tuned, the model is ensured not to forget the knowledge of the old class, and the recognition ability of the model to the new class is increased, but the new class has only a small amount of labeled samples, it is difficult to adjust the model as a whole, and at the same time, the efficient feature extraction network obtained by the model under a large amount of training data is damaged, the specification provides a continuous small sample image classification method based on dynamic neural network expansion, the method comprises:

[0006] Step S100, training a basic feature extraction network through samples with known class data labels, specifically:

[0007] The feature extraction network comprises a feature extractor and a projection layer;

[0008] The feature extractor is used for feature extraction on the samples with known class data labels, and basic visual features are obtained; the basic visual features are projected into an expandable feature space through the feature projection layer, data obtained by projecting the basic visual features in the expandable feature space is defined as known classes, a blank area of the basic visual features which is not projected in the expandable feature space is reserved, and the basic feature extraction network is determined;

[0009] Step S200, when it is necessary to expand the basic feature extraction network, combining samples with new class data labels and samples with known class data labels to obtain expansion training samples;

[0010] Step S300, fixing the parameters of the feature extractor of the basic feature extraction network, and performing feature extraction on the expansion training samples to obtain known class visual feature data and new class visual feature data, and projecting the new class visual feature data into the expandable feature space through the projection layer, and setting the data projected from the new class visual feature data as a new class;

[0011] Step S400, aligning the new class and the known class in the expandable feature space, so that the training loss of the projection layer is lower than a preset threshold or takes a minimum value, and a trained feature extraction network is obtained;

[0012] Step S500, acquiring a to-be-classified image through an image acquisition device, extracting visual features of the to-be-classified image through the trained feature extraction network, and then classifying the visual features of the to-be-classified image through an incremental class prototype classifier to obtain an image classification result.

[0013] In some preferred embodiments, the projection layer is constructed based on a learnable matrix;

[0014] In some preferred embodiments, the projection layer projects the base visual features into the expandable feature space through the projection layer and defines the known classes;

[0015] In some preferred embodiments, when the base feature extraction network needs to be expanded, the projection layer re-divides the expansion parameters from the unused parameters in the learnable matrix and the unused parameters in the new learnable matrix, randomly initializes the expansion parameters, and trains the expansion parameters through the expansion training samples, so as to embed the new class visual feature data into the expandable feature space and align the new class with the known classes.

[0016] In some preferred embodiments, the new class is aligned with the known classes, specifically:

[0017] For the new class visual feature data in the expandable feature space, the training loss L of the projection layer cls is:

[0018]

[0019] wherein, represents the mathematical expectation of all samples x from the distribution, p is the probability, and y * is the true label corresponding to x.

[0020] In some preferred embodiments, after obtaining the expansion training samples, a step of screening representative data is further included, specifically:

[0021] The expansion training samples select one sample closest to the cluster center of each class as the representative data through the Kmeans clustering algorithm and input the base feature extraction network to be trained.

[0022] Another aspect of the present specification provides a continuous small sample image classification system based on dynamic neural network expansion, which comprises:

[0023] A preliminary training module configured to train a base feature extraction network through samples with known class data labels, specifically: the feature extraction network comprises a feature extractor and a projection layer;

[0024] The feature extractor extracts features of the samples with known class data labels to obtain base visual features; the base visual features are projected into an expandable feature space through the feature projection layer, the data projected by the base visual features in the expandable feature space is defined as known classes, the blank area of the base visual features not projected in the expandable feature space is reserved, and the base feature extraction network is determined.

[0025] The extended training sample acquisition module is configured to combine samples with new class data labels and samples with known class data labels to obtain extended training samples when the base feature extraction network needs to be extended.

[0026] The new class embedding module is configured to fix the parameters of the feature extractor of the base feature extraction network, extract features of the extended training samples to obtain known class visual feature data and new class visual feature data, project the new class visual feature data to the extensible feature space through the projection layer, and set the projected data of the new class visual feature data as the new class.

[0027] The class alignment module is configured to align the new class and the known class in the extensible feature space, so that the training loss of the projection layer is lower than a preset threshold or takes a minimum value, and obtain a trained feature extraction network.

[0028] The image classification module is configured to acquire a to-be-classified image through an image acquisition device, extract visual features of the to-be-classified image through the trained feature extraction network, and then classify the visual features of the to-be-classified image through the incremental class prototype classifier to obtain an image classification result.

[0029] In some preferred embodiments, the projection layer, in particular the projection layer constructed based on a learnable matrix, projects the visual features to the known classification in the extensible feature space.

[0030] A third aspect of the present specification provides an electronic device, comprising:

[0031] at least one processor; and

[0032] a memory communicatively connected with the at least one processor; wherein

[0033] The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned dynamic neural network expansion based continuous small sample image classification method.

[0034] A fourth aspect of the present specification provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned dynamic neural network expansion based continuous small sample image classification method.

[0035] The present specification has the following beneficial effects:

[0036] (1) The present specification uses a small amount of labeled samples to help the model incrementally learn the feature extraction ability of the new class by projecting visual features into an extensible feature space. The obtained image classification model truly realizes the ability to ensure that the model can cope with the catastrophic forgetting of old class knowledge while increasing the recognition ability of the model to new classes, i.e., accuracy, without damaging the efficient feature extraction network obtained by the model under a large amount of training data.

[0037] (2) The method of the present specification can be used in the case of dynamically emerging new classes over time. Even with only a small amount of labeled samples, the method can quickly understand and recognize newly emerging objects, and at the same time, it will not excessively forget all object classes that have appeared in the history task over time.

[0038] (3) The present specification uses dynamic neural network expansion technology. The model can dynamically adjust the feature space as the task increases, and adaptively align the features of old and new classes. At the same time, the class representative point extraction method based on clustering can effectively reduce the consumption of computing resources. The method of the present specification is simple and flexible, which can significantly improve the classification performance of new class objects and effectively improve the classification performance of historical class objects. BRIEF DESCRIPTION OF DRAWINGS

[0039] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the following drawings:

[0040] Figure 1 is a flowchart of a continuous small sample image classification method based on dynamic neural network expansion in an embodiment of the present specification;

[0041] Figure 2 is a schematic diagram of the principle of the continuous small sample image classification method based on dynamic neural network expansion in an embodiment of the present specification. DETAILED DESCRIPTION

[0042] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0043] It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0044] The specification provides a method for continuous small sample image classification based on dynamic neural network expansion. The method uses a method of projecting visual features into an expandable feature space, only uses a small number of labeled samples, and helps the model to incrementally learn the feature extraction capability of the new class. The obtained image classification model truly realizes the guarantee of not forgetting the old class knowledge while increasing the recognition ability of the model to the new class, and does not damage the efficient feature extraction network obtained by the model under a large amount of training data.

[0045] In order to more clearly illustrate the method for continuous small sample image classification based on dynamic neural network expansion of the specification, the following will be combined with Figure 1 and Figure 2 The steps in the embodiments of the specification will be described in detail.

[0046] The method for continuous small sample image classification based on dynamic neural network expansion of the first embodiment of the specification includes steps S100-S500, which are described in detail as follows:

[0047] For unlabeled data, the unlabeled samples are classified by a prototype-based nearest neighbor classifier.

[0048] In step S100, the basic feature extraction network is trained by samples with known class data labels, specifically:

[0049] The feature extraction network includes a feature extractor and a projection layer;

[0050] The feature extractor extracts features from samples with known class data labels to obtain basic visual features; the basic visual features are projected into an expandable feature space through the feature projection layer, and the data projected by the basic visual features in the expandable feature space is defined as known classes, and the blank area of the basic visual features in the expandable feature space is reserved to determine the basic feature extraction network.

[0051] The method can initially train the extraction capability of the deep neural network by a large amount of labeled sample known class data.

[0052] In this embodiment, the projection layer is constructed based on a learnable matrix.

[0053] In this embodiment, the projection layer projects the basic visual features into an expandable feature space through the projection layer and defines them as known classes.

[0054] As Figure 2 The blank part of the projection layer is the reserved unused parameters. In the subsequent expansion step, if a new class is added to the feature extraction network, the expansion parameters are drawn from the blank area (unused parameters), and the remaining part is still reserved as unused parameters.

[0055] Step S200: When it is necessary to expand the basic feature extraction network, combine the samples with new class data labels and the samples with known class data labels to obtain expanded training samples.

[0056] After obtaining the expanded training samples, the next step is to select representative data, specifically:

[0057] The extended training samples are selected using the K-means clustering algorithm, with the sample closest to the cluster center of each class being used as representative data and input into the basic feature extraction network to be trained.

[0058] For example, if we want to retain 5 samples for each known class, we cluster the labeled data of the known classes into 5 classes based on the basic visual features extracted by the feature extractor. Then, the sample closest to the cluster center in each cluster retains its original image as representative data. When training new classes in this specification, to ensure that the model does not forget the knowledge of old classes, a small number of old class samples need to be saved and trained simultaneously with the new class samples. Since the base classes contain a large amount of labeled data, in order to reduce memory consumption and ensure the uniformity of the training data distribution, we apply the K-means clustering algorithm once to the labeled data of each base class to extract representative data of each class, maximizing the preservation of the class characteristics of the base classes while reducing the consumption of computing resources.

[0059] Step S300: Fix the parameters of the feature extractor in the basic feature extraction network, and extract features from the expanded training samples to obtain known class visual feature data and new class visual feature data. Project these features onto the expandable feature space through a projection layer, and set the projected data of the new class visual feature data as the new class; for example... Figure 2 As shown, Figure 2 The blank areas in the projection layer represent unused parameters.

[0060] Step S400: Align the new classes in the expandable feature space with the known classes, so that the training loss of the projection layer is lower than a preset threshold or reaches the minimum value, and obtain the trained feature extraction network.

[0061] In this embodiment, when the projection layer needs to expand the basic feature extraction network, it re-divides the unused parameters in the learnable matrix into expanded parameters and unused parameters in the new learnable matrix, randomly initializes the expanded parameters, and trains the expanded parameters through expanded training samples, thereby embedding the new class of visual feature data into the expandable feature space and aligning the new class with the known class.

[0062] like Figure 2The lower half, the projection layer divides the expansion part to participate in the training, and the expansion part projects the new class visual feature data of the expanded training sample into the expandable feature space, that is, the visual features of the new class are added in the expandable feature space, and the alignment of the visual features of the new class and the visual features of the known classes is performed in the expandable feature space, that is, the new unused area is reserved, and the unused parameters in the projection layer are also reserved for subsequent expansion.

[0063] In this embodiment, each time the base feature extraction network is expanded, the expansion parameters and the reserved unused parameters are determined according to the data amount of the new identifiable class to be added.

[0064] In this embodiment, the alignment of the new class and the known class is specifically:

[0065] For the new class visual feature data in the expandable feature space, the training loss L of the projection layer cls is:

[0066]

[0067] Wherein, represents the mathematical expectation of all samples x from distribution, p is the probability, and y * is the true label corresponding to x.

[0068] In this embodiment, as Figure 2 shown, after the base feature extraction network is trained with training data with known class labels, the base class training of the class prototype classifier is needed, and after the feature extraction network is expanded and the new class is aligned with the known class, the base class training of the class prototype classifier is needed

[0069] Step S500, acquire the image to be classified through the image acquisition device, extract the visual features of the image to be classified through the trained feature extraction network, and then classify the visual features of the image to be classified through the incremental class prototype classifier to obtain the image classification result.

[0070] Although the above embodiment describes each step in the above order, those skilled in the art can understand that, in order to achieve the effect of the embodiment, different steps do not have to be executed in such order, and they can be executed simultaneously (in parallel) or in reverse order, and these simple changes are within the protection scope of the present specification.

[0071] The second embodiment of the present specification is a continuous small sample image classification system based on dynamic neural network expansion, which comprises:

[0072] The preliminary training module is configured to train a basic feature extraction network by samples with known class data labels, and specifically, the feature extraction network comprises a feature extractor and a projection layer.

[0073] The feature extractor is configured to extract features of the samples with known class data labels to obtain basic visual features, and the basic visual features are projected into an expandable feature space through the feature projection layer, and data of the basic visual features projected in the expandable feature space is defined as known classes, and a blank area of the basic visual features not projected in the expandable feature space is reserved to determine the basic feature extraction network.

[0074] In the embodiment, the projection layer, specifically a projection layer constructed based on a learnable matrix, is configured to project the visual features into the known classes of the expandable feature space.

[0075] The expansion training sample acquisition module is configured to combine samples with new class data labels and samples with known class data labels to obtain expansion training samples when the basic feature extraction network needs to be expanded.

[0076] The new class embedding module is configured to fix parameters of the feature extractor of the basic feature extraction network, extract features of the expansion training samples to obtain known class visual feature data and new class visual feature data, project the new class visual feature data into the expandable feature space through the projection layer, and set data projected by the new class visual feature data as a new class.

[0077] The class alignment module is configured to align the new class and the known class in the expandable feature space, so that the training loss of the projection layer is lower than a preset threshold or takes a minimum value, and a trained feature extraction network is obtained.

[0078] The image classification module is configured to acquire a to-be-classified image through an image acquisition device, extract visual features of the to-be-classified image through the trained feature extraction network, and then classify the visual features of the to-be-classified image through the incremental class prototype classifier to obtain an image classification result.

[0079] It should be noted that the above embodiment provides a continuous small sample image classification system based on dynamic neural network expansion, and only the division of the above functional modules is exemplified. In actual application, the above functions can be completed by different functional modules according to needs, that is, the modules or steps in the embodiments of the present specification are further decomposed or combined, for example, the modules of the above embodiments can be combined into one module, or can be further split into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present specification are only for distinguishing various modules or steps, and are not considered as improper limitation of the present specification.

[0080] An electronic device of the third embodiment of the present specification comprises:

[0081] at least one processor; and

[0082] a memory connected in communication with the at least one processor; wherein

[0083] The memory stores instructions executable by the processor, and the instructions are executed by the processor to implement the above-mentioned continuous small sample image classification method based on dynamic neural network expansion.

[0084] A computer readable storage medium of the fourth embodiment of the present specification stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned continuous small sample image classification method based on dynamic neural network expansion.

[0085] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the storage device and the processing device described above and the related description can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

[0086] Those skilled in the art should clearly understand that the modules and method steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. The programs corresponding to the software modules and method steps can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are performed by electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present specification.

[0087] The terms "first", "second", and the like are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0088] The term "comprising" or any other similar term is intended to encompass a non-exclusive inclusion, such that a process, method, article, or apparatus / device including a series of elements includes not only those elements expressly listed, but also other elements inherent in such process, method, article, or apparatus / device or latent in such process, method, article, or apparatus / device.

[0089] So far, the technical solution of the present specification has been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the present specification is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present specification, and the technical solutions after such changes or replacements will all fall within the protection scope of the present specification.

Claims

1. A persistent few-shot image classification method based on dynamic neural network extension, characterized in that, The method includes: The basic feature extraction network is trained using samples with known class labels, specifically as follows: The feature extraction network includes a feature extractor and a projection layer; The basic visual features are obtained by extracting features from samples with known class data labels using a feature extractor. The basic visual features are then projected onto an expandable feature space through a projection layer. The data projected by the basic visual features in the expandable feature space is defined as the known class. The blank areas of the basic visual features that are not projected in the expandable feature space are retained, and the basic feature extraction network is determined. When it is necessary to expand the basic feature extraction network, samples with new class data labels and samples with known class data labels are combined to obtain expanded training samples. The parameters of the feature extractor of the basic feature extraction network are fixed, and feature extraction is performed on the extended training samples to obtain known class visual feature data and new class visual feature data. These are then projected onto the scalable feature space through a projection layer, and the data after projection of the new class visual feature data is set as the new class. Align the new classes in the scalable feature space with the known classes to make the training loss of the projection layer lower than a preset threshold or take the minimum value, and obtain a well-trained feature extraction network. An image to be classified is acquired through an image acquisition device. The visual features of the image to be classified are extracted through the trained feature extraction network. Then, the visual features of the image to be classified are classified through an incremental class prototype classifier to obtain the image classification result.

2. The persistent few-shot image classification method based on dynamic neural network extension according to claim 1, characterized in that, The process of aligning the new class with known classes specifically involves: For novel visual feature data in a scalable feature space, the training loss of the projection layer L... cls for: in, Indicates that it is aimed at those from The expected value of all samples x from the distribution, where p is the probability and y is the expected value of the sample x. * The actual label corresponding to x.

3. The persistent few-shot image classification method based on dynamic neural network extension according to claim 1, characterized in that, After obtaining the expanded training samples, the next step is to select representative data, specifically: The extended training samples are selected using the K-means clustering algorithm, with the sample closest to the cluster center of each class being used as representative data and input into the basic feature extraction network to be trained.

4. A persistent few-shot image classification system based on dynamic neural network extension, characterized in that, The system includes: The initial training module is configured to train a basic feature extraction network using samples with known class data labels. Specifically, the feature extraction network includes a feature extractor and a projection layer. The basic visual features are obtained by extracting features from samples with known class data labels using a feature extractor. The basic visual features are then projected onto an expandable feature space through a feature projection layer. The data projected by the basic visual features in the expandable feature space is defined as the known class. The blank areas of the basic visual features that are not projected in the expandable feature space are retained, and the basic feature extraction network is determined. The extended training sample acquisition module is configured to combine samples with new class data labels and samples with known class data labels to obtain extended training samples when it is necessary to extend the basic feature extraction network. The new class embedding module is configured to fix the parameters of the feature extractor of the basic feature extraction network, extract features from the extended training samples to obtain known class visual feature data and new class visual feature data, project them to the scalable feature space through the projection layer, and set the data after the projection of the new class visual feature data as the new class. The class alignment module is configured to align new classes in the scalable feature space with known classes, so that the training loss of the projection layer is lower than a preset threshold or reaches the minimum value, thereby obtaining a well-trained feature extraction network. The image classification module is configured to acquire an image to be classified through an image acquisition device, extract visual features of the image to be classified through the trained feature extraction network, and then classify the visual features of the image to be classified through an incremental class prototype classifier to obtain an image classification result.

5. An electronic device, comprising: At least one processor; And a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor to implement the continuous few-shot image classification method based on dynamic neural network extension as described in any one of claims 1-3.

6. A computer-readable storage medium storing computer instructions for execution by the computer to implement the continuous few-sample image classification method based on dynamic neural network extension as described in any one of claims 1-3.

Images