Small sample image incremental classification method and device based on embedding enhancement and adaptation

Abstract

The present invention proposes a small sample image incremental classification method based on embedded enhancement and self-adaptation, the method comprising: obtaining an image incremental classification system, the system is used to perform classification tasks on images to be classified; obtaining images to be classified, and uploading them to the system for recognition, when the system recognition fails, obtaining a small number of images of the category as training samples, and calculating the target prototype through a feature pre-training module, and adaptively adjusting the target prototype and / or the original prototype through a hybrid relationship mapping module to update all prototypes in the system to achieve classification and recognition of the images to be classified in the category; when the system recognizes successfully, the images to be classified are classified and recognized through a feature pre-training module, a hybrid relationship mapping module and a classifier, and the classification result is output. The method is used to enhance the scalability of the classifier, and introduce a hybrid relationship mapping mechanism to optimize the prototype representation of the sample, so that the system is gradually applicable to the recognition of all visible class images.

Description

Technical Field

[0001] The present invention relates to the field of automatic recognition technology, and in particular to a small sample image incremental classification method and device based on embedding enhancement and self-adaptation. Background Art

[0002] Thanks to the availability of a large number of labeled data sets, deep learning technology has achieved remarkable success in many computer vision tasks. Manually labeling data is an expensive and time-consuming process, and there are many types of images, making it almost impossible to fully label all possible image types at once. Therefore, most current classification algorithms are developed in a closed, static environment and are developed for one or several specific categories. However, actual scenes are usually dynamic, open, and non-stationary. With the continuous emergence of new category samples, classification algorithms require a large amount of new category labeled data and old data to retrain the model, resulting in high costs. At the same time, the amount of data for new categories is often relatively small, and the scarcity of data will cause the model to be severely overfitted, which is not conducive to incremental learning. The main research programs related to image classification are:

[0003] Solution 1) Incremental learning: Aims to be able to handle a sufficient amount of new data that continuously appears in the real world, and retain or even integrate and optimize old knowledge while learning new knowledge. Most research methods store limited old class samples or learn from old classes based on loss to prevent forgetting previous tasks. These methods work well when there is enough new sample data, but perform poorly when there are fewer incremental samples. The data shortage problem in small sample incremental learning will further aggravate the problem of knowledge forgetting and overfitting.

[0004] Solution 2) Small sample learning: The goal is to enable the model to classify unseen images when trained only from rare labeled training examples. Most studies train model parameters through meta-learning, metric learning, or optimization learning, and allow the model to quickly adapt to new small sample data. However, the model focuses more on capturing features that are useful for the current task, while discarding data features that are discriminative for previous tasks or future tasks, which is not conducive to incremental learning.

[0005] Solution 3) Incremental small sample learning: aims to incrementally acquire the ability to classify new categories without forgetting old categories from a small amount of sample data. There are two main approaches in current research: one is to fine-tune the embedding model and unify the classifier with a small amount of new sample data, but fine-tuning the network in a new session will cause the knowledge of old categories to be forgotten; the other is to decouple the learning of the embedding representation and the classifier, and only update the classifier when learning a new task, but freezing the embedding representation network lacks adaptability to the feature embedding of subsequent small sample incremental tasks, which is not conducive to adaptive learning of new samples.

[0006] Therefore, in the prior art, most of the research work has not yet proposed an effective method to alleviate the problem of dynamic development of real scenes, which limits the development of deep learning technology in related fields. In view of the shortcomings of the prior art, the present invention designs a small sample image incremental classification method and device based on embedding enhancement and adaptation on the basis of decoupling the embedding representation and the classifier in solution 3). Different from it, we believe that well-generalized feature embedding is crucial for subsequent small sample incremental tasks, so we further enhance the feature extraction network; at the same time, a hybrid relationship mapping method is used to adaptively adjust the prototype representation and the query data feature embedding representation. Summary of the invention

[0007] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0008] To this end, the first object of the present invention is to propose a small sample image incremental classification method based on embedding enhancement and adaptation, so as to avoid complete retraining of the model when new categories of images appear, greatly reduce computing resource overhead and promote the long-term operation of the system.

[0009] The second object of the present invention is to propose a small sample image incremental classification device based on embedding enhancement and adaptation.

[0010] A third object of the present invention is to provide a non-transitory computer-readable storage medium.

[0011] A fourth object of the present invention is to provide a computer program product.

[0012] To achieve the above object, a first embodiment of the present invention provides a method, comprising:

[0013] Acquire an image incremental classification system, wherein the image incremental classification system is used to perform an incremental classification task on the image to be classified;

[0014] Acquire an image to be classified, and upload the image to be classified to the image incremental classification system for identification;

[0015] When the image incremental classification system fails to recognize, a small number of images of the category are obtained from the images to be classified as training samples, and the training samples are calculated by a feature pre-training module to obtain a target prototype, and the target prototype and / or the original prototype are adaptively adjusted by a hybrid relationship mapping module to update all prototypes in the image incremental classification system, so as to realize the classification and recognition of the images to be classified of unrecognized categories through the adjusted image incremental classification system;

[0016] When the image incremental classification system recognizes successfully, classification and recognition are performed according to the image to be classified through the feature pre-training module, the hybrid relationship mapping module and the NCM classifier, and the classification result is output.

[0017] Optionally, in one embodiment of the present invention, the image acquisition incremental classification system comprises:

[0018] Select an embedding representation model based on the characteristics of the data set and the task, and combine self-supervised learning and attention mechanism to form a feature pre-training module, and pre-train the feature pre-training module of the model based on the image classification and annotation results;

[0019] Obtaining an output result of the feature pre-training module;

[0020] Based on the output results of the feature pre-training module, the training of the image incremental classification system is completed.

[0021] Optionally, in one embodiment of the present invention, the output result of obtaining the feature pre-training module may include:

[0022] The output result of the feature pre-training module does not reach the expected accuracy. The hyperparameters of the feature pre-training module are adjusted, and the feature pre-training module is further pre-trained.

[0023] The output result of the feature pre-training module reaches the expected accuracy, the parameters of the feature pre-training module are frozen, and the pseudo-incremental episodes are selected by the pseudo-incremental episode selection module to train the hybrid relationship mapping module and the NCM classifier.

[0024] Optionally, in one embodiment of the present invention, the pseudo-incremental plot selection module includes:

[0025] The pseudo base class includes: generating a support set S in N-way K-shot form from the basic data set in each iteration b , the corresponding query set Q b It is formed by sampling from N categories, each category contains E query samples, where N and E are positive integers, E>N, the query set is different from the samples in the support set, and can be expressed as (S b ,Q b ) to indicate;

[0026] Pseudo-incremental class, the pseudo-incremental class includes: rotating each sample of the pseudo-base class by 270 degrees, which can be used (S i ,Q i ) to indicate.

[0027] Optionally, in one embodiment of the present invention, the hybrid relationship mapping module includes:

[0028] According to the pseudo-incremental plot, the feature pre-training module F is used to extract the feature representation of the pseudo base class and the pseudo-incremental class. For the support set feature representation, we use the mean vector to calculate a prototype vector for each class as the initial weight of the classifier.

[0029]

[0030] Wherein, c represents the category, the pseudo base category and the pseudo increment category have a total of 2N categories, s j For S b ∪S i A sample of For j We merge the prototype representations and query set embedding representations of the pseudo base class and pseudo incremental class together to obtain the prototype representation set and query set embedding representation set of all categories, and use M c and X q express;

[0031] Prototype Self-Mapping (PSP), wherein PSP adaptively adjusts all prototypes by establishing a global dependency between the original prototype representation and the new prototype representation; wherein the input of PSP is in the form of a triple of (Query, Key, Value), and Query, Key, and Value share the same input source M c , PSP can be expressed as:

[0032] Query = M c W Q Key=M c W K Value = M c W V

[0033]

[0034] d is the dimension of Query, W Q / W K / W V are the learnable parameters of the three linear projection layers. The original prototype is projected into the shared metric space. The relationship matrix between the prototype representations in the shared space is obtained by softmax normalization. The relationship matrix is ​​used as a weight coefficient to aggregate the information from all prototype representations in Value and merged with the original prototype to obtain the updated prototype M. c ′;

[0035] Query set cross mapping (QCP), which establishes correlation between the query set embedding representation and each prototype to adapt to the current classification task; the query set embedding representation needs to be classified by the distance between it and the prototype representation. To this end, we introduce the query set embedding representation X q To prototype M c The cross-mapping of ′ adjusts the query set embedding representation so that the query set samples can better adapt to the target classification task. The formula is as follows:

[0036]

[0037] Optimization training, wherein the optimization training calculates the semantic difference between the embedding representation of all query sets and the prototype through a cosine similarity function, classifies using a nearest mean (NCM) classifier, and optimizes model parameters using a cross entropy loss function.

[0038] Optionally, in one embodiment of the present invention, the training of the hybrid relationship mapping module and the NCM classifier includes:

[0039] The training results of the hybrid relationship mapping module and the NCM classifier did not reach the expected accuracy, so the hyperparameters were adjusted, and the pseudo-incremental class was selected through the pseudo-incremental episode selection module to continue training the hybrid relationship mapping module and the NCM classifier;

[0040] The training results of the hybrid relationship mapping module and the NCM classifier reach the expected accuracy, the parameters of the hybrid relationship mapping module are frozen, and the process ends.

[0041] To achieve the above-mentioned purpose, the second aspect of the present invention proposes a small sample image incremental classification device based on embedding enhancement and adaptation, comprising:

[0042] A first acquisition module is used to acquire an image incremental classification system, where the image incremental classification system is used to perform an incremental classification task on the image to be classified;

[0043] A second acquisition module is used to acquire the image to be classified and upload the image to be classified to the image incremental classification system for identification;

[0044] A first classification module is used to obtain a small number of images of the category from the images to be classified as training samples when the image incremental classification system fails to recognize, and calculate the target prototypes from the training samples through a feature pre-training module, and adaptively adjust the target prototype and / or the original prototype through a hybrid relationship mapping module to update all prototypes in the image incremental classification system, so as to realize the classification and recognition of the images to be classified of unrecognized categories through the adjusted image incremental classification system;

[0045] The second classification module is used to perform classification and recognition based on the image to be classified through the feature pre-training module, the hybrid relationship mapping module and the NCM classifier when the image incremental classification system successfully recognizes it, and output the classification result.

[0046] Optionally, in one embodiment of the present invention, the image acquisition incremental classification system comprises:

[0047] Select an embedding representation model based on the characteristics of the data set and the task, and combine self-supervised learning and attention mechanism to form a feature pre-training module, and pre-train the feature pre-training module of the model based on the image classification and annotation results;

[0048] Obtaining an output result of the feature pre-training module;

[0049] Based on the output results of the feature pre-training module, the training of the image incremental classification system is completed.

[0050] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the small sample image incremental classification method based on embedded enhancement and adaptation described in the first aspect of the present invention is implemented.

[0051] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the small sample image incremental classification method based on embedded enhancement and adaptation described in the first aspect of the present invention.

[0052] In summary, the present invention proposes a method, apparatus, computer device and non-temporary computer-readable storage medium for incremental classification of small sample images based on embedding enhancement and adaptation, so that the model is adaptable to better handle new and old categories, so that the system can quickly adapt to the embedding expression of new categories from small sample data, incrementally have the ability to recognize new samples, avoid complete retraining of the new model to reduce a large amount of computing resource overhead and promote the long-term operation of the system.

[0053] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0055] Figure 1A schematic diagram of a flow chart of a small sample image incremental classification method based on embedding enhancement and adaptation provided by an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of the framework of an image incremental classification system provided by the present invention;

[0057] Figure 3 A structural diagram of an image incremental classification model provided by the present invention;

[0058] Figure 4 A training flow chart of an image incremental classification system provided by the present invention;

[0059] Figure 5 A flow chart of a small sample image incremental classification method based on embedding enhancement and adaptation provided by the present invention;

[0060] Figure 6 A schematic diagram of the structure of a small sample image incremental classification device based on embedding enhancement and adaptation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0062] The following describes a method and device for incremental classification of small sample images based on embedded enhancement and adaptation according to an embodiment of the present invention with reference to the accompanying drawings.

[0063] Figure 1 A flowchart of a small sample image incremental classification method based on embedding enhancement and adaptation provided by an embodiment of the present invention.

[0064] like Figure 1 As shown, the small sample image incremental classification method based on embedding enhancement and adaptation includes the following steps:

[0065] Step S1: Acquire an image incremental classification system, where the image incremental classification system is used to perform incremental classification tasks on images to be classified.

[0066] In one embodiment of the present invention, the image incremental classification system is composed of four modules including: a feature pre-training module, a pseudo-incremental scenario selection module, a hybrid relationship mapping module and an NCM classifier.

[0067] In one embodiment of the present invention, the feature pre-training module can select the ResNet series model that combines self-supervised learning and attention mechanism as the backbone network of the feature embedding module to obtain an embedding representation network with generalization ability and certain discrimination ability. The feature pre-training module F takes the image I as input, generates a two-dimensional feature vector: x=F(I), and performs classification training through the cosine classifier. Freezing the feature pre-training module parameters during the new task learning process can minimize the problem of forgetting old knowledge and overfitting of new data.

[0068] Among them, in one embodiment of the present invention, self-supervised learning can learn a more complete data space structure, extract more comprehensive information, learn a more robust and generalized feature representation between new and old categories, and use rotation prediction as an auxiliary task to rotate each training sample in the base class by 0, 90 and 180 degrees to expand the original training set, expanding the original K-category problem into a new 3K-category problem:

[0069] I′ i =Rotate(I i ,θ),θ∈{0°,90°,180°})

[0070] I i represents the i-th input image, θ represents the image rotation degree, I′ i Represents the rotated image, and the rotated sample is assigned a new label Y′ i , the label can be automatically generated.

[0071] Furthermore, in one embodiment of the present invention, the attention mechanism can better utilize the unique discriminative features of sparse samples when processing small sample incremental tasks, obtain more representative prototype representations, maintain an effective balance between the generalization and discriminability of the feature pre-training module, and facilitate decision-making between new and old categories.

[0072] Specifically, in one embodiment of the present invention, the attention mechanism captures long-range dependencies without being limited to adjacent points by calculating the interaction information between any two positions of the feature map through non-local calculation, so that more information can be integrated to obtain the self-attention map MA(x i ):

[0073]

[0074] Among them, x represents the feature map to be classified, x i Represents the information of the current focus position, x j represents the global information. θ, φ and g are all 1×1 convolution operations. represents the normalization operation. The final attention feature map becomes:

[0075]

[0076] Among them, W z is a learnable weight matrix implemented by a 1*1 convolution operation. i Represents a residual connection that adds contextual information back to the original feature map for enhancement.

[0077] In one embodiment of the present invention, the pseudo-incremental plot selection module has two important concepts: pseudo base class and pseudo incremental class, both of which are composed of support set and query set. Pseudo incremental samples are new classes that cannot be identified during the training process of feature pre-training module to help the training of hybrid relationship mapping module. Since rotating data at a larger angle may cause the synthetic image to lose some semantic information, the base class image is rotated 270 degrees as a pseudo incremental class.

[0078] Specifically, in one embodiment of the present invention, in each iteration, a support set S in the form of N-way K-shot is generated using the basic data set. b , the corresponding query set Q b It is sampled from N categories, each category contains E query samples. The query set is different from the samples in the support set. We use (S b ,Q b ) is called a pseudo base class. Each sample of the pseudo base class is rotated 270 degrees to form a pseudo increment class. We use (S i ,Q i ) to indicate.

[0079] In one embodiment of the present invention, the pseudo-incremental scenario selection module can solve the problem of limited data volume and difficulty in fully training the hybrid relationship mapping module.

[0080] In one embodiment of the present invention, the hybrid relational mapping module includes prototype self-projection (PSP), query set cross-projection (QCP) and optimization training.

[0081] Specifically, in one embodiment of the present invention, the hybrid relationship mapping module can use the feature pre-training module F to extract feature representations of the pseudo base class and the pseudo incremental class according to the pseudo incremental plot, and use the mean vector for the support set feature representation to calculate a prototype vector for each class as the initial weight of the classifier.

[0082]

[0083] Among them, c represents the category, pseudo base category and pseudo incremental category, a total of 2N categories, s j For Sb ∪S i A sample of For j The prototype representations and query set embeddings of the pseudo base class and pseudo incremental class are merged together to obtain the prototype representation set and query set embedding set of all categories, which are respectively represented by M c and X q express.

[0084] To model the complex interactions between prototypes and with query set embeddings, we develop PSP and QCP modules based on Transformer and c and X q The PSP and QCP modules are trained to guide the updating of prototype vectors and query set embedding representations to adapt to the global classification task.

[0085] In one embodiment of the present invention, the prototype self-mapping (PSP) adaptively adjusts all prototypes by establishing a global dependency relationship between the original prototype representation and the target prototype representation. The input of PSP is in the form of a triple (Query, Key, Value). Query, Key and Value share the same input source M c , PSP can be expressed as:

[0086] Query = M c W Q Key=M c W k Value = M c W V

[0087]

[0088] d is the dimension of Query, W Q / W K / W V are the learnable parameters of the three linear projection layers, projecting the original prototype representation into the shared metric space. The similarity scores between the prototype representations in the shared space are obtained through softmax normalization, i.e., the relationship matrix. We use the relationship matrix as a weight coefficient to aggregate the information from all prototype representations in Value and fuse it with the original prototype representation to obtain the updated prototype representation M c ′.

[0089] And, in one embodiment of the present invention, query set cross mapping (QCP). After the prototype is updated, by introducing the query set embedding representation X q To prototype M c′ is a cross-mapping process that establishes correlation between the query set embedding representation and each prototype, and adjusts the embedding representation of the query data so that the query set samples can better adapt to the target classification task. The formula is as follows:

[0090]

[0091] Based on this query set sample, the context information of the global task can be enriched.

[0092] Furthermore, in one embodiment of the present invention, the optimization training adopts a prototype-based objective loss function.

[0093] Specifically, in one embodiment of the present invention, all query sets are embedded using the cosine similarity function d(·,·) and all prototypes The semantic difference between them is analyzed and classified using the nearest congruent mean (NCM) classifier:

[0094]

[0095] Use queryset class labels y c , the cross entropy loss can be derived on the 2N categories of the current task as the target loss function for training:

[0096]

[0097] After learning, the parameters in the hybrid relational mapping module are frozen and deployed in the actual incremental session. The pseudo base class prototypes in the calculation process are replaced by the NCM classifier weights (the original prototypes of all previously seen categories), and the pseudo incremental class prototypes are replaced by the category prototypes in the actual incremental task. c represents the set of prototype vectors of all currently visible categories. After this round of tasks is completed, the NCM classifier weights are replaced with M c It is saved and used as the original prototype for the next incremental task.

[0098] Based on the above description, for ease of understanding, Figure 2 A schematic diagram of the framework of an image incremental classification system provided by the present invention, Figure 3 A structural diagram of an image incremental classification model provided by the present invention and Figure 4 This is a training flow chart of an image incremental classification system provided by the present invention.

[0099] Step S2: Obtain the image to be classified, and upload the image to be classified to the image incremental classification system for recognition.

[0100] Step S3: When the incremental image classification system fails to recognize, a small number of images of the category are obtained from the images to be classified as training samples, and the training samples are calculated through the feature pre-training module to obtain the target prototype, and the target prototype and / or the original prototype are adaptively adjusted through the hybrid relationship mapping module to update all prototypes in the incremental image classification system, so as to realize the classification and recognition of the images to be classified of unrecognized categories through the adjusted incremental image classification system.

[0101] Step S4: When the image incremental classification system successfully recognizes, classification and recognition are performed according to the image to be classified through the feature pre-training module, the hybrid relationship mapping module and the NCM classifier, and the classification result is output.

[0102] Based on the above description, for ease of understanding, Figure 5 This is a flow chart of a small sample image incremental classification method based on embedded enhancement and adaptation provided by the present invention.

[0103] In summary, the present invention proposes a small sample image incremental classification method based on embedding enhancement and adaptation, which makes the model adaptable to better handle new and old categories, so that the system can quickly adapt to the embedding expression of new categories from a small number of sample data, incrementally have the ability to recognize new samples, avoid complete retraining of the new model to reduce a large amount of computing resource overhead and promote the long-term operation of the system.

[0104] Figure 6 A schematic diagram of the structure of a small sample image incremental classification device based on embedding enhancement and adaptation provided by an embodiment of the present invention.

[0105] like Figure 6 As shown, the small sample image incremental classification device based on embedding enhancement and adaptation includes the following modules:

[0106] A first acquisition module is used to acquire an image incremental classification system, where the image incremental classification system is used to perform incremental classification tasks on the image to be classified;

[0107] The second acquisition module is used to acquire the image to be classified and upload the image to be classified to the image incremental classification system for recognition;

[0108] The first classification module is used to obtain a small number of images of the category to be classified from the images to be classified as training samples when the image incremental classification system fails to recognize, and calculate the target prototypes through the feature pre-training module using the training samples, and adaptively adjust the target prototype and / or the original prototype through the hybrid relationship mapping module to update all prototypes in the image incremental classification system, so as to realize the classification and recognition of the images to be classified of unrecognized categories through the adjusted image incremental classification system;

[0109] The second classification module is used to perform classification and recognition based on the image to be classified through the feature pre-training module, the hybrid relationship mapping module and the NCM classifier when the image incremental classification system successfully recognizes it, and output the classification result.

[0110] In one embodiment of the present disclosure, an image incremental classification system is obtained, including:

[0111] Select an embedding representation model based on the characteristics of the data set and task, and combine self-supervised learning and attention mechanism to form a feature pre-training module. Pre-train the model's feature pre-training module based on the image classification and annotation results.

[0112] Obtaining an output result of the feature pre-training module;

[0113] Based on the output results of the feature pre-training module, the training of the image incremental classification system is completed.

[0114] In summary, the present invention proposes a small sample image incremental classification device based on embedding enhancement and adaptation, so that the model has adaptability to better handle new and old categories, so that the system can quickly adapt to the embedding expression of new categories from a small number of sample data, and incrementally have the ability to recognize new samples. The model avoids complete retraining of the new model to reduce a large amount of computing resource overhead and promote the long-term operation of the system.

[0115] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the method for incremental classification of small sample images based on embedded enhancement and adaptation as described in the first aspect embodiment of the present application is implemented.

[0116] To achieve the above-mentioned objectives, the fourth aspect embodiment of the present application proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method of small sample image incremental classification based on embedded enhancement and adaptation described in the first aspect embodiment of the present application.

[0117] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0118] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0119] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0120] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0121] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0122] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0123] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0124] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A small sample image incremental classification method based on embedding enhancement and adaptation, characterized in that: The following steps are involved: Acquire an image incremental classification system, wherein the image incremental classification system is used to perform an incremental classification task on the image to be classified; Acquire an image to be classified, and upload the image to be classified to the image incremental classification system for identification; When the image incremental classification system fails to recognize, a small number of images of the category are obtained from the images to be classified as training samples, and the training samples are calculated by a feature pre-training module to obtain a target prototype, and the target prototype and / or the original prototype are adaptively adjusted by a hybrid relationship mapping module to update all prototypes in the image incremental classification system, so as to realize the classification and recognition of the images to be classified of unrecognized categories through the adjusted image incremental classification system; When the image incremental classification system successfully recognizes, the image to be classified is classified and recognized through the feature pre-training module, the hybrid relationship mapping module and the NCM classifier, and the classification result is output; The hybrid relationship mapping module comprises: According to the pseudo-incremental plot, use the feature pre-training module To extract the feature representation of the pseudo base class and pseudo incremental class, for the support set feature representation, we use the mean vector to calculate a prototype vector for each class as the initial weight of the classifier. , in, Represents a category, the pseudo base class and the pseudo increment class have a total of 2N categories, for A sample of for feature embedding; we merge the prototype representation and query set embedding representation of the pseudo base class and pseudo incremental class together to obtain the prototype representation set and query set embedding representation set of all categories, respectively. and express; Prototype self-mapping PSP, the PSP adaptively adjusts all prototypes by establishing a global dependency between the original prototype representation and the new prototype representation; wherein the input of PSP adopts ( , , ), , and Sharing the same input source , PSP can be expressed as: for The dimension of / / are the learnable parameters of the three linear projection layers, projecting the original prototypes into a shared metric space, via Normalization obtains the relationship matrix between each prototype representation in the shared space, and the relationship matrix is ​​used as a weight coefficient to aggregate The information represented by all prototypes in the image is combined with the original prototype to obtain the updated prototype. ; Query set cross-mapping QCP, which establishes correlation between the query set embedding representation and each prototype to adapt to the current classification task; the query set embedding representation needs to be classified by the distance between it and the prototype representation. To this end, we introduce the query set embedding representation To prototype The cross-mapping adjusts the query set embedding representation so that the query set samples can better adapt to the target classification task. The formula is as follows: ； Optimization training, wherein the optimization training calculates the semantic difference between the embedding representation of all query sets and the prototype through a cosine similarity function, classifies using a nearest mean (NCM) classifier, and optimizes model parameters using a cross entropy loss function.

2. The method according to claim 1, characterized in that The image acquisition incremental classification system comprises: Select an embedding representation model based on the characteristics of the data set and the task, and combine self-supervised learning and attention mechanism to form a feature pre-training module, and pre-train the feature pre-training module based on the image classification and annotation results; Obtaining an output result of the feature pre-training module; Based on the output results of the feature pre-training module, the training of the image incremental classification system is completed.

3. The method according to claim 2, characterized in that The output result of obtaining the feature pre-training module may include: The output result of the feature pre-training module does not reach the expected accuracy. The hyperparameters of the feature pre-training module are adjusted, and the feature pre-training module is further pre-trained. The output result of the feature pre-training module reaches the expected accuracy, the parameters of the feature pre-training module are frozen, and the pseudo-incremental episodes are selected by the pseudo-incremental episode selection module to train the hybrid relationship mapping module and the NCM classifier.

4. The method according to claim 3, characterized in that The pseudo-incremental plot selection module comprises: The pseudo base class includes: generating a support set in N-way K-shot form from the basic data set in each iteration , the corresponding query set It is formed by sampling from N categories, each category contains E query samples, where N and E are positive integers, E>N, and the query set is different from the samples in the support set. To express; Pseudo-incremental class, the pseudo-incremental class includes: rotating each sample of the pseudo-base class by 270 degrees, which can be used To express.

5. The method according to claim 3, characterized in that The training of the hybrid relationship mapping module and the NCM classifier includes: The training results of the hybrid relationship mapping module and the NCM classifier did not reach the expected accuracy, so the hyperparameters were adjusted, and the pseudo-incremental class was selected through the pseudo-incremental episode selection module to continue training the hybrid relationship mapping module and the NCM classifier; The training results of the hybrid relationship mapping module and the NCM classifier reach the expected accuracy, the parameters of the hybrid relationship mapping module are frozen, and the process ends.

6. A small sample image incremental classification device based on embedding enhancement and adaptation, characterized in that: Includes the following modules: A first acquisition module is used to acquire an image incremental classification system, where the image incremental classification system is used to perform an incremental classification task on the image to be classified; A second acquisition module is used to acquire the image to be classified and upload the image to be classified to the image incremental classification system for identification; A first classification module is used to obtain a small number of images of the category from the images to be classified as training samples when the image incremental classification system fails to recognize, and calculate the target prototypes through the feature pre-training module using the training samples, and adaptively adjust the target prototype and / or the original prototype through the hybrid relationship mapping module to update all prototypes in the image incremental classification system, so as to realize the classification and recognition of the images to be classified of unrecognized categories through the adjusted image incremental classification system; A second classification module is used to perform classification and recognition according to the image to be classified through a feature pre-training module, a hybrid relationship mapping module and an NCM classifier when the image incremental classification system successfully recognizes the image, and output a classification result; The hybrid relationship mapping module comprises: According to the pseudo-incremental plot, use the feature pre-training module To extract the feature representation of the pseudo base class and pseudo incremental class, for the support set feature representation, we use the mean vector to calculate a prototype vector for each class as the initial weight of the classifier. , in, Represents a category, the pseudo base class and the pseudo increment class have a total of 2N categories, for A sample of for feature embedding; we merge the prototype representation and query set embedding representation of the pseudo base class and pseudo incremental class together to obtain the prototype representation set and query set embedding representation set of all categories, respectively. and express; Prototype self-mapping PSP, the PSP adaptively adjusts all prototypes by establishing a global dependency between the original prototype representation and the new prototype representation; wherein the input of PSP adopts ( , , ), , and Sharing the same input source , PSP can be expressed as: for The dimension of / / are the learnable parameters of the three linear projection layers, projecting the original prototypes into a shared metric space, via Normalization obtains the relationship matrix between each prototype representation in the shared space, and the relationship matrix is ​​used as a weight coefficient to aggregate The information represented by all prototypes in the image is combined with the original prototype to obtain the updated prototype. ; Query set cross-mapping QCP, which establishes correlation between the query set embedding representation and each prototype to adapt to the current classification task; the query set embedding representation needs to be classified by the distance between it and the prototype representation. To this end, we introduce the query set embedding representation To prototype The cross-mapping adjusts the query set embedding representation so that the query set samples can better adapt to the target classification task. The formula is as follows: ； Optimization training, wherein the optimization training calculates the semantic difference between the embedding representation of all query sets and the prototype through a cosine similarity function, classifies using a nearest mean (NCM) classifier, and optimizes model parameters using a cross entropy loss function.

7. The device according to claim 6, characterized in that The image acquisition incremental classification system comprises: Select an embedding representation model based on the characteristics of the data set and the task, and combine self-supervised learning and attention mechanism to form a feature pre-training module, and pre-train the feature pre-training module of the model based on the image classification and annotation results; Obtaining an output result of the feature pre-training module; Based on the output results of the feature pre-training module, the training of the image incremental classification system is completed.

8. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images