Construction method of increased-class sample classification model and related equipment thereof

Abstract

The invention relates to a construction method of an increased-class sample classification model and related equipment thereof. The method comprises the following steps: acquiring a trained basic sample classification model; acquiring an increased-class sample comprising increased-class sample feature data and an increased-class category label, and processing the increased-class sample feature data by adopting the basic feature representation model to obtain feature representation of an increased-class category prototype corresponding to the increased-class category label; on the basis of the similarity distance between the feature representations of the added category prototype and the plurality of basic category prototypes, calibrating the feature representation of the added category prototype to obtain the calibrated feature representation of the added category prototype; and adding the calibration feature representation of the increased class prototype as the feature representation of the increased class prototype of the increased class label into a basic classifier model of the basic sample classification model to obtain an increased class sample classification model. Through the method and the device, the problems of disastrous forgetting and over-fitting of an increment small sample classification system in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for constructing an augmented sample classification model and related equipment. Background Art

[0002] In real-world scenarios, data generally arrives in the form of data streams over time, so this requires machine learning models to be able to continuously acquire the ability to incrementally learn new tasks based on new data and not forget old tasks. Such tasks that require incremental learning of models are generally referred to as incremental learning tasks. Incremental learning tasks generally require that the data of old tasks cannot be saved or only a very small amount of old task data can be saved. General neural network models will have catastrophic forgetting problems when incrementally learning new tasks. In addition, due to restrictions such as privacy issues, only a limited amount of data can be collected for a certain category in a new task, and this limited data generally cannot guarantee the normal training of the model (generally no more than 10 samples per category), because if general model training is performed at this time, serious overfitting problems will occur. Therefore, the incremental small sample classification system needs to overcome the catastrophic forgetting problem and the overfitting problem caused by small sample data.

[0003] There is no effective solution to the problems of catastrophic forgetting and overfitting in incremental small sample classification systems in related technologies. Summary of the invention

[0004] The method for constructing an augmented sample classification model and related devices provided by the embodiments of the present invention at least solve the problems of catastrophic forgetting and overfitting in the incremental small sample classification system in the related art.

[0005] A method for constructing a classification model for increased class samples, comprising:

[0006] Acquire a trained basic sample classification model, wherein the basic sample classification model includes a basic feature representation model and a basic classifier model, and the basic classifier model includes feature representations of a plurality of basic category prototypes corresponding to a plurality of basic category labels respectively;

[0007] Acquire an augmented class sample including augmented class sample feature data and an augmented class category label, process the augmented class sample feature data using the basic feature representation model, and obtain a feature representation of an augmented class category prototype corresponding to the augmented class category label;

[0008] Based on the similarity distance between the feature representations of the augmented category prototype and the plurality of basic category prototypes, calibrating the feature representation of the augmented category prototype to obtain a calibrated feature representation of the augmented category prototype;

[0009] The calibration feature representation of the augmented class prototype is used as the feature representation of the augmented class prototype of the augmented class label, and is added to the basic classifier model of the basic sample classification model to obtain the augmented class sample classification model.

[0010] In some of the embodiments, the types of samples used to train the basic sample classification model and the augmented class samples are one of the following: text, image, or video.

[0011] In some of the embodiments, the basic classification model uses a nearest neighbor classifier based on prototype feature representation and cosine similarity.

[0012] In some embodiments, the trained basic sample classification model is trained by the following steps:

[0013] From the basic category dataset Randomly sample batch data with sample size B;

[0014] Input batch data into the basic feature representation model f to obtain the feature representation φ of the initial basic category prototype of basic category i in the training sample i ;

[0015] Calculate the samples of base category i in the batch data (x i ,y i ) Data x i The feature representation of the initial basic category prototype φ i , to the true label y i The corresponding feature representation of the true base class prototype is φ yi The cosine similarity distance of , and the cross entropy loss value is calculated based on the cosine similarity distance;

[0016] Based on the cross entropy loss value of all samples in the batch data, the gradient update and back propagation of the basic sample classification model are performed.

[0017] In some of the embodiments, the cross entropy loss value is expressed as: Where ||·|| is the L2 norm of the vector and τ is a hyperparameter that adjusts the sharpness of the distribution.

[0018] In some embodiments, obtaining an augmented class sample including augmented class sample feature data and an augmented class category label, processing the augmented class sample feature data using the basic feature representation model, and obtaining a feature representation of an augmented class category prototype corresponding to the augmented class category label includes:

[0019] Get the number of samples included as K j The additional class samples of the additional class j;

[0020] Input the augmented class samples into the basic feature representation model to obtain the feature representation corresponding to the augmented class samples;

[0021] Calculate K for the augmented category j j The average of the feature representations of samples is taken as the feature representation φ of the prototype of augmented category j j .

[0022] In some embodiments, based on the similarity distance between the feature representations of the augmented category prototype and the plurality of basic category prototypes, calibrating the feature representation of the augmented category prototype to obtain the calibrated feature representation of the augmented category prototype includes:

[0023] Calculate the average feature representation of all samples belonging to base category i as the feature representation φ of the prototype of base category i i ;

[0024] Normalize the feature representation of each basic category prototype and the feature representation of each enhanced category prototype respectively to obtain the normalized feature representation of the basic category i prototype and the normalized feature representation of the prototype of augmented category j

[0025] Calculate the normalized feature representation of the prototype of augmented category j With each normalized feature representation of the base class i prototype The cosine similarity vector α j ;

[0026] The cosine similarity vector α j Normalize it to get the similarity weight w of the feature representation of the prototype of the augmented category j relative to the feature representation of each basic category prototype j ;

[0027] Use similarity weight w j The feature representation of the normalized basic category prototype is weighted to obtain the calibration term Δφ of the feature representation of the prototype of the augmented category j j ;

[0028] The calibration item of the feature representation of the prototype of the augmented category j is normalized to obtain the normalized calibration item of the feature representation of the prototype of the augmented category j

[0029] Calibration term using the normalized feature representation of the augmented class j prototype The feature representation φ of the prototype of augmented category j j Perform calibration to obtain the calibration feature representation of the prototype of the augmented category

[0030] In some embodiments, the normalization of the feature representation and the calibration item of the feature representation is performed by L2 norm normalization; the cosine similarity vector α j Use softmax function for normalization.

[0031] An image classification system, characterized in that it includes: a camera and an image classification device, wherein the image classification device includes: a processor and a memory, the memory is used to store augmented samples including augmented sample feature data and augmented category labels, images or videos captured by the camera, and a program; the program includes instructions, and when the instructions are executed by the processor, the processor executes the above-mentioned method.

[0032] A non-transitory machine-readable medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above method.

[0033] The method for constructing an augmented class sample classification model and the related device provided by the embodiment of the present invention are as follows: obtaining a trained basic sample classification model, wherein the basic sample classification model includes a basic feature representation model and a basic classifier model, and the basic classifier model includes feature representations of multiple basic class prototypes corresponding to multiple basic class labels respectively; obtaining augmented class samples including augmented class sample feature data and augmented class category labels, using the basic feature representation model to process the augmented class sample feature data, and obtaining feature representations of augmented class category prototypes corresponding to augmented class category labels; calibrating the feature representation of the augmented class category prototype based on the similarity distance between the feature representations of the augmented class category prototype and multiple basic class prototypes, and obtaining a calibrated feature representation of the augmented class category prototype; adding the calibrated feature representation of the augmented class category prototype as the feature representation of the augmented class category prototype of the augmented class category label to the basic classifier model of the basic sample classification model, and obtaining the augmented class sample classification model. The embodiment of the present invention at least solves the problems of catastrophic forgetting and overfitting in the incremental small sample classification system in the related art, and improves the classification effect of the incremental small sample classification system.

[0034] The details of one or more embodiments of the invention are set forth in the following drawings and description so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0036] Figure 1It is a flowchart of a method for constructing an augmented sample classification model according to an embodiment of the present invention.

[0037] Figure 2 It is a structural diagram of a system for constructing an augmented sample classification model according to an embodiment of the present invention.

[0038] Figure 3 Schematic diagram of the structure of the image classification system according to the embodiment of the present invention.

[0039] Figure 4 Schematic diagram of the structure of the electronic device of this embodiment. DETAILED DESCRIPTION

[0040] Embodiments of the present embodiment will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present embodiment are shown in the accompanying drawings, it should be understood that the present embodiment can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, which are instead provided for a more thorough and complete understanding of the present embodiment. It should be understood that the drawings and embodiments of the present embodiment are only for exemplary purposes and are not intended to limit the scope of protection of the present embodiment.

[0041] The incremental small sample classification method of this embodiment is based on the classification representation of prototypes, that is, the feature representation of a class is obtained by averaging all the features of the samples of that class. The prototype-based classification representation can be a basic feature space representation designed for incremental learning, or continuously optimize new category prototypes in new tasks. In the training of the prototype-based classification representation model, if the feature representation model is frozen after the model pre-training is completed, and then only the classifier is updated on the new task, although the trained model can achieve good performance in terms of the average accuracy of all categories, the accuracy of the model on the incremental category is often low. In addition, the categories in the new task are generally as important as or even more important than the categories in the old task. This is because the newer tasks in time are more likely to be the tasks that you want to pay more attention to in practical applications.

[0042] In view of the shortcoming that the incremental small sample classification method cannot pay attention to the performance of new tasks, this embodiment provides an incremental small sample classification method based on lossless calibration, which performs pre-training on a large number of pre-collected basic categories to obtain a feature representation model with good feature representation capabilities. When a new task with only a small number of samples arrives, the model construction method provided in this embodiment can calibrate the biased feature prototype of the augmented category through the feature prototype of the basic category without training. The feature prototype of the augmented category after lossless calibration can not only improve the average accuracy of all categories, but also improve the accuracy of the model on the augmented category.

[0043] This embodiment provides a method for constructing a classification model for augmented samples. Figure 1is a flow chart of the method for constructing the augmented sample classification model of this embodiment. Figure 1 As shown, the process includes the following steps:

[0044] Step S101, obtaining a trained basic sample classification model, wherein the basic sample classification model includes a basic feature representation model and a basic classifier model, and the basic classifier model includes feature representations of multiple basic category prototypes corresponding to multiple basic category labels respectively.

[0045] Step S102, obtaining an augmented class sample including augmented class sample feature data and an augmented class category label, and processing the augmented class sample feature data using a basic feature representation model to obtain a feature representation of an augmented class category prototype corresponding to the augmented class category label.

[0046] Step S103, based on the similarity distance between the feature representations of the augmented category prototype and the plurality of basic category prototypes, calibrate the feature representation of the augmented category prototype to obtain a calibrated feature representation of the augmented category prototype.

[0047] Step S104, adding the calibration feature representation of the augmented class prototype as the feature representation of the augmented class prototype of the augmented class label to the basic classifier model of the basic sample classification model to obtain the augmented class sample classification model.

[0048] Through the above steps, after obtaining a basic sample classification model such as one obtained by normal training using a large amount of sample data, the parameters of the basic feature representation model are frozen, and the feature representation of the augmented category prototype is obtained using the basic feature representation model. Then, the feature representation of the augmented category prototype is calibrated using the similarity distance between the feature representations of the augmented category prototype and multiple basic category prototypes, so that the augmented category prototype obtains a calibrated feature representation that can represent the augmented category after calibration, and then the calibrated feature representation of the augmented category is added to the basic classifier model as a newly added category, so that the basic sample classification model has the ability to classify the augmented category. The above construction method does not affect the original basic sample classification ability, that is, it avoids the catastrophic forgetting problem. At the same time, the process of adding the classification ability of the augmented category does not require incremental training of any parameters of the model, but obtains the calibrated feature representation of the augmented category by considering the similarity distance between the feature representations, thereby avoiding the overfitting problem caused by small sample training on the basis of achieving accurate classification of the augmented category.

[0049] In this embodiment, the types of samples and augmented class samples used to train the basic sample classification model are consistent. These samples can be of text, image or video type. Text type samples include text feature data and the label of the text; image or video type samples include image or video feature data, as well as the annotation box corresponding to the image or video and its label. Taking the augmented class as an example, for each augmented class label, there can be one or more augmented class samples, and these one or more augmented class samples can obtain the feature representation of one or more augmented class samples corresponding to each other through the basic feature representation model. For the case where an augmented class label only includes one augmented class sample, the feature representation of the augmented class sample can be used as the augmented class prototype corresponding to the augmented class label. For the case where an augmented class label includes multiple augmented class samples, the average of the feature representation of each augmented class sample can be used as the augmented class prototype corresponding to the augmented class label. Among them, the augmented class prototype essentially refers to the feature representation used to represent a certain label category, which is referred to as "the feature representation of the augmented class prototype corresponding to the augmented class label" in this embodiment, and is referred to as "the augmented class prototype" for short.

[0050] The similarity distance in the above step S103 includes but is not limited to any one of the distances such as Euclidean distance, cosine similarity distance, Hamming distance, and Pearson correlation coefficient. In some embodiments, the basic classification model uses a nearest neighbor classifier based on prototype feature representation and cosine similarity, that is, the similarity distance in the above step S103 is a cosine similarity distance, so that the similarity distance values ​​are all limited to the same threshold range [0,1], simplifying the processing of the distance.

[0051] The above-mentioned method for constructing the classification model of augmented samples can be divided into three main parts:

[0052] Part 1: Collect a large amount of basic category data and annotate it, and obtain a feature representation model with good feature extraction capabilities by performing supervised pre-training on the basic category data.

[0053] In the first part, a large amount of data for basic pre-training is collected, wherein the number of categories of the basic data is sufficient (generally more than 50 categories), and the amount of samples collected for each category is sufficient (generally there are hundreds to thousands of samples for each class); the sample classification model is decoupled into two parts: a feature representation model and a classifier model. For the pre-training process of the feature representation model, this embodiment adopts a supervised classification task based on a cross entropy loss function as a training target as a pre-training target, and uses the cosine similarity distance between the feature of each sample and the feature prototype of its corresponding category as a measure of the similarity of the sample to a certain class. The pre-training of the feature representation model adopts a training method of forward inputting samples and back-propagating gradients to update network parameters. After the pre-training is completed, all classification centers of the basic categories are replaced with the feature prototypes of the corresponding categories, that is, the average of all sample features under a certain category is used as the category feature representation of the category.

[0054] In the second part, a biased augmented class prototype is constructed using small sample data.

[0055] In the second part, new task data arrives in the form of small samples, that is, each augmented category has only a few samples. Since the feature representation model obtained by pre-training in the first part (called the basic feature representation model) is trained to obtain the feature representation of the basic samples, rather than the feature representation of the augmented samples, the feature representation of the augmented samples obtained by the basic feature representation model, and the augmented category prototypes further obtained based on the feature representation of these augmented samples, have a certain deviation from the ideal prototype of the augmented samples, which is called a biased prototype. For the acquisition of biased prototypes, in this part, the feature representation model obtained by the basic pre-training model is first fixed so that it is not updated when the augmented category data arrives, so as to overcome the "catastrophic forgetting" problem that the model will suffer when processing streaming data. For each augmented category in a new task, the average of the feature representations of all samples under the augmented category is first used as the biased prototype of the augmented category.

[0056] In the third part, the biased augmented category prototype is losslessly calibrated by using the approximately unbiased basic category feature representation, that is, the calibrated feature representation of the augmented category prototype in the above step S103 is obtained.

[0057] In the third part: Due to the small number of samples of the augmented class, the biased prototype constructed for the augmented class cannot well characterize the feature representation of the augmented class, and thus cannot obtain good augmented class performance. This part uses the large number of unbiased basic class prototypes with good representation obtained in the first part to calibrate the biased prototype of the augmented class. Specifically, for each biased prototype of the augmented class, the cosine similarity between the biased prototype and all the basic class prototype vectors is first calculated (for example, the two prototypes are first normalized and then the dot product operation is performed), and the distance from the biased prototype to all the basic class prototype vectors is obtained. This distance can be approximately regarded as the similarity relationship between the two categories. Then, the similarity vector from the augmented class prototype to all the basic class prototypes is passed through the softmax function to obtain the normalized weight. The weight is used to weight all the basic class prototypes to obtain the calibration item of the feature representation of the augmented class prototype. Finally, the calibration item of the feature representation of the augmented class prototype is added to the feature representation of the augmented class prototype to obtain the calibrated feature representation of the augmented class prototype after calibration. All augmented categories can be subjected to the same operation to obtain losslessly calibrated augmented category prototypes. The calibrated augmented category prototypes and the basic category prototypes obtained in the basic pre-training module are simultaneously used as the classification centers of all categories in the classifier to obtain a sample classification model that can classify the basic category and augmented category.

[0058] The above three parts are described in detail below.

[0059] The first part, collecting a large amount of basic category data and pre-training, includes the following steps:

[0060] Step 100: collect a large amount of basic category data as training data for training the basic feature representation model and obtaining the basic category prototype, annotate the data, and obtain the annotated basic category data. The category set of the basic category dataset is In this embodiment, the number of categories is more than 50, and the number of samples in each category is sufficient (hundreds to thousands of samples) to ensure normal training.

[0061] Step 101, for the model used for the image classification task, decouple it into two parts: a basic feature representation model f and a basic category classification model φ, that is, the model M = {f, φ}, the feature representation model f contains parameters θ f .

[0062] Step 102, randomly initialize the model M, wherein each row vector of the basic category classification model φ is regarded as a feature representation of a category.

[0063] Step 103, from the basic category dataset Randomly sample batches of data with a sample size of B for training.

[0064] Step 104: input the sampled batch data into the feature representation model f to obtain the feature representation R of the training sample. The feature representation R in this embodiment is represented as In matrix form, R is represented as a feature matrix of shape B×D, where For sample x i The corresponding D-dimensional feature representation.

[0065] Step 105: represent the basic category i as the feature representation of the initial basic category prototype That is, a row vector in the basic category classification model φ.

[0066] Step 106, calculate the cross entropy loss and back propagate. For the classification model, a nearest neighbor classifier based on cosine similarity is used. Sample (x i ,y i ) to the initial category representation φ of category j j The distance can be written as: Where ||·|| is the L2 norm of the vector, and τ is a hyperparameter that adjusts the sharpness of the distribution. Based on this, based on the cosine similarity metric, the sample (x i ,y i ) can be written as: in For the sample (x i ,y i )’s true label y i The feature representation of the corresponding initial base category prototype.

[0067] Step 107, calculate the loss values ​​of all samples in the same batch of data and complete gradient update and back propagation; repeat steps 103 to 106, that is, perform multiple iterations of optimization to obtain a better pre-trained model.

[0068] The second part, using small sample data to construct biased augmented class prototypes, includes the following steps:

[0069] Step 200: Change the initial category representation of the basic category to a unified feature representation of the basic category prototype, that is, for each basic category i, the average of the sample features of all categories i is counted as the feature representation of the basic category prototype Where N i is the number of samples of basic category i.

[0070] Step 201: For the data of the new task, it can be represented as N augmented categories, and the number of samples of augmented category j is K. j , (generally K j ≤10), that is

[0071] Step 202, fix the parameters θ of the basic feature representation model f , all new task data Input the basic feature representation model to obtain the feature representation R of the corresponding augmented category.

[0072] Step 203: for each category j in the current category task, calculate the K j The average of the feature representations of the samples of the augmented category j is taken as the biased prototype representation of its category, that is:

[0073] The third part, lossless calibration of biased augmented class prototypes, includes the following steps:

[0074] Step 300, obtain the basic category prototype in the model It is used as the basic calibration basis for the biased prototype of the class.

[0075] Step 301, obtain a biased prototype set of N augmented categories of the current task

[0076] Step 302, respectively normalize the basic category prototype and the increased category biased prototype, that is: for any Calculate separately: Where ||·|| is the L2 norm of the vector.

[0077] Step 303: for each additional class j, the standard biased prototype (The standard biased prototype is the normalized biased prototype), and its difference from the standard basic category i prototype is calculated respectively. Similarity In essence, it still represents the cosine similarity distance between the original two unnormalized original prototypes, where τ0 is a hyperparameter for adjusting the distribution.

[0078] Step 304: for each additional class j, the standard biased prototype After step 304, the similarity between the prototype and all basic category prototypes can be obtained, which can be recorded as a similarity vector

[0079] Step 305: For the standard biased prototype of the added category j The similarity vector α with the standard prototype of base category i j , use the softmax function for length The similarity vector α j Do normalization to further obtain the similarity weight w j .

[0080] Step 306, use w j The normalized basic category prototype is weighted to obtain the biased increase category prototype φ j Prototype characteristic calibration items

[0081] Step 307, normalize the prototype calibration item: Where ||·|| is the L2 norm of the vector.

[0082] Step 308, for each biased label of the augmented class, the prototype feature calibration item in step 306 is calculated, and the original biased class prototype of each calibration item is used for calibration to obtain the final calibrated augmented class prototype.

[0083] Step 309: For each augmented category of the new task, execute steps 301 to 308 in sequence to obtain the corresponding calibrated prototype, use the calibrated prototype as the feature representation of the augmented category and add it to the classifier of the model to obtain the final classification model for the incremental small sample image classification task.

[0084] In the above embodiment, the samples may be preprocessed, including normalization, rotation and augmentation of the image data, so as to increase the diversity of the sample space and improve the generalization ability of the model.

[0085] The gradient update in the above embodiment includes: calculating the output by inputting batch data into the model network, using the loss function to calculate the loss between the output and the true label, obtaining the gradient of the network parameters by differentiating the network parameters by the loss value, and finally back-propagating the gradient to update the network parameters, and finally the model iteration converges to the local optimal solution.

[0086] This embodiment also provides a system for constructing an augmented sample classification model. Figure 2 is a schematic diagram of the structure of the system for constructing the augmented sample classification model of this embodiment. Figure 2 As shown, the system includes: a training module 21 for collecting a large amount of basic category data and performing pre-training, a construction module 22 for constructing biased prototype features of the augmented category using small sample data of the augmented category, and a calibration module 23 for achieving lossless calibration of the biased prototype of the augmented category.

[0087] Training module 21: Collect a large amount of data for basic pre-training, where the number of categories of basic data is large enough (generally more than 50 categories), and the number of samples collected for each category is large enough (generally there are hundreds to thousands of samples for each class); the sample classification model is decoupled into two parts: the feature representation model and the classifier model. For the pre-training process of the feature representation model, this embodiment adopts a supervised classification task based on the cross entropy loss function as the training target as the pre-training target, and takes the cosine similarity distance between the feature of each sample and the feature prototype of its corresponding category as a measure of the similarity between the sample and a certain class. The pre-training of the feature representation model adopts a training method of forward inputting samples and back-propagating gradients to update network parameters. After the pre-training is completed, all classification centers of the basic category are replaced with the feature prototypes of the corresponding category, that is, the average of all sample features under a certain category is used as the category feature representation of the category.

[0088] Construction module 22: For the acquisition of biased prototypes, in this module, the feature representation model obtained from the basic pre-training model is first fixed so that it is not updated when the augmented category data arrives, so as to overcome the "catastrophic forgetting" problem that the model will suffer when processing streaming data. For each augmented category in a new task, the average feature representation of all samples under the augmented category is first used as the biased prototype of the augmented category.

[0089] Calibration module 23: This module uses a large number of well-represented unbiased basic category prototypes obtained in the first part to calibrate the biased prototypes of the augmented category. Specifically, for each biased prototype of the augmented category, the cosine similarity between the biased prototype and all basic category prototype vectors is first calculated (for example, the two prototypes are first normalized and then a dot product operation is performed), and the distance from the biased prototype to all basic category prototype vectors is obtained, which can be approximately regarded as the similarity relationship between the two categories. Then, the similarity vector from the augmented category prototype to all basic category prototypes is subjected to a softmax function to obtain a normalized weight, and the weight is used to weight all basic category prototypes to obtain the calibration item of the feature representation of the augmented category prototype. Finally, the calibration item of the feature representation of the augmented category prototype is added to the feature representation of the augmented category prototype to obtain the calibration feature representation of the augmented category prototype after calibration. All augmented categories can obtain losslessly calibrated augmented category prototypes through the same operation. The calibrated augmented category prototype and the basic category prototype obtained in the basic pre-training module are simultaneously used as the classification centers of all categories in the classifier to obtain a sample classification model that can classify the basic category and the augmented category.

[0090] The embodiment of the present invention also provides an image classification system. Figure 3 : is a schematic diagram of the structure of the image classification system of this embodiment. Figure 3As shown, the image classification system includes a camera 31 and an image classification device 32. The image classification device 32 includes: at least one processor; and a memory connected to the at least one processor. The memory stores augmented samples including augmented sample feature data and augmented category labels, images or videos collected by the camera, and a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, it is used to enable the electronic device to perform the method of the embodiment of the present invention.

[0091] After the image classification device constructs an augmented class sample classification model through the above-mentioned augmented class sample classification model construction method, it can classify the image or video captured by the camera into basic categories and augmented categories based on the augmented class sample classification model, and output the classification results.

[0092] An embodiment of the present invention further provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present invention.

[0093] refer to Figure 4 , a block diagram of an electronic device that can be used as an image classification device of an embodiment of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0094] like Figure 4 As shown, the electronic device includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In RAM 403, various programs and data required for the operation of the electronic device can also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0095] Multiple components in the electronic device are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 can be any type of device that can input information to the electronic device, and the input unit 406 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 407 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 can include but is not limited to a disk, an optical disk. The communication unit 409 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0096] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) processors, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 402 and / or a communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).

[0097] The computer programs for implementing the methods of the embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0098] In the context of an embodiment of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, a device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0099] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0100] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0101] The various steps described in the method implementation methods provided in the embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0102] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0103] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A method for constructing a classification model for augmented samples, characterized in that include: Acquire a trained basic sample classification model, wherein the basic sample classification model includes a basic feature representation model and a basic classifier model, and the basic classifier model includes feature representations of a plurality of basic category prototypes corresponding to a plurality of basic category labels respectively; Acquire an augmented class sample including augmented class sample feature data and an augmented class category label, process the augmented class sample feature data using the basic feature representation model, and obtain a feature representation of an augmented class category prototype corresponding to the augmented class category label; Based on the similarity distance between the feature representations of the augmented category prototype and the plurality of basic category prototypes, calibrating the feature representation of the augmented category prototype to obtain a calibrated feature representation of the augmented category prototype; The calibration feature representation of the augmented class prototype is used as the feature representation of the augmented class prototype of the augmented class label, and is added to the basic classifier model of the basic sample classification model to obtain the augmented class sample classification model.

2. The method according to claim 1, characterized in that The types of samples used to train the basic sample classification model and the augmented class samples are one of the following: text, image, or video.

3. The method according to claim 1, characterized in that The basic classification model adopts a prototype-based feature representation and a cosine similarity nearest neighbor classifier.

4. The method according to claim 1, characterized in that: The trained basic sample classification model is trained through the following steps: From the basic category dataset Randomly sample batch data with sample size B; Input batch data into the basic feature representation model f to obtain the feature representation R of the initial basic category prototype of basic category i in the training sample i ; Calculate the samples of base category i in the batch data (x i ,y i ) Data x i The feature representation R of the initial basic category prototype i , to the true label y i The corresponding feature representation of the true base class prototype is φ yi The cosine similarity distance of , and the cross entropy loss value is calculated based on the cosine similarity distance; Based on the cross entropy loss value of all samples in the batch data, the gradient update and back propagation of the basic sample classification model are performed.

5. The method according to claim 4, characterized in that The cross entropy loss value is expressed as: Where ||·|| is the L2 norm of the vector and τ is a hyperparameter that adjusts the sharpness of the distribution.

6. The method according to claim 1, characterized in that Acquiring an augmented class sample including augmented class sample feature data and an augmented class category label, processing the augmented class sample feature data using the basic feature representation model, and obtaining a feature representation of an augmented class category prototype corresponding to the augmented class category label comprises: Get the number of samples included as K j The additional class samples of the additional class j; Input the augmented class samples into the basic feature representation model to obtain the feature representation corresponding to the augmented class samples; Calculate K for the augmented category j j The average of the feature representations of samples is taken as the feature representation φ of the prototype of augmented category j j .

7. The method according to claim 6, characterized in that Based on the similarity distance between the feature representations of the augmented category prototype and the plurality of basic category prototypes, calibrating the feature representation of the augmented category prototype to obtain the calibrated feature representation of the augmented category prototype includes: Calculate the average feature representation of all samples belonging to base category i as the feature representation φ of the prototype of base category i i ; Normalize the feature representation of each basic category prototype and the feature representation of each enhanced category prototype respectively to obtain the normalized feature representation of the basic category i prototype and the normalized feature representation of the prototype of augmented category j Calculate the normalized feature representation of the prototype of augmented category j With each normalized feature representation of the base class i prototype The cosine similarity vector α j ; The cosine similarity vector α j Normalize it to get the similarity weight w of the feature representation of the prototype of the augmented category j relative to the feature representation of each basic category prototype j ; Use similarity weight w j The feature representation of the normalized basic category prototype is weighted to obtain the calibration term Δφ of the feature representation of the prototype of the augmented category j j ; The calibration item of the feature representation of the prototype of the augmented category j is normalized to obtain the normalized calibration item of the feature representation of the prototype of the augmented category j Calibration term using the normalized feature representation of the augmented class j prototype The feature representation φ of the prototype of augmented category j j Perform calibration to obtain the calibration feature representation of the prototype of the augmented category 8. The method according to claim 7, characterized in that The normalization of feature representation and calibration items of feature representation adopts L2 norm normalization, and the cosine similarity vector α j Use softmax function for normalization.

9. An image classification system, characterized in that include: A camera and an image classification device, wherein the image classification device comprises: a processor and a memory, the memory being used to store augmented samples comprising augmented sample feature data and augmented category labels, images or videos captured by the camera, and a program; the program comprising instructions, which, when executed by the processor, cause the processor to execute a method according to any one of claims 1 to 8.

10. A non-transitory machine-readable medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

Images