Double-teacher brain-like distillation continual learning image classification method, device and storage medium

Abstract

The application discloses a double-teacher brain-like distillation continuous learning image classification method, defines a visual language pre-training model as E0, and constructs an initial visual prototype according to the E0 on image data; the E0 is used as a first teacher model; a student model is constructed from the E0 and pre-training model weight parameters are loaded; the student model is trained for the mth task, and the student model after training is recorded as E m ; when the student model is trained for the m+1th task, the E m is used as a second teacher model, and the student model is trained from the E m and loads the weight; the visual prototype is updated in a sliding mode with batch data in the training process; a classification loss L m and an instance prototype alignment loss L ce are calculated according to the E con ; an instance prototype similarity distillation loss is calculated according to the E m‑1 and the E0, the average value is used as a final distillation loss L dis ; the L ce , the L con and the L dis are combined to obtain an optimization target; the double-teacher distillation is used to simultaneously relieve the forgetting of catastrophic forgetting and the forgetting of zero sample migration ability.

Description

TECHNICAL FIELD

[0001] The application relates to an image classification method, in particular to a double-teacher brain-like distillation continual learning image classification method, equipment and a storage medium. BACKGROUND

[0002] At present, continual learning refers to a kind of machine learning task in which a model needs to continuously learn multiple tasks, and the model needs to maintain old task information and continuously learn new tasks without the participation of old task data (or strictly limit the number) in joint training. Knowledge distillation refers to a kind of machine learning method, which includes a student model and a teacher model. The distillation method compares the responses of the models to the same input, so that the response of the student model approaches the teacher model to learn its knowledge. According to the number of teacher models, it is divided into single-teacher distillation and multi-teacher distillation.

[0003] In recent years, with the proposal of the Transformer architecture and the large-scale use of pre-training models in the visual field, the image classification technology of continual learning has made great progress. Further, people have proposed visual language pre-training models based on visual pre-training models. However, the continual learning of pre-training visual language models not only faces catastrophic forgetting in the training process, but also faces the forgetting of zero-shot transfer ability. Ordinary continual learning methods are difficult to prevent the forgetting of zero-shot transfer ability, and the combination of distillation methods and continual learning based on pre-training visual language models is a solution.

[0004] The existing method mainly realizes the distillation between the old model and the current model by using auxiliary data sets. This method not only needs additional data set storage space, which is limited in some scenarios. In addition, the existing method adopts a single-teacher distillation paradigm, which only uses the initial model or the model of the previous task for distillation, so there are limitations in alleviating catastrophic forgetting and zero-shot transfer ability forgetting. SUMMARY

[0005] The application provides a double-teacher brain-like distillation continual learning image classification method, equipment and a storage medium to solve the technical problems in the prior art.

[0006] The technical solution adopted by the application to solve the technical problems in the prior art is:

[0007] A double-teacher brain-like distillation continual learning image classification method, the method comprising the following steps:

[0008] Step 1: Compile the training image sample set; divide the image sample set into multiple task image sample sets according to the task category, and each task image sample set is divided into a training dataset and a test dataset; the training dataset of each task includes multiple category image sample sets; each category image sample set includes multiple labeled image samples and corresponding label and category semantic information;

[0009] Step 2: Construct a visual language pre-trained model to obtain initial visual prototypes from image data. The visual language pre-trained model includes a visual encoder and a text encoder. The visual encoder processes image data, encoding images into vector data. The text encoder processes text data, encoding text into vector data. Features are extracted from all image samples in the training dataset for each task using the visual language pre-trained model. The initial visual prototype for each category is obtained by calculating the average feature value of image samples of the same category in the same task.

[0010] Step 3: Define the visual language pre-trained model as E0; use E0 as the first teacher model; construct the student model, making its structure the same as the visual language pre-trained model, and load the weight parameters of the visual language pre-trained model.

[0011] Step 4: Train the student model for the m-th task, where m = 1, 2, ..., T, and T is the number of tasks. The trained student model is denoted as E. m When training the student model for the (m+1)th task, E m As a second teacher model, the student model originates from E m Load weights for training; enable the visual prototype to slide and update as batch data is processed during training;

[0012] Step 5: Calculate the classification loss L based on Em. ce Alignment loss L with instance prototype con According to E m-1 The similarity distillation loss between the instance prototype and E0 is calculated, and their average value is used as the final distillation loss L. dis Classification loss L ce Instance prototype alignment loss L con Final distillation loss L dis The optimization objective is obtained by weighted combination;

[0013] Step 6: Repeat steps 4 and 5 until the optimization objectives of all training tasks have reached the set values, and the training ends.

[0014] Step 7: Use the trained model to classify the image to be classified.

[0015] Furthermore, in step 5, the calculation formula for the optimization objective is as follows:

[0016] L = L ce + 0.2L con + 0.2L dis ;

[0017] In the formula:

[0018] L is an optimization target;

[0019] L ce is a classification loss;

[0020] L con is an instance prototype alignment loss;

[0021] L dis is a final distillation loss.

[0022] Further, in step 2, the calculation formula of the visual prototype of each category is as follows:

[0023]

[0024] In the formula:

[0025] c is the sample category serial number;

[0026] is the feature extracted by the visual language pre-training model for the jth image of the cth sample;

[0027] is the initial visual prototype of the cth sample;

[0028] K is the sample quantity of the cth sample;

[0029] Norm() represents a normalization function.

[0030] Further, in step 4, the method of updating the visual prototype with batch data in the training process includes the following method steps:

[0031] Let the sliding update calculation formula be as follows:

[0032]

[0033] In the formula:

[0034] is the visual prototype after the tth sliding update of the visual prototype of the cth sample;

[0035] is the visual prototype after the (t+1)th sliding update of the visual prototype of the cth sample;

[0036] t is the sliding number;

[0037] c represents the sample category number;

[0038] The features extracted by the current model for the j-th image in this batch;

[0039] B represents the sample size for this batch;

[0040] Norm() represents the normalization function;

[0041] γ is the sliding coefficient.

[0042] Furthermore, in step 5, the method for calculating the instance prototype alignment loss includes the following steps:

[0043] Input the c-th class sample data of the m-th task into the student model to obtain the visual features of the c-th class sample data. Using the visual features, text features, and the visual prototype updated by sliding, construct the cross-modal instance prototype alignment loss. The calculation formula for the instance prototype alignment loss is as follows:

[0044]

[0045] In the formula:

[0046] L con For instance prototype alignment loss;

[0047] c represents the sample category number;

[0048] m is the task number;

[0049] for A set;

[0050] The visual prototype is the result of the t-th sliding update of the visual prototype of the c-th sample.

[0051] H c Let be the visual text feature matrix for category c;

[0052] H m This represents the visual text feature matrix output by the student model for the m-th task.

[0053] P m Let be the prototype matrix for the m-th task;

[0054] h is Any element in;

[0055] g is Any element except h;

[0056] f is H m ∪Pm Any element except h;

[0057] τ is the temperature coefficient;

[0058] sim is the cosine similarity function.

[0059] Furthermore, in step 5, the formula for calculating the final distillation loss is as follows:

[0060] Let m be the task number; when m is greater than or equal to 2:

[0061]

[0062]

[0063]

[0064] When m equals 1:

[0065] In the formula:

[0066] The instance prototype similarity distillation loss is calculated based on E0;

[0067] According to E i-1 Calculate the instance prototype similarity distillation loss;

[0068] L dis This is the final distillation loss;

[0069] () represents the Frobenius norm;

[0070] sim() represents the cosine similarity function;

[0071] H m This represents the visual text feature matrix output by the student model for the m-th task.

[0072] This represents the visual text feature matrix output by the first teacher model for the m-th task.

[0073] This represents the visual text feature matrix output by the second teacher model for the m-th task.

[0074] P m Let be the prototype matrix for the m-th task.

[0075] Furthermore, in step 5, the classification loss is calculated using the cross-entropy loss function. The formula for calculating the classification loss is as follows:

[0076] L ceCE(Z m , Y m );

[0077] wherein:

[0078] m is a task number;

[0079] L ce is a classification loss;

[0080] Z m is a logits matrix output by a student model corresponding to the mth task;

[0081] Y m is a label matrix corresponding to the mth task;

[0082] CE() represents a cross-entropy loss function.

[0083] The application further provides a device for a double-teacher brain-inspired distillation lifelong image classification method, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program and realize the steps of the double-teacher brain-inspired distillation lifelong image classification method as described above when executing the computer program.

[0084] The application further provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the double-teacher brain-inspired distillation lifelong image classification method as described above.

[0085] The application has the following advantages and positive effects:

[0086] (1) The application introduces a prototype in the method for lifelong classification based on visual language pre-training model and knowledge distillation, and fully utilizes multi-modal information; (2) The application only needs to use a training data set and a previous model weight, distills limited data in a brain-inspired manner, and does not need to use an additional data set; (3) The application adopts two teacher models to continue double-teacher distillation, and simultaneously relieves the forgetting of catastrophic forgetting and the forgetting of zero-shot transfer ability. BRIEF DESCRIPTION OF DRAWINGS

[0087] Figure 1 is a workflow diagram of the double-teacher brain-inspired distillation lifelong image classification method of the application. DETAILED DESCRIPTION

[0088] The application will be described in detail below with reference to the accompanying drawings and in combination with embodiments, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the application, and are not used to limit the application.

[0089] The following are Chinese interpretations of English words, English abbreviations and English phrases:

[0090] Please refer to Figure 1 A double-teacher brain-like distillation continuous learning image classification method, comprising the following steps:

[0091] Step 1, compile a training image sample set; the image sample set is divided into multiple task image sample sets according to task categories, and each task image sample set is divided into a training data set and a test data set; the training data set of each task includes multiple category image sample sets; each category image sample set includes multiple image samples with labels and corresponding labels and category semantic information;

[0092] Step 2, construct a visual language pre-training model for obtaining initial visual prototypes from image data; the visual language pre-training model includes a visual encoder and a text encoder; the visual encoder is used to process image data and encode images into vector data; the text encoder is used to process text data and encode text into vector data; features of all image samples in the training data set in each task are extracted by using the visual language pre-training model; the initial visual prototype of each category is obtained by calculating the average value of the features of the image samples of the same category in the same task;

[0093] Step 3, define the visual language pre-training model as E0; E0 is used as a first teacher model; a student model is constructed, which has the same structure as the visual language pre-training model and loads the weight parameters of the visual language pre-training model;

[0094] Step 4, train the student model for the mth task, m = 1, 2,..., T, T is the number of tasks, and the trained student model is denoted as Em; when the student model is trained for the m+1th task, Em is used as a second teacher model, and the student model is trained by loading the weight from Em; the visual prototype is updated by sliding with batch data during the training process; m m m

[0095] Step 5, calculate the classification loss Lc and the instance prototype alignment loss Lp according to Em; calculate the instance prototype similarity distillation loss according to Em and E0, average it and take it as the final distillation loss Ld; combine the classification loss Lc, the instance prototype alignment loss Lp and the final distillation loss Ld by weighting to obtain an optimization target; m ce con m-1 dis ce con dis

[0096] Step 6, repeat steps 4 to 5 until the optimization target of all training tasks reaches a set value, and the training is completed.​​​​​​​​​​​

[0097] Step 7, using the trained model to classify the images to be classified.

[0098] Preferably, in step 5, the calculation formula of the optimization target is as follows:

[0099] L = L ce + 0.2L con + 0.2L dis ;

[0100] In the formula:

[0101] L is the optimization target;

[0102] L ce is the classification loss;

[0103] L con is the instance prototype alignment loss;

[0104] L dis is the final distillation loss.

[0105] Preferably, in step 2, the calculation formula of the visual prototype of each category is as follows:

[0106]

[0107] In the formula:

[0108] c is the sample category serial number;

[0109] is the feature extracted by the visual language pre-training model for the jth image of the cth sample;

[0110] is the initial visual prototype of the cth sample;

[0111] K is the sample quantity of the cth sample;

[0112] Norm() represents a normalization function.

[0113] Preferably, in step 4, the method of updating the visual prototype with batch data in the training process includes the following method steps:

[0114] Let the sliding update calculation formula be as follows:

[0115]

[0116] In the formula:

[0117] is the visual prototype after the tth sliding update of the visual prototype of the cth sample;

[0118] the visual prototype of the c-th sample after the t+1-th sliding update of the visual prototype;

[0119] t is the number of sliding times;

[0120] c is the sample category serial number;

[0121] is the feature extracted by the current model for the j-th image in the batch;

[0122] B is the number of samples in the batch;

[0123] Norm() represents a normalization function;

[0124] γ is a sliding coefficient.

[0125] Preferably, in step 5, the method for calculating the instance prototype alignment loss comprises the following method steps:

[0126] inputting the c-th sample data of the m-th task into the student model to obtain the visual feature of the c-th sample data, and constructing a cross-modal instance prototype alignment loss by using the visual feature of the c-th sample data, the text feature and the visual prototype after the sliding update; the calculation formula of the instance prototype alignment loss is as follows:

[0127]

[0128] In the formula:

[0129] L con is the instance prototype alignment loss;

[0130] c is the sample category serial number;

[0131] m is the task serial number;

[0132] is the set of

[0133] is the visual prototype of the c-th sample after the t-th sliding update of the visual prototype;

[0134] H c is the visual-text feature matrix of category c;

[0135] H m is the visual-text feature matrix output by the student model for the m-th task;

[0136] P m is the prototype matrix of the m-th task;

[0137] h is any element in ​​

[0138] g is any element except h;

[0139] f is H m ∪P m any element except h;

[0140] τ is a temperature coefficient;

[0141] sim is a cosine similarity function.

[0142] Preferably, in step 5, the final distillation loss is calculated as follows:

[0143] Let m be the task number; when m is greater than or equal to 2:

[0144]

[0145]

[0146]

[0147] When m is equal to 1:

[0148] In the formula:

[0149] is the instance prototype similarity distillation loss calculated according to E0;

[0150] is the instance prototype similarity distillation loss calculated according to E i-1 ;

[0151] L dis is the final distillation loss;

[0152] () represents the Frobenius norm, also known as the F norm;

[0153] sim() represents the cosine similarity function;

[0154] H m is the visual text feature matrix output by the student model corresponding to the mth task;

[0155] is the visual text feature matrix output by the first teacher model corresponding to the mth task;

[0156] is the visual text feature matrix output by the second teacher model corresponding to the mth task;

[0157] P m is the prototype matrix of the mth task.

[0158] Preferably, in step 5, the classification loss is calculated by using a cross-entropy loss function, and the calculation formula of the classification loss is as follows:

[0159] L ce = CE(Z m , Y m );

[0160] In the formula:

[0161] m is the task number;

[0162] L ce is the classification loss;

[0163] Z m is the logits matrix output by the student model corresponding to the mth task; logits usually refers to the original and processed scores or scores of the output layer of the model.

[0164] Y m is the label matrix corresponding to the mth task;

[0165] CE() represents the cross-entropy loss function.

[0166] The application further provides a device for a double-teacher brain-like distillation continuous learning image classification method, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program and realize the steps of the double-teacher brain-like distillation continuous learning image classification method as described above when executing the computer program.

[0167] The application further provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the double-teacher brain-like distillation continuous learning image classification method as described above.

[0168] The working flow and working principle of the application are further described below by using a preferred embodiment of the application:

[0169] The application provides a double-teacher brain distillation continuous learning image classification method, which combines a visual language pre-training model and a double-teacher knowledge distillation continuous learning image classification method. The method introduces a prototype to fully mine multi-modal information, and the method comprises a visual and text encoder of the visual language pre-training model, an instance prototype alignment module and a brain-inspired similarity distillation module. The instance prototype alignment module utilizes cross-modal contrast learning to shorten the distance between image semantic instances and prototypes of the same category and lengthen the distance between different categories, and the brain-inspired similarity distillation module simulates the learning process of human beings. During the learning process, the human brain tends to analyze similar knowledge and unfamiliar knowledge in the knowledge to be learned according to the existing knowledge, and focuses on learning the latter. In the learning of the student model to the teacher model, the student model does not directly distill the instances, but distills the similarity between the instances and the prototypes, and the teacher model comprises an initial model and a model of a previous task, and the prototype serves as a standard for measurement. The student model focuses on learning the unfamiliar knowledge with a large deviation between the similarities obtained by the student model and the teacher model. The application is implemented by the following technical solutions.

[0170] S1: constructing an image classification task and initializing a visual prototype and a training model: taking the training model as a student model.

[0171] S11: preparing a data set and dividing the data set into T tasks according to categories, and dividing a training set and a test set for each task. Taking the mth task as an example, the training data set of the mth task comprises N categories, each category comprises K labeled image samples, corresponding labels and category semantic information, and is denoted as wherein:

[0172] w represents an image serial number; w = 1, 2, …, N x K;

[0173] T m denotes the training data set of the mth task;

[0174] denotes the wth image sample of the mth task;

[0175] denotes the label of the wth image sample of the mth task;

[0176] denotes the semantic information of the wth image sample of the mth task;

[0177] S12: Construct a visual language pre-training model. The visual language pre-training model can be constructed by downloading an existing visual language pre-training model in the prior art. The constructed visual language pre-training model is frozen and used as a first teacher model. The visual language pre-training model is usually composed of two main components: a visual encoder and a language encoder. The visual encoder is used to process image data and convert images into vector representations, while the language encoder is used to process text data and encode text into vectors. The visual language pre-training model is pre-trained on large-scale data to gradually adjust weights and parameters, so that the visual language pre-training model can effectively capture the relationship between images and text. Finally, through fine-tuning for downstream task training, the visual language pre-training model can exhibit excellent performance on specific tasks.

[0178] Features are extracted from all images in the training data set using the frozen visual language pre-training model, and the following results are obtained: Features extracted by the visual language pre-training model for the jth image of the cth sample.

[0179] S13: Calculate the initial visual prototype of each class in the training data set. This is achieved by averaging the features by class. Taking class c as an example, the expression is as follows:

[0180] The calculation formula of the visual prototype of each class is as follows:

[0181]

[0182] In the formula:

[0183] c is the sample class number;

[0184] is the feature extracted by the visual language pre-training model for the jth image of the cth sample;

[0185] is the initial visual prototype of the cth sample;

[0186] K is the number of samples of the cth sample;

[0187] Norm() represents a normalization function.

[0188] S14: The initial structure of the student model is consistent with the structure of the frozen visual language pre-training model. The weights are loaded from the frozen visual language pre-training model to prepare for training the student model.

[0189] S2: Update the visual prototype according to the current task, train the student model, calculate the classification loss and instance prototype alignment loss, and use the student model trained in the previous task as a second teacher model. Select the initial model and the model of the previous task as teacher models to calculate the instance prototype similarity distillation loss. The specific steps are as follows:

[0190] S21: input the current batch data into the model to obtain the visual features of the batch, and perform sliding update on the visual prototype, and perform sliding update on the visual prototype of the cth sample data.

[0191] Let the sliding update calculation formula be as follows:

[0192]

[0193] In the formula:

[0194] is the visual prototype after the tth sliding update on the visual prototype of the cth sample;

[0195] is the visual prototype after the (t+1) th sliding update on the visual prototype of the cth sample;

[0196] t is the sliding number;

[0197] c is the sample category serial number;

[0198] is the feature extracted by the current model for the jth image in the batch;

[0199] B is the number of samples in the batch;

[0200] Norm() represents a normalization function;

[0201] γ is the sliding coefficient.

[0202] One model training is completed by iterating multiple times, and the data processed in one iteration process is a batch of data.

[0203] S22: use the visual features, text features and updated visual prototypes of the current batch Construct a cross-modal contrast loss function. Taking category c as an example, the expression is as follows:

[0204] Input the cth sample data of the mth task into the student model to obtain the visual features of the cth sample data, and use the visual features, text features and sliding updated visual prototypes of the cth sample data to construct a cross-modal instance prototype alignment loss. The calculation formula of the instance prototype alignment loss is as follows:

[0205]

[0206] In the formula:

[0207] L con is the instance prototype alignment loss;

[0208] c is the sample category serial number;

[0209] m is the task number;

[0210] for A set;

[0211] H represents c and A collection;

[0212] H m ∪P m H represents m With P m A collection;

[0213] express Elements other than h;

[0214] H m ∪P m / h represents H m ∪P m Elements other than h;

[0215] The visual prototype is the result of the t-th sliding update of the visual prototype of the c-th sample.

[0216] H c Let be the visual text feature matrix for category c;

[0217] H m This represents the visual text feature matrix output by the student model for the m-th task.

[0218] P m Let be the prototype matrix for the m-th task;

[0219] h is Any element in;

[0220] g is Any element except h;

[0221] f is H m ∪P m Any element except h;

[0222] τ is the temperature coefficient;

[0223] sim is the cosine similarity function.

[0224] S23: Calculate the brain-inspired distillation loss L using the visual features, textual features, and visual prototypes of the current batch, along with two teacher models. dis The calculation formula is as follows:

[0225] Let m be the task number; when m is greater than or equal to 2:

[0226]

[0227]

[0228]

[0229] When m is equal to 1:

[0230] In the formula:

[0231] is the instance prototype similarity distillation loss calculated according to E0;

[0232] is the instance prototype similarity distillation loss calculated according to E i-1 ;

[0233] L dis is the final distillation loss;

[0234] () represents the Frobenius norm (F norm);

[0235] sim() represents the cosine similarity function;

[0236] H m is the visual-text feature matrix output by the student model corresponding to the mth task;

[0237] is the visual-text feature matrix output by the first teacher model corresponding to the mth task,

[0238] is the visual-text feature matrix output by the second teacher model corresponding to the mth task;

[0239] P m is the prototype matrix of the mth task.

[0240] S24: The classification loss of the model on the mth task is:

[0241] The cross-entropy loss function is used to calculate the classification loss, and the calculation formula of the classification loss is as follows:

[0242] L ce = CE(Z m , Y m );

[0243] In the formula:

[0244] m is the task number;

[0245] Lce is a classification loss

[0246] Z m is a logits matrix output by the student model corresponding to the mth task;

[0247] Y m is a label matrix corresponding to the mth task;

[0248] CE() represents a cross-entropy loss function.

[0249] S25: combine the classification loss L ce , the instance prototype alignment loss L con , and the final distillation loss L dis according to certain weights to obtain an optimization target; according to the calculation result of the optimization target, perform back propagation and update the model parameters.

[0250] L = L ce + 0.2L con + 0.2L dis .

[0251] In the formula:

[0252] L is an optimization target;

[0253] L ce is a classification loss;

[0254] L con is an instance prototype alignment loss;

[0255] L dis is a final distillation loss.

[0256] According to S21, S22, and S24, the prototype is updated, the instance prototype alignment loss and the classification loss are calculated, wherein the sliding parameter γ is generally initialized as 0.98, and is reduced by 0.06 each time until it is reduced to 0.02. The temperature parameter τ is generally set to 0.2.

[0257] S3: repeat the above S2 step until the optimization target of all training tasks reaches the set value, and the training is completed;

[0258] When training the first task, that is, when m is equal to 1: according to S21, S22, S23, and S24, the prototype is updated, the instance prototype alignment loss, the distillation loss, and the classification loss are calculated. The distillation loss is calculated according to the following formula:

[0259]

[0260] Set an appropriate learning rate to update the student model by back propagation, repeat 1000 iterations, and complete the training to save the student model of the first task.

[0261] Then the second task is trained. The second and subsequent tasks are distilled by double teacher distillation in the distillation loss calculation of S23, when m is greater than or equal to 2: the distillation loss is calculated according to the following formula.

[0262]

[0263]

[0264]

[0265] Optimization target: L = L ce + 0.2L con + 0.2L dis ;

[0266] The model is updated by back propagation according to the appropriate learning rate set according to the optimization target, and the training is completed after 1000 iterations, and the model of the second task is saved.

[0267] The above operation is repeated until all tasks are trained.

[0268] S4: Load the model parameters saved during the training of each task, use the test set to construct a classification task, and test the student model.

[0269] The above-described embodiments are only used to illustrate the technical ideas and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot be limited to the patent scope of the present application only by the present embodiment, that is, any equivalent changes or modifications made according to the spirit disclosed by the present application still fall within the patent scope of the present application.

Claims

1. A dual-teacher brain-like distillation continual learning image classification method, characterized in that, The method comprises the following steps: Step 1, preparing a training image sample set; the image sample set is divided into multiple task image sample sets according to task categories, each task image sample set is divided into a training data set and a test data set; the training data set of each task includes multiple category image sample sets; each category image sample set includes multiple image samples with labels and corresponding labels and category semantic information; Step 2, constructing a visual language pre-training model for obtaining initial visual prototypes from image data; the visual language pre-training model includes a visual encoder and a text encoder; the visual encoder is used for processing image data and encoding images into vector data; the text encoder is used for processing text data and encoding text into vector data; features of all image samples in the training data set in each task are extracted by using the visual language pre-training model; The initial visual prototype of each category is obtained by calculating the average value of the features of the image samples of the same category in the same task; Step 3, defining the visual language pre-training model as E0; taking E0 as a first teacher model; constructing a student model, so that the structure of the student model is the same as that of the visual language pre-training model, and the weight parameters of the visual language pre-training model are loaded; Step 4, training the student model for the mth task, m = 1, 2, …, T, T is the number of tasks, and the trained student model is recorded as E m ; when the student model is trained for the m+1th task, E m is loaded as the second teacher model, and the student model is trained from E m ; the visual prototype is updated in the training process with the batch data. Step 5, according to E m Calculate the classification loss L ce Alignment loss L with instance prototype con According to E m-1 The similarity distillation loss between the instance prototype and E0 is calculated, and their average value is used as the final distillation loss L. dis Classification loss L ce Instance prototype alignment loss L con Final distillation loss L dis The optimization objective is obtained by weighted combination; Step 6, repeating steps 4 to 5 until the optimization target of all training tasks reaches the set value, and the training is completed; Step 7, using the trained model to classify the images to be classified; In step 5, the method for calculating the instance prototype alignment loss comprises the following method steps: The mth task cth sample data is input into the student model to obtain the visual features of the cth sample data, and the cross-modal instance prototype alignment loss is constructed by using the visual features, text features and the sliding updated visual prototype of the cth sample data; the calculation formula of the instance prototype alignment loss is as follows: In the formula: L con For instance prototype alignment loss; C is the sample category serial number; M is the task serial number; a set of the set of a visual prototype updated for the cth slide of the visual prototype of the cth sample; H c is a visual text feature matrix of class c; H m is the visual text feature matrix output by the student model for the mth task; P m Prototype matrix for the mth task; h is any of the elements of g is any element except h; f is H m ∪P m any element of h except h; Tau is the temperature coefficient; Sim is the cosine similarity function.

2. The dual-teacher brain-inspired distillation continual learning image classification method according to claim 1, characterized in that, In step 5, the calculation formula of the optimization target is as follows: L = L ce + 0.2L con + 0.2L dis ; In the formula: L is the optimization target; L ce is a classification loss; L con instance prototype alignment loss; L dis Loss to final distillation.

3. The dual-teacher brain-storming continual learning image classification method of claim 1, wherein, In step 2, the calculation formula of the visual prototype of each category is as follows: In the formula: C is the sample category serial number; is the feature extracted by the visual language pre-training model for the cth type of sample jth image; Initial visual prototype for Category C sample; K is the sample quantity of the cth sample; Norm() represents a normalization function.

4. The dual-teacher brain-storming continual learning image classification method of claim 1, wherein, In step 4, the method for sliding updating the visual prototype with batch data in the training process comprises the following method steps: The sliding update calculation formula is as follows: In the formula: a visual prototype updated for the t-th time for the visual prototype of the c-th sample; a visual prototype of the (t+1)th sliding update to the visual prototype of the cth sample; T is the sliding number; C is the sample category serial number; is the feature extracted by the current model for the jth image in the batch; B is the sample quantity of the batch; Norm() represents a normalization function; Gamma is the sliding coefficient.

5. The dual-teacher brain-inspired distillation continual learning image classification method according to claim 1, characterized in that, In step 5, the calculation formula of the final distillation loss is as follows: Let m be the task serial number; when m is greater than or equal to 2: When m is equal to 1: In the formula: instance prototype similarity distillation loss calculated according to E0; For example, the similarity loss function can be defined as follows: i-1 An example prototype similarity distillation loss is computed; L dis Loss to final distillation; () denotes the Frobenius norm; Sim() represents a cosine similarity function; H m is the visual text feature matrix corresponding to the mth task output by the student model; is the visual text feature matrix output by the first teacher model for the mth task; is the visual text feature matrix output by the second teacher model for the mth task; P m Prototype matrix for the mth task.

6. The dual-teacher brain-inspired distillation continual learning image classification method according to claim 1, characterized in that, In step 5, the classification loss is calculated by using a cross-entropy loss function, and the calculation formula of the classification loss is as follows: L ce = CE(Z m , Y m ); In the formula: M is the task serial number; L ce is a classification loss; Z m is the logits matrix output by the student model for the mth task; Y m a label matrix corresponding to the mth task; CE() represents a cross-entropy loss function.

7. A device of a dual-teacher brain-like distillation continual learning image classification method, comprising a memory and a processor, characterized in that, The memory is used for storing a computer program; the processor is used for executing the computer program and realizing the double-teacher brain distillation continuous learning image classification method steps in any one of claims 1 to 6 when the computer program is executed.

8. A storage medium storing a computer program, characterized by The computer program is executed by the processor to implement the steps of the dual-teacher brain-inspired distillation continual learning image classification method according to any one of claims 1 to 6.

Images