A continual learning image classification method facing open environment

Abstract

This invention discloses a continuous learning image classification method for open environments. First, a multi-task model, a new dataset, and historical datasets for the image classification task are acquired. Next, out-of-domain sample detection is used to divide the new dataset into in-domain and out-of-domain datasets, generating sub-task labels for the in-domain dataset. Then, in-domain sample detection is performed on the out-of-domain dataset, generating sub-task labels for the out-of-domain dataset. Subsequently, the new dataset and sub-task labels are merged into a labeled dataset, and the labeled dataset and historical datasets are merged into a training dataset. Finally, the training dataset is input into the model for incremental training, and a playback strategy is used to save some data. During prediction, sub-task labels are generated for images, and the model classifies images based on the images and sub-task labels. Compared with existing technologies, this invention fully considers the diversity and unknown nature of data in open environments, improving image classification performance in continuous learning scenarios.

Description

Technical Field

[0001] This invention relates to a continuous learning image classification method for open environments, belonging to the fields of computer vision and continuous learning in computer technology, and is particularly suitable for open scenarios where training data is continuously added and its distribution may change at any time. Background Technology

[0002] Image classification is a classic task in computer vision, with numerous applications in security, e-commerce, and autonomous driving. With the development of machine learning and the rise of deep learning, machine learning models have achieved performance comparable to or even surpassing human capabilities in solving single image classification tasks. However, traditional machine learning methods only perform well on single tasks and cannot learn continuously like humans. Existing research has found that complex machine learning models, such as neural networks, rapidly lose knowledge of previously learned tasks when learning new image classification tasks due to changes in the distribution of training data—a phenomenon known as catastrophic forgetting.

[0003] Continuous learning is a field of machine learning proposed to address the problem of catastrophic forgetting. Existing continuous learning methods can be broadly categorized into three types: regularization-based methods, replay-based methods, and structure-based methods. Regularization-based methods add a regularization term to the loss function to penalize drastic changes in model parameters. Replay-based methods save a small number of old task samples to review old tasks while the model learns new ones. Structure-based methods add extra model structure when learning new tasks to minimize the sharing of model parameters between new and old tasks.

[0004] However, current continuous learning methods rely on several assumptions when solving image classification tasks. Each incremental training data session comes entirely from the same task, one that the model has not yet learned. All training data corresponds to known tasks, and there is no new training data for older tasks. In real-world continuous learning image classification tasks, models often undergo incremental training at fixed intervals or after collecting a sufficient amount of image data. This means that each collected training data session may come from different tasks, and the tasks corresponding to these training data are unknown. Furthermore, in real-world scenarios, even learned tasks may receive new data. Due to the openness of data in real-world applications, existing continuous learning methods cannot effectively prevent catastrophic forgetting in such scenarios. Therefore, there is an urgent need for a continuous learning image classification method for open environments to address the continuous learning image classification problem in real-world applications. Summary of the Invention

[0005] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] To address the problems and shortcomings of existing technologies, this invention aims to provide a continuous learning image classification method for open environments. For the continuous learning image classification problem in open environments, this method first identifies samples that do not belong to the currently learned image classification subtask through out-of-domain sample detection. Then, it determines whether the aforementioned samples belong to the already learned image classification subtask through in-domain sample detection, and generates subtask labels for all new data. During training, an out-of-domain loss function is used to make the model more sensitive to out-of-domain samples, thereby obtaining more accurate subtask labels. During prediction, subtask labels are first generated based on the input samples, and then the categories are output, improving the model's image classification accuracy and solving the problems mentioned in the background technology.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] This invention discloses a continuous learning image classification method for open environments, which mainly includes the following steps:

[0009] Step 1: Obtain the multi-task model f, the new dataset D, and the historical dataset D corresponding to the image classification task. h ;

[0010] Step 2: Perform out-of-domain sample detection on the currently learned image classification subtask, and divide the newly added dataset D into in-domain dataset D. ind and external dataset D ood Generate a domain-specific data subtask tag set L ind ;

[0011] Step 3, for the external dataset D ood Perform in-domain sample detection to determine whether the image classification subtask sample is a learned image classification subtask, and generate an out-of-domain data subtask label set L. ood ;

[0012] Step 4: Combine the newly added dataset D with the domain-specific data subtask tag set L. ind , Extraterritorial data subtask tag set L ood Merge into labeled dataset D l The labeled dataset D l and the historical dataset D h Merge into training dataset D train ;

[0013] Step 5, transfer the training dataset D train Input the multi-task model f for incremental training;

[0014] Step 6, for the training dataset D train A representative dataset D is generated by saving a portion of the data using a playback strategy. e and use the representative dataset D e Update the historical dataset D h ;

[0015] Step 7: Obtain the image to be classified and generate a sub-task label set for it. The multi-task model f can output the image classification result based on the image to be classified and the sub-task label set.

[0016] Furthermore, in step 2, the newly added dataset D is divided into intra-domain dataset D based on the detection of out-of-domain samples. ind and external dataset D ood Generate a domain-specific data subtask tag set L ind Specifically, it includes the following steps:

[0017] Step 2.1: Obtain the most recently learned subtask t of the multi-task model f;

[0018] Step 2.2: Extract image samples sequentially from the newly added dataset D;

[0019] Step 2.3: Input the sample into the multi-task model f to obtain the maximum probability value p output by the multi-task model f on the subtask t. t,i ;

[0020] Step 2.4: Use the out-of-domain sample discrimination formula in conjunction with the maximum probability value p. t,i Determine whether the sample is an out-of-domain sample, and then add it to the out-of-domain dataset D. ood and the domain dataset D ind ;

[0021] Step 2.5, for the domain dataset D ind In each sample, subtask labels are generated within the domain. i Mark the subtasks within the domain as l i Add the domain-specific data subtask tag set L ind And output the external dataset D ood .

[0022] Furthermore, in step 3, the out-of-domain dataset D... ood Perform in-domain sample detection to determine whether the image classification subtask sample is a learned image classification subtask, and generate an out-of-domain data subtask label set L.ood Specifically, it includes the following steps:

[0023] Step 3.1: Obtain the set T of all currently known subtasks, and calculate the set T of subtasks to be detected. d ;

[0024] Step 3.2, in the out-of-domain dataset D ood Each out-of-domain sample is extracted sequentially from the set of subtasks to be detected, T. d Extract each subtask t sequentially n ;

[0025] Step 3.3, combine the out-of-domain samples and the subtask t n The input is given to the multi-task model f, and the result is obtained from the multi-task model f in the subtask t. n The maximum out-of-domain probability value output above

[0026] Step 3.4: Use the correlation calculation formula combined with the maximum probability value outside the domain. Calculate the out-of-domain samples in the subtask t n correlation on

[0027] Step 3.5, calculate the correlation. Subtask t that yields the maximum value max And the maximum correlation of the out-of-domain samples across all the subtasks to be detected. And compare them;

[0028] Step 3.6: Generate corresponding out-of-domain subtask labels for the out-of-domain samples. o And mark the extra-domain subtask as l o Subtask tag set L for adding the data from outside the domain ood Output later.

[0029] Furthermore, in step 4, the labeled dataset D is obtained by merging. l The steps are as follows:

[0030] Step 4.1, for the image samples in the newly added dataset D, in the domain data subtask label set L ind Find the corresponding intradomain subtask marker l i Forming intradomain triples (x i ,y i ,l i );

[0031] Step 4.2, in the extra-domain data subtask tag set L ood In the middle, find the corresponding extra-domain subtask marker l oForming an extradomain triple (x i ,y i ,l o );

[0032] Step 4.3: Add the intra-domain and extra-domain triples to the labeled dataset D. l And output it.

[0033] Furthermore, in step 5, for the training dataset D train The steps for incremental training of the multi-task model f are as follows:

[0034] Step 5.1, add a new task t to the multi-task model f. N+1 and initialize its model structure;

[0035] Step 5.2: Update the set T of all currently known subtasks, and update the number N of all currently known subtasks;

[0036] Step 5.3, using the cross-entropy loss function L ce extraterritorial loss function L ood and l2-regularized loss function L norm Construct the total loss function L;

[0037] Step 5.4: Calculate the gradient using the total loss function L and update the model until convergence.

[0038] Furthermore, in step 6, the training dataset D... train A representative dataset D is generated by saving a portion of the data using a playback strategy. e and use the representative dataset D e Update the historical dataset D h The specific steps are as follows:

[0039] Step 6.1, obtain the training dataset D train The total number of image categories C, and the historical dataset D h Size M;

[0040] Step 6.2, from the training dataset D train Random selection Save a sample of that category, and combine all the saved samples to form a representative dataset D. e ;

[0041] Step 6.3, update the historical dataset D h For the representative dataset D e .

[0042] Furthermore, in step 7, the image to be classified is acquired and a subtask label set is generated for it. The multi-task model f can then output the image classification result based on the image to be classified and the subtask label set. The specific steps are as follows:

[0043] Step 7.1: Generate subtask labels l for the acquired image to be classified. test ;

[0044] Step 7.2, label the image to be classified and the subtask. test Input the multi-task model f, and obtain the multi-task model f in subtask l test The probability distribution q of the output ltest ;

[0045] Step 7.3, output the probability distribution q. ltest The image category with the highest probability.

[0046] Furthermore, the out-of-domain loss function L described in step 5.3 ood The definition is,

[0047]

[0048] Where N is the number of all current subtasks, II is the indicator function, and m is the hyperparameter. For the training set D train All subtasks in the middle are marked as t k A collection of image samples For the sample set The total number of samples in the middle. For the sample set The o-th sample in the multi-task model f and subtask t j The probability distribution of the output, The probability distribution The entropy.

[0049] Furthermore, the cross-entropy loss function L described in step 5.3 ce The definition is,

[0050]

[0051] Where B is the training dataset D train The total number of samples, p i The triples composed of the image samples are the corresponding categories y output by the multi-task model f and the sub-task labels. i The probability of.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] This invention provides a continuous learning image classification method for open environments. It primarily uses out-of-domain sample detection to divide a newly added dataset into in-domain and out-of-domain datasets, generating sub-task labels for the in-domain dataset. It also performs in-domain sample detection on the out-of-domain dataset to generate sub-task labels for the out-of-domain dataset. The newly added dataset and sub-task labels are merged into a labeled dataset, and the labeled dataset is merged with historical datasets to form a training dataset. Finally, the training dataset is input into the model for incremental training, and a playback strategy is used to save some training data. During training, the out-of-domain loss function is included as a term in the total loss function. By encouraging the model to output a more uniform probability distribution on out-of-domain data, the generated sub-task labels are more accurate. This invention fully considers the diversity and unknowns of data in continuous learning image classification problems in practical application scenarios. By utilizing the differences in the model's output on images of different tasks, it solves the image classification problem in open environments that traditional continuous learning image classification techniques have not considered, and improves the model's image classification performance in continuous learning scenarios. Attached Figure Description

[0054] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application.

[0055] In the attached diagram:

[0056] Figure 1 : This is a flowchart of the steps of the present invention;

[0057] Figure 2 This is a schematic diagram of the steps for detecting out-of-domain samples in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the steps for in-domain sample detection in an embodiment of the present invention. Detailed Implementation

[0059] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0060] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0061] This invention discloses a continuous learning image classification method for open environments. The following will describe this disclosure in detail with reference to the accompanying drawings and embodiments.

[0062] Reference Figure 1 As shown, the main steps include the following:

[0063] Step 1: Obtain the multi-task model f, the new dataset D, and the historical dataset D corresponding to the image classification task. h ;

[0064] Step 2: Perform out-of-domain sample detection on the currently learned image classification subtask, and divide the newly added dataset D into in-domain dataset D. ind and external dataset D ood Generate a domain-specific data subtask tag set L ind ;

[0065] Step 3, for the external dataset D ood Perform in-domain sample detection to determine whether the image classification subtask samples are from a learned image classification subtask, and generate an out-of-domain data subtask label set L. ood ;

[0066] Step 4: Combine the newly added dataset D with the domain-specific data subtask label set L. ind , Extraterritorial data subtask tag set L ood Merge into labeled dataset D l , label dataset D l and historical dataset D h Merge into training dataset D train ;

[0067] Step 5, transfer the training dataset D train Input the multi-task model f for incremental training;

[0068] Step 6, for the training dataset D train A representative dataset D is generated by saving a portion of the data using a playback strategy. e and use the representative dataset D e Update historical dataset D h ;

[0069] Step 7: Obtain the image to be classified and generate a sub-task label set for it. The multi-task model f can output the image classification result based on the image to be classified and the sub-task label set.

[0070] Specifically, we use the ResNet model as the multi-task model f in this study. The multi-task model f includes a shared feature extraction structure to extract image feature vectors, and several classifiers that map the feature vectors to probability distributions, with one classifier for each task. The multi-task model f corresponding to the image classification task has already learned all known tasks so far, and the newly added dataset D is represented as D = *(x i ,y i ), i = 1, 2, ..., N s}, historical dataset D h Represented as D h =*(x h ,y h ,t h ),h=1,2,…,M}. Where, x i and x h Represents an image vector, y i and y h Indicates the category label to which it belongs, t h N represents the subtask label corresponding to the image. s M represents the size of the new dataset D, and M represents the size of the historical dataset D. h The size of the sub-task is determined. Out-of-domain sample detection is used to identify samples that do not belong to the currently learned image classification sub-task, while in-domain sample detection is used to determine whether an image sample belongs to a learned image classification sub-task. The sub-task label set of the image to be classified includes the in-domain data sub-task label set L. ind and the tag set L of the extra-domain data subtask ood .

[0071] Reference Figure 2 As shown, in step 2, the newly added dataset D is divided into intra-domain dataset D based on the detection results of out-of-domain samples. ind and external dataset D ood And generate a domain-specific data subtask tag set L ind Specifically, it includes the following steps.

[0072] Step 2.1: Obtain the most recently learned subtask t of the multi-task model f.

[0073] Step 2.2: Extract image samples sequentially from the newly added dataset D.

[0074] Step 2.3: Input the sample into the multi-task model f to obtain the maximum probability value p output by the multi-task model f on the subtask t. t,i .

[0075] Step 2.4: Use the out-of-domain sample discrimination formula combined with the maximum probability value p t,i Determine whether a sample is from outside the domain, and then add it to the outside dataset D.ood and the domain dataset D ind .

[0076] Step 2.5, for the domain dataset D ind In each sample, subtask labels are generated within the domain. i Mark subtasks within the domain as l i Join the domain data subtask tag set L ind And output the out-of-domain dataset D ood .

[0077] Specifically, out-of-domain sample detection is performed on the most recently learned subtask t. Since subtask t has just been learned, the model's performance on this subtask is not affected by catastrophic forgetting. Compared to other subtasks, out-of-domain sample detection on subtask t achieves better detection accuracy. Each image sample (x) is sequentially traversed from the newly added dataset D. i ,y i ), where x i For sample input, y i Let be the sample category, and ... i ,y i The input is fed into the model, and the probability distribution of the classifier output for the corresponding subtask t is found. The maximum probability value in this discrete probability distribution is denoted as p. t,i Then, determine whether a sample is from outside the region using the out-of-region sample discrimination formula. The out-of-region sample discrimination formula is expressed as:

[0078]

[0079] Where, p t,avg For historical dataset D h δ is the average of the maximum probability values ​​output by the multi-task model f and subtask t for all image samples with subtask label t. ood This is a hyperparameter. When the formula result is 1, the sample (x) i ,y i If the sample is outside the domain, then it is identified as a sample from outside the domain. i ,y i Add to external dataset D ood When the formula result is 0, the sample (x) i ,y i If a sample is identified as a sample within the domain, then the sample (x) will be... i ,y i Add the domain dataset D ind Repeat step 2.4 until each sample (x) i ,y iAll samples have been determined to be out-of-domain samples. The principle behind this determination is that the maximum probability of the model outputting a sample within the domain is often higher than the maximum probability of it outputting a sample outside the domain. Then, for the in-domain dataset D... ind Each sample generates a subtask label l i =t, mark the subtask as l i Subtask tag set L for adding in-domain data ind Here we assume that the in-domain samples and the training data for subtask t follow the same distribution, therefore we define the in-domain dataset D. ind Let t be the subtask label for all samples in the dataset. Based on the above steps, we can then divide the newly added dataset D into domain datasets Di. ind and external dataset D ood Simultaneously, generate a domain-specific data subtask tag set L. ind .

[0080] Reference Figure 3 As shown, in step 3, the off-domain dataset D... o Perform in-domain sample detection to determine whether the image classification subtask samples are from a learned image classification subtask, and generate an out-of-domain data subtask label set L. ood Specifically, it includes the following steps:

[0081] Step 3.1: Obtain the set T of all currently known subtasks, and calculate the set T of subtasks to be detected. d .

[0082] Step 3.2, in the external dataset D ood Each out-of-domain sample is extracted sequentially, forming the subtask set T to be detected. d Extract each subtask t sequentially n .

[0083] Step 3.3, combine the out-of-domain samples and subtask t n Inputting a multi-task model f, we obtain the multi-task model f in subtask t. n The maximum out-of-domain probability value p of the output tn,i .

[0084] Step 3.4: Use the correlation calculation formula combined with the maximum out-of-domain probability value p tn,i Calculate out-of-domain samples in subtask t n correlation on

[0085]

[0086] Step 3.5, Calculate the correlation Subtask t that yields the maximum value max And the maximum correlation of current out-of-domain samples across all subtasks to be detected. And compare them.

[0087] Step 3.6: Generate corresponding out-of-domain subtask labels for out-of-domain samples. o And mark the out-of-domain subtasks as l o Subtask tag set L for adding data from outside the domain ood Output later.

[0088] Specifically, for the domain dataset D ind After generating subtask labels, it is also necessary to add labels to the out-of-domain dataset D. ood Generate subtask labels. Due to the external dataset D... ood The image samples in the dataset have already been identified as out-of-domain samples on subtask t, therefore computation needs to be performed on other learned subtasks. The current set of all known subtasks is T = *t1,t2,…,t N}, where N is the total number of all current subtasks. The set of subtasks to be detected is calculated as T. d =T-*t}, in the set of subtasks to be detected T d Extract each subtask t sequentially n From the external dataset D ood Iterate through each image sample (x) in turn. i ,y i ), and the image samples (x) i ,y i The input is given to the model, and for each subtask t other than subtask t... n Find the subtask t corresponding to the model n The probability distribution output by the classifier is used to obtain the largest out-of-domain maximum probability value in this discrete probability distribution, denoted as [missing value]. Then calculate the correlation according to the correlation calculation formula. The formula for calculating correlation is expressed as follows:

[0089]

[0090] Where, p tn,avg For the historical dataset D h In the multi-task model f, all image samples labeled t for subtasks are related to subtask t. n The maximum probability average of the output. Although the calculation is performed on subtask t. n In the above, we still used historical data from subtask t, for the same reason that the model will not be affected by catastrophic forgetting on the most recently learned data.

[0091] It is worth noting that the image sample labeled t for subtask t n In this context, it refers to samples from outside the domain. Therefore, we consider the maximum probability p from outside the domain to be... tn,iCompared to the maximum probability average p tn,avg The larger the value, the higher the correlation. The larger the sample (x) is. i ,y i The more likely it is to be a subtask t n The samples within the domain above can be used to define samples (x). i ,y i Generate subtask labels. Calculate the maximum relevance. And the subtask t that maximizes the correlation. max The formula is:

[0092]

[0093]

[0094] In particular, if sample (x) i ,y i If the correlation is low across all learned subtasks, then the sample (x) is considered to have low correlation across all learned subtasks. i ,y i The image samples (x) follow an unknown distribution, i.e., the image samples (x) i ,y i This belongs to a new task. With δ ind Comparison, where δ ind This is a hyperparameter. If... Then, generate corresponding subtask labels for out-of-domain samples. i Represented as l i =t max Otherwise, it is an out-of-domain sample (x) i ,y i Generate corresponding subtask markers. o Represented as l o =N+1. This allows you to mark subtasks as l. o Adding the tag set L of the extra-domain data subtask ood This generates the external dataset D. ood After the subtasks are labeled, all samples in the newly added dataset D have obtained the corresponding subtask labels.

[0095] In step 4, the newly added dataset D is compared with the domain data subtask label set L. ind , Extraterritorial data subtask tag set L ood Merge into labeled dataset D l The specific process is as follows:

[0096] Step 4.1: For the image samples in the newly added dataset D, label the data in the domain subtask set L. ind Find the corresponding intradomain subtask marker l iForming intradomain triples (x i ,y i ,l i ).

[0097] Step 4.2, in the extra-domain data subtask tag set L ood In the middle, find the corresponding extra-domain subtask marker l o Forming an extradomain triple (x i ,y i ,l o ).

[0098] Step 4.3: Add the intra-domain and extra-domain triples to the labeled dataset D. l And output it.

[0099] In step 5, for the training dataset D train The steps for incremental training of the multi-task model f are as follows:

[0100] Step 5.1, add a new task t to the multi-task model f. N+1 It also initializes the model structure.

[0101] Step 5.2: Update the set T of all known subtasks and update the number N of all known subtasks.

[0102] Step 5.3, using the cross-entropy loss function L ce extraterritorial loss function L ood and l2-regularized loss function L norm Construct the total loss function L.

[0103] Step 5.4: Calculate the gradient using the total loss function L and update the model until convergence.

[0104] Specifically, the first step is to check whether there is any training data belonging to a new, unlearned task, i.e., to examine the training dataset D. train Does the set of known subtasks, T, exist? If so, update the known subtask set T and the total number of subtasks N, add the classifier corresponding to the new task to the model, and initialize it. Then, train the model using the total loss function L, where L = L ce +λ1L ood +λ2L norm L ce Let L be the cross-entropy loss function. ood Let L be the out-of-domain loss function. norm The loss function is L2-regularized, where λ1 and λ2 are weight hyperparameters. The cross-entropy loss function is L... ce The definition is,

[0105]

[0106] Where B is the training dataset D train The total number of samples, p i It is the corresponding category y output by the triples composed of image samples in the multi-task model f and the sub-task labels. i The probability of.

[0107] Out-of-domain loss function L ood The definition is,

[0108]

[0109] Where N is the number of all current subtasks, II is the indicator function, and m is the hyperparameter. For training set D train All subtasks in the middle are marked as t k A collection of image samples For sample set The total number of samples in the middle, q tj,o For sample set The o-th sample in the multi-task model f and sub-task t j The probability distribution of the output, H(q) tj,o ) represents the probability distribution q tj,o The entropy. The out-of-domain loss function L. ood The model is encouraged to output a more uniform probability distribution on out-of-domain data, thus making the generated subtask labels more accurate.

[0110] In step 6, the training dataset D train A representative dataset D is generated by saving a portion of the data using a playback strategy. e and use the representative dataset D e Update historical dataset D h The specific steps are as follows:

[0111] Step 6.1, Obtain the training dataset D train The total number of image categories C, and the size M of the historical dataset Dh.

[0112] Step 6.2, from the training dataset D train Random selection Save a sample of that category, and combine all the saved samples to form a representative dataset D. e .

[0113] Step 6.3, Update the historical dataset D h To represent dataset D e .

[0114] Specifically, the historical dataset D can be... hThe size M is set to 2000. If the current training dataset D... train The dataset contains image data across 20 categories, i.e., C=20. Therefore, for each category, from the training dataset D... train Random selection Save one image sample of this category.

[0115] In step 7, the image to be classified is obtained and a subtask label set is generated for it. The multi-task model f can output the image classification result based on the image to be classified and the subtask label set. The specific steps are as follows:

[0116] Step 7.1: Generate subtask labels for the acquired images to be classified. test .

[0117] Step 7.2, label the image to be classified and the subtask. test Input a multi-task model f, and obtain the multi-task model f in subtask l test The probability distribution q of the output ltest .

[0118] Step 7.3, output probability distribution q ltest The image category with the highest probability.

[0119] Specifically, after receiving the image x to be classified, the process first determines whether image x is an out-of-domain sample according to step 2. If it is an out-of-domain sample, the subtask label corresponding to image x is then set to l. test =t; otherwise, continue with step 3 to perform in-domain sample detection and obtain the corresponding subtask label l. test If the subtask is marked l test If it is a task that has already been learned, then the test labeled dataset is input into the multi-task model f to obtain the classification result; if the subtask label l test This is a task that has not been learned before, so you can mark the subtask as l. test To maximize the similarity of the subtask in step 3, prediction is then performed. Alternatively, image x can be considered an out-of-domain sample for the entire image classification task and rejected for prediction. Here, the subtask is labeled l. test Including extra-domain subtask markers o and domain subtask marker l i .

[0120] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A continuous learning image classification method for open environments, characterized in that, Includes the following steps: Step 1: Obtain the multi-task model corresponding to the image classification task. Add a new dataset and historical datasets ; Step 2: Perform out-of-domain sample detection on the currently learned image classification subtask, and add the new dataset. Divided into intra-domain datasets and extra-domain datasets Generate a domain-specific data subtask tag set. ; Step 3, for the extra-domain dataset Perform in-domain sample detection to determine whether the image classification subtask sample is a learned image classification subtask, and generate an out-of-domain data subtask label set. ; Step 4, add the new dataset With the domain-specific data subtask tag set Extraterritorial data subtask tag set Merge into labeled datasets The labeled dataset and the historical dataset Merge into training dataset ; Step 5, the training dataset Input the multi-task model Perform incremental training; Step 6, for the training dataset A representative dataset is generated by saving a portion of the data using a playback strategy. and the representative dataset Update the historical dataset ; Step 7: Obtain the image to be classified and generate a sub-task label set for it; the multi-task model The image classification result can be output based on the image to be classified and the subtask label set; Step 2 also includes the following steps: Step 2.1, obtain the multi-task model. The most recent learning subtask ; Step 2.2, in the newly added dataset Image samples are extracted sequentially and input into the multi-task model. The multi-task model is obtained. In the subtask The maximum probability value of the output ; Step 2.3: Use the out-of-domain sample discrimination formula in conjunction with the maximum probability value. Determine whether the sample is an out-of-domain sample, and then add it to the out-of-domain dataset accordingly. and the dataset within the domain ; Step 2.4, for the dataset within the domain In each sample, generate subtask labels within the domain. Mark the subtasks within the domain. Add to the domain-specific data subtask tag set And output the extra-domain dataset ; The out-of-domain sample discrimination formula in step 2.3 is expressed as follows: in, For the historical dataset All subtasks are tagged as Image samples in the multi-task model sub-tasks The average of the maximum probability values ​​output above. This is a hyperparameter.

2. The continuous learning image classification method for open environments according to claim 1, characterized in that: In step 3, the off-domain dataset Perform in-domain sample detection to determine whether the image classification subtask sample is a learned image classification subtask, and generate an out-of-domain data subtask label set. Specifically, it includes the following steps: Step 3.1: Obtain the set of all currently known subtasks. Calculate the set of subtasks to be detected ; Step 3.2, in the extra-domain dataset Each out-of-domain sample is extracted sequentially from the set of subtasks to be detected. Extract each subtask sequentially ; Step 3.3, combine the out-of-domain samples and the subtask Input to the multi-task model The multi-task model is obtained. In the subtask The maximum out-of-domain probability value output above ; Step 3.4: Use the correlation calculation formula combined with the maximum probability value outside the domain. Calculate the out-of-domain samples in the subtask correlation on ; Step 3.5, calculate the correlation. Subtasks to obtain the maximum value And the maximum correlation of the out-of-domain samples across all the subtasks to be detected. And compare them; Step 3.6: Generate corresponding out-of-domain subtask labels for the out-of-domain samples. And mark the extra-domain subtasks. Subtask tag set for adding the extra-domain data Output later.

3. The continuous learning image classification method for open environments according to claim 2, characterized in that, The labeled dataset is obtained by merging in step 4. The steps are as follows: Step 4.1, for the newly added dataset Image samples in the data subtask tag set within the domain Find the corresponding intra-domain subtask marker Forming intradomain triples ; Step 4.2, in the extra-domain data subtask tag set Find the corresponding extra-domain subtask marker Forming extraterritorial triples ; Step 4.3: Add the intra-domain and extra-domain triples to the labeled dataset. And output it.

4. The continuous learning image classification method for open environments according to claim 3, characterized in that, In step 5, for the training dataset Input the multi-task model The steps for incremental training are as follows: Step 5.1, for the multi-task model Add a task and initialize its model structure; Step 5.2, update the set of all currently known subtasks. Update the number of all current subtasks. ; Step 5.3, using the cross-entropy loss function extraterritorial loss function and - Regularization loss function Construct the total loss function ; Step 5.4, using the total loss function Calculate the gradient and update the model until convergence.

5. The continuous learning image classification method for open environments according to claim 4, characterized in that, In step 6, the training dataset A representative dataset is generated by saving a portion of the data using a playback strategy. and the representative dataset Update the historical dataset The specific steps are as follows: Step 6.1, Obtain the training dataset. Total number of image categories and the historical dataset Size ; Step 6.2, from the training dataset Random selection Save each sample of that category, and combine all the saved samples into a representative dataset. ; Step 6.3, update the historical dataset. For the representative dataset .

6. The continuous learning image classification method for open environments according to claim 5, characterized in that: In step 7, the image to be classified is acquired and a sub-task label set is generated for it; the multi-task model The image classification result can be output based on the image to be classified and the subtask label set. The specific steps are as follows: Step 7.1: Generate subtask labels for the acquired image to be classified. ; Step 7.2: Label the image to be classified and the subtask. Input the multi-task model To obtain the multi-task model In subtask probability distribution of the output ; Step 7.3: Output the probability distribution. The image category with the highest probability.

7. The continuous learning image classification method for open environments according to claim 4, characterized in that: The out-of-domain loss function described in step 5.3 The definition is, in, This represents the number of all current subtasks. For indicator functions, For hyperparameters, For training set All subtasks are marked as A collection of image samples For sample set The total number of samples in the middle. For sample set The first in One sample in the multi-task model sub-tasks The probability distribution of the output, The probability distribution The entropy.

8. The continuous learning image classification method for open environments according to claim 4, characterized in that: The cross-entropy loss function described in step 5.3 The definition is, in, For the training dataset The total number of samples, The triples composed of the image samples in the multi-task model Corresponding category to the output on the subtask tag The probability of.

Images