Neural collapse theory-based pre-training model class incremental learning identification method

Abstract

According to the pre-training model class incremental learning recognition method based on the neural collapse theory, the NC theory is introduced into feature adaptation of the pre-training model, and a novel normal form is provided to solve downstream tasks through the pre-training model. And an ETF classifier is constructed, so that the feature representation can be guided to adapt to the optimal structure. The ETF classifier not only meets the NC2 characteristics in the neural collapse, but also can improve the instability in the training process by fixing the optimal structure in advance. An FCM module is introduced to compress various features to a specified class prototype, intra-class variability is reduced, NC1 features of neural collapse are met, and feature representation is aligned with an optimal classifier structure by supervising a learning process, so that feature representation with higher discrimination is obtained. Incremental learning training is carried out on the pre-training model, and then the trained model is used for executing the target recognition task, so that the recognition effect can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of data classification and recognition, and specifically relates to a recognition method of incremental learning of a pre-trained model based on neural breakdown theory. Background Art

[0002] Class-Incremental Learning (CIL) is a learning model in which an artificial intelligence system continuously learns new tasks without forgetting previously learned tasks. These tasks have non-overlapping sets of categories. In CIL, the model needs to gradually learn new categories and maintain the ability to recognize old categories while learning new categories. However, due to the update of model parameters, the feature representations between different incremental tasks will shift, resulting in a decrease in performance on previous tasks. This phenomenon is called catastrophic forgetting.

[0003] To address the challenges in CIL, researchers have introduced pre-trained models to leverage their universal feature representations trained on large-scale datasets. However, the universal feature representations of pre-trained models may not be adaptive and discriminative for various incremental categories, as these incremental categories may be beyond the distribution range of the pre-training dataset. In addition, since incremental categories cannot be learned at once, the relationship between categories cannot be optimally constructed, resulting in a lack of discriminative feature representations for downstream tasks.

[0004] The Neural Collapse theory explains an elegant phenomenon between the last layer of features and the classifier in deep learning classification models, which can be summarized into the following three characteristics:

[0005] (1) Variability Collapse (NC1): The variability of intra-class features collapses to zero. (2) Convergence to Simplex ETF (NC2): The class mean features converge to the simplex equiangular tight frame (ETF), achieving equal length, equal angles, and maximum distance in the feature space. (3) Self-Duality (NC3): The linear classifier converges to the class mean features until global scaling.

[0006] Existing pre-trained model based class incremental learning methods (PTM-CIL) can be mainly divided into three categories: prompt-based methods, representation-based methods and model hybrid-based methods.

[0007] (1) Prompt-based methods. These methods use prompt adjustment strategies to slightly update the feature representation of PTM, but the adaptability of feature representation is limited due to the limited learnable parameters. For example, Wang et al. adopted a prompt adjustment strategy in their study, but this method is limited by limited parameters and cannot effectively adapt to incremental categories. (2) Representation-based methods. These methods try to use the general representation of PTM to construct classifiers, but they perform poorly when faced with downstream incremental datasets that are inconsistent with the distribution of pre-training datasets. For example, Zhou et al. used adapters to change the general representation, but the continuous update of the adapter still faces the problem of catastrophic forgetting. (3) Model-mixture methods. These methods design a set of models during the learning process and use model merging and model integration strategies to represent all tasks, but increase the memory cost of model parameters. For example, Wang et al. designed multiple models to merge different tasks, but this method brings additional computational burden.

[0008] It can be seen that the prior art has the following defects:

[0009] (1) Domain gap problem: In the existing technology, there is a significant domain gap between the pre-training dataset and the downstream task dataset, which leads to limited performance of the pre-trained model when directly applied to downstream tasks. The domain gap is mainly reflected in the distribution, category, and features of the data, which affects the generalization ability of the model.

[0010] (2) Feature redundancy and inefficiency: Existing pre-trained models often contain a large number of redundant features, which may not be completely useful in downstream tasks and may even have a negative impact on performance. In addition, the feature representation of the model may not be compact and efficient enough, resulting in a waste of computing resources.

[0011] (3) Poor adaptability between classifier and feature representation: Traditional pre-training methods usually optimize classifier and feature representation independently, lacking effective alignment and adaptive adjustment between the two. This makes it difficult for the classifier to adapt to the changing feature space in category incremental learning, thus affecting the overall performance.

[0012] (4) Poor model scalability: Some existing incremental learning methods have deficiencies in model scalability and are difficult to adapt to a substantial increase in the number of categories, which limits the effectiveness of the model in processing large-scale data.

[0013] In summary, the above defects of the prior art lead to poor classification effect of the model for classification tasks. Summary of the invention

[0014] In order to solve the above problems existing in the prior art, the present application provides a recognition method of incremental learning of pre-trained models based on neural breakdown theory. The technical problem to be solved by the present application is achieved through the following technical solutions:

[0015] A recognition method of incremental learning of a pre-trained model based on neural breakdown theory includes:

[0016] S100, obtaining a task data set and a pre-trained model that has been constructed, wherein the task data set includes multiple tasks and their corresponding data; each task corresponds to multiple samples with category labels; the pre-trained model includes: a pre-trained backbone model, a task-related adapter, an FCM module, and an ETF classifier;

[0017] S200, in the training adaptation stage, the pre-training model is trained using all samples corresponding to the first task in the task data set to obtain an adapted training model;

[0018] S300, in the incremental learning stage, freezing the parameters of the modules other than the FCM module in the adaptive training model, using all sample features of all current tasks except the first task combined with the sample features of the first task, performing incremental learning on the adaptive training model to obtain a recognition model;

[0019] S400, using a recognition model to recognize an image corresponding to a target recognition task, and obtaining a category to which the target belongs.

[0020] Beneficial effects:

[0021] The present application provides a recognition method for incremental learning of pre-trained models based on the theory of neural breakdown, introduces the theory of NC into the feature adaptation of pre-trained models, and provides a novel paradigm to use pre-trained models to solve downstream tasks. In addition, the present invention constructs an ETF classifier that can guide the feature representation to adapt to the optimal structure. The ETF classifier not only meets the NC2 characteristics in neural breakdown, but also can solve the instability in the training process by pre-fixing the optimal structure. The FCM module is introduced to compress multiple features onto a specified class prototype, reduce the intra-class variability, and meet the NC1 characteristics of neural breakdown. Through the supervised learning process, the feature representation is aligned with the optimal classifier structure to obtain a more discriminative feature representation. The present application performs incremental learning training on the pre-trained model, and then uses the trained model to perform target recognition tasks to improve the recognition effect.

[0022] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1It is a flowchart of a recognition method of incremental learning of a pre-trained model based on neural breakdown theory provided by the present application;

[0024] Figure 2 It is a schematic diagram of the training process of the pre-trained model provided in this application;

[0025] Figure 3 It is a schematic diagram of the geometric principle of the equiangular tight frame (ETF) provided in this application;

[0026] Figure 4 It is a schematic diagram comparing the method provided by the present application with the prior art method;

[0027] Figure 5 This is the effect diagram of identifying the target in the drone aerial photography or surveillance image provided by this application;

[0028] Figure 6 This is an effect diagram of identifying targets in satellite remote sensing images provided by this application. DETAILED DESCRIPTION

[0029] The present application is further described in detail below in conjunction with specific embodiments, but the implementation methods of the present application are not limited thereto.

[0030] First, the application background of this application is introduced.

[0031] Class-Incremental Learning (CIL) has significant application value in the field of target recognition, especially in the processing of radar images and remote sensing images. For example, in the military field, in military target recognition, especially in dealing with dynamic and complex battlefield environments. With the emergence of new targets (such as drones, stealth fighters, new missiles, etc.), traditional target recognition systems face the challenge of reduced recognition accuracy. Incremental learning can quickly learn the characteristics of new targets without forgetting existing targets, thereby improving the adaptability and real-time response capabilities of the recognition system. In multi-sensor data fusion, electronic warfare interference, and cross-modal target recognition, incremental learning helps the system to continuously optimize and ensure efficient recognition.

[0032] like Figure 1 As shown, the present application provides a recognition method of incremental learning of a pre-trained model based on neural breakdown theory, including:

[0033] S100, obtaining a task data set and a pre-trained model, wherein the task data set includes multiple tasks and their corresponding data; each task corresponds to multiple samples carrying category labels;

[0034] refer to Figure 2As shown, the pre-trained model of the present application includes: a pre-trained backbone model, a task-related adapter, an FCM module and an ETF classifier. The pre-trained backbone network is the basic part of the model, which is usually pre-trained on a large data set and is used to extract preliminary features of the input data. The task-related adapter is used to bridge the domain gap between the pre-trained data set and the downstream data set, so that the model can better adapt to different downstream tasks. The feature compression module is designed to compress feature representations, reduce storage and computational costs, and improve the performance of the model. ETF stands for Isometric Tight Frame, which is a special matrix structure used to build an optimal classifier.

[0035] The geometry of the equiangular tight frame (ETF) is shown in Figure 3 As shown, ETFs The space consists of K vectors that satisfy a specific geometric relationship, that is, the angles between them are equal and optimal. This geometric structure enables the ETF classifier to construct the optimal classification boundary. The figure shows how four vectors (representing four categories) form a regular simplex, which is equivalent to the ETF in the neural collapse phenomenon. The ETF classifier can use the neural collapse phenomenon to guide the transfer of feature representations and construct the optimal classifier structure. In all incremental learning processes, the pre-trained backbone is frozen to maintain the generalization of feature representations.

[0036] This application involves incremental learning, which is a learning mode in which the model continuously learns new tasks on new categories. The task dataset contains T tasks, and the task dataset is represented by {D1, D2, ..., D T},in is the sample set of the t-th task, containing nt samples, xi represents the ith sample, yi represents the category label of the ith sample, and the category sets of different tasks do not overlap. The category set is represented by C = {Ci,…,C T}, the total number of all visible categories is Kt. The goal of incremental learning is to learn a unified classifier that can accurately classify all visible categories. Consistent with the traditional PTM-CIL method, this application defines a pre-trained backbone model to be used as the initialization of the feature extractor f(·) and the linear classifier h(·).

[0037] Neural collapse refers to a phenomenon on balanced data, which reveals that there is a simplified geometric structure between the features of the last layer of the model and the classifier. This phenomenon is manifested as the close clustering of features of the same category, resulting in simplified classifier weights. Figure 3 As shown in the figure, a simple equiangular tight frame (ETF) refers to A matrix consisting of K vectors in space satisfies the following properties:

[0038]

[0039] in, is a matrix that allows rotation and satisfies UTU=I K , I K is the identity matrix, 1 K is a vector of all 1s.

[0040] All column vectors in E have the same l2 norm, and the inner product between any pair of vectors is

[0041]

[0042] Among them, for Its value is 1 when k1=k2, otherwise it is 0.

[0043] Since neural collapse describes the optimal geometric structure between the last layer of features in the model and the classifier, this application proposes to fix a learnable classifier to the structure constructed by neural collapse to pre-allocate this optimality. Next, this application adopts an ETF classifier, constructs an ETF when initializing the classifier and fixes it during training. This application hopes that ETF can guide the transfer of general feature representation to the optimal classifier structure.

[0044] By observing neural collapse, the optimal classifier structure can be constructed to guide the migration of feature representations; this application decides to use the ETF classifier to solve the PTM-CIL problem. Therefore, this application first introduces a task-related adapter to bridge the domain gap between the pre-training dataset and the downstream task. Then, the feature compression module is introduced to compress the features into the specified category prototype, reduce the variability within the category, and align them with a specific simple equiangular tight frame. Finally, the optimal structure alignment method is proposed to supervise the feature compression process. The overall method framework is as follows Figure 2 shown.

[0045] Due to the domain gap between the pre-training dataset and the dataset of the downstream task, the general feature representation of the pre-training model may not be discriminative for the downstream incremental tasks. Therefore, this application introduces a lightweight adapter adjustment in the training adaptation stage to bridge the domain gap.

[0046] The pre-trained backbone model includes L Transformer blocks, each of which includes a self-attention module and a multi-layer perceptron MLP layer; in order to maintain the generalization ability of the pre-trained model, an adapter module is learned for each MLP layer as a bypass branch. That is, the task-related adapter includes L adaptive adapter modules; each multi-layer perceptron layer is provided with an adaptive adapter module as a bypass branch, which is used to modulate the downstream task information into the generalized feature representation in the Transformer block; the adaptive adapter module adapter is a bottleneck module, including a down-projection layer A nonlinear activation function σ and an up-projection layer The adaptive adapter module adjusts the output of the adaptive adapter module to:

[0047] m o =σ(m i W down )W up +MLP(m i );

[0048] Among them, mi and mo are the input and output of the MLP layer respectively, the set of all L adaptive adapter modules is A, and the adaptation embedding function with the set A is f(x; A), Indicates the dimension of the lower projection layer.

[0049] Freeze the pre-trained weights during task-related adaptation and only optimize the adapter. In other learning steps, both the pre-trained model weights and the adapter are frozen to ensure the general feature representation is capable of representing unseen categories.

[0050] Since the ETF classifier is introduced as a classifier to guide the optimization of feature representation, this application needs to transfer the feature representation to a specific category prototype to follow the characteristics of neural collapse. Therefore, this application introduces a feature compression module (FCM) to compress various features. After the feature extractor, a feature compression module P(·) is added to convert the general feature representation into features that adapt to the pre-assigned ETF category prototype. The feature P(·) consists of two layers of MLP and a batch normalization (BatchNorm) layer. In the incremental learning step, the feature compression module will be continuously fine-tuned to continuously adapt to the pre-assigned category prototype, and the adjustment process is expressed as:

[0051]

[0052] Where zi is the input feature of the FCM module, It is the feature output by the FCM module.

[0053] In order to alleviate the overfitting of the ETF classifier, this application introduces a mixup strategy to increase feature diversity, which can be expressed as:

[0054]

[0055] Where λ is a random hyperparameter that balances samples xi and xj, Represents the diversity characteristics after mixing, Represents the mixed sample labels.

[0056] Then the cross entropy is used to optimize the learnable parameter P(·), which is recorded as:

[0057]

[0058] S200, in the training adaptation stage, the pre-training model is trained using all samples corresponding to the first task in the task data set to obtain an adapted training model;

[0059] As an optional implementation of the present application, S200 includes:

[0060] S210, inputting all samples corresponding to the first task in the task data set into the pre-trained backbone model, so that the pre-trained backbone model outputs a coarse-grained feature representation, and using the task-related adapter to adjust the domain gap of the coarse-grained feature to obtain a first feature representation of all samples of the first task;

[0061] S220, inputting the first feature representation into the FCM module to convert it into a second feature representation adapted to the pre-assigned ETF category prototype; and inputting the second feature representation into the ETF classifier to obtain the categories of all tasks corresponding to the first task;

[0062] S230, calculating a target loss function using the categories of all samples corresponding to the first task, and adjusting the parameters of the task-related adapter and the FCM module in the pre-trained model using the target loss function;

[0063] S240, replacing the task-related adapter and FCM module in the pre-trained model with the task-related adapter and FCM module after adjusting the parameters, and repeating the process from S110 to S120 once to obtain a trained pre-trained model.

[0064] This implementation trains the task-related adapter and FCM integrated into the pre-trained model and optimizes their parameters using the target loss function L. The purpose of this step is to enable the pre-trained model to generate appropriate feature representations for the first task (D1).

[0065] In a specific implementation of the present application, after S140, the recognition method of incremental learning of the pre-trained model based on neural breakdown theory further includes:

[0066] a, storing the second feature representation output by the FCM module in the trained pre-trained model into the first memory;

[0067] b. Store the mean and covariance of the categories of all samples corresponding to the first task in the second memory.

[0068] Since the pre-trained model and the adapter are frozen in the incremental step, this application needs to supervise the feature compression process to ensure that the feature migration meets the characteristics of neural collapse. In addition, this application also hopes to capture the category relationship between different task categories. Therefore, this application designs two feature playback strategies to introduce the features of the previous task in the current training step.

[0069] Full Memory (FM): Build memory M FM To store the features z of the samples.

[0070] Class Memory (CM): Build memory M CM To collect the mean μ of all sample categories.

[0071] The complete storage of this application is stored in the first memory, and the category storage is stored in the second memory.

[0072] Since the pre-trained model provides well-distributed and generalizable representations, each category tends to exhibit a unimodal distribution and can be naturally modeled as a Gaussian distribution. This application uses a reparameterization strategy to replay the features of previous tasks. By generating features from category preservation, this application can obtain rich simulation features for each category to achieve optimal structural alignment, where the number of replays ρ is set to 256 in all experiments.

[0073] Thanks to two feature replay strategies, this application can redistribute the weights of the ETF classifier according to the number of seen categories to maintain the optimal classifier structure. In order to compress various features, this application needs to align the features with the specified corresponding category weights ei. Due to the large capacity gap between the randomly reallocated category weights and the feature representation, optimal classifier alignment becomes challenging. The batch normalization layer (Batchnorm layer) of the feature compression module can minimize collapse (Tian, ​​Krishnan, and Isola 2019), making fewer singular values ​​approach zero to meet characteristic.

[0074] Therefore, this application explores the use of soft maximum loss to achieve optimal structural alignment. In this way, the loss can be adjusted to compensate for the poor alignment characteristics that may occur when the capacity gap is large. For simplicity, this application uses a simple LogSum function, expressed as:

[0075]

[0076] Where α is the smoothing factor, ek is the weight of the corresponding category, and Wp is the weight of the feature compression module P. Experiments show that the soft maximum loss has better alignment performance than the mean square error (MSE) function.

[0077] Therefore, the objective loss function is expressed as:

[0078]

[0079] Where δ is set to 10, τ is the temperature coefficient, α is the smoothing factor, ek is the weight of the corresponding category, and Wp is the weight of the feature compression module.

[0080] This implementation constructs or updates the full preservation (FM) or category preservation (CM) matrices based on the feature representation generated by the previous FCM. These matrices are used to store the feature information of the current task. They rely on the selection and integration of the various basic components in the pre-trained model. At the same time, the stored feature information will directly affect the adaptation effect of subsequent tasks and the performance of the overall model.

[0081] S300, in the incremental learning stage, freezing the parameters of the modules other than the FCM module in the adaptive training model, using all sample features of all current tasks except the first task combined with the sample features of the first task, performing incremental learning on the adaptive training model to obtain a recognition model;

[0082] In a specific implementation of the present application, S300 includes:

[0083] S310, freezing the parameters of the pre-trained backbone model, the task-related adapter, and the ETF classifier in the trained pre-trained model, and setting the parameters of the FCM module to change with the task adaptability, to obtain an incremental model;

[0084] S320, for any current task except the first task in the task data set, all samples of the current task are input into the incremental model to output a first feature representation through the pre-trained backbone model, extract a second feature representation from the first memory, and send it to the FCM module together with the first feature representation of the current task to obtain a second feature representation of the task, and store it in the first memory;

[0085] When processing a new task (Dt, t>1), first replay or integrate the feature information of the previous task extracted in step 2 (through FM / CM). This ensures that the model can utilize previously learned knowledge and promotes knowledge transfer and cumulative learning.

[0086] When processing a new task (Dt, t>1), this application freezes the parameters of the backbone model, task-related adapters, and ETF classifiers, and only further updates the parameters of the FCM optimized in step 2 to adapt to the characteristics of the new task. This allows the model to quickly adapt to new tasks while maintaining the memory of old tasks.

[0087] S330, inputting the second feature representation of the current task in S320 into the ETF classifier to obtain the categories of all samples corresponding to the current task, and storing the mean and covariance of the categories in the second memory;

[0088] S340, calculating a target loss function using the categories of all samples corresponding to the current task in S330, and adjusting the parameters of the task-related adapter and the FCM module in the pre-trained model using the target loss function;

[0089] S350, replacing the task-related adapter and FCM module in the pre-trained model with the task-related adapter and FCM module after adjusting parameters, and repeating the process from S320 to S340 until all tasks in the task data set are completed, thereby obtaining a recognition model, a first memory storing the second feature representation of all tasks, and a second memory of the category mean and covariance of the samples.

[0090] Repeat the process from S320 to S340 until all tasks (D1 to D T ) are processed. In each iteration, the model is constantly learning new task features while keeping the memory of old task knowledge. This step is the core of continuous learning and optimization of the model. Through continuous iteration on the entire dataset sequence, the model not only learns the specific features of each task, but also gradually develops a universal representation ability across tasks. This ability enables the model to achieve faster adaptation and higher performance on new tasks. Overall optimization is the key to improving model performance, which relies on the synergy and continuous optimization of all previous steps.

[0091] This application constructs an equiangular tight frame (ETF) classifier, which is an optimal classifier structure that can guide feature representation to adapt to the optimal structure. The ETF classifier not only satisfies the NC2 property in neural collapse (class mean features converge to an equiangular tight frame), but also avoids the instability of the classifier during training by pre-fixing the optimal structure.

[0092] The Feature Compression Module (FCM) is introduced to compress multiple features onto a specified class prototype, reducing intra-class variability and satisfying the NC1 property of neural collapse (intra-class feature variability tends to zero). The Feature Compression Module aligns the feature representation with the optimal classifier structure through a supervised learning process, thereby obtaining a more discriminative feature representation.

[0093] refer to Figure 4 As shown, Figure 4 This is a simplified comparison diagram of the prior art method and the method of the present application. Since the categories of different steps cannot be learned simultaneously and the objective function may not be appropriate, the classifier structure is not optimal for all categories seen. That is, the geometric structure displayed will gradually present a form with dense features. This method uses the neural collapse theory to allocate the optimal classifier structure before model optimization to guide the adaptation of feature representation. This allows the model to maintain the characteristics of balanced feature distribution after the introduction of new classes.

[0094] S400, using a recognition model to recognize an image corresponding to a target recognition task, and obtaining a category to which the target belongs.

[0095] For example, in radar images, with the emergence of new targets (such as drones, new ships, aircraft, etc.), the target recognition system needs to continuously update its recognition model without forgetting known targets. For example, an air monitoring system can use the method of this application to incrementally learn the characteristics of new aircraft or drones, while a maritime monitoring system needs to cope with the emergence of new ships or vessels. Incremental learning can ensure that the system maintains the ability to recognize traditional targets while recognizing new targets, thereby effectively coping with complex and dynamic environments.

[0096] In remote sensing images, target categories are also constantly changing, especially in areas such as agricultural monitoring, urban planning, and disaster response. With the emergence of new crop types, building forms, or natural disaster types, remote sensing systems need to incrementally learn these new categories. For example, using the method of this application in agricultural monitoring can help the remote sensing system gradually adapt to emerging crop types, or in urban expansion monitoring, the method of this application can timely identify newly built buildings and roads. In post-disaster response, incremental learning can help the system identify disaster-stricken areas and new changes in a timely manner during monitoring, and maintain effective recognition capabilities for historical data.

[0097] In addition, in applications such as unmanned driving, drones, and environmental monitoring, class incremental learning also plays an important role. Autonomous driving systems need to deal with ever-changing traffic environments. The method of this application helps the system adapt to new types of vehicles, pedestrians, and traffic signs while maintaining the ability to detect existing targets. When drones perform tasks such as agricultural monitoring and wildlife protection, they also need to identify newly emerging species or environmental changes through incremental learning.

[0098] refer to Figure 5 , this application can be applied to target recognition in drone aerial photography or surveillance images, refer to Figure 6 , this application can also be applied to target recognition in satellite remote sensing images. Figure 5 and Figure 6 It can be seen that the present application can locate the target in the image and classify it accurately.

[0099] Most methods in the prior art, whether they are hint-based methods, representation methods or model hybrid methods, do not explicitly use the theory of neural collapse to guide feature representation and classifier optimization. This application introduces the neural collapse theory into feature adaptation of pre-trained models (PTMs) for the first time, providing a novel paradigm to use pre-trained models to solve downstream tasks.

[0100] It is worth noting that the terms "first" and "second" in this application are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0101] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components or steps.

[0102] The above content is a further detailed description of the present application in combination with specific preferred implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, which should be deemed to fall within the scope of protection of the present application.

Claims

1. A recognition method for incremental learning of pre-trained models based on neural breakdown theory, characterized in that: include: S100, obtaining a task data set and a pre-trained model, wherein the task data set includes a plurality of tasks and their corresponding data; Each task corresponds to multiple samples with category labels; The pre-trained model includes: a pre-trained backbone model, a task-related adapter, an FCM module and an ETF classifier; S200, in the training adaptation stage, the pre-training model is trained using all samples corresponding to the first task in the task data set to obtain an adapted training model; S300, in the incremental learning stage, freezing the parameters of the modules other than the FCM module in the adaptive training model, using all sample features of all current tasks except the first task combined with the sample features of the first task, performing incremental learning on the adaptive training model to obtain a recognition model; S400, using a recognition model to recognize an image corresponding to a target recognition task, and obtaining a category to which the target belongs.

2. The recognition method of incremental learning of pre-trained model based on neural breakdown theory according to claim 1 is characterized in that: The task data set includes T tasks, and the task data set is represented as {D1, D2, ..., D T },in is the sample set of the t-th task, containing nt samples, xi represents the ith sample, yi represents the category label of the ith sample, and the category sets of different tasks do not overlap. The category set is represented by C = {Ci,…,C T }, the total number of all visible categories is Kt.

3. The recognition method of incremental learning of pre-trained model based on neural breakdown theory according to claim 2 is characterized in that: S200 includes: S210, inputting all samples corresponding to the first task in the task data set into the pre-trained backbone model, so that the pre-trained backbone model outputs a coarse-grained feature representation, and using the task-related adapter to adjust the domain gap of the coarse-grained feature to obtain a first feature representation of all samples of the first task; S220, inputting the first feature representation into the FCM module to convert it into a second feature representation adapted to the pre-assigned ETF category prototype; and inputting the second feature representation into the ETF classifier to obtain the categories of all tasks corresponding to the first task; S230, calculating a target loss function using the categories of all samples corresponding to the first task, and adjusting the parameters of the task-related adapter and the FCM module in the pre-trained model using the target loss function; S240, replacing the task-related adapter and FCM module in the pre-trained model with the task-related adapter and FCM module after adjusting parameters, and repeating the process from S110 to S120 once to obtain a trained pre-trained model.

4. The recognition method of incremental learning of pre-trained model based on neural breakdown theory according to claim 3 is characterized in that: After S240, the recognition method of incremental learning of the pre-trained model based on neural breakdown theory further includes: The second feature representation output by the FCM module in the trained pre-trained model is stored in the first memory; The mean and covariance of the categories of all samples corresponding to the first task are stored in the second memory.

5. The recognition method of incremental learning of pre-trained model based on neural breakdown theory according to claim 4 is characterized in that: S300 includes: S310, freezing the parameters of the pre-trained backbone model, the task-related adapter, and the ETF classifier in the trained pre-trained model, and setting the parameters of the FCM module to change with the task adaptability, to obtain an incremental model; S320, for any current task except the first task in the task data set, all samples of the current task are input into the incremental model to output a first feature representation through the pre-trained backbone model, extract a second feature representation from the first memory, and send it to the FCM module together with the first feature representation of the current task to obtain a second feature representation of the task, and store it in the first memory; S330, inputting the second feature representation of the current task in S320 into the ETF classifier to obtain the categories of all samples corresponding to the current task, and storing the mean and covariance of the categories in the second memory; S340, calculating a target loss function using the categories of all samples corresponding to the current task in S330, and adjusting the parameters of the task-related adapter and the FCM module in the pre-trained model using the target loss function; S350, replacing the task-related adapter and FCM module in the pre-trained model with the task-related adapter and FCM module after adjusting parameters, and repeating the process from S320 to S340 until all tasks in the task data set are completed, thereby obtaining a recognition model, a first memory storing the second feature representation of all tasks, and a second memory of the category mean and covariance of the samples.

6. The recognition method of incremental learning of pre-trained models based on neural breakdown theory according to any one of claims 1 to 5, characterized in that: The pre-trained backbone model includes L Transformer blocks, each of which includes a self-attention module and a multi-layer perceptron MLP layer; the task-related adapter includes L adaptive adapter modules; each multi-layer perceptron layer is provided with an adaptive adapter module as a side branch, and the side branch is used to modulate downstream task information into a generalized feature representation in the Transformer block; the adaptive adapter module includes a down-projection layer A nonlinear activation function σ and an up-projection layer The adaptive adapter module adjusts the output of the adaptive adapter module to: m o =σ(m i W down )W up +MLP(m i ); Among them, mi and mo are the input and output of the MLP layer respectively, the set of all L adaptive adapter modules is A, and the adaptation embedding function with the set A is f(x; A), Indicates the dimension of the lower projection layer.

7. The recognition method of incremental learning of pre-trained models based on neural breakdown theory according to any one of claims 1 to 5, characterized in that: The feature compression module includes two layers of MLP and batch normalization layer; the feature compression module will be continuously fine-tuned to continuously adapt to the pre-assigned category prototypes, and the adjustment process is expressed as: zi=f(xi); Where zi is the input feature of the FCM module, It is the feature output by the FCM module.

8. The recognition method of incremental learning of pre-trained models based on neural breakdown theory according to any one of claims 1 to 5, characterized in that: The ETF classifier introduces a hybrid strategy in the classification process to increase feature diversity, which is expressed as: Where λ is a random hyperparameter that balances samples xi and xj, Represents the diversity characteristics after mixing, Represents the mixed sample labels.

9. The recognition method of incremental learning of pre-trained models based on neural breakdown theory according to any one of claims 3 or 5, characterized in that: The objective loss function is expressed as: Where δ is set to 10, τ is the temperature coefficient, α is the smoothing factor, ek is the weight of the corresponding category, and Wp is the weight of the feature compression module.

Images