A continuous evolutionary learning method based on consistent hybrid expert hint generator

Abstract

The present invention discloses a continuous evolutionary learning method based on a consistent hybrid expert prompt generator, which relates to the field of pattern recognition technology. By introducing an orthogonal projection mechanism, it is ensured that the prompt generator can generate prompts consistent with the corresponding old tasks during the new task training process, thereby effectively preventing catastrophic forgetting and improving the stability and anti-forgetting ability of the model in multi-task learning. In addition, the prompt generator proposed by the present invention can dynamically select experts to adapt to different task requirements and realize efficient knowledge sharing and stability maintenance. The present invention solves the problem that the existing methods have not effectively solved the interference with the old tasks during the new task training process, resulting in the model being prone to serious catastrophic forgetting when switching tasks.

Description

Technical Field

[0001] The invention relates to the technical field of pattern recognition, and in particular to a continuous evolutionary learning method based on a consistent mixed expert prompt generator. Background Art

[0002] Deep learning models often face the challenge of "catastrophic forgetting" when dealing with continuous learning tasks, that is, forgetting the knowledge of previous tasks in the process of learning new tasks. To address this problem, the mixture of experts (MoE) model has demonstrated strong adaptability and scalability in dealing with multi-task learning by virtue of its ability to dynamically select suitable experts. However, existing MoE methods still have limitations in maintaining knowledge consistency between tasks and preventing forgetting. Specifically, existing methods mostly rely on freezing or gradually expanding expert networks, and lack effective mechanisms to ensure the consistency of the prompt generator between new and old tasks, resulting in representation drift and knowledge forgetting when the model switches between different tasks. In addition, although some orthogonal projection techniques can reduce the interference of new tasks on old tasks, they are mostly targeted at simple neural network structures and have not yet been effectively applied to complex mixed expert models.

[0003] In recent years, research in the field of continuous learning has focused on solving the problem of "catastrophic forgetting" when models learn new tasks. Among them, the tuning method based on the prompt generator has attracted much attention, especially in the visual Transformer (ViTs). Existing methods usually retrieve task-related prompts through a prompt pool and maintain model stability by freezing old prompts or incrementally training new prompts. However, it is difficult for such methods to theoretically ensure the consistency of prompt generation, which makes it easy for representation drift and forgetting to occur when switching tasks. Some methods improve stability through slow updates, first task adaptation, or enhanced task differentiation capabilities, but they still cannot completely solve the problems of consistency and catastrophic forgetting between tasks.

[0004] On the other hand, the mixed expert model has been applied to continuous learning by dynamically selecting experts to improve the adaptability and knowledge sharing ability of the model in a multi-task environment. However, existing MoE methods usually rely on expanding or freezing experts and fail to solve the problem of consistency in prompt generation between tasks. To this end, some studies have introduced orthogonal projection techniques to reduce forgetting by projecting gradients to orthogonal directions of old task features, but these methods are mainly used for simple models and are difficult to apply in complex MoE structures.

[0005] According to the above analysis, the existing continuous learning technology based on prompt tuning has not effectively solved the problem of interference with old tasks during new task training, which makes the model prone to severe catastrophic forgetting when switching tasks. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a continuous evolutionary learning method based on a consistent hybrid expert prompt generator, which solves the problem that the existing methods have not effectively solved the interference of old tasks during the training of new tasks, resulting in the model being prone to serious catastrophic forgetting when switching tasks.

[0007] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a continuous evolutionary learning method based on a consistent hybrid expert prompt generator, comprising the following steps:

[0008] S1: correlation matrix for initializing the cue generator;

[0009] S2: Get the training data set for the current task;

[0010] S3: Determine whether the current task is the first task, if yes, proceed to step S4, otherwise proceed to step S7;

[0011] S4: Input the input features of the current task into the prompt generator and ViT model for forward propagation, and calculate the classification loss;

[0012] S5: Based on the classification loss, the gradients of the routing network weights and the candidate expert matrix in the prompt generator are calculated by back propagation to update the corresponding parameters;

[0013] S6: Determine whether the current task has reached the maximum number of iterations. If not, return to step S3; otherwise, proceed to step S7:

[0014] S7: Save the parameters of the prompt generator and update the two non-centered covariance matrices using the training dataset and the trained correlation matrix;

[0015] S8: performing singular value decomposition on the two updated non-centered covariance matrices respectively, and selecting the basis vectors of the corresponding null space based on the decomposition results;

[0016] S9: Update the null space projection matrix according to the basis vectors of the null space, and introduce a relaxation factor for the null space projection matrix;

[0017] S10: Based on the updated null space projection matrix, the gradients of the routing network weights and the candidate expert matrix are projected into the null space of the old task features, and the routing network weights and the candidate expert matrix of the prompt generator are updated using the learning rate;

[0018] S11: Determine whether the current task is the last task. If not, return to step S2. Otherwise, complete the learning of all tasks and output the updated routing network weights and candidate expert matrix of the prompt generator to achieve continuous evolutionary learning based on the consistent hybrid expert prompt generator.

[0019] Furthermore, the correlation matrix of the initialization prompt generator in S1 includes the routing network weight matrix , Candidate Expert Matrix , the first non-centered covariance matrix , the second non-centered covariance matrix , the first null space projection matrix and the second null space projection matrix , the formula is:

[0020] ,

[0021] ,

[0022] ,

[0023] in, is the field of real numbers, is the input feature dimension, is the length of the prompt, is the number of experts, is the identity matrix.

[0024] Furthermore, the training data set of the current task in S2 for:

[0025]

[0026] in, is the training sample, is the category label corresponding to the training sample, For the task, For category.

[0027] Furthermore, the S4 includes the following sub-steps:

[0028] S41: Input the input features to the prompt generator, calculate the gate value through the routing network weight matrix, and select the front The gate value is used as an expert, and the formula is:

[0029]

[0030] in, is the gate value, is the activation function, is the input feature;

[0031]

[0032]

[0033] in, is a function, represents the variable used to mark the state of each gate value, For the A gate value;

[0034] S42: Select the front The gate values ​​are normalized and the normalized results are used together with the candidate expert matrix Perform weighted merging to generate the final prompt , the formula is:

[0035]

[0036]

[0037] in, The gating vector generated for normalization;

[0038] S43: Change the prompt and input features Input to the corresponding layer of the ViT model for forward propagation, the formula is:

[0039]

[0040] in, is the output feature of the corresponding layer of the ViT model, Represents the ViT layer, Tag the image;

[0041] S44: Input features Input into the classifier connected to the ViT model, get the prediction result, and calculate the cross entropy loss , the formula is:

[0042]

[0043] in, is the total number of categories, For Category The real label, For the model category The predicted probability of .

[0044] Furthermore, in S5, the gradient of the routing network weight and the candidate expert matrix in the prompt generator is calculated by back propagation, and the formula is:

[0045]

[0046]

[0047] in, is the gradient of the routing network weight, is the gradient of the candidate expert matrix, is the symbol of partial derivative.

[0048] Furthermore, in S7, the two non-centralized covariance matrices are updated using the training data set and the trained correlation matrix, including the following sub-steps:

[0049] S71: Input feature matrix recorded during the forward propagation process according to the current task Update the first non-centered covariance matrix, the formula is:

[0050]

[0051]

[0052] in, are the input features of different samples, is the updated first non-centered covariance matrix, is the original first non-centered covariance matrix, with the superscript is the transpose of the matrix;

[0053] S72: The gating vector matrix output by the routing network recorded during the forward propagation process according to the current task Update the second non-centered covariance matrix, the formula is:

[0054]

[0055]

[0056] in, is the gating vector of different samples, is the updated second non-centered covariance matrix, is the original second uncentered covariance matrix.

[0057] Furthermore, the S8 includes the following sub-steps:

[0058] S81: Perform singular value decomposition on the two updated non-centralized covariance matrices respectively, the formula is:

[0059]

[0060]

[0061] in, and is the left singular vector matrix and the right singular vector matrix of the first non-centered covariance matrix decomposition, is the first diagonal singular value matrix, and is the left singular vector matrix and the right singular vector matrix of the second non-centered covariance matrix decomposition, is the second diagonal singular value matrix;

[0062] S82: Based on the decomposition result, select the right singular vector corresponding to the minimum singular value as the basis vector of the null space. The formula is:

[0063]

[0064]

[0065] in, is the basis vector of the null space corresponding to the input feature, is the basis vector of the null space corresponding to the gating vector, is the right singular vector whose singular value in the input feature is less than the threshold, is the right singular vector whose singular value in the gated vector is less than the threshold, is the number of singular vectors selected from the input features, is the number of singular vectors selected in the gating vector.

[0066] Furthermore, in S9, the null space projection matrix is ​​updated according to the basis vector of the null space, and a relaxation factor is introduced into the null space projection matrix, and the formula is:

[0067]

[0068]

[0069] in, is the null space projection matrix for updating the routing network weight gradients, is the null space projection matrix for the candidate expert gradient update, is the relaxation factor.

[0070] Furthermore, in S10, the gradient of the routing network weight and the candidate expert matrix is ​​projected into the null space of the old task features, and the formula is:

[0071]

[0072]

[0073] in, is the routing network weight projected into the null space of the old task features, is the candidate expert matrix projected into the null space of the old task features;

[0074] Use the learning rate to update the routing network weights and candidate expert matrix of the prompt generator, the formula is:

[0075]

[0076]

[0077] in, is the updated routing network weight, is the original routing network weight, is the learning rate, is the updated candidate expert matrix, is the original candidate expert matrix.

[0078] The beneficial effect of the present invention is that the present invention accurately projects the updated gradient of the prompt generator into the orthogonal subspace of the old task features through the orthogonal projection technology. This strategy effectively avoids the interference of the new task learning on the old task feature representation, thereby significantly reducing the catastrophic forgetting phenomenon. The prompt generator proposed in the present invention can dynamically select experts to adapt to different task requirements and realize efficient knowledge sharing and stability maintenance. This method ensures that the specific instance representation of the old task remains stable when learning a new task, which theoretically guarantees the stability of the model. In four categories of incremental learning benchmark tests, the method of the present invention achieved a significant improvement in accuracy and a significant reduction in forgetting rate under the class incremental continuous learning setting, proving its effectiveness and superiority in the field of continuous learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 Flowchart of a continuous evolutionary learning method based on a consistent hybrid expert hint generator. DETAILED DESCRIPTION

[0080] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0081] The present invention solves the problem of forgetting old task knowledge caused by new task training in continuous learning by designing a prompt generator based on consistent hybrid experts. The method introduces an orthogonal projection mechanism to ensure that the prompt generator can generate prompts consistent with the corresponding old tasks during the new task training process, thereby effectively preventing catastrophic forgetting and improving the stability and anti-forgetting ability of the model in multi-task learning. In addition, the prompt generator can dynamically select experts to adapt to different task requirements and achieve efficient knowledge sharing and stability maintenance. In order to overcome the shortcomings of the prior art, the present invention provides a continuous evolutionary learning method based on a hybrid expert prompt generator, which updates the parameters of the prompt generator in the null space to ensure the consistency of the prompts during the new task training process, thereby effectively preventing catastrophic forgetting and improving the stability of continuous learning. Specifically, the present invention is implemented on 10 tasks of the public data set CIFAR-100, combined with the Vision Transformer (ViT) model pre-trained on ImageNet-21k, and fine-tuned by a hybrid expert prompt generator. Specifically includes the following steps:

[0082] like Figure 1 As shown, a continuous evolutionary learning method based on a consistent hybrid expert prompt generator includes the following steps:

[0083] S1: correlation matrix for initializing the cue generator;

[0084] S2: Get the training data set for the current task;

[0085] S3: Determine whether the current task is the first task, if yes, proceed to step S4, otherwise proceed to step S7;

[0086] S4: Input the input features of the current task into the prompt generator and ViT model for forward propagation, and calculate the classification loss;

[0087] S5: Based on the classification loss, the gradients of the routing network weights and the candidate expert matrix in the prompt generator are calculated by back propagation to update the corresponding parameters;

[0088] S6: Determine whether the current task has reached the maximum number of iterations. If not, return to step S3; otherwise, proceed to step S7:

[0089] S7: Save the parameters of the prompt generator and update the two non-centered covariance matrices using the training dataset and the trained correlation matrix;

[0090] S8: performing singular value decomposition on the two updated non-centered covariance matrices respectively, and selecting the basis vectors of the corresponding null space based on the decomposition results;

[0091] S9: Update the null space projection matrix according to the basis vectors of the null space, and introduce a relaxation factor for the null space projection matrix;

[0092] S10: Based on the updated null space projection matrix, the gradients of the routing network weights and the candidate expert matrix are projected into the null space of the old task features, and the routing network weights and the candidate expert matrix of the prompt generator are updated using the learning rate;

[0093] S11: Determine whether the current task is the last task. If not, return to step S2. Otherwise, complete the learning of all tasks and output the updated routing network weights and candidate expert matrix of the prompt generator to achieve continuous evolutionary learning based on the consistent hybrid expert prompt generator.

[0094] The S1 initializes the relevant matrix of the prompt generator, including the routing network weight matrix , Candidate Expert Matrix , the first non-centered covariance matrix , the second non-centered covariance matrix , the first null space projection matrix and the second null space projection matrix , the formula is:

[0095] ,

[0096] ,

[0097] ,

[0098] in, is the real number domain (real number space), indicating that the elements of the matrix belong to the real number set, is the input feature dimension, is the length of the prompt, is the number of experts, is the identity matrix.

[0099] The router network weight matrix is ​​randomly initialized. The router is responsible for generating gating values ​​based on the input features and selecting the most appropriate expert to generate prompts. During initialization, all weights should satisfy zero mean and small variance to facilitate subsequent gradient descent optimization.

[0100] The candidate expert matrix is ​​randomly initialized, and each expert will have a set of unique prompts that will be fine-tuned and updated during task learning. Each row of the matrix represents a prompt generated by an expert, which can be a standard normal distribution in the initial state.

[0101] Initialize two uncentered covariance matrices to capture the distribution changes of input features and gating vectors between tasks. Specifically, Used to reflect the covariance of input features, Used to record the covariance of the gate vector. These two matrices will be used in the subsequent null space projection.

[0102] Initialize two null space projection matrices. The projection matrix is ​​used to project the gradient into the null space of the old task during task training to avoid interference with the learned features of the old task. These two projection matrices will be adjusted and optimized in the subsequent gradient update step.

[0103] Get the current task from the CIFAR-100 dataset The training dataset .

[0104] The training dataset of the current task in S2 for:

[0105]

[0106] in, is the training sample, is the category label corresponding to the training sample, For the task, For category.

[0107] The CIFAR-100 dataset is divided into 10 tasks, each task contains different categories. In each task, the input features are fed into the prompt generator and the ViT model for processing.

[0108] The S4 comprises the following sub-steps:

[0109] S41: Input the input features to the prompt generator, calculate the gate value through the routing network weight matrix, and select the front The gate value is used as an expert, and the formula is:

[0110]

[0111] in, is the gate value, is the activation function, is the input feature;

[0112] is a length of A vector indicating the current input feature The probability of selection among experts is calculated by calculating the gate value for each input feature. , we can select the expert that is most suitable for the current task. In order to further reduce the computational overhead and highlight the important experts, we select The maximum gate value;

[0113]

[0114]

[0115] in, is a function used to convert the gate value The elements in the are divided into two categories: the selected gate values ​​and other gate values ​​that are not selected, represents the variable used to mark the state of each gate value, For the A gate value;

[0116] function Used to The large gate value is set to negative infinity so that it is zero after the soft maximization operation. This operation ensures that only The most important experts were activated;

[0117] S42: Select the front The gate values ​​are normalized and the normalized results are used together with the candidate expert matrix Perform weighted merging to generate the final prompt , the formula is:

[0118]

[0119]

[0120] in, The gating vector generated for normalization;

[0121] S43: Change the prompt and input features Input to the corresponding layer of the ViT model for forward propagation, the formula is:

[0122]

[0123] in, is the output feature of the corresponding layer of the ViT model, Represents the ViT layer, Tag the image;

[0124] The ViT model consists of multiple layers, each of which applies prompts to adjust the feature extraction process. For each layer, the input includes the image tag , Category Tags and prompt , they jointly participate in the forward propagation, and the final category label will be sent to the classifier for final classification;

[0125] S44: Input features Input to the classifier connected to the ViT model. The classifier is connected after the ViT model to classify the output features of the ViT model, obtain the prediction results, and calculate the cross entropy loss , the formula is:

[0126]

[0127] in, is the total number of categories, For Category The real label, For the model category The predicted probability of .

[0128] Cross entropy loss is the main indicator to measure the difference between the model output and the true label. This loss will be used for back propagation to guide the model to update parameters.

[0129] After the forward pass is completed and the loss is calculated, the gradients of the prompt generator are calculated through backpropagation.

[0130] In S5, the gradient of the routing network weight and the candidate expert matrix in the prompt generator is calculated by back propagation, and the formula is:

[0131]

[0132]

[0133] in, is the gradient of the routing network weight, is the gradient of the candidate expert matrix, is the symbol of partial derivative.

[0134] S6 determines whether the maximum number of iterations has been reached. Specifically, during the training process of each task, the present invention sets a maximum number of iterations. To ensure that the model can stop and save parameters after sufficient training. Determine whether the preset number of iterations has been reached If the number of iterations does not reach , then continue the cycle of S4 and S5.

[0135] Maximum iteration stop condition: If the current task The number of iterations reaches , then stop the training of the current task and save the prompt generator parameters of the current task, including the router weights and the candidate expert matrix At this point, the model has sufficiently fine-tuned these parameters through many iterations, and has maintained stability on old tasks via null space projections at each update.

[0136] Once the maximum number of iterations is reached, save the trained router weight matrix and the candidate expert matrix , which will be used in future tasks. The saved parameters include the router weights and prompts fine-tuned during training to ensure that the prompt generator can generate stable task-related prompts.

[0137] In S7, the two non-centralized covariance matrices are updated using the training data set and the trained correlation matrix, including the following steps:

[0138] S71: Input feature matrix recorded during the forward propagation process according to the current task Update the first non-centered covariance matrix, the formula is:

[0139]

[0140]

[0141] in, are the input features of different samples, is the updated first non-centered covariance matrix, is the original first non-centered covariance matrix, with the superscript is the transpose of the matrix;

[0142] S72: The gating vector matrix output by the routing network recorded during the forward propagation process according to the current task Update the second non-centered covariance matrix, the formula is:

[0143]

[0144]

[0145] in, is the gating vector of different samples, is the updated second non-centered covariance matrix, is the original second uncentered covariance matrix.

[0146] The updated uncentered covariance matrix is ​​used to capture the correlation between the input features of the current task and the expert’s selection, thereby maintaining the stability of the model during task switching.

[0147] The S8 comprises the following sub-steps:

[0148] S81: Perform singular value decomposition on the two updated non-centralized covariance matrices respectively, the formula is:

[0149]

[0150]

[0151] in, and is the left singular vector matrix and the right singular vector matrix of the first non-centered covariance matrix decomposition, is the first diagonal singular value matrix, and is the left singular vector matrix and the right singular vector matrix of the second non-centered covariance matrix decomposition, is the second diagonal singular value matrix;

[0152] The goal of this decomposition is to extract the right singular vectors corresponding to the singular values ​​close to zero from the covariance matrix, which will be used to construct the null space basis:

[0153] Select the right singular vector corresponding to the singular value close to zero: According to the singular value decomposition result, from and The smallest singular value close to zero is selected, and the right singular vectors corresponding to these singular values ​​are considered to be the null space basis of the covariance matrix. The null space basis vectors are used for subsequent gradient projection to ensure that the features of the old task are not disturbed when the gradient is updated;

[0154] S82: Based on the decomposition result, select the right singular vector corresponding to the minimum singular value as the basis vector of the null space. The formula is:

[0155]

[0156]

[0157] in, is the basis vector of the null space corresponding to the input feature, is the basis vector of the null space corresponding to the gating vector, is the right singular vector whose singular value in the input feature is less than the threshold, is the right singular vector whose singular value in the gated vector is less than the threshold, is the number of singular vectors selected from the input features, is the number of singular vectors selected in the gating vector.

[0158] and The basis of the approximate null space of the input features and the gating vector, respectively, represents the directions of information that are not important in the old task. In subsequent gradient updates, the gradients projected into these null spaces will not affect the features and expert selection of the old task.

[0159] In S9, the null space projection matrix is ​​updated according to the basis vector of the null space, and a relaxation factor is introduced for the null space projection matrix, and the formula is:

[0160]

[0161]

[0162] in, is the null space projection matrix for updating the routing network weight gradients, is the null space projection matrix for the candidate expert gradient update, is the relaxation factor.

[0163] These projection matrices are used to project the gradients into the null space of the old task, ensuring that the gradient updates of the new task do not interfere with the features and gating selections of the old task. In order to enhance the plasticity of learning new tasks while maintaining stability, a relaxation factor can be introduced for the projection matrix of the candidate expert , Controls the strength of the projection matrix. When it approaches 1, the model pays more attention to stability. When it approaches 0, the plasticity of the model is enhanced.

[0164] In order to maintain the consistency of the features of the old tasks and avoid interference of the new tasks on the old tasks, the present invention calculates the gradient and Projected into the null space of the old task. In this way, the updated gradient direction is guaranteed to be orthogonal to the subspace of the old task input features and gate vectors, thus avoiding interference with the old task.

[0165] In S10, the gradient of the routing network weight and the candidate expert matrix is ​​projected into the null space of the old task feature, and the formula is:

[0166]

[0167]

[0168] in, is the routing network weight projected into the null space of the old task features, is the candidate expert matrix projected into the null space of the old task features;

[0169] The learning rate is used to update the routing network weights and candidate expert matrix of the prompt generator. This update ensures that the prompt generator can continue to learn the characteristics of the new task while maintaining the stability of the old task. The formula is:

[0170]

[0171]

[0172] in, is the updated routing network weight, is the original routing network weight, is the learning rate, is the updated candidate expert matrix, is the original candidate expert matrix.

[0173] Determine whether all tasks are completed: Determine the current task Is it the last task? If it is the last task, it means that the learning of all tasks has been completed and the next step is entered; if it is not the last task, it returns to step S2 and continues to learn the next task.

[0174] Output prompt generator parameters: When all tasks After completion, the final optimized hint generator parameters are output, including the router weight matrix and the candidate expert matrix These parameters have been trained on multiple tasks and are able to generate task-relevant stable cues and ensure robustness against forgetting of old tasks.

[0175] In the inference phase, the optimized prompt generator is used to generate corresponding prompts for the test samples, and the ViT model is used for classification prediction. The process of generating prompts is the same as that during training, ensuring the model's adaptability to new and old tasks.

[0176] In one embodiment of the present invention, experiments were conducted on four classes of incremental continuous learning, 10-task CIFAR-100, 20-task CIFAR-100, 10-task ImageNet-R, and 10-task DomainNet-200. The specific experimental results are shown in Table 1, which evenly divide the data set into 10 or 20 tasks for continuous learning experiments. In the table, "VPT-baseline" and "CLIP-baseline" represent sequential baseline methods using VPT and CLIP models, respectively, which are sequentially fine-tuned without mixed experts (MoE) and orthogonal projection. In contrast, "VPT-CPG" and "CLIP-CPG" represent the models proposed by the present invention, using a mixed expert prompt generator and using a null space projection matrix to ensure consistency and anti-forgetting ability between tasks. The upper bound represents the ideal performance of jointly training all tasks, which can be used as the upper limit of the performance of continuous learning. As can be seen from the table, the method of the present invention is significantly better than the baseline model in all experiments, especially in the continuous learning task, the accuracy of the model using the method of the present invention is improved by 6% to 10%, and the forgetting rate is reduced by 11% to 20%. The specific experimental results and comparisons are shown in Table 1, which demonstrate the effectiveness of the method of the present invention and its obvious advantages in anti-forgetting ability.

[0177] Table 1 Comparison with baseline methods

[0178]

[0179] The comparison between the present invention and other methods on four types of incremental continuous learning tasks is shown in Table 2. It can be seen from the table that the present method has achieved the best performance at present.

[0180] Table 2 Comparison with other methods

[0181]

[0182] The performance of the embodiment of the present invention on 50-task CIFAR-100 is shown in Table 3:

[0183] Table 3 Comparison of implementation examples with other methods and baseline methods on 50-task CIFAR-100

[0184]

[0185] Among them, VPT-baseline is the baseline method of sequential training, VPT-CPG is an embodiment of the present invention, and the others are existing advanced methods. It can be seen that this embodiment achieves an accuracy improvement of 21.38% and a forgetting rate reduction of 27.86% compared with the baseline, and exceeds other existing methods.

[0186] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the invention.

Claims

1. A continuous evolutionary learning method based on a consistent hybrid expert prompt generator, characterized in that: The following steps are involved: S1: correlation matrix for initializing the cue generator; S2: Get the training data set for the current task; S3: Determine whether the current task is the first task, if yes, proceed to step S4, otherwise proceed to step S7; S4: Input the input features of the current task into the prompt generator and ViT model for forward propagation, and calculate the classification loss; S5: Based on the classification loss, the gradients of the routing network weights and the candidate expert matrix in the prompt generator are calculated by back propagation to update the corresponding parameters; S6: Determine whether the current task has reached the maximum number of iterations. If not, return to step S3; otherwise, proceed to step S7: S7: Save the parameters of the prompt generator and update the two non-centered covariance matrices using the training dataset and the trained correlation matrix; S8: performing singular value decomposition on the two updated non-centered covariance matrices respectively, and selecting the basis vectors of the corresponding null space based on the decomposition results; S9: Update the null space projection matrix according to the basis vectors of the null space, and introduce a relaxation factor for the null space projection matrix; S10: Based on the updated null space projection matrix, the gradients of the routing network weights and the candidate expert matrix are projected into the null space of the old task features, and the routing network weights and the candidate expert matrix of the prompt generator are updated using the learning rate; S11: Determine whether the current task is the last task. If not, return to step S2. Otherwise, complete the learning of all tasks and output the updated routing network weights and candidate expert matrix of the prompt generator to achieve continuous evolutionary learning based on the consistent hybrid expert prompt generator. The S4 comprises the following sub-steps: S41: Input the input features to the prompt generator, calculate the gate value through the routing network weight matrix, and select the front The gate value is used as an expert, and the formula is: in, is the gate value, is the activation function, is the input feature, is the routing network weight matrix, is the field of real numbers, is the number of experts; in, is a function, represents the variable used to mark the state of each gate value, For the A gate value; S42: Select the front The gate values ​​are normalized and the normalized results are used together with the candidate expert matrix Perform weighted merging to generate the final prompt , the formula is: in, is the gating vector generated by normalization, is the input feature dimension, is the length of the prompt; S43: Change the prompt and input features Input to the corresponding layer of the ViT model for forward propagation, the formula is: in, is the output feature of the corresponding layer of the ViT model, Represents the ViT layer, Tag the image; S44: Input features Input into the classifier connected to the ViT model, get the prediction result, and calculate the cross entropy loss , the formula is: in, is the total number of categories, For Category The real label, For the model category The predicted probability of .

2. A continuous evolutionary learning method based on a consistent hybrid expert prompt generator according to claim 1, characterized in that: The S1 initializes the relevant matrix of the prompt generator, including the routing network weight matrix , Candidate Expert Matrix , the first non-centered covariance matrix , the second non-centered covariance matrix , the first null space projection matrix and the second null space projection matrix , the formula is: ， ， ， in, is the field of real numbers, is the input feature dimension, is the length of the prompt, is the number of experts, is the identity matrix.

3. The continuous evolutionary learning method based on a consistent hybrid expert prompt generator according to claim 1, characterized in that: The training dataset of the current task in S2 for: in, is the training sample, is the category label corresponding to the training sample, For the task, For category.

4. The continuous evolutionary learning method based on a consistent hybrid expert prompt generator according to claim 1, characterized in that: In S5, the gradient of the routing network weight and the candidate expert matrix in the prompt generator is calculated by back propagation, and the formula is: in, is the gradient of the routing network weight, is the gradient of the candidate expert matrix, is the symbol of partial derivative.

5. The continuous evolutionary learning method based on a consistent hybrid expert prompt generator according to claim 4, characterized in that: In S7, the two non-centralized covariance matrices are updated using the training data set and the trained correlation matrix, including the following steps: S71: Input feature matrix recorded during the forward propagation process according to the current task Update the first non-centered covariance matrix, the formula is: in, are the input features of different samples, is the updated first non-centered covariance matrix, is the original first non-centered covariance matrix, with the superscript is the transpose of the matrix, For the task; S72: The gate vector matrix output by the routing network recorded during the forward propagation process according to the current task Update the second non-centered covariance matrix, the formula is: in, is the gating vector of different samples, is the updated second non-centered covariance matrix, is the original second uncentered covariance matrix.

6. The continuous evolutionary learning method based on a consistent hybrid expert prompt generator according to claim 5, characterized in that: The S8 comprises the following sub-steps: S81: Perform singular value decomposition on the two updated non-centralized covariance matrices respectively, the formula is: in, and is the left singular vector matrix and the right singular vector matrix of the first non-centered covariance matrix decomposition, is the first diagonal singular value matrix, and is the left singular vector matrix and the right singular vector matrix of the second non-centered covariance matrix decomposition, is the second diagonal singular value matrix; S82: Based on the decomposition result, select the right singular vector corresponding to the minimum singular value as the basis vector of the null space. The formula is: in, is the basis vector of the null space corresponding to the input feature, is the basis vector of the null space corresponding to the gating vector, is the right singular vector whose singular value in the input feature is less than the threshold, is the right singular vector whose singular value in the gated vector is less than the threshold, is the number of singular vectors selected from the input features, is the number of singular vectors selected in the gating vector.

7. A continuous evolutionary learning method based on a consistent hybrid expert prompt generator according to claim 6, characterized in that: In S9, the null space projection matrix is ​​updated according to the basis vector of the null space, and a relaxation factor is introduced for the null space projection matrix, and the formula is: in, is the null space projection matrix for updating the routing network weight gradients, is the null space projection matrix for the candidate expert gradient update, is the relaxation factor, is the identity matrix.

8. The continuous evolutionary learning method based on a consistent hybrid expert prompt generator according to claim 7, characterized in that: In S10, the gradient of the routing network weight and the candidate expert matrix is ​​projected into the null space of the old task feature, and the formula is: in, is the routing network weight projected into the null space of the old task features, is the candidate expert matrix projected into the null space of the old task features; Use the learning rate to update the routing network weights and candidate expert matrix of the prompt generator, the formula is: in, is the updated routing network weight, is the original routing network weight, is the learning rate, is the updated candidate expert matrix, is the original candidate expert matrix.

Images