Decoupling double-speed self-adaptive test method, device, equipment, medium and product

Abstract

The invention relates to a decoupling double-speed self-adaptive test method and device, equipment, a medium and a product. The method comprises the following steps: acquiring a tested image sample and a template cue word; wherein the template cue word is set based on the category of the trained task; obtaining a pre-trained target decoupling double-speed adaptive model, and performing feature extraction on the image sample and the template cue word according to the target decoupling double-speed adaptive model to obtain image features and text features; wherein the pre-trained target decoupling double-speed self-adaptive model is obtained through a decoupling double-speed self-adaptive fine tuning step; performing similarity calculation on the image features and the text features to obtain similarity data; and testing the target decoupling double-speed adaptive model according to the similarity data to obtain a test result. By adopting the method, the plasticity and the stability of the model can be effectively balanced under the condition of data imbalance, and the test performance of the head class and the tail class is considered at the same time.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a decoupled dual-speed adaptive testing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep neural networks have achieved remarkable results in tasks such as image recognition. Continuous learning algorithms have been proposed, aiming to endow models with human-like continuous learning capabilities, enabling them to continuously learn new knowledge while retaining old knowledge when faced with dynamically expanding tasks, achieving a balance between plasticity and stability. This results in good performance on both new and old tasks. Most existing continuous learning methods typically assume that the amount of data across tasks is balanced. However, real-world data often exhibits a long-tail distribution: only a few categories have sufficient samples (called the head class), while most categories have scarce samples (called the tail class). When the number of training samples is significantly imbalanced across tasks, many tasks struggle to learn effectively due to a severe lack of training samples and are more prone to forgetting, making it even more difficult to balance the model's plasticity and stability. Furthermore, the extremely poor performance of the tail class limits the overall performance of the model. Therefore, traditional continuous learning methods often exhibit significant performance degradation when faced with extremely imbalanced data distributions. Although the DAP method considers the obstacles to model training caused by the interaction between data imbalance and continuous task expansion, its performance still needs improvement. Therefore, there is an urgent need for an imbalanced continuous learning method that can effectively balance the plasticity and stability of the model under imbalanced data conditions, while taking into account the performance of the head class and the tail class. Summary of the Invention

[0003] Based on this, it is necessary to provide a decoupled dual-speed adaptive testing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can effectively balance the plasticity and stability of a model under conditions of data imbalance, in order to address the above-mentioned technical problems.

[0004] Firstly, this application provides a decoupled dual-speed adaptive testing method, including:

[0005] Obtain test image samples and template prompts; wherein, the template prompts are set based on the categories of the trained task;

[0006] A pre-trained target decoupled dual-speed adaptive model is obtained, and features are extracted from the image samples and the template prompt words based on the target decoupled dual-speed adaptive model to obtain image features and text features; wherein, the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step;

[0007] Similarity data is obtained by calculating the similarity between the image features and the text features.

[0008] The target decoupled dual-speed adaptive model was tested based on the similarity data, and the test results were obtained.

[0009] In one embodiment, the decoupling dual-speed adaptive fine-tuning step includes:

[0010] Acquire training image data and training prompts for multiple tasks;

[0011] The text encoder is fine-tuned based on the training image data, the training prompts, and the initial visual language model to obtain a candidate visual language model.

[0012] The image encoder is fine-tuned based on the preset dual-speed adaptor, the training image data, the training prompts, and the candidate visual language model to obtain the target decoupled dual-speed adaptive model.

[0013] In one embodiment, the initial visual language model is fine-tuned using a text encoder based on the training image data and the training cue words to obtain a candidate visual language model, including:

[0014] The initial visual language model is subjected to image encoder parameter freezing to obtain an initial frozen model;

[0015] Based on the initial freezing model, feature extraction is performed on the training image data and the training prompt words to obtain initial image features and initial text features;

[0016] The initial image features and the initial text features are used to calculate the similarity to obtain the image-text matching score;

[0017] The cross-entropy loss is calculated based on the image-text matching score to obtain loss data. The parameters of the initial frozen model are then fine-tuned based on the loss data to obtain the candidate visual language model.

[0018] In one embodiment, the preset dual-speed adaptor includes: a fast adaptor, a slow adaptor, and a task modulation fusion unit; the step of fine-tuning the candidate visual language model using an image encoder based on the preset dual-speed adaptor, the training image data, and the training cue words to obtain the target decoupled dual-speed adaptive model includes:

[0019] The text encoder parameters of the candidate visual language model are frozen to obtain the candidate frozen model;

[0020] The candidate frozen model is fine-tuned using an image encoder based on the training image data, the fast adaptor, the slow adaptor, and the training cue words to obtain fast fine-tuning results and slow fine-tuning results; wherein the hyperparameters of the fast adaptor are greater than the hyperparameters of the slow adaptor.

[0021] The fast fine-tuning result and the slow fine-tuning result are modulated by the task modulation fusion device to obtain the image encoder fine-tuning result;

[0022] The candidate freezing model is fine-tuned based on the fine-tuning results of the image encoder to obtain the target decoupled dual-speed adaptive model.

[0023] In one embodiment, testing the target decoupled dual-speed adaptive model based on the similarity data to obtain test results includes:

[0024] If all tasks corresponding to the similarity data have been tested and completed, the test accuracy of the target decoupled dual-speed adaptive model on the historical tasks that have been tested and completed is obtained to obtain the final accuracy.

[0025] The cumulative test accuracy is obtained by recording the test accuracy of the target decoupled dual-speed adaptive model on all current tasks after each training of a historical task is completed.

[0026] The average accuracy is obtained by averaging the cumulative test accuracy; the test result includes the final accuracy and the average accuracy.

[0027] In one embodiment, the method further includes:

[0028] Obtain fast hyperparameters and slow hyperparameters; wherein the fast hyperparameters are greater than the slow hyperparameters;

[0029] Based on the preset initial adapter, the fast hyperparameters, and the slow hyperparameters, an adapter is designed to obtain a fast adapter and a slow adapter.

[0030] The fast adaptor and the slow adaptor are combined according to the task modulation fusion device to obtain the preset dual-speed adaptor; wherein, the task modulation fusion device is used to calculate the modulation coefficient, and the modulation coefficient is used to adjust the output weight of the fast adaptor and the slow adaptor in the preset dual-speed adaptor.

[0031] Secondly, this application also provides a decoupled dual-speed adaptive test device, comprising:

[0032] An acquisition module is used to acquire image samples and template prompts for the test; wherein the template prompts are set based on the categories of the trained task.

[0033] The feature extraction module is used to obtain a pre-trained target decoupled dual-speed adaptive model, and to extract features from the image samples and the template prompt words based on the target decoupled dual-speed adaptive model to obtain image features and text features; wherein, the acquisition of the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step;

[0034] The similarity calculation module is used to calculate the similarity between the image features and the text features to obtain similarity data;

[0035] The testing module is used to test the target decoupled dual-speed adaptive model based on the similarity data and obtain the test results.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0037] Obtain test image samples and template prompts; wherein, the template prompts are set based on the categories of the trained task;

[0038] A pre-trained target decoupled dual-speed adaptive model is obtained, and features are extracted from the image samples and the template prompt words based on the target decoupled dual-speed adaptive model to obtain image features and text features; wherein, the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step;

[0039] Similarity data is obtained by calculating the similarity between the image features and the text features.

[0040] The target decoupled dual-speed adaptive model was tested based on the similarity data, and the test results were obtained.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0042] Obtain test image samples and template prompts; wherein, the template prompts are set based on the categories of the trained task;

[0043] A pre-trained target decoupled dual-speed adaptive model is obtained, and features are extracted from the image samples and the template prompt words based on the target decoupled dual-speed adaptive model to obtain image features and text features; wherein, the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step;

[0044] Similarity data is obtained by calculating the similarity between the image features and the text features.

[0045] The target decoupled dual-speed adaptive model was tested based on the similarity data, and the test results were obtained.

[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0047] Obtain test image samples and template prompts; wherein, the template prompts are set based on the categories of the trained task;

[0048] A pre-trained target decoupled dual-speed adaptive model is obtained, and features are extracted from the image samples and the template prompt words based on the target decoupled dual-speed adaptive model to obtain image features and text features; wherein, the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step;

[0049] Similarity data is obtained by calculating the similarity between the image features and the text features.

[0050] The target decoupled dual-speed adaptive model was tested based on the similarity data, and the test results were obtained.

[0051] The aforementioned decoupled dual-speed adaptive testing method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire test image samples and template prompts; wherein the template prompts are set based on the categories of the trained task; acquire a pre-trained target decoupled dual-speed adaptive model, and extract features from the image samples and template prompts based on the target decoupled dual-speed adaptive model to obtain image features and text features; wherein the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step; similarity is calculated on the image features and text features to obtain similarity data; and the target decoupled dual-speed adaptive model is tested based on the similarity data to obtain test results. Therefore, a pre-trained target-decoupled dual-speed adaptive model is used to extract features from image samples and template prompts. This method abandons the randomly initialized linear classifier and instead uses stable category semantic anchors in the target-decoupled dual-speed adaptive model to classify image features of image samples. This compensates for the performance degradation caused by the extreme scarcity of tail class features in imbalanced continuous learning scenarios. The dual-speed adaptor enables the model to quickly acquire new task knowledge while reducing the forgetting of old knowledge. The strategy of decoupling the learning text image encoder in the target-decoupled dual-speed adaptive model further reduces the bias caused by long-tail data and mitigates catastrophic forgetting. Finally, the target-decoupled dual-speed adaptive model is tested based on the similarity data to obtain test results for verifying the model's performance. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an application environment diagram of the decoupled dual-speed adaptive testing method in one embodiment;

[0054] Figure 2 This is a flowchart illustrating the decoupling of the dual-speed adaptive testing method in one embodiment;

[0055] Figure 3 This is a flowchart illustrating the decoupling of the dual-speed adaptive testing method in one embodiment;

[0056] Figure 4 This is a schematic diagram of the overall DFSA architecture for decoupling the dual-speed adaptive test method in another embodiment;

[0057] Figure 5This is a schematic diagram illustrating the performance verification of five representative tasks in another embodiment of the decoupled dual-speed adaptive testing method.

[0058] Figure 6 This is a schematic diagram of visual language fine-tuning for the decoupled dual-speed adaptive testing method in another embodiment;

[0059] Figure 7 This is a schematic diagram of the test process for the decoupled dual-speed adaptive test method in another embodiment;

[0060] Figure 8 This is a schematic diagram illustrating the performance changes of the decoupled dual-speed adaptive testing method in another embodiment, fine-tuned in three scenarios on the ImageNet-R-TICL dataset.

[0061] Figure 9 This is a structural block diagram of a decoupled dual-speed adaptive test device in one embodiment;

[0062] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] The decoupled dual-speed adaptive testing method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 acquires test image samples and template prompts; the template prompts are set based on the category of the trained task; acquires a pre-trained target decoupling dual-speed adaptive model, and extracts features from the image samples and template prompts based on the target decoupling dual-speed adaptive model to obtain image features and text features; the pre-trained target decoupling dual-speed adaptive model is generated through a decoupling dual-speed adaptive fine-tuning step; similarity is calculated between the image features and text features to obtain similarity data; and the target decoupling dual-speed adaptive model is tested based on the similarity data to obtain test results. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0065] In one exemplary embodiment, such as Figure 2 As shown, a decoupled dual-speed adaptive testing method is provided, which is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 202 to 208. Wherein:

[0066] Step 202: Obtain test image samples and template prompts; wherein, the template prompts are set based on the categories of the trained task.

[0067] The test image samples are image data from all the tasks that have been seen. For example, after training is completed for Task t, the model will be tested on all the tasks that have been seen, that is, on the test set of Task 0 to Task t.

[0068] In some embodiments, template prompts can be formed by obtaining the category names and image samples contained in all currently trained tasks and combining the category names with a preset template. Template prompts can also be obtained in other ways, but are not limited to this.

[0069] In this embodiment, by acquiring image samples and template prompts of the trained task for testing, the classification accuracy of the model in each round of training can be accurately measured, and the performance changes of the model after each round of training can be evaluated, thereby improving the real-time performance and effectiveness of model training monitoring.

[0070] Step 204: Obtain the pre-trained target decoupled dual-speed adaptive model, and extract features from the image samples and template prompts based on the target decoupled dual-speed adaptive model to obtain image features and text features; wherein, the pre-trained target decoupled dual-speed adaptive model is generated through the decoupled dual-speed adaptive fine-tuning step.

[0071] Among them, the pre-trained target decoupled dual-speed adaptive model is designed to address the imbalanced continuous learning task. The target decoupled dual-speed adaptive model is pre-trained using the Decoupled Fast-Slow Adaptation (DFSA) method. The target decoupled dual-speed adaptive model is obtained by decoupling and step-by-step fine-tuning the initial visual language model based on the preset dual-speed adaptor and decoupling strategy in the decoupled dual-speed adaptive fine-tuning step. It can be used to solve the problem of the gradual performance decline of the initial visual language model due to forgetting and tail tasks. Tail tasks refer to tasks with insufficient samples, i.e., continuous learning tasks with insufficient samples.

[0072] In some embodiments, image features are extracted from image samples using the image encoder of the target decoupled dual-speed adaptive model, and text features are extracted from template prompt words using the text encoder of the target decoupled dual-speed adaptive model.

[0073] In this embodiment, feature extraction is performed using the image encoder and text encoder of the pre-trained target decoupled dual-speed adaptive model. This allows for the accurate extraction of image and text features for the current task, which can then be used for precise classification of image and text data.

[0074] In some embodiments, the decoupled dual-speed adaptive fine-tuning step includes: acquiring training image data and training prompts for multiple tasks; fine-tuning the text encoder based on the training image data, training prompts, and an initial visual language model to obtain a candidate visual language model; and fine-tuning the image encoder based on a preset dual-speed adaptor, training image data, training prompts, and the candidate visual language model to obtain a target decoupled dual-speed adaptive model.

[0075] Among them, multiple tasks can refer to multiple tasks in imbalanced continuous learning, training image data refers to the image data of the current training task, and training prompt words are prompt text composed of the category name of the current training task and a preset template.

[0076] In some embodiments, such as Figure 5 As shown, Figure 5 Performance validation curves for five representative training tasks are shown. Figure 5 This study experimentally compared two decoupled training strategies: fine-tuning the image encoder first and then the text encoder, and fine-tuning the text encoder first and then the image encoder. The performance changes of five training tasks under these two strategies were compared. The experiments showed that the performance degradation was slower and more stable under the strategy of fine-tuning the text encoder first and then the image encoder.

[0077] In this embodiment, our decoupling learning strategy for training text encoder fine-tuning and image encoder fine-tuning is as follows: Figure 4 As shown in Stages I and II, the first stage fine-tunes the text encoder while keeping the image encoder frozen and not updated. The second stage keeps the text encoder frozen and not updated while fine-tuning the image encoder. This strategy first fine-tunes the text encoder, which has a faster convergence speed, and then freezes it, thereby establishing more stable semantic anchors for each category. Then, it fine-tunes the image encoder, which has a slower convergence speed. Stable semantic anchors make the classification boundary more robust to long-tail data, alleviating the serious bias caused by long-tail data to the classification boundary, and facilitating better alignment of the feature space, thus reducing catastrophic forgetting.

[0078] In some embodiments, the initial visual language model is fine-tuned using a text encoder based on training image data and training prompts to obtain a candidate visual language model. This includes: freezing the image encoder parameters of the initial visual language model to obtain an initial frozen model; extracting features from the training image data and training prompts based on the initial frozen model to obtain initial image features and initial text features; calculating the similarity between the initial image features and initial text features to obtain an image-text matching score; calculating the cross-entropy loss based on the image-text matching score to obtain loss data; and fine-tuning the parameters of the initial frozen model based on the loss data to obtain a candidate visual language model.

[0079] Among them, image encoder parameter freezing refers to freezing the parameters of the image encoder in the model to ensure that the parameters of the image encoder do not change during subsequent fine-tuning learning. The image-text matching score can be the logit, and the loss data can be the loss calculation gradient used to update the parameters that need to be fine-tuned.

[0080] In some embodiments, such as Figure 6 As shown, Figure 6 This demonstrates the specific steps involved in fine-tuning the text encoder parameters of the model, including feature extraction, similarity calculation, cross-entropy loss calculation, and parameter fine-tuning. It also shows how the model converges, until a well-trained candidate visual language model is obtained. Figure 6 The current task in the training context refers to the task currently being trained.

[0081] In some embodiments, the preset dual-speed adaptor includes: a fast adaptor, a slow adaptor, and a task modulation fusion unit; fine-tuning the candidate visual language model using an image encoder based on the preset dual-speed adaptor, training image data, and training prompts to obtain a target decoupled dual-speed adaptive model includes: freezing the text encoder parameters of the candidate visual language model to obtain a candidate frozen model; fine-tuning the candidate frozen model using an image encoder based on training image data, the fast adaptor, the slow adaptor, and training prompts to obtain a fast fine-tuning result and a slow fine-tuning result; wherein the hyperparameters of the fast adaptor are greater than the hyperparameters of the slow adaptor; modulating the fast fine-tuning result and the slow fine-tuning result using the task modulation fusion unit to obtain an image encoder fine-tuning result; and fine-tuning the candidate frozen model based on the image encoder fine-tuning result to obtain the target decoupled dual-speed adaptive model.

[0082] It should be noted that the working process of the preset dual-speed adaptor is as follows: Figure 4 As shown in the Image Encoder section, the Fast Adapter is used to quickly but briefly acquire knowledge during fine-tuning, while the Slow Adapter is used to gradually accumulate and consolidate knowledge during fine-tuning. The Task-modulated weighting dynamically balances the contributions of the Fast Adapter and the Slow Adapter to obtain the final output of the image encoder.

[0083] In some embodiments, the image encoder fine-tuning learning of the candidate frozen model based on training image data, a fast adaptor, a slow adaptor, and training cue words involves using a fast adaptor to quickly but briefly acquire knowledge during the fine-tuning process, and a slow adaptor to gradually accumulate and consolidate knowledge during the fine-tuning process.

[0084] It should be noted that having higher hyperparameters for the fast adaptor than for the slow adaptor ensures that the fast adaptor can efficiently acquire knowledge for new tasks, while the slow adaptor can more stably consolidate knowledge and reduce forgetting. The resulting fast fine-tuning result is the output data of the fast adaptor, and the slow fine-tuning result is the output data of the slow adaptor. The task modulation fusion unit generates modulation coefficients based on the fast and slow fine-tuning results. These modulation coefficients are then used to combine the fast and slow adaptors in an end-to-end manner to obtain the image encoder fine-tuning result, enabling the rapid acquisition of knowledge for new tasks while maintaining long-term stability.

[0085] In this embodiment, combined with Figure 4The overall architecture of DFSA, as shown in the figure, involves multiple tasks, such as Task 0, Task t, and Task T, which arrive sequentially and are fed into the model for training. Based on the pre-trained CLIP model, all its pre-trained model parameters are frozen during fine-tuning. An adapter is used to fine-tune the pre-trained text encoder, and a pre-set Dual Speed ​​Adapter is used to fine-tune the image encoder. Similarity is calculated between image features and text features of each category to generate the discrimination result for the input image. Assume the model has been trained on all tasks up to task t, and the currently arriving task is Task t. Processing Task t is divided into two stages. The first stage, as shown... Figure 4 As shown in Stage I, all parameters of the image encoder are frozen, and only the parameters of the text encoder that need to be updated are fine-tuned, according to... Figure 6 The training process is shown in the diagram. The second stage, as... Figure 4 As shown in Stage II, all parameters of the text encoder are frozen, and only the parameters in the image encoder that need to be updated are fine-tuned, according to... Figure 6 The training is performed according to the process shown.

[0086] Step 206: Calculate the similarity between image features and text features to obtain similarity data.

[0087] The similarity calculation can be performed using cosine similarity, i.e., the CLIP (Cosine Similarity) model, or other similarity calculation methods used for classification, and is not limited to these.

[0088] In some embodiments, image features and text features are calculated using the CLIP model to obtain similarity data, i.e., image-text matching scores, which are used to reflect the degree of matching and accuracy between the image and the text, but are not limited thereto.

[0089] Step 208: Test the target decoupled dual-speed adaptive model based on similarity data and obtain the test results.

[0090] Among them, testing refers to the process of testing the image classification task. Based on similarity data, the model's judgment result on the input sample can be obtained.

[0091] In some embodiments, the target decoupled dual-speed adaptive model is tested based on similarity data to obtain test results, including: obtaining the test accuracy of the target decoupled dual-speed adaptive model on the historical tasks that have been tested and completed, when all tasks corresponding to the similarity data have been tested and completed, to obtain the final accuracy; obtaining the test accuracy of the target decoupled dual-speed adaptive model on all current tasks after each historical task has been trained, to obtain the cumulative test accuracy; calculating the mean of the cumulative test accuracy to obtain the average accuracy; the test results include the final accuracy and the average accuracy.

[0092] In some embodiments, such as Figure 7 As shown, the DFSA method based on this scheme was tested on three imbalanced continuous learning datasets: ImageNet-R-TICL, CIFAR100-TICL, and TinyImageNet-TICL. Experiments were conducted on each dataset under three scenarios: TICL-Descending, TICL-Ascending, and TICL-Shuffled. Based on the final accuracy and average accuracy, it can be determined that the DFSA method of this scheme achieved good results in all scenarios. The tables show the comparison methods and the Top-1 accuracy (%). The continuous learning performance is evaluated using two metrics: final accuracy and average accuracy. One represents the average accuracy of the model after training on the last task and testing it on all tasks; the other is the average accuracy. This means that during the model's training of each task sequentially, the test accuracy on all tasks is recorded after each task is completed, until all tasks are trained and tested. The average of all the recorded accuracies is then used to obtain the metric. More specifically, if there are a total of T tasks, the test accuracy of the model on all tasks encountered after training the t-th task is denoted as . Then the final accuracy is , Furthermore, the DFSA method is labeled with (Ours). Tables 1, 2, and 3 show the results for the ImageNet-R-TICL, CIFAR100-TICL, and TinyImageNet-TICL datasets, respectively. As can be seen from the following tables, our method achieves superior performance.

[0093] Table 1. Comparison of classification accuracy (Top1 accuracy%) in three scenarios on the ImageNet-R-TICL dataset.

[0094]

[0095] Table 2. Comparison of classification accuracy (Top1 accuracy%) in three scenarios on the CIFAR100-TICL dataset.

[0096]

[0097] Table 3. Comparison of classification accuracy (Top1 accuracy%) in three scenarios on the TinyImageNet-TICL dataset.

[0098]

[0099] In the aforementioned decoupled dual-speed adaptive testing method, test image samples and template prompts are acquired; the template prompts are set based on the categories of the trained task; a pre-trained target decoupled dual-speed adaptive model is acquired, and features are extracted from the image samples and template prompts based on the target decoupled dual-speed adaptive model to obtain image features and text features; the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step; similarity is calculated between the image features and text features to obtain similarity data; and the target decoupled dual-speed adaptive model is tested based on the similarity data to obtain test results. Therefore, a pre-trained target-decoupled dual-speed adaptive model is used to extract features from image samples and template prompts. This method abandons the randomly initialized linear classifier and instead uses stable category semantic anchors in the target-decoupled dual-speed adaptive model to classify image features of image samples. This compensates for the performance degradation caused by the extreme scarcity of tail class features in imbalanced continuous learning scenarios. The dual-speed adaptor enables the model to quickly acquire new task knowledge while reducing the forgetting of old knowledge. The strategy of decoupling the learning text image encoder in the target-decoupled dual-speed adaptive model further reduces the bias caused by long-tail data and mitigates catastrophic forgetting. Finally, the target-decoupled dual-speed adaptive model is tested based on the similarity data to obtain test results for verifying the model's performance.

[0100] In one exemplary embodiment, such as Figure 3 As shown, the decoupled dual-speed adaptive test method further includes steps 302 to 306. Wherein:

[0101] Step 302: Obtain the fast hyperparameters and the slow hyperparameters; wherein, the fast hyperparameters are greater than the slow hyperparameters.

[0102] Here, fast hyperparameters refer to the hyperparameters in the fast adaptor. Slow hyperparameters refer to the hyperparameters in a slow adaptor. ,and To ensure that the fast adaptor can efficiently acquire knowledge for new tasks, the slow adaptor can more stably consolidate knowledge and reduce forgetting.

[0103] In some embodiments, fast hyperparameters and slow hyperparameters can be determined based on various experiments to ensure that the dual-speed adaptor combining the fast and slow adaptors can be fine-tuned to produce a model that performs best in experiments. Alternatively, fast hyperparameters and slow hyperparameters can be obtained through other empirical settings or data analysis.

[0104] Step 304: Design an adaptor based on the preset initial adaptor, fast hyperparameters, and slow hyperparameters to obtain a fast adaptor and a slow adaptor.

[0105] The working process of the Dual Speed ​​Adapter is as follows: Figure 4 The Image Encoder section is shown below.

[0106] In some embodiments, a standard adapter can be used as an example to design a fine-tuning model, and the process is shown in formula (1):

[0107] (1);

[0108] in Represents image features. and These represent the weights of the upsampling and downsampling linear layers, respectively. This represents the activation function. Based on this adapter, a fast adaptor was designed: and slow adaptors: As shown in formulas (2) and (3).

[0109] (2);

[0110] (3);

[0111] in, and This represents two hyperparameters, and To ensure that the fast adaptor can efficiently acquire knowledge for new tasks, the slow adaptor can more stably consolidate knowledge and reduce forgetting.

[0112] Step 306: Combine the fast adaptor and the slow adaptor according to the task modulation fusion to obtain a preset dual-speed adaptor; wherein, the task modulation fusion is used to calculate the modulation coefficient, and the modulation coefficient is used to adjust the output weight of the fast adaptor and the slow adaptor in the preset dual-speed adaptor.

[0113] Here, "combining" refers to combining the outputs of the fast adaptor and the slow adaptor in an end-to-end manner.

[0114] In some embodiments, the task modulation fusion unit, such as Figure 4 Task-modulated weighting in [the context of image features]. , For the size of a batch, The feature dimension is represented by calculating the feature representation: And calculate the modulation coefficients according to the formula (4) of the task modulation fusion unit. :

[0115] (4)

[0116] MLP stands for Multilayer Perceptron. This represents the activation function. The final output of the final dual-speed adaptor. As shown in formula (5):

[0117] (5)

[0118] In this way, the fast and slow adaptors are combined in an end-to-end manner through modulation coefficients, enabling long-term stability while rapidly acquiring knowledge of new tasks.

[0119] In some embodiments, for a better understanding of the scheme of this application, the following are provided: Figure 8 The experimental results tested in some specific embodiments are described below:

[0120] like Figure 8 As shown, Figure 8 Demonstrates the performance of the ImageNet-R-TICL dataset in three scenarios. Figure 8 The paper demonstrates the sequential performance of various methods on ImageNet-R-TICL. In the descending order training scenario, the head tasks (tasks with sufficient samples) arrive first, and all methods perform well initially. However, due to model forgetting and insufficient samples in the tail tasks (tasks with limited samples), performance gradually declines. The proposed DFSA method exhibits the slowest decline, demonstrating its superior robustness against forgetting. In the ascending order training scenario, the initial task samples are fewer, causing most methods to perform poorly in the early stages. As larger tasks with more samples arrive, performance on these tasks is higher, leading to an overall increase in average accuracy. DFSA effectively utilizes text-guided image encoder learning, allowing the model to maintain good performance throughout the process.

[0121] Table 4 Ablation experiments of each component

[0122]

[0123] Table 4 presents the ablation experiment results for the contributions of each component of DFSA. In the decoupled learning process between the text encoder and the image encoder, Image-Text indicates fine-tuning the image encoder first, followed by fine-tuning the text encoder, while Text-Image represents the opposite. The Text-Image order consistently outperforms the Image-Text order in all scenarios, with a significant accuracy improvement in Ascending mode. The metrics improved by 1.79%, indicating that obtaining a stable decision boundary before feature alignment is crucial for continuous learning tasks with long-tailed data. Introducing the Dual Speed ​​Adapter (FSA) further improved model performance. For image-text order, FSA improves performance in both descending and asscending scenarios, but only in the shuffled scenario. The metrics decreased by 0.15%. Finally, combining the Text-Image order with FSA yielded optimal results in all scenarios, highlighting the complementary roles of the two components (dual-speed adaptor and task modulation fusionist) in DFSA.

[0124] In one specific embodiment, the process of testing the target decoupled dual-speed adaptive model is an incremental learning process. For example, there are T tasks (1, 2, 3, ..., T) for testing, which arrive sequentially. After a task arrives, the target decoupled dual-speed adaptive model is trained on the data (image samples and template prompts) of that task. Specifically, following the decoupled dual-speed adaptive testing method, after training on each task, the model is tested on all currently seen tasks.

[0125] For example, after training on task t, the model will be tested on tasks 1-t that it has already seen, obtaining the test accuracy ACC_t after learning task t. (This ACC_t is the test result on a combined test set of the test sets of the learned tasks 1, 2, 3, ..., t; it can also be seen as testing on the test sets of 1, 2, 3, ..., t separately, and then averaging these test accuracies). The final accuracy is the performance obtained by testing on all the currently seen tasks after learning the last task, which is ACC_T. Furthermore, the average accuracy is the average of ACC_1, ACC_2, ACC_3, ..., ACC_T.

[0126] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0127] Based on the same inventive concept, this application also provides a decoupled dual-speed adaptive testing device for implementing the decoupled dual-speed adaptive testing method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the decoupled dual-speed adaptive testing device provided below can be found in the limitations of the decoupled dual-speed adaptive testing method described above, and will not be repeated here.

[0128] In one exemplary embodiment, such as Figure 9 As shown, a decoupled dual-speed adaptive testing device is provided, comprising: an acquisition module 901, a feature extraction module 902, a similarity calculation module 903, and a testing module 904, wherein:

[0129] The acquisition module 901 is used to acquire image samples and template prompts for the test; wherein the template prompts are set based on the categories of the trained task.

[0130] The feature extraction module 902 is used to obtain a pre-trained target decoupled dual-speed adaptive model and to extract features from image samples and template prompts based on the target decoupled dual-speed adaptive model to obtain image features and text features; wherein, the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step;

[0131] The similarity calculation module 903 is used to calculate the similarity between image features and text features to obtain similarity data;

[0132] Test module 904 is used to test the target decoupled dual-speed adaptive model based on similarity data and obtain test results.

[0133] In some embodiments, the apparatus further includes: a decoupled dual-speed adaptive fine-tuning module, configured to acquire training image data and training prompts for multiple tasks; fine-tune a text encoder based on the training image data, training prompts, and an initial visual language model to obtain a candidate visual language model; and fine-tune an image encoder based on a preset dual-speed adaptor, training image data, training prompts, and the candidate visual language model to obtain a target decoupled dual-speed adaptive model.

[0134] In some embodiments, the decoupled dual-speed adaptive fine-tuning module is further used to freeze the image encoder parameters of the initial visual language model to obtain an initial frozen model; extract features from the training image data and training prompts based on the initial frozen model to obtain initial image features and initial text features; calculate the similarity between the initial image features and initial text features to obtain an image-text matching score; calculate the cross-entropy loss based on the image-text matching score to obtain loss data; and fine-tune the parameters of the initial frozen model based on the loss data to obtain a candidate visual language model.

[0135] In some embodiments, the preset dual-speed adaptor includes a fast adaptor, a slow adaptor, and a task modulation fusion unit; the decoupled dual-speed adaptive fine-tuning module is further used to freeze the text encoder parameters of the candidate visual language model to obtain a candidate frozen model; the candidate frozen model is fine-tuned using image encoder based on training image data, the fast adaptor, the slow adaptor, and training prompts to obtain fast fine-tuning results and slow fine-tuning results; wherein the hyperparameters of the fast adaptor are greater than the hyperparameters of the slow adaptor; the fast fine-tuning results and slow fine-tuning results are modulated using the task modulation fusion unit to obtain image encoder fine-tuning results; the candidate frozen model is fine-tuned based on the image encoder fine-tuning results to obtain the target decoupled dual-speed adaptive model.

[0136] In some embodiments, the testing module 904 is further configured to, when all tasks corresponding to the similarity data have been tested, obtain the test accuracy of the target decoupled dual-speed adaptive model on the historical tasks that have been tested, and obtain the final accuracy; obtain the test accuracy of the target decoupled dual-speed adaptive model on all current tasks after each training of a historical task, and obtain the cumulative test accuracy; calculate the mean of the cumulative test accuracy to obtain the average accuracy; the test results include the final accuracy and the average accuracy.

[0137] In some embodiments, the apparatus further includes: a dual-speed adaptor construction module, configured to acquire fast hyperparameters and slow hyperparameters; wherein the fast hyperparameters are greater than the slow hyperparameters; to design an adaptor based on a preset initial adaptor, the fast hyperparameters, and the slow hyperparameters to obtain a fast adaptor and a slow adaptor; and to combine the fast adaptor and the slow adaptor based on a task modulation fusion module to obtain a preset dual-speed adaptor; wherein the task modulation fusion module is used to calculate modulation coefficients, and the modulation coefficients are used to adjust the output weights of the fast adaptor and the slow adaptor in the preset dual-speed adaptor.

[0138] In the aforementioned decoupled dual-speed adaptive testing device, test image samples and template prompts are acquired; the template prompts are set based on the categories of the trained task; a pre-trained target decoupled dual-speed adaptive model is acquired, and features are extracted from the image samples and template prompts based on the target decoupled dual-speed adaptive model to obtain image features and text features; the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step; similarity is calculated between the image features and text features to obtain similarity data; the target decoupled dual-speed adaptive model is tested based on the similarity data to obtain test results. Therefore, a pre-trained target-decoupled dual-speed adaptive model is used to extract features from image samples and template prompts. This method abandons the randomly initialized linear classifier and instead uses stable category semantic anchors in the target-decoupled dual-speed adaptive model to classify image features of image samples. This compensates for the performance degradation caused by the extreme scarcity of tail class features in imbalanced continuous learning scenarios. The dual-speed adaptor enables the model to quickly acquire new task knowledge while reducing the forgetting of old knowledge. The strategy of decoupling the learning text image encoder in the target-decoupled dual-speed adaptive model further reduces the bias caused by long-tail data and mitigates catastrophic forgetting. Finally, the target-decoupled dual-speed adaptive model is tested based on the similarity data to obtain test results for verifying the model's performance.

[0139] Each module in the aforementioned decoupled dual-speed adaptive test device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0140] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores a pre-trained target decoupled dual-speed adaptive model. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a decoupled dual-speed adaptive testing method.

[0141] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the decoupled dual-speed adaptive testing method described above.

[0143] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the decoupled dual-speed adaptive testing method described above.

[0144] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the decoupled dual-speed adaptive testing method described above.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A decoupled dual-speed adaptive testing method, characterized in that, The method includes: Obtain test image samples and template prompts; wherein, the template prompts are set based on the categories of the trained task; A pre-trained target decoupled dual-speed adaptive model is obtained, and features are extracted from the image samples and the template prompt words based on the target decoupled dual-speed adaptive model to obtain image features and text features; wherein, the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step; Similarity data is obtained by calculating the similarity between the image features and the text features. The target decoupled dual-speed adaptive model was tested based on the similarity data, and the test results were obtained.

2. The method according to claim 1, characterized in that, The decoupling dual-speed adaptive fine-tuning step includes: Acquire training image data and training prompts for multiple tasks; The text encoder is fine-tuned based on the training image data, the training prompts, and the initial visual language model to obtain a candidate visual language model. The image encoder is fine-tuned based on the preset dual-speed adaptor, the training image data, the training prompts, and the candidate visual language model to obtain the target decoupled dual-speed adaptive model.

3. The method according to claim 2, characterized in that, Based on the training image data and the training prompts, the initial visual language model is fine-tuned using a text encoder to obtain a candidate visual language model, including: The initial visual language model is subjected to image encoder parameter freezing to obtain an initial frozen model; Based on the initial freezing model, feature extraction is performed on the training image data and the training prompt words to obtain initial image features and initial text features; The initial image features and the initial text features are used to calculate the similarity to obtain the image-text matching score; The cross-entropy loss is calculated based on the image-text matching score to obtain loss data. The parameters of the initial frozen model are then fine-tuned based on the loss data to obtain the candidate visual language model.

4. The method according to claim 2, characterized in that, The preset dual-speed adaptor includes: a fast adaptor, a slow adaptor, and a task modulation fusion unit; the step of fine-tuning the candidate visual language model using an image encoder based on the preset dual-speed adaptor, the training image data, and the training cue words to obtain the target decoupled dual-speed adaptive model includes: The text encoder parameters of the candidate visual language model are frozen to obtain the candidate frozen model; The candidate frozen model is fine-tuned using an image encoder based on the training image data, the fast adaptor, the slow adaptor, and the training cue words to obtain fast fine-tuning results and slow fine-tuning results; wherein the hyperparameters of the fast adaptor are greater than the hyperparameters of the slow adaptor. The fast fine-tuning result and the slow fine-tuning result are modulated by the task modulation fusion device to obtain the image encoder fine-tuning result; The candidate freezing model is fine-tuned based on the fine-tuning results of the image encoder to obtain the target decoupled dual-speed adaptive model.

5. The method according to claim 1, characterized in that, The step of testing the target decoupled dual-speed adaptive model based on the similarity data to obtain test results includes: If all tasks corresponding to the similarity data have been tested and completed, the test accuracy of the target decoupled dual-speed adaptive model on the historical tasks that have been tested and completed is obtained to obtain the final accuracy. The cumulative test accuracy is obtained by recording the test accuracy of the target decoupled dual-speed adaptive model on all current tasks after each training of a historical task is completed. The average accuracy is obtained by averaging the cumulative test accuracy; the test results include the final accuracy and the average accuracy.

6. The method according to claim 2, characterized in that, The method further includes: Obtain fast hyperparameters and slow hyperparameters; wherein the fast hyperparameters are greater than the slow hyperparameters; Based on the preset initial adapter, the fast hyperparameters, and the slow hyperparameters, an adapter is designed to obtain a fast adapter and a slow adapter. The fast adaptor and the slow adaptor are combined according to the task modulation fusion device to obtain the preset dual-speed adaptor; wherein, the task modulation fusion device is used to calculate the modulation coefficient, and the modulation coefficient is used to adjust the output weight of the fast adaptor and the slow adaptor in the preset dual-speed adaptor.

7. A decoupled dual-speed adaptive testing device, characterized in that, The device includes: An acquisition module is used to acquire image samples and template prompts for the test; wherein the template prompts are set based on the categories of the trained task. The feature extraction module is used to obtain a pre-trained target decoupled dual-speed adaptive model, and to extract features from the image samples and the template prompt words based on the target decoupled dual-speed adaptive model to obtain image features and text features; wherein, the acquisition of the pre-trained target decoupled dual-speed adaptive model is generated through a decoupled dual-speed adaptive fine-tuning step; The similarity calculation module is used to calculate the similarity between the image features and the text features to obtain similarity data; The testing module is used to test the target decoupled dual-speed adaptive model based on the similarity data and obtain the test results.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images