Model continuous learning test method for image-text data processing, medium and equipment

Abstract

The invention provides a model continuous learning test method for image-text data processing, a medium and equipment, and the method comprises the steps: obtaining image-text data which comprises the combination of a plurality of groups of images and characters; respectively inputting the image-text data into the first to-be-tested model and the second to-be-tested model; matrix entropies and matrix conditional entropies of the first to-be-tested model and the second to-be-tested model at different moments of the first continuous learning task and the second continuous learning task are obtained; and calculating an old knowledge understanding index and a new knowledge extension index of continuous learning based on the corresponding matrix entropy and the matrix conditional entropy. According to the model continuous learning test method for image-text data processing provided by the invention, the matrix entropy is used as model expression, and the matrix conditional entropy of the two to-be-tested models is calculated, so that old knowledge understanding for a preorder task after a learning task and new knowledge extension indexes for the preorder task after the learning task are obtained; and the performance of the continuous learning algorithm of the image-text data in a multi-mode environment can be quantified.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to the field of training model technology for graphic and text data processing. Background Art

[0002] Learning is fundamental to the ability of intelligent systems to adapt to dynamic environments. Humans and other organisms demonstrate powerful adaptive capabilities by continuously acquiring, updating, accumulating, and utilizing knowledge. The desire for AI systems to adapt in a similar manner has driven research on continuous learning. Within the field of AI, continuous learning is a key research topic, aiming to enable machine learning models to effectively maintain and update existing knowledge while continuously receiving new data. With the increasing prevalence of multimodal data, evaluating and optimizing the continuous learning capabilities of these models when processing complex data such as images and text has become increasingly important.

[0003] Continuous learning (CL) refers to learning from a dynamic data distribution. The typical setting for continuous learning is to learn content one by one and behave as if the content were observed simultaneously. Continuous learning is also often referred to as incremental learning or lifelong learning, emphasizing the practical challenges in real-world environments. In practice, training samples from different distributions arrive sequentially. Continuous learning models need to learn the corresponding task with limited or no access to previous training samples and perform well on their test set. In continuous learning, training samples can arrive incrementally, in batches, or simultaneously.

[0004] Continuous learning requires addressing three key issues: First, knowledge retention. We hope that after learning Task 1, the model will retain most of the knowledge from Task 1 after learning Task 2, maintaining test accuracy on Task 1. Second, knowledge transfer. We hope that the model that has learned Task 1 will be able to apply knowledge learned from other tasks, so that it can perform well on Task 2 even without learning Task 2. Third, model expansion. Generally speaking, as the model needs to learn more and more tasks, the model parameters need to expand to a certain extent, but we hope that this expansion of model parameters is efficient.

[0005] The quality of a continuous learning algorithm is primarily measured by observing its performance on certain benchmarks. A continuous learning benchmark is essential for research on continuous learning algorithms. It typically includes a collection of suitable continuous tasks to test a continuous learning algorithm's ability to learn new tasks and combat catastrophic forgetting. These benchmarks typically include single-modality tasks such as language understanding and image recognition. There are two general approaches to designing a continuous learning benchmark: one is to split existing tasks into non-overlapping subtasks, and the other is to integrate different types of tasks.

[0006] In recent years, with the development of multimodal pre-training models, some new benchmarks have begun to emerge to evaluate the performance of these models in the continuous learning process, especially in terms of transfer performance to downstream tasks. However, this evaluation method has certain limitations because it cannot fully reflect the actual performance of the model in a multimodal environment.

[0007] Application Contents

[0008] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a model continuous learning test method, medium and equipment for graphic data processing, which is used to quantify the performance of the continuous learning algorithm of graphic data in a multimodal environment.

[0009] To achieve the above-mentioned purpose and other related purposes, the present application provides a model continuous learning testing method for graphic data processing, the method comprising: obtaining graphic data, wherein the graphic data comprises a combination of multiple groups of pictures and texts; inputting the graphic data into a first model to be tested and a second model to be tested, respectively; obtaining the matrix entropy and matrix conditional entropy of the first model to be tested and the second model to be tested at different times of the first continuous learning task and the second continuous learning task, respectively; and calculating the old knowledge comprehension index and the new knowledge expansion index of the continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first model to be tested and the second model to be tested, respectively.

[0010] In one embodiment of the present application, the obtaining of the matrix entropy and matrix conditional entropy of the first model to be tested and the second model to be tested at different times of the first continuous learning task and the second continuous learning task includes: obtaining the first representation vector matrix and the second representation vector matrix output by the first model to be tested and the second model to be tested at the checkpoints at different times of the first continuous learning task and the second continuous learning task; obtaining the first normalized covariance matrix and the second normalized covariance matrix corresponding to the graphic data based on the first representation vector matrix and the second representation vector matrix; obtaining the corresponding matrix entropy and matrix conditional entropy based on the first normalized covariance matrix and the second normalized covariance matrix.

[0011] In one embodiment of the present application, the first normalized covariance matrix and the second normalized covariance matrix are obtained as follows:

[0012]

[0013] Among them, Σ M；D (i, j) is the element in the i-th row and j-th column of the normalized covariance matrix, M is the model to be tested, D is the data set, N is the number of samples in the data set, z=[z1,…,z N] is the representation vector matrix output by the dataset D after being processed by the model M, z s is the representation vector of the sth sample, is the vector z s The decentralized value is calculated as is the value of the i-th dimension of the representation vector corresponding to the s-th sample after decentralization, is the value of the j-th dimension of the representation vector corresponding to the s-th sample after decentralization.

[0014] In one embodiment of the present application, the matrix entropy is calculated as follows:

[0015]

[0016] The matrix joint entropy is calculated as:

[0017]

[0018] ⊙ refers to Hadamard;

[0019] The matrix conditional entropy is calculated as follows:

[0020]

[0021] in, is the first model to be tested, M' is the first model to be tested, is the matrix entropy, is the first normalized covariance matrix, is the second normalized covariance matrix, is the matrix conditional entropy of the first normalized covariance matrix under the condition that the second normalized covariance matrix is ​​known, is the matrix conditional entropy of the first normalized covariance matrix under the condition that the first normalized covariance matrix is ​​known, is the matrix joint entropy of the first normalized covariance matrix and the second normalized covariance matrix, is the matrix joint entropy of the first normalized covariance matrix and the first normalized covariance matrix.

[0022] In one embodiment of the present application, the method for obtaining the old knowledge understanding indicator of continuous learning is:

[0023]

[0024] The method for obtaining the new knowledge expansion indicator of continuous learning is as follows:

[0025] In one embodiment of the present application, it also includes: obtaining an elasticity-stability trade-off index based on the old knowledge understanding index of the continuous learning and the new knowledge expansion index of the continuous learning; the elasticity-stability trade-off index is the sum of the old knowledge understanding index of the continuous learning and the new knowledge expansion index of the continuous learning.

[0026] To achieve the above-mentioned purpose and other related purposes, the present application also provides a model continuous learning test system for processing graphic and text data, including: a graphic and text data acquisition module, used to acquire graphic and text data, wherein the graphic and text data includes a combination of multiple groups of pictures and texts; a model processing module, used to input the graphic and text data into a first model to be tested and a second model to be tested, respectively, to obtain the matrix entropy and matrix conditional entropy of the first model to be tested and the second model to be tested at different times of the first continuous learning task and the second continuous learning task, respectively, and calculate the old knowledge understanding index and the new knowledge expansion index of the continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first model to be tested and the second model to be tested, respectively.

[0027] To achieve the above-mentioned purpose and other related purposes, the present application also provides an electronic terminal, comprising: a memory for storing a computer program; a processor for running the computer program to implement the steps of the model continuous learning and testing method for graphic data processing as described above.

[0028] To achieve the above-mentioned purpose and other related purposes, the present application also provides a computer storage medium storing program instructions, which, when executed, implement the steps of the model continuous learning and testing method for graphic data processing as described above.

[0029] To achieve the above-mentioned purpose and other related purposes, the present application also provides a computer program product, characterized in that the computer program product includes computer program code, and when the computer program code is run on a computer, the computer implements the method described above.

[0030] As described above, the model continuous learning test method, medium and device for graphic data processing of the present application have the following characteristics:

[0031] Beneficial effects:

[0032] The model continuous learning test method for graphic data processing provided in this application uses matrix entropy as the model representation. By calculating the matrix conditional entropy of two models to be tested, the old knowledge understanding of the previous task after the learning task and the new knowledge expansion index of the previous task after the learning task are obtained. This can quantify the performance of the continuous learning algorithm of graphic data in a multimodal environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0034] Figure 1 Shown is a schematic diagram of the overall process of a model continuous learning and testing method for graphic and text data processing in one embodiment of the present application;

[0035] Figure 2 Shown is a flow chart of obtaining matrix entropy and matrix conditional entropy in a model continuous learning and testing method for graphic data processing in one embodiment of the present application;

[0036] Figure 3 Shown is a schematic diagram of the principle of continuous learning of a model continuous learning test method for graphic data processing in one embodiment of the present application;

[0037] Figure 4 Shown is a schematic diagram of a method for obtaining an old knowledge comprehension indicator for continuous learning in a continuous learning testing method for a model for graphic and text data processing in one embodiment of the present application;

[0038] Figure 5 Shown is a schematic diagram of a method for obtaining new knowledge expansion indicators for continuous learning in a continuous learning testing method for a model for graphic and text data processing in an embodiment of the present application;

[0039] Figure 6 Shown is a principle block diagram of a model continuous learning and testing system for graphic and text data processing in one embodiment of the present application;

[0040] Figure 7 2 is a principle block diagram of an electronic terminal in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following describes the implementation of this embodiment through specific examples. Those skilled in the art can easily understand the other advantages and effects of this embodiment from the content disclosed in this specification. This embodiment can also be implemented or applied through different specific implementations. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this embodiment. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless there is a conflict.

[0042] The model continuous learning test method for graphic data processing in this embodiment is based on CLiMB, a multimodal continuous learning benchmark, which includes multiple language and vision datasets as well as single language and single vision datasets. The multimodal task serves as upstream training data for downstream single modal tasks. In the upstream training stage, CLiMB tests the ability of the algorithm to migrate to unseen tasks after a series of training tasks, as well as its ability to resist forgetting tasks seen in the past. In the downstream transfer learning stage, CLiMB mainly tests the low-sample transfer learning ability of the algorithm.

[0043] To study the anti-forgetting and knowledge transfer capabilities of continuous learning algorithms on multimodal tasks, CLiMB includes four vision-language tasks, five language-only tasks, and four image-only tasks. The multimodal vision-language tasks are used for upstream continuous learning, while the unimodal tasks are used for downstream transfer learning.

[0044] During the upstream continuous learning phase, the model M to be tested sequentially learns K multimodal tasks using CL algorithm A. While the model cannot access the data for tasks 1…i-1 when learning task i, algorithm A can allocate memory space to store data from previous tasks. This upstream phase primarily tests two aspects of the model's performance: knowledge transfer from previous tasks to new ones, and its ability to resist forgetting for previously seen tasks.

[0045] In the downstream transfer learning stage, we mainly test the low-sample transfer learning ability of model M on downstream unimodal tasks after learning upstream multimodal task i. We test this ability by fine-tuning the encoder of task i checkpoint.

[0046] The performance of upstream tasks is measured by upstream knowledge transfer index and forgetting transfer index. The calculation of upstream knowledge transfer index is based on the score of pre-trained model fine-tuned directly on upstream task i.

[0047] and the score of the continuous learning algorithm A after fine-tuning on tasks 1…i and random benchmarks

[0048] The calculation formula is:

[0049] After the model has learned task j, the forgetting transfer index for the previous task i is calculated using the following formula:

[0050]

[0051] in The model is tested directly on task i after learning upstream tasks 1…j through the continuous learning algorithm A (without fine-tuning).

[0052] These two metrics effectively quantify the transfer learning algorithm's ability to combat forgetting and its impact on future tasks. They fill a gap in the benchmarks for continuous learning algorithms in multimodal pre-training and establish a framework for studying the multimodal performance of continuous learning algorithms. They systematically evaluate how quickly upstream continuous learning generalizes to new multimodal and unimodal tasks.

[0053] After testing existing continuous learning algorithms using the aforementioned continuous learning benchmark indicators, it was found that these algorithms failed to have a positive impact on the learning of task i after the model learned from task 1 to task i-1. Therefore, this embodiment considers whether the continuous learning algorithm can generate positive gains for the data representation layer. However, current methods generally treat the model as a whole and fail to effectively distinguish the performance of the continuous learning algorithm in terms of the representation layer, which has become a bottleneck for further improving the algorithm's effectiveness.

[0054] This embodiment provides a model continuous learning test method, medium, and device for graphic data processing, which are used to quantify the performance of a continuous learning algorithm for graphic data in a multimodal environment.

[0055] The principles and implementation methods of the model continuous learning test method, medium and equipment for graphic data processing of this embodiment will be explained in detail below, so that those skilled in the art can understand the model continuous learning test method, medium and equipment for graphic data processing of this embodiment without creative work.

[0056] Specifically, this application provides a model continuous learning and testing method for graphic data processing. Figure 1 Shown is a schematic diagram of the overall process of the model continuous learning test method for graphic data processing in one embodiment of the present application; Figure 1 As shown, the method includes the following steps S100 to S200.

[0057] Step S100, obtaining graphic and text data, wherein the graphic and text data includes a combination of multiple groups of pictures and texts;

[0058] Step S200, inputting the graphic data into the first model to be tested and the second model to be tested respectively;

[0059] Step S300, obtaining matrix entropy and matrix conditional entropy of the first model to be tested and the second model to be tested at different times in the first continuous learning task and the second continuous learning task respectively;

[0060] Step S400 , calculating an old knowledge understanding index and a new knowledge expansion index of continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first model to be tested and the second model to be tested, respectively.

[0061] The model continuous learning test method for graphic data processing of this embodiment is improved based on the multimodal pre-training continuous learning benchmark CLiMB. Specifically, under the setting of vision-language tasks, common continuous learning (CL) algorithms cannot promote knowledge transfer and often have a negative impact on the learning of future tasks. The purpose of the model continuous learning test method for graphic data processing of this embodiment is to design indicators that can quantify the performance of continuous learning (CL) algorithms at the data representation layer.

[0062] Unlike existing techniques that use the model's scores on each task to calculate forgetting and knowledge transfer, this embodiment's continuous learning testing method for image and text data processing utilizes matrix entropy as a model representation. By calculating the matrix conditional entropy of the two models under test, it derives metrics for understanding the previous knowledge of the image and text data in the previous task after learning task i and for expanding the new knowledge of the image and text data in the previous task after learning task i.

[0063] The following describes in detail the above steps S100 and S400 in the model continuous learning test method for graphic and text data processing of this embodiment.

[0064] Step S100: Obtain graphic and text data, where the graphic and text data includes multiple sets of images and text. For example, one set of image and text may include a picture of a dog and a description: "This is a picture." In this embodiment, multiple sets of graphic and text data including multiple sets of image and text are constructed as a data test set.

[0065] Step S200: input the graphic data into the first model to be tested and the second model to be tested respectively.

[0066] The first and second models under test in this embodiment are both pre-trained models, ViLT (Vision-and-Language Transformer), used in the CLiMB multimodal continuous learning benchmark. During the upstream training phase, CLiMB primarily tests the algorithm's ability to transfer to unseen tasks after a series of training tasks, as well as its ability to resist forgetting previously seen tasks. During the downstream transfer learning phase, CLiMB primarily tests the algorithm's low-sample transfer learning capabilities.

[0067] Step S300: Obtain matrix entropy and matrix conditional entropy of the first model to be tested and the second model to be tested at different times in the first continuous learning task and the second continuous learning task, respectively.

[0068] Figure 2 Shown is a flow chart of obtaining matrix entropy and matrix conditional entropy in a model continuous learning test method for graphic data processing in an embodiment of the present application; Figure 2As shown, in a specific implementation of this embodiment, obtaining the matrix entropy and matrix conditional entropy of the first model to be tested and the second model to be tested at different times of the first continuous learning task and the second continuous learning task includes:

[0069] Step S310: Obtain a first representation vector matrix and a second representation vector matrix output by the first model to be tested and the second model to be tested at checkpoints at different times of the first continuous learning task and the second continuous learning task, respectively;

[0070] Step S320: acquiring a first normalized covariance matrix and a second normalized covariance matrix corresponding to the graphic data based on the first representation vector matrix and the second representation vector matrix;

[0071] Step S330: Obtain corresponding matrix entropy and matrix conditional entropy based on the first normalized covariance matrix and the second normalized covariance matrix.

[0072] Step S400 , calculating an old knowledge understanding index and a new knowledge expansion index of continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first model to be tested and the second model to be tested, respectively.

[0073] Figure 3 Shown is a schematic diagram of the principle of continuous learning of a model continuous learning test method for graphic data processing in one embodiment of the present application; Figure 3 As shown, the representation vectors of the two models to be tested (usually the models are located at different checkpoints of the continuous learning algorithm) on the reference dataset are extracted, the covariance matrix is ​​calculated, and the matrix entropy and conditional entropy of the matrix are calculated using the covariance matrix. Preferably, the matrix entropy and conditional entropy of the two model checkpoints on the reference dataset are used to calculate the old knowledge understanding and new knowledge acquisition indicators in this embodiment.

[0074] In a specific implementation of this embodiment, the first normalized covariance matrix and the second normalized covariance matrix are obtained as follows:

[0075]

[0076]

[0077] Among them, Σ M；D (i, j) is the element in the i-th row and j-th column of the normalized covariance matrix, M is the model to be tested, D is the data set, N is the number of samples in the data set, z=[z1,…,z N ] is the representation vector matrix output by the dataset D after being processed by the model M, z s is the representation vector of the sth sample, is the vector z s The decentralized value is calculated as is the value of the i-th dimension of the representation vector corresponding to the s-th sample after decentralization, is the value of the j-th dimension of the representation vector corresponding to the s-th sample after decentralization.

[0078] In a specific implementation of this embodiment, the matrix entropy is calculated as follows:

[0079]

[0080] The matrix conditional entropy is calculated as follows:

[0081] H(K m+1 |K1,…,K m )=H(K1,K2,…,K m+1 )-H(K1,K2,…K m ).

[0082] The matrix joint entropy is calculated as:

[0083]

[0084] ⊙ refers to Hadamard.

[0085] Therefore, the matrix entropy of the first model M to be tested and the second model M' to be tested is The corresponding matrix conditional entropy is

[0086] in, is the first model to be tested, M' is the first model to be tested, is the matrix entropy, is the first normalized covariance matrix, is the second normalized covariance matrix. is the matrix conditional entropy of the first normalized covariance matrix under the condition that the second normalized covariance matrix is ​​known, is the matrix conditional entropy of the first normalized covariance matrix under the condition that the first normalized covariance matrix is ​​known, is the matrix joint entropy of the first normalized covariance matrix and the second normalized covariance matrix, is the matrix joint entropy of the first normalized covariance matrix and the first normalized covariance matrix.

[0087] Figure 4 Shown is a schematic diagram of a method for obtaining the old knowledge understanding index of continuous learning in the continuous learning test method of the model for graphic data processing in one embodiment of the present application; Figure 4As shown, in a specific implementation of this embodiment, the method for obtaining the old knowledge understanding indicator of continuous learning is:

[0088]

[0089] Figure 5 Shown is a schematic diagram of a method for obtaining new knowledge expansion indicators for continuous learning in a continuous learning test method for a model used for graphic data processing in an embodiment of the present application; Figure 5 As shown, the method for obtaining the new knowledge expansion indicator of continuous learning is:

[0090] In a specific implementation method of this embodiment, it also includes: obtaining an elasticity-stability trade-off index based on the old knowledge understanding index of the continuous learning and the new knowledge expansion index of the continuous learning; the elasticity-stability trade-off index is the sum of the old knowledge understanding index of the continuous learning and the new knowledge expansion index of the continuous learning.

[0091] That is, the elasticity-stability trade-off index of the calculation model is calculated by the old knowledge understanding and new knowledge expansion indicators, and the calculation formula is the old knowledge understanding index OKU + the new knowledge expansion index NKE.

[0092] The training results of the model continuous learning test method for graphic data processing in this embodiment for upstream tasks are shown in Tables 1 to 4.

[0093] Table 1 Continuous Learning Algorithm: Elastic Weight Consolidation (EWC) Upstream Task Test Results

[0094]

[0095] Table 2 Continuous Learning Algorithm: Sequential Fine-tuning (Seqential-FT) Upstream Task Test Results

[0096]

[0097] Table 3 Continuous Learning Algorithm: Experience-Replay Upstream Task Test Results

[0098]

[0099] Table 4 Continuous Learning Algorithm: Freeze-Bottom-9layers Upstream Task Test Results

[0100]

[0101] The English definitions and data descriptions in the above table are as follows:

[0102] Task test task (dataset), Score Type scoring type, Forg forgetting score, Repre representation layer score, Repeat-Head reuses the original classification head, Re-Head retrains the classification head.

[0103] The indicators represented by the data points are as follows:

[0104] The triples of data points with the rating type Forg (forgotten rating) are [original task rating, current task rating, rating ratio].

[0105] The triples of data points with the scoring type of Repre (representation layer scoring) are (old knowledge understanding index, new knowledge expansion index, elasticity).

[0106] There are five Task test tasks, including VQAv2 (Visual Question Answering v2), NLVR2 (Natural Language for Visual Reasoning 2), SNLI-VE (Stanford Natural Language Inference for Visual Entailment), VCR (Visual Common Reasoning), and BMC (Bone Marrow Cytomorphology).

[0107] The model continuous learning test method for graphic data processing of this embodiment is used to test common continuous learning algorithms (Elastic Weight Consolidation, Freeze bottom k layers, Sequential Finetuning, Experience Replay). Repre represents the three indicators defined in this embodiment: old knowledge understanding, new knowledge expansion and plasticity. Forg represents the forgetting indicator defined by the CLiMB scheme. The old knowledge understanding and new knowledge expansion indicators are calculated by the conditional entropy of the covariance matrix of the calculation model on the same reference task through the formula. The same row in Tables 1 to 4 above uses the same reference task. For example, the first row of each data table uses VQAv2 as the reference task.

[0108] The datasets used in this example include VQAv2, NLVR, SNLI-VE, VCR, and BMC (Bone Marrow Cytomorphology). BMC is a bone marrow cytomorphology classification dataset containing over 170,000 de-identified, expert-annotated cell images from bone marrow smears of 945 patients using May-Grünwald-Giemsa / Pappenheim staining.

[0109] In the following description, this embodiment refers to the reference task as the old task, and the task after the reference task as the new task. As can be seen from Tables 1 to 4, for the old knowledge understanding index, a value close to 1 means that the model retains most of the representations of the old task after fine-tuning the subsequent tasks, so that when the retrained classification head (re-head) is tested on the old task, the model can restore most of the performance of the old task. An old knowledge understanding index (Oldknowledge usage) less than 1 means that after the model is fine-tuned for the subsequent tasks, a part of the representation of the old task is lost, so that the result of re-testing the old task by retraining the classification head has an observable gap with the performance of the model that has just been trained on the old task.

[0110] For the new knowledge extension metric, a value greater than 0 means that the model has gained measurable representations from learning the new task, and the accuracy of the task has also improved. A value less than 0 means that the model has not gained any new knowledge from training on the new task, and the accuracy on the reference task will not improve accordingly.

[0111] The training results of the model continuous learning test method for graphic data processing in this embodiment for downstream tasks are shown in Tables 5 to 8.

[0112] Table 5 Continuous learning algorithm: Elastic Weight Consolidation (EWC) downstream task test results and NKE trend

[0113]

[0114] Table 6 Continuous Learning Algorithm: Sequential Fine-tuning (Seqential-FT) Downstream Task Test Results and NKE Trends

[0115]

[0116] Table 7 Continuous Learning Algorithm: Experience-Replay Downstream Task Test Results and NKE Trends

[0117]

[0118] Table 8 Continuous Learning Algorithm: Freeze-Bottom-9layers Downstream Task Test Results and NKE Trends

[0119]

[0120] The descriptions of Tables 5 to 8 are as follows:

[0121] There are five upstream continuous learning tasks, including VQAv2 (Visual Question Answering v2, second version), NLVR2 (Natural Language for Visual Reasoning 2, natural language dataset for visual reasoning 2), SNLI-VE (Stanford Natural Language Inference for Visual Entailment), VCR (Visual Common Reasoning), and BMC (Bone Marrow Cytomorphology).

[0122] NKE represents the new knowledge acquisition index, NKE(Cifar-10) represents the new knowledge acquisition index using the Cifar-10 dataset as the reference dataset D, OKU represents the old knowledge understanding, Cifar-10 represents the accuracy score of the Cifar-10 test set for the downstream task, and FashionMNIST represents the accuracy score of the FashionMNIST test set for the downstream task.

[0123] Language data set:

[0124] SST-2 indicates the test accuracy of the single-language dataset SST-2 when the downstream task is used, and IMDB indicates the test accuracy of the single-language dataset IMDB when the downstream task is used.

[0125] On the downstream tasks Cifar-10 and FashionMNIST, the changing trend of NKE is basically the same as the changing trend of downstream task accuracy, that is, when NKE is positive at a certain moment, the downstream task accuracy from the previous moment to the current moment is increasing. This trend is first more obvious in the Freeze-bottom-9-layers algorithm. The data points marked in yellow are in line with this trend. Because Freeze-bottom-9-layers is significantly different from the other algorithms, the change in downstream task accuracy is uncertain when NKE changes slightly. Therefore, among the other algorithms, after using the unified reference data Cifar-10 in this embodiment, the change in NKE can also be correctly reflected in the change in downstream task accuracy.

[0126] like Figure 4 and Figure 5 As shown, the specific implementation process of this embodiment is as follows:

[0127] First, a set of images and text is fed into the Vision-Language Transformer (ViLT) models M and M', which have been fine-tuned by the continuous learning tasks T and T', respectively. This set of images and text contains several pairs of images and text, such as a picture of a dog and a description: "This is a picture."

[0128] After obtaining the representation vectors z and z' output by the model at different checkpoints at the time of the continuous learning task, their normalized covariance matrices ∑ M；D and ∑ M′；D The calculation formula is:

[0129]

[0130] in

[0131] Then according to the two normalized covariance matrices ∑ M；D and ∑ M′；D , the new knowledge acquisition index NKE of this embodiment is calculated using the formula of matrix entropy and matrix conditional entropy.

[0132] The calculation formula of matrix entropy is:

[0133]

[0134] The calculation formula of matrix conditional entropy is:

[0135] H(K m+1 |K1,…,K m )=H(K1,K2,…,K m+1 )-H(K1,K2,…,Km ).

[0136] in:

[0137] H(K1,…,K T )=H(K1☉K2☉…☉K T ),

[0138] ⊙ refers to Hadamard.

[0139] Finally, according to Figure 4 and Figure 5 These two indicators are calculated using the NKE and OKU formulas in .

[0140] The protection scope of the model continuous learning and testing method for graphic data processing described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing, or replacing steps in the existing technology based on the principles of the present application are included in the protection scope of the present application.

[0141] This embodiment also provides a model continuous learning and testing system for graphic and text data processing. Figure 6 Shown is a principle block diagram of a model continuous learning test system for graphic data processing in one embodiment of the present application; Figure 6 As shown, the model continuous learning and testing system for graphic and text data processing includes: a graphic and text data acquisition module and a model processing module.

[0142] Specifically, the graphic data acquisition module is used to acquire graphic data, wherein the graphic data includes a combination of multiple groups of pictures and texts; the model processing module is used to input the graphic data into the first model to be tested and the second model to be tested respectively, obtain the matrix entropy and matrix conditional entropy of the first model to be tested and the second model to be tested at different times of the first continuous learning task and the second continuous learning task respectively, and calculate the old knowledge understanding index and the new knowledge expansion index of continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first model to be tested and the second model to be tested respectively.

[0143] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0144] It should also be understood that the division of modules in the embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present application may be integrated into a single processor, or may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0145] Figure 7 : is a schematic block diagram of an electronic terminal provided in an embodiment of the present application. Figure 7 As shown, the electronic terminal includes: at least one processor 401, a memory 402, at least one network interface 403 and a user interface 405. The various components in the electronic terminal are coupled together via a bus system 404. It is understood that the bus system 404 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 404 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 404 is not shown in FIG. Figure 7 Various buses are labeled as bus systems.

[0146] The user interface 405 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0147] It will be appreciated that the memory 402 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM) or a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0148] The memory 402 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 400. Examples of such data include: any executable program for operating on the electronic terminal 400, such as an operating system 4021 and an application 4022; the operating system 4021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 4022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The implementation of the XX method provided in the embodiment of the present invention can be included in the application 4022.

[0149] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 401 or by software instructions. The above processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor 401 may be a microprocessor or any conventional processor. The steps of the accessory optimization method provided in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium located in a memory. The processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0150] In an exemplary embodiment, the electronic terminal 400 may be configured to execute the aforementioned method using one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs).

[0151] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: a computer program code, which, when executed on a computer, causes the computer to execute Figures 1 to 5 The method of any of the embodiments shown.

[0152] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code, and when the program code is run on a computer, the computer executes Figures 1 to 5 The method of any of the embodiments shown.

[0153] As used in this specification, the terms "component," "module," "system," and the like are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a computing device and a computing device can be a component. One or more components can reside in a process and / or an execution thread, and a component can be located on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures stored thereon. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component on a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0154] Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, electronic terminals and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed systems, electronic terminals, and methods can be implemented in other ways. For example, the electronic terminal embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the electronic terminal or unit can be electrical, mechanical, or other forms.

[0157] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0159] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (program) are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable electronic terminal. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state drives (SSDs)).

[0160] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store program codes.

[0161] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0162] In summary, the model continuous learning testing method for graphic data processing provided by this application utilizes matrix entropy as a model representation. By calculating the matrix conditional entropy of two models under test, it derives indicators of the understanding of previous knowledge from the previous task after the learning task and the expansion of new knowledge from the previous task after the learning task. This method can quantify the performance of continuous learning algorithms for graphic data in multimodal environments. Therefore, this application effectively overcomes the various shortcomings of the existing technology and has high industrial application value.

[0163] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A model continuous learning test method for graphic data processing, characterized in that: The method comprises: Acquire graphic data, wherein the graphic data includes a combination of multiple groups of pictures and texts; Inputting the graphic data into the first model to be tested and the second model to be tested respectively; Obtain matrix entropy and matrix conditional entropy of the first model to be tested and the second model to be tested at different times of the first continuous learning task and the second continuous learning task respectively; An old knowledge understanding index and a new knowledge expansion index of continuous learning are calculated based on the matrix entropy and the matrix conditional entropy corresponding to the first model to be tested and the second model to be tested, respectively.

2. The model continuous learning and testing method for graphic and text data processing according to claim 1 is characterized in that: The obtaining of the matrix entropy and the matrix conditional entropy of the first model to be tested and the second model to be tested at different times of the first continuous learning task and the second continuous learning task respectively comprises: Obtaining a first representation vector matrix and a second representation vector matrix respectively output by the first model to be tested and the second model to be tested at checkpoints at different times of the first continuous learning task and the second continuous learning task; Acquire a first normalized covariance matrix and a second normalized covariance matrix corresponding to the graphic data based on the first representation vector matrix and the second representation vector matrix; Corresponding matrix entropy and matrix conditional entropy are obtained based on the first normalized covariance matrix and the second normalized covariance matrix.

3. The model continuous learning and testing method for graphic data processing according to claim 2 is characterized in that: The first normalized covariance matrix and the second normalized covariance matrix are obtained as follows: Among them, Σ M；D (i, j) is the element in the i-th row and j-th column of the normalized covariance matrix, M is the model to be tested, D is the data set, N is the number of samples in the data set, z = [z1,…,z N ] is the representation vector matrix output by the dataset D after being processed by the model M, z s is the representation vector of the sth sample, is the vector z s The decentralized value is calculated as is the value of the i-th dimension of the representation vector corresponding to the s-th sample after decentralization, is the value of the j-th dimension of the representation vector corresponding to the s-th sample after decentralization.

4. The model continuous learning test method for graphic data processing according to claim 2 or 3, characterized in that: The matrix entropy is calculated as follows: The matrix joint entropy is calculated as: ⊙ refers to Hadamard; The matrix conditional entropy is calculated as follows: in, is the first model to be tested, M' is the first model to be tested, is the matrix entropy, is the first normalized covariance matrix, is the second normalized covariance matrix, is the matrix conditional entropy of the first normalized covariance matrix when the second normalized covariance matrix is ​​known, is the matrix conditional entropy of the first normalized covariance matrix under the condition that the first normalized covariance matrix is ​​known, is the matrix joint entropy of the first normalized covariance matrix and the second normalized covariance matrix, is the matrix joint entropy of the first normalized covariance matrix and the first normalized covariance matrix.

5. The model continuous learning and testing method for graphic and text data processing according to claim 4 is characterized in that: The method of obtaining the old knowledge understanding index of the continuous learning is as follows: The method for obtaining the new knowledge expansion index of the continuous learning is as follows:

6. The model continuous learning and testing method for graphic and text data processing according to claim 1 is characterized in that: Also includes: A flexibility-stability trade-off index is obtained based on the old knowledge understanding index of the continuous learning and the new knowledge expansion index of the continuous learning; the flexibility-stability trade-off index is the sum of the old knowledge understanding index of the continuous learning and the new knowledge expansion index of the continuous learning.

7. A model continuous learning and testing system for graphic and text data processing, characterized in that: include: A graphic data acquisition module, used to acquire graphic data, wherein the graphic data includes a combination of multiple groups of pictures and texts; A model processing module is used to input the graphic data into the first model to be tested and the second model to be tested respectively, obtain the matrix entropy and matrix conditional entropy of the first model to be tested and the second model to be tested at different times of the first continuous learning task and the second continuous learning task respectively, and calculate the old knowledge comprehension index and the new knowledge expansion index of continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first model to be tested and the second model to be tested respectively.

8. An electronic terminal, characterized in that: include: comprising a memory for storing a computer program; A processor is used to run the computer program to implement the steps of the model continuous learning testing method for graphic data processing as described in any one of claims 1 to 6.

9. A computer storage medium storing program instructions, characterized in that: When the program instructions are executed, the steps of the model continuous learning test method for graphic data processing as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product includes computer program codes, and when the computer program codes are executed on a computer, the computer implements the steps of the model continuous learning and testing method for graphic and text data processing as described in any one of claims 1 to 6.

Images