Model continual learning test method for text data processing, medium and device

Abstract

The application provides a model continuous learning test method, medium and equipment for image-text data processing, the method comprises the following steps: obtaining image-text data, wherein the image-text data comprises a combination of multiple groups of pictures and texts; inputting the image-text data into a first to-be-tested model and a second to-be-tested model respectively; obtaining the matrix entropy and the matrix conditional entropy of the first to-be-tested model and the second to-be-tested model at different time points of a first continuous learning task and a second continuous learning task respectively; and calculating an old knowledge understanding index and a new knowledge expansion index of continuous learning based on the corresponding matrix entropy and matrix conditional entropy respectively. The model continuous learning test method for image-text data processing provided by the application uses matrix entropy as model performance, calculates the matrix conditional entropy of two to-be-tested models, obtains the old knowledge understanding of the previous task after the learning task and the new knowledge expansion index of the previous task after the learning task, and can quantify the performance of the continuous learning algorithm of the image-text data in a multi-modal environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to the technical field of training model for processing graphic-text data. BACKGROUND

[0002] Learning is the foundation of intelligent systems adapting to dynamic environments. Humans and other living beings demonstrate strong adaptability by continuously acquiring, updating, accumulating, and utilizing knowledge. It is desirable for artificial intelligence systems to adapt in a similar manner, which has prompted research into continual learning. In the field of artificial intelligence, continual learning is a key research topic that aims to enable machine learning models to effectively maintain and update existing knowledge while continuously receiving new data. With the increasing popularity of multi-modal data, it is particularly important to evaluate and optimize the continual learning capabilities of these models when processing complex data such as graphic-text.

[0003] Continual learning (CL) refers to learning from dynamic data distributions. The typical setting of continual learning is to learn content one by one and behave as if these contents are observed simultaneously. Continual learning is also commonly known as incremental learning or lifelong learning, emphasizing practical challenges in real-world environments. In practice, training samples of different distributions arrive one after another, and continual learning models need to learn corresponding tasks without access to old training samples or with limited access. In continual learning, training samples can arrive in batches in an incremental manner or simultaneously.

[0004] Continual learning needs to address three problems: first, knowledge retention, which requires a model that has learned task one to retain most of the knowledge of task one after learning task two, i.e., to maintain the test accuracy on task one. Second, knowledge transfer, which requires a model that has learned task one to be able to transfer knowledge even without learning task two, i.e., to perform well on task two. Third, model expansion, which generally requires the expansion of model parameters to learn more and more tasks, but we want the expansion of model parameters to be efficient.

[0005] The performance of a continual learning algorithm is mainly evaluated by observing its performance on certain benchmark programs. Continual learning benchmarks are necessary for the research of continual learning algorithms, as they typically gather suitable consecutive tasks for testing the ability of continual learning algorithms to learn new tasks and combat "catastrophic forgetting". These benchmark tests usually include single-modal tasks such as language understanding and image recognition. There are generally two approaches to designing a continual learning benchmark: one is to split existing tasks into mutually exclusive subtasks, and the other is to integrate different types of tasks.

[0006] In recent years, with the development of multi-modal pre-training models, some new benchmarks have emerged to evaluate the performance of these models in the process of continual learning, especially in the transfer performance of downstream tasks. However, this evaluation method has certain limitations as it cannot fully reflect the actual performance of the model in a multi-modal environment.

[0007] SUMMARY

[0008] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a model continual learning test method for image-text data processing, medium and equipment, which is used to quantify the performance of the continual learning algorithm of image-text data in a multi-modal environment.

[0009] To achieve the above-mentioned purpose and other related purposes, the present application provides a model continual learning test method for image-text data processing, the method comprising: obtaining image-text data, wherein the image-text data contains a combination of multiple groups of pictures and text; inputting the image-text data into a first to-be-tested model and a second to-be-tested model respectively; obtaining matrix entropy and matrix conditional entropy of the first to-be-tested model and the second to-be-tested model at different time points of a first continual learning task and a second continual learning task respectively; calculating an old knowledge understanding index and a new knowledge expansion index of continual learning based on the matrix entropy and the matrix conditional entropy corresponding to the first to-be-tested model and the second to-be-tested model respectively.

[0010] In an embodiment of the present application, the obtaining of the matrix entropy and the matrix conditional entropy of the first to-be-tested model and the second to-be-tested model at different time points of the first continual learning task and the second continual learning task comprises: obtaining first representation vector matrices and second representation vector matrices output by the first to-be-tested model and the second to-be-tested model at different checkpoint time points of the first continual learning task and the second continual learning task respectively; obtaining first normalized covariance matrices and second normalized covariance matrices corresponding to the image-text data based on the first representation vector matrices and the second representation vector matrices; obtaining corresponding matrix entropy and matrix conditional entropy based on the first normalized covariance matrices and the second normalized covariance matrices.

[0011] In an embodiment of the present application, the first normalized covariance matrix and the second normalized covariance matrix are obtained in the following manner:

[0012]

[0013] wherein, M；D (i,j) is the element of the i-th row and the j-th column of the normalized covariance matrix, M is the to-be-tested model, D is the data set, N is the number of samples in the data set, z=[z1,…,z Nis a representation vector matrix output by the data set D after being processed by the model M, z s is a representation vector of the s-th sample, is a representation vector of the s-th sample, s is a decentered value, and the calculation formula is is a value of the i-th dimension of the representation vector corresponding to the s-th sample after being decentered, is a value of the j-th dimension of the representation vector corresponding to the s-th sample after being decentered.

[0014] In an embodiment of the present application, the calculation method of the matrix entropy is as follows:

[0015]

[0016] The calculation method of the matrix joint entropy is as follows:

[0017]

[0018] ⊙ indicates Hadamard product;

[0019] The calculation method of the matrix conditional entropy is as follows:

[0020]

[0021] wherein, is a first to-be-tested model, M' is a second to-be-tested model, is a matrix entropy, is a first normalized covariance matrix, is a second normalized covariance matrix, is a matrix conditional entropy of the first normalized covariance matrix under the condition that the second normalized covariance matrix is known, is a matrix conditional entropy of the first normalized covariance matrix under the condition that the first normalized covariance matrix is known, is a matrix joint entropy of the first normalized covariance matrix and the second normalized covariance matrix, is a matrix joint entropy of the first normalized covariance matrix and the first normalized covariance matrix.

[0022] In an embodiment of the present application, the acquisition method of the old knowledge understanding index of the continuous learning is as follows:

[0023]

[0024] The acquisition method of the new knowledge expansion index of the continuous learning is as follows:

[0025]

[0026] In an embodiment of the present application, the method further comprises: obtaining an elasticity-stability trade-off indicator based on the old knowledge understanding indicator of the continuous learning and the new knowledge expansion indicator of the continuous learning; and the elasticity-stability trade-off indicator is a sum of the old knowledge understanding indicator of the continuous learning and the new knowledge expansion indicator of the continuous learning.

[0027] To achieve the above object and other related objects, the present application further provides a model continuous learning test system for image-text data processing, comprising: an image-text data acquisition module, configured to acquire image-text data, wherein the image-text data comprises a combination of multiple groups of pictures and texts; and a model processing module, configured to input the image-text data into a first to-be-tested model and a second to-be-tested model respectively, acquire matrix entropy and matrix conditional entropy of the first to-be-tested model and the second to-be-tested model at different time points of a first continuous learning task and a second continuous learning task respectively, and calculate an old knowledge understanding indicator and a new knowledge expansion indicator of continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first to-be-tested model and the second to-be-tested model respectively.

[0028] To achieve the above object and other related objects, the present application further provides an electronic terminal, comprising: a memory configured to store a computer program; and a processor configured to run the computer program to implement the steps of the model continuous learning test method for image-text data processing.

[0029] To achieve the above object and other related objects, the present application further provides a computer storage medium, which stores program instructions, and the program instructions are executed to implement the steps of the model continuous learning test method for image-text data processing.

[0030] To achieve the above object and other related objects, the present application further provides a computer program product, characterized in that the computer program product comprises computer program code, and when the computer program code is run on a computer, the computer is caused to implement the method.

[0031] As described above, the model continuous learning test method for image-text data processing, medium and device provided by the present application have the following beneficial effects:

[0032] The model continuous learning test method for image-text data processing provided by the present application uses matrix entropy as model performance, calculates the matrix conditional entropy of two to-be-tested models, obtains an old knowledge understanding indicator of a previous task after a learning task and a new knowledge expansion indicator of the previous task after the learning task, and can quantify the performance of a continuous learning algorithm of image-text data in a multi-modal environment. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings.

[0034] Figure 1 The overall flowchart of the model continuous learning test method for processing graphic and text data in an embodiment of the present application is shown.

[0035] Figure 2 The flowchart of obtaining matrix entropy and matrix conditional entropy in the model continuous learning test method for processing graphic and text data in an embodiment of the present application is shown.

[0036] Figure 3 The principle diagram of continuous learning in the model continuous learning test method for processing graphic and text data in an embodiment of the present application is shown.

[0037] Figure 4 The acquisition method diagram of the old knowledge understanding index of continuous learning in the model continuous learning test method for processing graphic and text data in an embodiment of the present application is shown.

[0038] Figure 5 The acquisition method diagram of the new knowledge expansion index of continuous learning in the model continuous learning test method for processing graphic and text data in an embodiment of the present application is shown.

[0039] Figure 6 The principle block diagram of the model continuous learning test system for processing graphic and text data in an embodiment of the present application is shown.

[0040] Figure 7 The principle block diagram of the electronic terminal in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] The implementation manners of the embodiments will be described below through specific and concrete examples. Those skilled in the art can easily understand other advantages and effects of the embodiments from the content disclosed in the present specification. The embodiments can also be implemented or applied through other different specific implementation manners, and each detail in the present specification can be modified or changed based on different views and applications without departing from the spirit of the embodiments. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0042] The model continuous learning test method for image-text data processing of the embodiment is based on CLiMB, a multi-modal continuous learning benchmark, which includes multiple language-visual data sets and single language and single visual data sets. Among them, the multi-modal task is used as the upstream training data of the downstream single-modal task. In the upstream training stage, the CLiMB tests the ability of the algorithm to migrate to a never-seen task after a series of training tasks, and the ability to resist forgetting the past seen tasks. In the downstream transfer learning stage, the CLiMB mainly tests the low-sample transfer learning ability of the algorithm.

[0043] In order to study the anti-forgetting and knowledge transfer ability of the continuous learning algorithm on the multi-modal task, CLiMB includes 4 visual-lingual tasks, 5 only language tasks and 4 only image tasks. Among them, the multi-modal visual-lingual task is used for upstream continuous learning, and the single-modal task is used for downstream transfer learning stage.

[0044] In the upstream continuous learning stage, the model M to be tested uses the CL algorithm A to learn K multi-modal tasks in turn. The model cannot obtain the data of tasks 1…i-1 when learning task i, but the algorithm A can allocate memory space to store the data of the previous tasks. In the upstream stage, the performance of the model is mainly tested in two aspects: one is the knowledge transfer property from old tasks to new tasks, and the other is the anti-forgetting property of the already seen tasks.

[0045] In the downstream transfer learning stage, the low-sample transfer learning ability of the model M after learning the upstream multi-modal task i on the downstream single-modal task is mainly tested. We test this ability by fine-tuning the encoder of the task i checkpoint.

[0046] The upstream task performance is measured by the upstream knowledge transfer index and the forgetting transfer index. The calculation of the upstream knowledge transfer index is based on the score of the pre-trained model directly fine-tuned on the upstream task i

[0047] and the score of the model fine-tuned on tasks 1…i using the continuous learning algorithm A and the random benchmark

[0048] The formula is:

[0049] The forgetting transfer index of the model after learning task j for the previous task i is calculated by the following formula:

[0050]

[0051] Wherein is obtained by directly testing the model after learning the upstream tasks 1…j using the continuous learning algorithm A on task i (without fine-tuning).

[0052] The two indicators above quantify the performance of the transfer learning algorithm against forgetting and the gain effect on future tasks, fill the blank of the multi-modal pre-training of the continuous learning algorithm benchmark, and create a framework for researching the performance of continuous learning algorithms on multi-modal. Systematically evaluate how upstream continuous learning quickly generalizes to new multi-modal and single-modal tasks.

[0053] After testing the existing continuous learning algorithms by the above continuous learning benchmark indicators, it is found that these algorithms fail to have a positive impact on the learning of task i after the model has learned tasks 1 to task i-1. Therefore, the embodiment considers whether the continuous learning algorithm can have a positive gain on the data representation layer. However, the current method generally regards the model as a whole and fails to effectively distinguish the performance of the continuous learning algorithm in the representation layer, which becomes a bottleneck for further improving the algorithm effect.

[0054] The embodiment provides a model continuous learning test method, medium and equipment for image-text data processing for quantifying the performance of continuous learning algorithms for image-text data in a multi-modal environment.

[0055] The principles and implementation modes of the model continuous learning test method, medium and equipment for image-text data processing of the embodiment will be described in detail below, so that those skilled in the art can understand the model continuous learning test method, medium and equipment for image-text data processing of the embodiment without creative labor.

[0056] Specifically, the present application provides a model continuous learning test method for image-text data processing, Figure 1 The figure shows the overall flowchart of the model continuous learning test method for image-text data processing in an embodiment of the present application; as Figure 1 As shown, the method comprises the following steps S100 to S200.

[0057] Step S100, acquiring image-text data, wherein the image-text data comprises a combination of multiple groups of pictures and texts;

[0058] Step S200, inputting the image-text data into a first to-be-tested model and a second to-be-tested model, respectively;

[0059] Step S300, acquiring the matrix entropy and the matrix conditional entropy of the first to-be-tested model and the second to-be-tested model at different time points of the first continuous learning task and the second continuous learning task, respectively;

[0060] Step S400, calculating the old knowledge understanding index and the new knowledge expansion index of continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first to-be-tested model and the second to-be-tested model, respectively.

[0061] The model continuous learning test method for image-text data processing of the embodiment is improved based on the multi-modal pre-training continuous learning benchmark CLiMB. Specifically, in the visual-linguistic task setting, common continuous learning (CL) algorithms cannot promote knowledge transfer and actually often have a negative impact on the learning of future tasks. The purpose of the model continuous learning test method for image-text data processing of the embodiment is to design an index that can quantify the performance of continuous learning (CL) algorithms at the data representation level.

[0062] Unlike the prior art, which calculates forgetting and knowledge transfer using model scores on each task, the model continuous learning test method for image-text data processing of the embodiment uses matrix entropy as the model performance. By calculating the matrix conditional entropy of the two models to be tested, the old knowledge understanding of the image-text data in the previous tasks after learning task i and the new knowledge expansion index of the image-text data in the previous tasks after learning task i are obtained.

[0063] The above steps S100 and S400 in the model continuous learning test method for image-text data processing of the embodiment are described in detail below.

[0064] Step S100, obtaining image-text data, wherein the image-text data contains a plurality of groups of picture and text combinations. For example, one group of picture and text combinations is a picture of a dog and a description: "This is a picture". In the embodiment, a plurality of groups of image-text data containing a plurality of groups of picture and text combinations are constructed as a data test set.

[0065] Step S200, inputting the image-text data into a first model to be tested and a second model to be tested, respectively.

[0066] The first model to be tested and the second model to be tested of the embodiment are both pre-trained models ViLT (Vision-and-Language Transformer) used in the CLiMB multi-modal continuous learning benchmark. In the upstream training stage, CLiMB mainly tests the ability of the algorithm to migrate to a never-seen task after a series of training tasks and the ability to resist forgetting the past seen tasks. In the downstream transfer learning stage, CLiMB mainly tests the low-sample transfer learning ability of the algorithm.

[0067] Step S300, obtaining the matrix entropy and the matrix conditional entropy of the first model to be tested and the second model to be tested at different time points of the first continuous learning task and the second continuous learning task, respectively.

[0068] Figure 2 The flowchart shows the process of obtaining the matrix entropy and the matrix conditional entropy in the model continuous learning test method for image-text data processing of the embodiment; as Figure 2As shown, in one specific implementation of the embodiment, the obtaining respectively by the first to-be-tested model and the second to-be-tested model of the matrix entropy and the matrix conditional entropy of the matrix at different time instants of the first and second continuous learning tasks comprises:

[0069] Step S310, obtaining first and second representation vector matrices output by the first and second to-be-tested models at checkpoints at different time instants of the first and second continuous learning tasks respectively;

[0070] Step S320, obtaining first and second normalized covariance matrices corresponding to the text-image data based on the first and second representation vector matrices;

[0071] Step S330, obtaining corresponding matrix entropy and matrix conditional entropy based on the first and second normalized covariance matrices.

[0072] Step S400, calculating old knowledge understanding and new knowledge expansion indexes of continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first and second to-be-tested models respectively.

[0073] Figure 3 Fig. 1 shows a schematic diagram of the principle of continuous learning of a model for text-image data processing according to an embodiment of the present application; as Figure 3 As shown, representation vectors of two to-be-tested models (usually, the models are at checkpoints at different time instants of a continuous learning algorithm) on a reference data set are extracted, a covariance matrix is calculated, and the matrix entropy and the conditional entropy of the matrix are calculated through the covariance matrix. Preferably, the old knowledge understanding and the new knowledge acquisition indexes in the embodiment are calculated through the matrix entropy and the conditional entropy of the two model checkpoints on the reference data set.

[0074] In one specific implementation of the embodiment, the first and second normalized covariance matrices are obtained in the following manner:

[0075]

[0076] wherein, Σ M；D (i,j) is an element in the i-th row and the j-th column of the normalized covariance matrix, M is a to-be-tested model, D is a data set, N is the number of samples in the data set, z = [z1, …, zN] is a representation vector matrix output by the data set D after being processed by the model M, z N is a representation vector of the s-th sample, s is a value after the representation vector z s is decentered, and the calculation formula is ​ is the value of the i-th dimension of the representation vector corresponding to the s-th sample after decentralization, is the value of the j-th dimension of the representation vector corresponding to the s-th sample after decentralization.

[0077] In one specific implementation manner of the embodiment, the calculation manner of the matrix entropy is:

[0078]

[0079] The calculation manner of the matrix conditional entropy is:

[0080] H(K m+1 |K1,…,K m )=H(K1,K2,…,K m+1 )-H(K1,K2,…,K m )

[0081] The calculation manner of the matrix joint entropy is:

[0082]

[0083] ⊙ indicates Hadamard product.

[0084] Therefore, the matrix entropy of the first to-be-tested model M and the second to-be-tested model M' is The corresponding matrix conditional entropy is

[0085] wherein, M is the first to-be-tested model, M' is the second to-be-tested model, is the matrix entropy, is the first normalized covariance matrix, is the second normalized covariance matrix. is the matrix conditional entropy of the first normalized covariance matrix given the second normalized covariance matrix, is the matrix conditional entropy of the first normalized covariance matrix given the first normalized covariance matrix, is the matrix joint entropy of the first normalized covariance matrix and the second normalized covariance matrix, is the matrix joint entropy of the first normalized covariance matrix and the first normalized covariance matrix.

[0086] Figure 4 Fig. 1 shows a schematic diagram of an old knowledge understanding index acquisition manner of continuous learning in a model continuous learning test method for graphic data processing in an embodiment of the present application; as shown in the figure, Figure 4 In one specific implementation manner of the embodiment, the old knowledge understanding index acquisition manner of continuous learning is:

[0087]

[0088] Figure 5 FIG. 1 shows a schematic diagram of a method for obtaining a new knowledge expansion indicator of continuous learning in a model continuous learning test method for image-text data processing according to an embodiment of the present application. As shown in FIG. 1, the method for obtaining the new knowledge expansion indicator of continuous learning comprises the following steps: Figure 5

[0089] In a specific implementation of the embodiment, the method further comprises: obtaining an elasticity-stability trade-off indicator based on the old knowledge understanding indicator of continuous learning and the new knowledge expansion indicator of continuous learning; and the elasticity-stability trade-off indicator is a sum of the old knowledge understanding indicator of continuous learning and the new knowledge expansion indicator of continuous learning.

[0090] That is, the elasticity-stability trade-off indicator of the model is calculated by the old knowledge understanding indicator and the new knowledge expansion indicator, and the calculation formula is OKU+NKE.

[0091] The model continuous learning test method for image-text data processing according to the embodiment is used to test the training results of the upstream tasks, and the results are shown in Tables 1-4.

[0092] Table 1: Test results of the upstream tasks using the Elastic Weight Consolidation (EWC) continuous learning algorithm

[0093]

[0094] Table 2: Test results of the upstream tasks using the Seqential-FT continuous learning algorithm

[0095]

[0096] Table 3: Test results of the upstream tasks using the Experience-Replay continuous learning algorithm

[0097]

[0098] Table 4: Test results of the upstream tasks using the Freeze-Bottom-9layers continuous learning algorithm

[0099]

[0100] The English definitions and data explanations in the above tables are as follows:

[0101] ​Task test task (dataset), Score Type score type, Forg forgetting score, Repre representation layer score, Repeat-Head reuse original classification head, Re-Head retrain classification head.

[0102] The indicators represented by the data points are as follows:

[0103] The triplets of data points with Score Type Forg (forgetting score) are [original task score, current task score, score ratio] respectively.

[0104] The triplets of data points with Score Type Repre (representation layer score) are (old knowledge understanding indicator, new knowledge expansion indicator, flexibility) respectively.

[0105] Among them, Task test task has five, VQAv2 (Visual Question Answering v2), NLVR2 (Natural Language for Visual Reasoning 2), SNLI-VE (Stanford Natural Language Inference for Visual Entailment), VCR (Visual Common Reasoning), and BMC (Bone Marrow Cytomorphology).

[0106] The model of the image-text data processing of the embodiment is used to test the common continuous learning algorithm (Elastic Weight Consolidation, Freeze bottom k layers, Sequential Finetuning, Experience Replay). Repre represents the three indicators defined in the embodiment: old knowledge understanding, new knowledge expansion, and plasticity. Forg represents the forgetting indicator defined in the CLiMB scheme. The old knowledge understanding and new knowledge expansion indicators are calculated by the conditional entropy of the covariance matrix of the model on the same reference task through the formula. The same reference task is used in the same row in the above Tables 1 to 4, for example, the first row of each data table uses VQAv2 as the reference task.

[0107] The data sets used in this embodiment include VQAv2, NLVR, SNLI-VE, VCR and BMC (Bone Marrow Cytomorphology). BMC is a bone marrow cytology classification dataset containing over 170,000 de-identified, expert-annotated cell images from bone marrow smears of 945 patients, using May-Grünwald-Giemsa / Pappenheim staining.

[0108] In the following description, the embodiment will refer to the task as the old task and the task after the reference task as the new task. As can be seen from Tables 1 to 4, for the old knowledge understanding index, a value close to 1 means that the model retains most of the old task representation after fine-tuning on the subsequent task, so that the model can restore most of the performance of the old task when re-training the classification head for the old task. And the old knowledge usage index (Old knowledge usage) less than 1 means that the model loses part of the representation of the old task after fine-tuning on the subsequent task, so that the re-test result of the re-trained classification head on the old task has an observable gap with the performance of the model just trained on the old task.

[0109] For the new knowledge expansion index, a value greater than 0 means that the model obtains measurable representation from the learning of the new task, so that the accuracy of the task is also improved. And a value less than 0 means that the model does not obtain new knowledge from the training of the new task, so that the accuracy on the reference task will not be improved.

[0110] The model continuous learning test method for image-text data processing of the embodiment is shown in Tables 5 to 8.

[0111] Table 5 Continuous learning algorithm: downstream task test results and NKE trends under elastic weight consolidation (EWC)

[0112]

[0113] Table 6 Continuous learning algorithm: downstream task test results and NKE trends under sequential fine-tuning (Seqential-FT)

[0114]

[0115] Table 7 Continuous learning algorithm: downstream task test results and NKE trends under experience replay (Experience-Replay)

[0116]

[0117] Table 8. Continual learning algorithm: downstream task test results and NKE trends for freeze-bottom-9 layers

[0118]

[0119] The explanations of Tables 5-8 are as follows:

[0120] There are five upstream continual learning tasks, VQAv2 (Visual Question Answering v2), NLVR2 (Natural Language for Visual Reasoning 2), SNLI-VE (Stanford Natural Language Inference for Visual Entailment), VCR (Visual Common Reasoning), and BMC (Bone Marrow Cytomorphology).

[0121] NKE represents the new knowledge acquisition index, NKE (Cifar-10) represents the new knowledge acquisition index with Cifar-10 dataset as the reference dataset D, OKU represents the old knowledge understanding, cifar-10 represents the accuracy score of the downstream task for the cifar-10 test set, and fashionmnist represents the accuracy score of the downstream task for the fashionmnist test set.

[0122] Language dataset:

[0123] SST-2 represents the test accuracy (Accuracy) of the downstream task for the single language dataset SST-2, and IMDB represents the test accuracy (Accuracy) of the downstream task for the single language dataset IMDB.

[0124] On the downstream tasks Cifar-10 and FashionMNIST, the change trend of NKE is basically the same as the change trend of the accuracy of the downstream task, that is, when NKE is positive at a certain time, the accuracy of the downstream task from the last time to the present time is increased. This change trend is first more obvious in the Freeze-bottom-9-layers algorithm, and the yellow data points are consistent with this trend. Because Freeze-bottom-9-layers is significantly different from the other several algorithms, the change of the downstream task accuracy is not certain when NKE changes little. Therefore, in the other several algorithms, after using the unified reference data Cifar-10, the change of NKE can also correctly reflect the change of the downstream task accuracy.

[0125] As shown in Figure 4 and Figure 5 , the specific implementation process of the embodiment is as follows:

[0126] First, a group of pictures and texts are input into the Vision-Language Transformer (ViLT) models M and M' after the continuous learning tasks T and T' are fine-tuned, and the group of pictures and texts contains several pairs of pictures and a piece of text, for example, a picture of a dog and a description: "This is a picture".

[0127] After obtaining the representation vectors z and z' output by the models at different time points of the continuous learning task, the normalized covariance matrices ∑ M；D and ∑ M′；D are calculated respectively. The calculation formula is:

[0128]

[0129] Among them,

[0130] Then, according to the two normalized covariance matrices ∑ M；D and ∑ M′；D , the new knowledge acquisition index NKE of the embodiment is calculated by using the formula of matrix entropy and matrix conditional entropy.

[0131] The calculation formula of matrix entropy is:

[0132]

[0133] The calculation formula of matrix conditional entropy is:

[0134] H(K m+1 |K1,…,K m )=H(K1,K2,…,K m+1 )-H(K1,K2,…,Km )

[0135] wherein:

[0136] H(K1,..., K T ) = H(K1 0 K2 0... 0 K T )

[0137] 0 denotes Hadamard product.

[0138] Finally, the two indexes are calculated according to the formulas of NKE and OKU in Figure 4 and Figure 5 .

[0139] The protection scope of the model continuous learning test method for image-text data processing described in the embodiments of the present application is not limited to the execution order of the steps listed in the embodiments, and any scheme realized by increasing, reducing or replacing the steps of the prior art according to the principles of the present application is included in the protection scope of the present application.

[0140] The embodiments also provide a model continuous learning test system for image-text data processing, Figure 6 which is a principle block diagram of the model continuous learning test system for image-text data processing in an embodiment of the present application; as Figure 6 shown, the model continuous learning test system for image-text data processing includes an image-text data acquisition module and a model processing module.

[0141] Specifically, the image-text data acquisition module is configured to acquire image-text data, wherein the image-text data contains combinations of multiple groups of pictures and texts; and the model processing module is configured to input the image-text data into a first to-be-tested model and a second to-be-tested model respectively, acquire matrix entropy and matrix conditional entropy of the first to-be-tested model and the second to-be-tested model at different time points of a first continuous learning task and a second continuous learning task respectively, and calculate an old knowledge understanding index and a new knowledge expansion index of continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first to-be-tested model and the second to-be-tested model respectively.

[0142] It should be understood that the specific processes of the respective modules performing the above-mentioned corresponding steps have been described in detail in the above-mentioned method embodiments, and for the sake of brevity, will not be repeated here.

[0143] It should also be understood that the division of the modules in the embodiments of the present application is illustrative, and is only a logical functional division. In actual implementation, there can be another division manner. In addition, the functional modules in each embodiment of the present application can be integrated in one processor, or can be physically separated, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0144] Figure 7 is a schematic block diagram of an electronic terminal provided by an embodiment of the present application. As shown in Figure 7 the electronic terminal includes at least one processor 401, a memory 402, at least one network interface 403 and a user interface 405. The various components in the electronic terminal are coupled together by a bus system 404. It can be understood that the bus system 404 is used to realize the connection communication between the components. The bus system 404 includes a data bus, a power supply bus, a control bus and a status signal bus in addition to the data bus. However, for the purpose of clear illustration, all the buses are marked as the bus system in Figure 7 .

[0145] The user interface 405 can include a display, a keyboard, a mouse, a trackball, a click gun, a key, a button, a touchpad or a touch screen, etc.

[0146] It can be understood that the memory 402 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM, Static Random Access Memory), synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory). The memory described in the embodiments of the present application is intended to include but not limited to these and any other suitable categories of memory.

[0147] The memory 402 in the embodiments of the present application is used to store various categories of data to support the operation of the electronic terminal 400. Examples of these data include any executable programs for operating on the electronic terminal 400, such as an operating system 4021 and an application program 4022. The operating system 4021 contains various system programs, such as a framework layer, a core library layer, a driver layer, etc., for realizing various basic services and processing hardware-based tasks. The application program 4022 can contain various application programs, such as a media player (Media Player), a browser (Browser), etc., for realizing various application services. The XX method provided by the embodiments of the present application can be included in the application program 4022.

[0148] The method disclosed in the embodiments of the present application can be applied to the processor 401 or implemented by the processor 401. The processor 401 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 401. The processor 401 described above can be a general processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor 401 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor 401 can be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiments of the present application, the steps can be directly embodied as hardware decoding processor for execution, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the foregoing method.

[0149] In the exemplary embodiments, the electronic terminal 400 can be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), or the like for executing the foregoing method.

[0150] According to the method provided in the embodiments of the present application, the present application further provides a computer program product, which comprises computer program code, when the computer program code runs on a computer, so that the computer executes Figures 1 to 5 the method of any of the embodiments shown.

[0151] According to the method provided in the embodiments of the present application, the present application further provides a computer readable storage medium, which stores program code, when the program code runs on a computer, so that the computer executes Figures 1 to 5 the method of any of the embodiments shown.

[0152] As used in this description, the terms "component," "module," "system", and the like are intended to refer to a computer-related entity, either hardware, software, a combination of hardware and software, software in execution, or software in execution. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and / or thread of execution and a component can be localized, co-resident, and / or distributed among one computer or distributed across multiple computers. Also, these components can execute from various computer readable media having various data structures stored thereon. The components can communicate by way of local and / or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and / or across a network such as the Internet with other systems via the signal).

[0153] Those of skill in the art would understand that the various illustrative logical blocks and steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or a combination of computer software and electronic hardware. The choice of hardware and software would depend on the specific application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0154] Those of skill in the art would understand that, for the described convenience and brevity, the specific working process of the system, electronic terminal and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0155] In several embodiments provided in the present application, it should be understood that the disclosed system, electronic terminal and method can be implemented in other ways. For example, the electronic terminal embodiments described above are merely illustrative, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between the electronic terminal or unit can be electrical, mechanical or other forms.

[0156] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0157] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be physically present separately, or two or more units can be integrated into one unit.

[0158] In the above embodiments, the functions of each functional unit can be implemented by software, hardware, firmware or any combination thereof, in whole or in part. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the flow or function according to the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable electronic terminals. Computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as high-density digital video disc (digital video disc, DVD), or semiconductor media (such as solid state disk (solid state disk, SSD) and the like.

[0159] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0160] The above merely describes the specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0161] In summary, the model continuous learning test method for image-text data processing provided by the present application uses matrix entropy as the model performance, calculates the matrix conditional entropy of two models to be tested, obtains the old knowledge understanding of the previous task after the learning task and the new knowledge expansion index of the previous task after the learning task, and can quantify the performance of the continuous learning algorithm of the image-text data in the multi-modal environment. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0162] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical idea disclosed in the present application should be covered by the claims of the present application.

Claims

1. A model continuous learning test method for image-text data processing, characterized in that: The method comprises: acquiring image-text data, wherein the image-text data comprises a combination of multiple groups of pictures and texts; inputting the image-text data into a first to-be-tested model and a second to-be-tested model respectively; acquiring matrix entropy and matrix conditional entropy of the first to-be-tested model and the second to-be-tested model at different time points of a first continuous learning task and a second continuous learning task respectively; calculating an old knowledge understanding index and a new knowledge expansion index of continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first to-be-tested model and the second to-be-tested model respectively; the calculation method of the matrix entropy is: the calculation method of the matrix joint entropy is: ⊙ refers to Hadamard product; the calculation method of the matrix conditional entropy is: wherein, is a first model under test, M' is a second model under test, is a matrix entropy, is a first normalized covariance matrix, is a second normalized covariance matrix, is a matrix conditional entropy of the first normalized covariance matrix given the second normalized covariance matrix, is a matrix conditional entropy of the first normalized covariance matrix given the first normalized covariance matrix, is a matrix joint entropy of the first normalized covariance matrix and the second normalized covariance matrix, is a matrix joint entropy of the first normalized covariance matrix and the first normalized covariance matrix. 2.The model continuous learning test method for image-text data processing according to claim 1, characterized in that: the acquisition of the matrix entropy and the matrix conditional entropy of the first to-be-tested model and the second to-be-tested model at different time points of the first continuous learning task and the second continuous learning task comprises: acquiring first representation vector matrices and second representation vector matrices output by the first to-be-tested model and the second to-be-tested model at different time points of the first continuous learning task and the second continuous learning task respectively; acquiring first normalized covariance matrices and second normalized covariance matrices corresponding to the image-text data based on the first representation vector matrices and the second representation vector matrices; acquiring corresponding matrix entropy and matrix conditional entropy based on the first normalized covariance matrices and the second normalized covariance matrices. 3.The model continuous learning test method for text data processing according to claim 2, characterized in that: the acquisition method of the first normalized covariance matrices and the second normalized covariance matrices is: wherein Σ M；D (i,j) is the element of the i-th row and j-th column of the normalized covariance matrix, M is the model to be tested, D is the data set, N is the number of samples in the data set, z=[z1,…,z N ] is the representation vector matrix output by the data set D after processing by the model M, z s is the representation vector of the s-th sample, is the value of the representation vector z s after decentralization, and the calculation formula is is the value of the i-th dimension of the representation vector corresponding to the s-th sample after decentralization, is the value of the j-th dimension of the representation vector corresponding to the s-th sample after decentralization.

4. The model continuous learning test method for image-text data processing according to claim 1, characterized in that: the acquisition method of the old knowledge understanding index of continuous learning is: the acquisition method of the new knowledge expansion index of continuous learning is: 5.The model continuous learning test method for image-text data processing according to claim 1, characterized in that: further comprising: acquiring an elasticity-stability trade-off index based on the old knowledge understanding index of continuous learning and the new knowledge expansion index of continuous learning; the elasticity-stability trade-off index is the sum of the old knowledge understanding index of continuous learning and the new knowledge expansion index of continuous learning.

6. A model continuous learning test system for image-text data processing, characterized in that: comprising: an image-text data acquisition module, configured to acquire image-text data, wherein the image-text data comprises a combination of multiple groups of pictures and texts; a model processing module, configured to input the image-text data into a first to-be-tested model and a second to-be-tested model respectively, acquire matrix entropy and matrix conditional entropy of the first to-be-tested model and the second to-be-tested model at different time points of a first continuous learning task and a second continuous learning task respectively, and calculate an old knowledge understanding index and a new knowledge expansion index of continuous learning based on the matrix entropy and the matrix conditional entropy corresponding to the first to-be-tested model and the second to-be-tested model respectively; the calculation method of the matrix entropy is: the calculation method of the matrix joint entropy is: ⊙ refers to Hadamard product; the calculation method of the matrix conditional entropy is: wherein, is a first model under test, M' is a second model under test, is a matrix entropy, is a first normalized covariance matrix, is a second normalized covariance matrix, is a matrix conditional entropy of the first normalized covariance matrix given the second normalized covariance matrix, is a matrix conditional entropy of the first normalized covariance matrix given the first normalized covariance matrix, is a matrix joint entropy of the first normalized covariance matrix and the second normalized covariance matrix, is a matrix joint entropy of the first normalized covariance matrix and the first normalized covariance matrix.

7. An electronic terminal, characterized in that comprising: a memory, configured to store a computer program; a processor, configured to run the computer program to realize the steps of the model continuous learning test method for image-text data processing according to any one of claims 1 to 5.

8. A computer storage medium storing program instructions, characterized in that: The program instructions, when executed, implement the steps of the model continuous learning test method for graphic text data processing as claimed in any one of claims 1 to 5.

9. A computer program product, characterised in that, The computer program product comprises computer program code, which, when executed on a computer, causes the computer to implement the steps of the model continuous learning test method for graphic text data processing as claimed in any one of claims 1 to 5.

Images