Intra- and inter-modal multi-modal curriculum learning methods, apparatuses, and devices

Abstract

This application provides a multimodal curriculum learning method, apparatus, and device based on intra-modal and inter-modal approaches. The method includes: processing multimodal data using a multimodal module containing multiple modal encoders to obtain task prediction results and sample representation results for each modal encoder; calculating the task loss gradient vector for each modal encoder based on the task prediction results, and calculating the modal loss gradient vector for each modal encoder based on the sample representation results; determining the relation value for each modal encoder based on the task loss gradient vector and the modal loss gradient vector; selecting the modal encoder with the largest relation value to learn from the task, and selecting the remaining modal encoders (excluding the one with the largest relation value) to learn from the modal. Based on the method provided in this application, mutual suppression between modalities is avoided, and the optimization level of each modal encoder and the overall optimization level of the multimodal module are improved.

Description

Technical Field

[0001] This application relates to the field of multimodal learning technology, and in particular to a method, apparatus and device for multimodal curriculum learning based on intramodal and intermodal learning. Background Technology

[0002] Multimodal learning is a method for modeling multimodal inputs, predicting and inferring results by processing and integrating data from different sources and modalities. Relying on the complementarity and alignment of information between modalities, multimodal learning not only improves the performance of machine learning models on tasks previously performed on a single modal basis (e.g., audio-assisted image classification), but also solves tasks that cannot be accomplished using a single modality alone (e.g., video question answering tasks that simultaneously consider video visuals and text).

[0003] However, existing modal encoders in multimodal learning cannot achieve the same level of optimization as their counterparts in unimodal learning. For example, in an audio-assisted image classification multimodal model, although its performance outperforms unimodal models with only image or only audio input, this performance improvement comes entirely from more information input, not from a better-optimized model. Conversely, the optimization level of modal encoders in multimodal models is far lower than that in unimodal models. Therefore, overcoming the problem of incomplete modal optimization in multimodal learning is a pressing technical challenge. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a multimodal course learning method, apparatus, and device based on intramodal and intermodal learning to overcome or at least partially solve the above problems.

[0005] A first aspect of this application discloses a multimodal curriculum learning method based on intramodal and intermodal learning, the method comprising:

[0006] A multimodal module containing multiple modal encoders is used to process multimodal data for a task, resulting in task prediction results and sample representation results for each modal encoder. The multimodal data includes single-modal data of various different data types.

[0007] Based on the task prediction results, calculate the task loss gradient vector for each modal encoder, and based on the sample characterization results, calculate the modal loss gradient vector for each modal encoder.

[0008] Based on the task loss gradient vector and the modal loss gradient vector, the relational value of each modal encoder is determined, and the relational value represents the learning direction of the modal encoder;

[0009] The modal encoder with the largest relation value is selected to learn from the task, and the remaining modal encoders other than the modal encoder with the largest relation value are selected to learn from the modality.

[0010] Optionally, based on the task prediction results, the task loss gradient vector for each modality encoder is calculated, including:

[0011] Based on the predicted and actual results of the task, the task loss is obtained;

[0012] The gradient of the task loss is calculated as the backpropagation gradient in each modal encoder, and used as the task loss gradient vector for each modal encoder.

[0013] Optionally, based on the sample characterization results, the modal loss gradient vector for each modal encoder is calculated, including:

[0014] Single-modal data is input into the corresponding modality teacher model for processing to obtain the knowledge distillation result of the single-modal data. The teacher model is a pre-trained modality encoder.

[0015] Based on the sample characterization results and the knowledge distillation results, the distillation loss is determined, whereby the distillation loss characterizes the difficulty of the single-modal data.

[0016] Based on the self-stepping learning method, with minimizing the modal loss as the optimization objective, the modal loss is obtained according to the distillation loss and the distillation loss weight, wherein the distillation loss weight is determined according to the mapping relationship between the distillation loss weight and the distillation loss.

[0017] The gradient of the modal loss is calculated as the backpropagation gradient in each modal encoder, and used as the modal loss gradient vector for each modal encoder.

[0018] Optionally, the modal loss includes a weighted distillation loss, and a first regularization term and a second regularization term;

[0019] The first regularization term uses the moving average of distillation loss as a coefficient. The first regularization term amplifies the weight of distillation loss that is less than the average and reduces the weight of distillation loss that is greater than the average.

[0020] The regularization coefficient of the second regularization term is predefined, and the second regularization term is used to control the distillation loss weight to tend to 1.

[0021] Optionally, based on the task loss gradient vector and the modal loss gradient vector, the relational value of each modal encoder is determined, including:

[0022] With the goal of achieving Pareto optimality for updating the task loss gradient vector and the modal loss gradient vector, a gradient objective function is constructed for each modal encoder.

[0023] The gradient objective function is solved to obtain the relational values ​​for each modal encoder.

[0024] Optionally, the modal encoder with the largest relational value learns from the task according to the following steps:

[0025] Based on the task prediction results and task loss weights, calculate the weighted task loss gradient vector;

[0026] The parameters of the modal encoder with the largest relational value are updated based on the weighted task loss gradient vector.

[0027] Optionally, the modal encoders other than the one with the largest relational value learn from the modes according to the following steps:

[0028] The weighted modal loss gradient is calculated based on the sample representation results of each modal encoder and the distillation loss weight corresponding to each modal encoder.

[0029] The parameters of the modal encoder are updated based on the weighted modal loss gradient.

[0030] Optionally, the multimodal course learning includes multiple learning rounds, and the method further includes:

[0031] After completing the current learning round, the relation values ​​of each modal encoder are recalculated to obtain new relation values;

[0032] In the next learning round, the modal encoder with the largest new relation value is selected to learn from the task, and the remaining modal encoders, excluding the modal encoder with the largest new relation value, are selected to learn from the modality.

[0033] A second aspect of this application discloses a multimodal course learning device based on intramodal and intermodal learning, the device comprising:

[0034] The processing module is used to perform task processing on multimodal data using a multimodal module containing multiple modal encoders, to obtain task prediction results and sample representation results of each modal encoder. The multimodal data includes single-modal data of various different data types.

[0035] The calculation module is used to calculate the task loss gradient vector of each modal encoder based on the task prediction results, and to calculate the modal loss gradient vector of each modal encoder based on the sample characterization results.

[0036] The determination module is used to determine the relational value of each modal encoder based on the task loss gradient vector and the modal loss gradient vector, wherein the relational value represents the learning direction of the modal encoder;

[0037] The selection module is used to select the modal encoder with the largest relation value to learn from the task, and to select the remaining modal encoders other than the modal encoder with the largest relation value to learn from the modality.

[0038] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the multimodal curriculum learning method based on intramodal and intermodal learning described in the first aspect of this application.

[0039] The embodiments of this application have the following advantages:

[0040] In this embodiment, both modal data and intermodal interactions are considered for multimodal learning. A multimodal module containing multiple modal encoders is used to process the multimodal data, obtaining task prediction results and sample representation results for each modal encoder. Then, based on the task prediction results, the task loss gradient vector for each modal encoder is calculated, and based on the sample representation results, the modal loss gradient vector for each modal encoder is calculated. Based on the task loss gradient vector and the modal loss gradient vector, a relational value for each modal encoder is determined, representing the learning direction of the modal encoder. The modal encoder with the largest relational value is selected to learn from the task, and the remaining modal encoders (excluding the one with the largest relational value) are selected to learn from the modal.

[0041] Because the learning direction of each modal encoder is determined based on relational values, each modal encoder can always obtain an optimization target suitable for itself, avoiding mutual suppression between modalities. Furthermore, intramodal learning allows each modal encoder to learn data within its own modality from easy to difficult, thereby improving the optimization level of each individual modal encoder; intermodal learning (i.e., learning from the task) determines the order in which modalities learn from the task, balancing the learning situation between modalities and improving the overall optimization level of the multimodal module, thus overcoming the problem of incomplete modal optimization in multimodal learning. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the steps of a multimodal curriculum learning method based on intramodal and intermodal learning, as provided in an embodiment of this application.

[0044] Figure 2 This is an overall architecture diagram of a multimodal curriculum learning method based on intramodal and intermodal learning provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the structure of a multimodal course learning device based on intramodal and intermodal learning provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0047] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] To address the issue of insufficient optimization of modal encoders in multimodal learning, relevant techniques mainly fall into two categories: 1) The first category considers that this problem arises because different modal encoders have different architectures and are adapted to different optimization strategies. A single optimization method cannot simultaneously adapt to multiple modal encoders, resulting in insufficient optimization of all encoders. Based on this idea, the Gradient-Blending (GB) method adds the loss values ​​of each modality to the original optimization objective, making the new optimization objective a weighted sum of multiple loss values. The additional loss values ​​and corresponding weights are used to optimize each encoder. The Uni-Modal Teacher (UMT) method adds the knowledge distillation loss value of a single-modal teacher model to the original optimization objective to guide the learning of each encoder. All of these methods improve the optimization effect of each modality by adding optimization objectives.

[0049] 2) The second type of method addresses this problem because different modalities have varying strengths and influences on the results. This can lead the model to optimize only in a way that favors the strong modality in order to improve final performance, resulting in a suppression effect of the strong modality on the weak modality, preventing the weak modality encoder from being fully optimized. For example, on some audio-assisted image classification datasets, the model may only need the audio to make accurate predictions and not the image modality at all. This causes the optimization process to be performed only on the audio encoder, leaving the image encoder under-optimized. Based on this idea, the Adaptive Tracking Factor (ATF) method dynamically adjusts the learning progress of the modalities, allowing weaker modalities to learn more times on the dataset; the Conditional Learning Speed ​​(CLS) method increases the learning rate of the weak modality and decreases the learning rate of the strong modality based on the learning performance of different modalities; and the On-the-fly Gradient Modulation with Generalization Enhancement (OGM-GE) method increases the gradient update of the weak modality and decreases the gradient update of the strong modality based on the output results of different modalities. The above methods all improve the optimization effect of each modality by balancing the learning speed and learning progress between modalities.

[0050] However, the two methods mentioned above still have the following two limitations. First, these methods focus primarily on the balance between modalities, neglecting the impact of intra-modal data on the modal encoder. Suppose an encoder within a modality consistently faces challenging or even noisy data—for example, distorted images or noisy audio—causing its output to be detrimental to the final prediction and inference. This modality becomes a weak modality, suppressed by other strong modalities and difficult to optimize effectively. Second, the methods rely on the performance of individual modalities to determine modal strength, limiting their application. For example, in audio-assisted image classification tasks, both the audio and image modalities can produce predictions; the more accurate modality indicates its greater influence, allowing for the identification of strong and weak modalities. However, in tasks like video question answering, predictions require consideration of both video content and text questions. Without predictions from individual modalities, the methods cannot determine strong and weak modalities, making optimization and balancing impossible.

[0051] In other words, the reason why existing technologies still have insufficiently optimized modal encoders is that: different modal encoders require different optimization strategies but are optimized by the same optimization strategy, and the model is only willing to optimize strong modes for the sake of overall performance, resulting in strong modes suppressing weak modes.

[0052] Therefore, to overcome the limitations of existing technologies, the applicant proposes the following technical concept: A course-based learning approach is adopted, progressing from simple to complex courses, while simultaneously considering modal data and intermodal interactions for multimodal course learning. This involves selectively selecting data for training within each modality and selectively choosing modality learning tasks among different modalities. First, a multimodal module containing multiple modal encoders is used to process the multimodal data, obtaining task prediction results and sample representation results for each modal encoder. Then, based on the task prediction results, the task loss gradient vector for each modal encoder is calculated, and based on the sample representation results, the modal loss gradient vector for each modal encoder is calculated. Based on the task loss gradient vector and the modal loss gradient vector, a relational value for each modal encoder is determined, representing the learning direction of the modal encoder. The modal encoder with the largest relational value is selected to learn from the task, and the remaining modal encoders (excluding the one with the largest relational value) are selected to learn from the modality.

[0053] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a multimodal curriculum learning method based on intramodal and intermodal learning, as provided in an embodiment of this application. Figure 1 As shown, the method may include steps S110 to S140:

[0054] Step S110: Use a multimodal module containing multiple modal encoders to process the multimodal data to obtain the task prediction results and the sample representation results of each modal encoder. The multimodal data includes single-modal data of various different data types.

[0055] In this embodiment, unimodal data can be images, audio, text, tables, etc., and multimodal data can be composed of unimodal data of various different data types. For example, multimodal data can be composed of images, audio, and text. For instance, a multimodal dataset can be represented as... N is the number of samples, M is the number of modalities, and x im Let y be the i-th sample of the m-th modality, where 1 ≤ m ≤ M, 1 ≤ i ≤ N. i Let x be the label (i.e., the true result) for the i-th sample. Depending on the type of multimodal data, x... im It can be an image, an audio clip, a line of text, a table, etc. i It could be a category, an answer, a match, etc.

[0056] A multimodal module contains multiple modal encoders, each processing one monomodal dataset. For example, a multimodal module for a video question-answering task involving video images and text questions might include a modal encoder for processing video images and a modal encoder for processing text questions. The goal of multimodal learning is to predict the outcome. This is achieved by using a multimodal module with multiple modal encoders to process the multimodal data. Each modal encoder processes its corresponding monomodal data, resulting in a sample representation for each encoder. The final task prediction result is then derived from these sample representations.

[0057] For example, task prediction results Represented as:

[0058]

[0059] Where f0 is a multimodal module with θ0 as a parameter, f m It is θ m A modal encoder with parameters, 1≤m≤M, f1(x i1 ;θ1) represents the sample representation result of the first modal encoder, f M (x iM ;θ M ) represents the sample representation result of the Mth modal encoder.

[0060] Step S120: Calculate the task loss gradient vector for each modal encoder based on the task prediction results, and calculate the modal loss gradient vector for each modal encoder based on the sample characterization results.

[0061] In this embodiment of the application, for each modal encoder in the multimodal module, the gradient during training is affected by two losses: task loss and modal loss. The gradients of the task loss and modal loss during backpropagation are respectively the gradient vector of the task loss. and modal loss gradient vector The task loss gradient vector is used for inter-modal learning, while the modal loss gradient vector is used for intra-modal learning.

[0062] In an optional embodiment, based on the task prediction result, the task loss gradient vector of each modality encoder is calculated, including steps A1 to A2:

[0063] Step A1: Based on the predicted and actual results of the task, obtain the task loss.

[0064] Step A2: Calculate the gradient of the task loss as it propagates back through each modal encoder, and use it as the task loss gradient vector for each modal encoder.

[0065] In this embodiment, the task prediction result is the prediction result of the multimodal module. Based on this task prediction result and the corresponding actual result, the task loss can be obtained. For example, the task loss L0 can be expressed as:

[0066]

[0067] Where l0 represents the task loss for a single sample, y i This represents the true result of the i-th sample. Let N represent the pre-defined task result for the i-th sample, where N is the number of samples. Then, based on the parameters of the modal encoder, the gradient of the task loss propagated back through each modal encoder is calculated to obtain the task loss gradient vector for each modal encoder.

[0068] In an optional embodiment, based on the sample characterization results, the modal loss gradient vector of each modal encoder is calculated, including steps B1 to B4:

[0069] Step B1: Input the single-modal data into the corresponding modality teacher model for processing to obtain the knowledge distillation result of the single-modal data. The teacher model is a pre-trained modality encoder.

[0070] Step B2: Based on the sample characterization results and the knowledge distillation results, determine the distillation loss, whereby the distillation loss characterizes the difficulty of the single-modal data.

[0071] Step B3: Based on the self-stepping learning method, with minimizing the modal loss as the optimization objective, the modal loss is obtained according to the distillation loss and the distillation loss weight. The distillation loss weight is determined according to the mapping relationship between the distillation loss weight and the distillation loss.

[0072] Step B4: Calculate the gradient of the modal loss as it propagates back through each modal encoder, and use it as the modal loss gradient vector for each modal encoder.

[0073] In this embodiment, during intramodal learning, it is necessary to determine the difficulty of the data in each modality within the modality. To enable this determination without relying on single-modal prediction results, knowledge distillation is employed, using a pre-trained single-modal teacher model g. m Its pre-trained parameters are φ m Furthermore, the distillation loss is used to determine the strength of each modality. Specifically, the distillation loss is characterized using the L2 norm distance, then sample x... im For the modal encoder f m 2-norm distance l im Represented as:

[0074]

[0075] A larger distillation loss indicates a greater discrepancy between the modal encoder's sample representation and the teacher model's output knowledge distillation result, suggesting a worse learning performance of the modal encoder on that modal data, and thus a higher difficulty level for that modal data. In intramodal learning, it's desirable to minimize the impact of more difficult modal data; therefore, difficulty and weight are incorporated as regularization terms into the optimization objective. A self-paced learning method is then employed to minimize the modal loss, calculating it as the optimization objective. Finally, based on the modal encoder's parameters, the gradient of this modal loss during backpropagation within the modal encoder is calculated to obtain the modal loss gradient vector of the modal encoder.

[0076] In one specific implementation, the modal loss includes a weighted distillation loss, a first regularization term, and a second regularization term; wherein, the first regularization term uses the moving average of the distillation loss as its coefficient, and the first regularization term amplifies the weight of distillation loss values ​​less than the average and reduces the weight of distillation loss values ​​greater than the average; the regularization term coefficient of the second regularization term is predefined, and the second regularization term is used to control the weight of the distillation loss to tend towards 1.

[0077] For example, the modal loss is expressed as:

[0078]

[0079] Among them, L m Indicates modal loss; w im l im λ represents the weighted distillation loss; m w im Let λ represent the first regularization term. m w im It can amplify the weights of distillation loss values ​​below the mean and reduce the weights of distillation loss values ​​above the mean, allowing the model to focus on learning the current simple samples; β m (logw im ) 2 Let β represent the second regularization term. m (logw im ) 2 This prevents the distillation loss weight from being too large or too small, making it more likely to be 1; N represents the number of samples in the multimodal data, w im Let l represent the distillation loss weight of the i-th sample in the m-th mode. im λ represents the distillation loss of the i-th sample in the m-th mode. m β represents the moving average of distillation losses. m This represents the coefficient of the predefined regularization term.

[0080] The mapping relationship between distillation loss weight and distillation loss is as follows: the larger the distillation loss, the smaller the corresponding distillation loss weight; conversely, the smaller the distillation loss, the larger the corresponding distillation loss weight. This mapping relationship is obtained by differentiating the modal loss, and is expressed as follows:

[0081]

[0082] Among them, w im λ represents the distillation loss weight of the i-th sample in the m-th mode. m β represents the moving average of distillation losses. m Let represent the coefficient of the regularization term, w represent the Lambert W function, and e be the natural base.

[0083] Step S130: Determine the relation value of each modal encoder based on the task loss gradient vector and the modal loss gradient vector, wherein the relation value represents the learning direction of the modal encoder.

[0084] In this embodiment, for each modal encoder in the multimodal module, the gradient during training is affected by both task loss and modal loss. Therefore, the relative relationship between the task loss gradient vector and the modal loss gradient vector reflects the learning situation of the modal encoder from the task and from the modality. If a modal encoder learns more from the modality, it indicates that it is suppressed by other modalities in multimodal learning and has difficulty learning from the task. Therefore, this modality is designated as a weak modality, allowing it to learn more from the modality first, and then gradually learn from the task later.

[0085] Furthermore, by leveraging knowledge from multi-objective optimization, the task loss gradient vector and modal loss gradient vector are analyzed. The relationship between the two gradient vectors is used to obtain the relational value of each modal encoder. This relational value represents the learning direction of the modal encoder. Based on the relational value, it can be determined whether the modal encoder should learn from the task or from the modality.

[0086] In one optional embodiment, determining the relational value of each modal encoder based on the task loss gradient vector and the modal loss gradient vector includes: constructing a gradient objective function for each modal encoder with the goal of satisfying Pareto optimality for updating the task loss gradient vector and the modal loss gradient vector; and solving the gradient objective function to obtain the relational value of each modal encoder.

[0087] For example, the gradient objective function is expressed as:

[0088]

[0089] Solving the gradient objective function yields the numerical relation α. m Represented as:

[0090]

[0091] Wherein, the relational value α m This reflects the relationship between the task loss gradient vector and the modality loss gradient vector, for example, α m When the gradient is →1, if the task loss gradient vector and the modal loss gradient vector form an acute angle, then the gradient from the task is greater than that from the distillation. If the task loss gradient vector and the modal loss gradient vector form an obtuse angle, then the gradient from the task is much greater than that from the distillation. This indicates that learning from the task is easier than learning from the teacher. Therefore, the relationship value α... m It can be used to indicate the learning direction of a modal encoder and determine whether to learn from the task or the modality.

[0092] In one alternative implementation, considering that gradient calculation requires backpropagation, and given the particularly large number of parameters in the modal encoder, the calculation of the loss value relative to the modal encoder gradient requires extremely high complexity and time-consuming computation. To further improve the computational efficiency of the relational values, the chain rule of derivatives is used to transform the task loss gradient vector... and modal loss gradient vector The calculation is replaced by the characterization result Z of the modal encoder. m =f m (x im ;θ m The gradient calculation outputs the task loss gradient vector. and output mode loss gradient vector This improves computational efficiency.

[0093] Step S140: Select the modal encoder with the largest relation value to learn from the task, and select the remaining modal encoders other than the modal encoder with the largest relation value to learn from the modality.

[0094] In this embodiment, a larger relation value indicates a greater similarity between the task loss gradient vector and the modal loss gradient vector, meaning the modal encoder can learn from both the task and the modality simultaneously. The modal encoder with the largest relation value is considered a strong modality, while the others are considered weak modalities. By selecting the modal encoder with the largest relation value to learn from the task, while other modal encoders learn from the modality (i.e., knowledge distillation from the teacher model), each modal encoder can always obtain optimizations suitable for it, and mutual suppression between modalities is avoided.

[0095] In practice, for the learning of the modal encoder with the largest correlation value, the parameters of that modal encoder are updated using the task loss gradient vector; for the learning of the remaining modal encoders, the parameters of their respective modal encoders are updated using their respective modal loss gradient vectors. For example, intra-modal and inter-modal multimodal curriculum learning can be represented as:

[0096]

[0097]

[0098]

[0099] Where, m * For the mode with the largest relational value, m′≠m * For other modalities, η is the learning rate. θ represents the parameter of the modal encoder with the largest relational value. m′ This represents the parameters of the remaining modal encoders.

[0100] In one optional implementation, the modal encoder with the largest relational value learns from the task by following these steps: calculating a weighted task loss gradient vector based on the task prediction result and the task loss weight; and updating the parameters of the modal encoder with the largest relational value based on the weighted task loss gradient vector.

[0101] For example, the modal encoder with the largest relation value can be represented from the task as:

[0102]

[0103] in, L represents the gradient vector of the output task loss. i0 For the task loss of the i-th sample, w i0 θ represents the task loss weight. j is the parameter of the modal encoder with the largest relational value, where j is equal to the number of the modal encoder with the largest relational value.

[0104] Specifically, the task loss weight is determined based on the mapping relationship between the task loss weight and the task loss. The task loss can also be expressed as:

[0105]

[0106] Where L0 represents the task loss; w i0 l i0 This represents the weighted task loss; λ0w i0 λ0w represents the third regularization term. i0It can amplify the weights of tasks with loss values ​​below the average and reduce the weights of tasks with loss values ​​above the average, allowing the model to focus on learning the current simple samples; β0(logw i0 ) 2 Denotes the fourth regularization term, β0(logw) i0 ) 2 This prevents the task loss weight from being too large or too small, making the task loss weight more likely to be 1; N represents the number of samples in the multimodal data, w i0 Let l represent the task loss weight for the i-th sample. i0 Let λ0 represent the task loss for the i-th sample, λ0 represent the moving average of the task loss, and β0 represent the predefined regularization coefficient.

[0107] By differentiating the task loss, the mapping relationship between the task loss weight and the task loss is obtained, which is expressed as:

[0108]

[0109] Where W represents the Lambert W function, and e is the natural base.

[0110] In one optional implementation, the modal encoders other than the modal encoder with the largest relational value learn from the modality by following these steps: calculating a weighted modal loss gradient based on the sample representation results of each modal encoder and the distillation loss weight corresponding to each modal encoder; and updating the parameters of the modal encoders based on the weighted modal loss gradient.

[0111] For example, the learning from modes by the remaining modal encoders can be represented as:

[0112]

[0113] in, The output modal loss gradient vector, l ij For the task loss of the i-th sample, w ij θ represents the distillation loss weights corresponding to the modal encoder. j For the parameters of the remaining modal encoders, j is equal to the number of the remaining modal encoders except for the modal encoder with the largest relational value.

[0114] In summary, intramodal and intermodal multimodal learning can be represented as:

[0115]

[0116] In one optional implementation, the multimodal curriculum learning includes multiple learning rounds, and the method further includes steps C1 to C2:

[0117] Step C1: After completing the learning of the current learning round, recalculate the relation values ​​of each modal encoder to obtain new relation values.

[0118] Step C2: In the next learning round, select the modal encoder with the largest new relation value to learn from the task, and select the remaining modal encoders other than the modal encoder with the largest new relation value to learn from the modality.

[0119] In this embodiment, to ensure that all modal encoders in the multimodal module can optimize at both intermodal (task-based) and intramodal (data) levels, the modal encoders learned from the task and modal are not fixed in each learning round. The relation values ​​for each modal encoder need to be recalculated in each learning round, and then the modal encoders learned from the task and modal are determined based on the new relation values. This ensures that each modal encoder always obtains optimizations suitable for it, and also avoids mutual suppression between modalities.

[0120] For example, for a weak modal encoder (i.e., the relation value is not the maximum), in the initial learning rounds, it learns more from the modality (i.e., knowledge distillation extracted from the teacher model). After several learning rounds, the relation value of the modal encoder is calculated to be the maximum. At this point, the modal encoder is a strong modal encoder, and then the strong modal encoder begins to learn from the task.

[0121] For example, Figure 2 This is an overall architecture diagram of a multimodal curriculum learning method based on intramodal and intermodal data, provided in an embodiment of this application. It explains the encoder incomplete optimization problem in multimodal learning from the perspective of intramodal data and intermodal interactions, and proposes a general multimodal curriculum learning framework that simultaneously considers intramodal data and intermodal balance. First, the multimodal data x... i The inputs are processed by the modal encoders in the multimodal module to obtain the task prediction results and the sample representation results of each modal encoder. Then, the task loss is determined based on the task prediction results and the actual results. io The distillation loss l is obtained based on the sample characterization results and knowledge distillation results. im (1≤m≤M); then, based on the task loss and distillation loss, the task loss gradient vector and the modal loss gradient vector of each modal encoder are determined respectively. Based on the task loss gradient vector and the modal loss gradient vector, the relation value of each modal encoder is determined. Then, the modal encoder with the largest relation value is selected to learn from the task, and the other modal encoders besides the modal encoder with the largest relation value are selected to learn from the modality.

[0122] Because the learning direction of each modal encoder is determined based on relational values, each modal encoder can always obtain an optimization objective suitable for itself, avoiding mutual suppression between modalities. Furthermore, intramodal learning allows each modal encoder to learn data within its own modality from easy to difficult, thereby improving the optimization level of each individual modal encoder; intermodal learning (i.e., learning from the task) determines the order in which modalities learn from the task, balancing the learning situation between modalities and improving the overall optimization level of the multimodal module, thus overcoming the problem of incomplete modal optimization in multimodal learning. In addition, this method can be applied to multimodal tasks where a single modality can produce prediction results and where a single modality cannot produce prediction results independently.

[0123] This application also provides a multimodal curriculum learning device based on intramodal and intermodal learning, referring to... Figure 3 As shown, Figure 3 This is a schematic diagram of a multimodal learning device based on intramodal and intermodal learning provided in an embodiment of this application. The device includes:

[0124] The processing module 310 is used to perform task processing on multimodal data using a multimodal module containing multiple modal encoders, to obtain task prediction results and sample representation results of each modal encoder. The multimodal data includes single-modal data of various different data types.

[0125] The calculation module 320 is used to calculate the task loss gradient vector of each modal encoder based on the task prediction result, and to calculate the modal loss gradient vector of each modal encoder based on the sample characterization result.

[0126] The determination module 330 is used to determine the relational value of each modal encoder based on the task loss gradient vector and the modal loss gradient vector, wherein the relational value represents the learning direction of the modal encoder;

[0127] Selection module 340 is used to select the modal encoder with the largest relation value to learn from the task, and to select the remaining modal encoders other than the modal encoder with the largest relation value to learn from the modality.

[0128] In one optional embodiment, the computing module includes:

[0129] The task loss module is used to obtain the task loss based on the predicted and actual results of the task.

[0130] The task gradient module is used to calculate the gradient of the task loss as it propagates back through each modal encoder, serving as the task loss gradient vector for each modal encoder.

[0131] In one optional embodiment, the computing module includes:

[0132] The distillation module is used to input single-modal data into the corresponding modality teacher model for processing, and obtain the knowledge distillation result of the single-modal data. The teacher model is a pre-trained modality encoder.

[0133] The distillation loss module is used to determine the distillation loss based on the sample characterization results and the knowledge distillation results, wherein the distillation loss characterizes the difficulty of the single-modal data;

[0134] The modal loss module is used to obtain the modal loss based on the distillation loss and distillation loss weights, with the optimization objective being minimizing the modal loss using a self-stepping learning method. The distillation loss weights are determined based on the mapping relationship between the distillation loss weights and the distillation loss.

[0135] The modal gradient module is used to calculate the gradient of the modal loss as it propagates back through each modal encoder, serving as the modal loss gradient vector for each modal encoder.

[0136] In one alternative embodiment, the modal loss includes a weighted distillation loss, and a first regularization term and a second regularization term;

[0137] The first regularization term uses the moving average of distillation loss as a coefficient. The first regularization term amplifies the weight of distillation loss that is less than the average and reduces the weight of distillation loss that is greater than the average.

[0138] The regularization coefficient of the second regularization term is predefined, and the second regularization term is used to control the distillation loss weight to tend to 1.

[0139] In one optional embodiment, the determining module includes:

[0140] A construction module is used to construct the gradient objective function for each modal encoder with the goal of achieving Pareto optimality for the updates of the task loss gradient vector and the modal loss gradient vector;

[0141] The solution module is used to solve the gradient objective function to obtain the relational values ​​for each modal encoder.

[0142] In an optional embodiment, the apparatus further includes a first learning module, which is used for the modal encoder with the largest relation value to learn from the task. The first learning module includes:

[0143] The first calculation submodule is used to calculate the weighted task loss gradient vector based on the task prediction result and the task loss weight.

[0144] The first update module is used to update the parameters of the modal encoder with the largest relational value based on the weighted task loss gradient vector.

[0145] In an optional embodiment, the apparatus further includes a second learning module, which is used for learning from the modes of the modal encoders other than the modal encoder with the largest relational value. The second learning module includes:

[0146] The second calculation submodule is used to calculate the weighted modal loss gradient based on the sample representation results of each modal encoder and the distillation loss weight corresponding to each modal encoder.

[0147] The second update module is used to update the parameters of the modal encoder based on the weighted modal loss gradient.

[0148] In one optional embodiment, the multimodal curriculum learning includes multiple learning rounds, and the device further includes:

[0149] The recalculation module is used to recalculate the relation values ​​of each modal encoder after completing the current learning round, and obtain new relation values.

[0150] The reselect module is used to select the modal encoder with the largest new relation value to learn from the task in the next learning round, and to select the remaining modal encoders other than the modal encoder with the largest new relation value to learn from the modality.

[0151] This application also provides an electronic device, see embodiments thereof. Figure 4 , Figure 4 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device 400 includes a memory 410 and a processor 420. The memory 410 and the processor 420 are connected via a bus for communication. The memory 410 stores a computer program that can run on the processor 420 to implement the steps of the multimodal course learning method based on intramodal and intermodal learning described in the embodiments of this application.

[0152] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0153] This application describes embodiments of methods, apparatus, and electronic devices according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0154] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0156] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0157] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0158] The foregoing has provided a detailed description of a multimodal course learning method, apparatus, and device based on intramodal and intermodal learning provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal curriculum learning method based on intra-modal and inter-modal learning, characterized in that, The method includes: A multimodal module containing multiple modal encoders is used to process multimodal data for a task, resulting in task prediction results and sample representation results for each modal encoder. The multimodal data includes single-modal data of various different data types, and the single-modal data includes at least: images, audio, text, and tables. Based on the task prediction results, calculate the task loss gradient vector for each modal encoder, and based on the sample characterization results, calculate the modal loss gradient vector for each modal encoder. Based on the task loss gradient vector and the modal loss gradient vector, the relational value of each modal encoder is determined, and the relational value represents the learning direction of the modal encoder; The modal encoder with the largest relation value is selected to learn from the task, and the remaining modal encoders other than the modal encoder with the largest relation value are selected to learn from the modality. Based on the sample characterization results, the modal loss gradient vector for each modal encoder is calculated, including: Single-modal data is input into the corresponding modality teacher model for processing to obtain the knowledge distillation result of the single-modal data. The teacher model is a pre-trained modality encoder. Based on the sample characterization results and the knowledge distillation results, the distillation loss is determined, whereby the distillation loss characterizes the difficulty of the single-modal data. Based on the self-stepping learning method, with minimizing the modal loss as the optimization objective, the modal loss is obtained according to the distillation loss and the distillation loss weight, wherein the distillation loss weight is determined according to the mapping relationship between the distillation loss weight and the distillation loss. The gradient of the modal loss is calculated as the backpropagation gradient in each modal encoder, and used as the modal loss gradient vector for each modal encoder.

2. The method according to claim 1, characterized in that, Based on the task prediction results, calculate the task loss gradient vector for each modality encoder, including: Based on the predicted and actual results of the task, the task loss is obtained; The gradient of the task loss is calculated as the gradient of the backpropagation in each modal encoder, and is used as the task loss gradient vector of each modal encoder.

3. The method according to claim 1, characterized in that, The modal loss includes a weighted distillation loss, and a first regularization term and a second regularization term; The first regularization term uses the moving average of distillation loss as a coefficient. The first regularization term amplifies the weight of distillation loss that is less than the average and reduces the weight of distillation loss that is greater than the average. The regularization coefficient of the second regularization term is predefined, and the second regularization term is used to control the distillation loss weight to tend to 1.

4. The method according to claim 1, characterized in that, Based on the task loss gradient vector and the modal loss gradient vector, determine the relational values ​​for each modal encoder, including: With the goal of achieving Pareto optimality for updating the task loss gradient vector and the modal loss gradient vector, a gradient objective function is constructed for each modal encoder. The gradient objective function is solved to obtain the relational values ​​for each modal encoder.

5. The method according to claim 1, characterized in that, The modal encoder with the largest relational value learns from the task according to the following steps: Calculate the weighted task loss gradient vector based on the task prediction result and the task loss weight. The parameters of the modal encoder with the largest relational value are updated based on the weighted task loss gradient vector.

6. The method according to claim 1, characterized in that, The modal encoders other than the one with the largest relational value learn from the modality according to the following steps: Based on the sample representation results of each modal encoder and the distillation loss weight corresponding to each modal encoder, the weighted modal loss gradient vector is calculated; The parameters of the modal encoder are updated based on the weighted modal loss gradient vector.

7. The method according to claim 1, characterized in that, The multimodal learning course includes multiple learning rounds, and the method further includes: After completing the current learning round, the relation values ​​of each modal encoder are recalculated to obtain new relation values; In the next learning round, the modal encoder with the largest new relation value is selected to learn from the task, and the remaining modal encoders, excluding the modal encoder with the largest new relation value, are selected to learn from the modality.

8. A multimodal curriculum learning device based on intramodal and intermodal learning, characterized in that, The device includes: The processing module is used to perform task processing on multimodal data using a multimodal module containing multiple modal encoders, to obtain task prediction results and sample representation results of each modal encoder. The multimodal data includes single-modal data of various different data types, and the single-modal data includes at least: images, audio, text and tables. The calculation module is used to calculate the task loss gradient vector of each modal encoder based on the task prediction results, and to calculate the modal loss gradient vector of each modal encoder based on the sample characterization results. The determination module is used to determine the relational value of each modal encoder based on the task loss gradient vector and the modal loss gradient vector, wherein the relational value represents the learning direction of the modal encoder; The selection module is used to select the modal encoder with the largest relation value to learn from the task, and to select the remaining modal encoders other than the modal encoder with the largest relation value to learn from the modality. The computing module includes: The distillation module is used to input single-modal data into the corresponding modality teacher model for processing, and obtain the knowledge distillation result of the single-modal data. The teacher model is a pre-trained modality encoder. The distillation loss module is used to determine the distillation loss based on the sample characterization results and the knowledge distillation results, wherein the distillation loss characterizes the difficulty of the single-modal data; The modal loss module is used to obtain the modal loss based on the distillation loss and distillation loss weights, with the optimization objective being minimizing the modal loss using a self-stepping learning method. The distillation loss weights are determined based on the mapping relationship between the distillation loss weights and the distillation loss. The modal gradient module is used to calculate the gradient of the modal loss as it propagates back through each modal encoder, serving as the modal loss gradient vector for each modal encoder.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal curriculum learning method based on intramodal and intermodal learning as described in any one of claims 1-7.

Images