A balanced multi-modal learning method based on knowledge distillation

Abstract

The application provides a balanced multi-modal learning method based on knowledge distillation, comprising the following steps: acquiring multi-modal data, determining a first source modality and a first target modality from the multi-modal data; encoding the first source modality to obtain a second target modality; encoding the second target modality to obtain a second source modality; calculating a reverse translation loss between the first source modality and the second source modality, and calculating a forward translation loss between the first target modality and the second target modality; obtaining a task loss of a reverse intermediate vector in completing a downstream task; balancing the training degrees of the second target modality and the second source modality, and obtaining a distillation loss through adaptive adjustment of modality training balancing; calculating a total loss, training a model by minimizing the total loss; and repeatedly performing steps S1-S7 until the model reaches a preset value. The application effectively alleviates the imbalance problem of inter-modal optimization, and improves the overall performance and robustness of the model.

Description

Technical Field

[0001] This invention relates to the fields of balanced learning and multimodal learning, specifically to a balanced multimodal learning method based on knowledge distillation. Background Technology

[0002] Multimodal learning, by establishing connections between different modalities and complementing their information, learns better feature representations, thereby improving model training effectiveness. It has been widely applied in fields such as medical diagnostic analysis, autonomous driving, video analysis, and emotion recognition. However, in actual training, it has been found that even when multimodal models outperform single-modal models, the information from these different modalities is still not fully utilized, resulting in an imbalance in optimization between modalities. Knowledge distillation refers to a method of transferring knowledge from a complex model to a simplified model to improve performance and reduce computational resource requirements.

[0003] To address the imbalance problem in multimodal learning, existing solutions mainly include regularization and gradient adjustment. To mitigate the imbalance from different perspectives, existing techniques employ various methods. One involves randomly dropping certain modalities during training as a regularization technique to adjust the learning speed between different modalities. Another approach is to use an additional classifier for a particular modality and its fusion modality, where the classifier obtains the optimal mixing weights by fitting the behavior. Yet another approach is to adjust the gradient to slow down the learning rate of a dominant modality, thereby reducing its inhibitory effect on other modalities.

[0004] However, existing technologies have the following shortcomings: traditional balanced multimodal learning methods require the introduction of additional neural modules, which makes the training process more complex and expensive; balanced multimodal learning methods combined with regularization ignore the accuracy of the task model, and the high randomness makes the training process of the model unstable and the model more difficult to interpret; balanced multimodal learning combined with gradient adjustment ignores the generality of the method and makes the model more difficult to interpret. Summary of the Invention

[0005] This invention provides a balanced multimodal learning method based on knowledge distillation.

[0006] One aspect of the present invention provides a balanced multimodal learning method based on knowledge distillation, comprising the following steps:

[0007] S1. Acquire multimodal data, and determine a first source mode and a first target mode from the multimodal data;

[0008] S2. Encode the first source mode to obtain the second target mode;

[0009] S3. Encode the second target mode to obtain the second source mode;

[0010] S4. Calculate the inverse translation loss between the first source mode and the second source mode, and calculate the forward translation loss between the first target mode and the second target mode;

[0011] S5. Obtain the task loss for completing the downstream task by obtaining the reverse intermediate vector;

[0012] S6. Balance the training levels of the second target mode and the second source mode, and obtain the distillation loss by adaptively adjusting the mode training balance.

[0013] S7. Calculate the total loss and train the model by minimizing the total loss;

[0014] S8. Repeat steps S1-S7 until the model reaches the preset value.

[0015] Furthermore, step S1 specifically includes the following steps:

[0016] The first source mode is forward encoded to obtain the forward intermediate vector;

[0017] The first target mode is obtained by forward encoding the forward intermediate vector.

[0018] Furthermore, step S2 specifically includes the following steps:

[0019] The first target mode is reverse encoded to obtain the reverse intermediate vector;

[0020] The second source mode is obtained by reverse decoding the inverse intermediate vector.

[0021] Furthermore, the balancing of the training levels of the second target modality and the second source modality includes:

[0022] Use the second objective modality to complete the downstream task and calculate the task effect;

[0023] Use the second source mode to complete the downstream task and calculate the task effect;

[0024] Comparing the overall performance of the two, if the generated target modality performs better in completing the task, it indicates that the encoder is trained better, meaning the source modality is trained to a higher degree. Conversely, if the decoder is trained worse, it indicates that the target modality is trained to a higher degree.

[0025] Furthermore, the distillation loss obtained by adaptively adjusting modal training balancing includes:

[0026] The predicted labels generated when calculating the task loss are used as soft labels for the teacher model.

[0027] The labels generated by the modalities with lower training levels in S6 are used as student labels for the student model.

[0028] Knowledge distillation was performed using soft labels and student labels, and the distillation loss for both was calculated.

[0029] Furthermore, the model is trained by minimizing the total loss, specifically by updating the model parameters by minimizing the total loss through backpropagation, in order to continuously improve the model's performance on various tasks and modalities.

[0030] The present invention has the following advantages over the prior art:

[0031] Superior Model Performance: This invention designs a balanced multimodal learning scheme based on knowledge distillation, which effectively alleviates the imbalance problem between modal optimizations and improves the overall performance and robustness of the model.

[0032] Excellent model versatility: This invention designs an adaptive modality training balancing scheme that can be embedded into existing multimodal fusion methods and model structures.

[0033] Lower model training overhead: The framework designed in this invention does not introduce additional neural modules, but optimizes the original model structure. This approach aims to reduce the computational overhead of balancing and improve the computational efficiency of balanced multimodal learning. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the method provided in the disclosed embodiments of the present invention.

[0035] Figure 2 This is a schematic diagram of the knowledge distillation process in the method provided in the disclosed embodiments of the present invention.

[0036] Figure 3 This is a schematic diagram of the multimodal learning process in the method provided in the embodiments of the present invention. Detailed Implementation

[0037] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0038] This disclosure provides a balanced multimodal learning method based on knowledge distillation, such as... Figure 1-3 As shown,

[0039] The steps include: S1, acquiring multimodal data, and determining a first source mode and a first target mode from the multimodal data;

[0040] S2. Encode the first source mode to obtain the second target mode;

[0041] S3. Encode the second target mode to obtain the second source mode;

[0042] S4. Calculate the inverse translation loss between the first source mode and the second source mode, and calculate the forward translation loss between the first target mode and the second target mode;

[0043] S5. Obtain the task loss for completing the downstream task by obtaining the reverse intermediate vector;

[0044] S6. Balance the training levels of the second target mode and the second source mode, and obtain the distillation loss by adaptively adjusting the mode training balance.

[0045] S7. Calculate the total loss and train the model by minimizing the total loss;

[0046] S8. Repeat steps S1-S7 until the model reaches the preset value.

[0047] In the first step, a modal input is selected as the source mode X. S Another mode X T As the target mode.

[0048] In the second step, during the forward encoding process, encoder f... e For the input mode X S Forward encoding is performed by the encoder f e Typically, a deep neural network generates a positive intermediate vector. This intermediate vector ε S→T Modal X was captured S The feature representation. Then the decoder f is used. d Receive the positive intermediate vector ε S→T The decoder f d Typically, this is a deep neural network, and it focuses on this positive intermediate vector ε. S→T Perform forward decoding to obtain the generated target mode. In this way, the model can translate the source mode into the corresponding target mode.

[0049] In the third step, reverse encoding is the inverse process of forward decoding, which generates the target mode. via encoder f e Reverse encoding yields a reverse intermediate vector. This vector ε T→S Captured the target mode of generation The feature representation. Then the decoder f is used. d Reverse decoding of the inverse intermediate vector generates the source mode. This step enables reverse translation from the target mode to the source mode, ensuring that the model can perform bidirectional conversion between different modes. In other words, this reverse intermediate vector captures the feature representations of both the target mode and the source mode, fusing the characteristic information of the two modes.

[0050] In the fourth step, the target mode X is compared. T With the generation of target modes The difference between them is used to calculate the forward translation loss. Forward translation loss measures the accuracy of the translation by comparing the difference between the generated target mode and the actual target mode; it measures the accuracy by comparing the source mode X. S With the source mode of generation The difference between them is used to calculate the back translation loss. This ensures the model learns an invertible mapping. These two losses are used to guide the model to learn better cross-modal feature representations.

[0051] The fifth step involves downstream tasks such as classification and regression, utilizing the inverse intermediate vector ε. T→S Execute downstream tasks Calculate task loss Typically calculated from the difference between predicted values ​​and actual labels, the model is guided to accurately complete the task, thereby enhancing the model's understanding and processing of cross-modal feature representations, i.e., learning better fusion representations.

[0052] In step six, when evaluating the degree of modality training, the generated target modality is compared. and the generation source mode The effectiveness of each entity performing downstream tasks is quantified, i.e. and This can include metrics such as classification accuracy and regression precision. By comparing the results, the encoder's performance can be evaluated. e and decoder f d The relative performance of the source and target modalities is used to understand their relative training levels. Then, the predicted labels generated from the inverse intermediate vectors are analyzed. As soft labels for the teacher model, this is a method that uses information from the generated modalities to guide training. The student model uses labels generated from less trained modalities to preserve the student model's training information on specific modalities. Distillation loss is calculated by comparing the output probability distributions of the teacher and student models. This ensures that the student model maintains high performance on the less trained modalities, thereby promoting the learning of the weaker modalities and alleviating the imbalance problem in the multimodal learning process.

[0053] In step seven, calculate the total loss of the model. This is the sum of the proportions of forward translation loss, backward translation loss, task loss, and distillation loss, where λ1, λ2, λ3, and λ4 can be determined based on the specific needs of the actual situation to minimize the total loss. Then, backpropagation is used to minimize the total loss. The model parameters are updated to continuously improve its performance across various tasks and modalities.

[0054] Step eight involves repeating steps one through seven, iterating multiple times to gradually optimize the model until the desired effect is achieved. The number of iterations and the convergence of the training directly affect the performance and generalization ability of the final model. The end of the training process can also be determined by setting the number of iterations or specifying a fixed model accuracy value.

[0055] This disclosure uses image-audio modalities as an example, and describes the balanced training of image-audio modalities based on knowledge distillation.

[0056] In multimodal sentiment analysis scenarios, data often contains multiple modalities, such as audio, text, video, and images. By comprehensively utilizing information from multiple input modalities, sentiment analysis can be performed more comprehensively, improving model accuracy. However, it has been found that even when a multimodal model outperforms its unimodal counterpart, the multimodal information is not fully utilized, exhibiting an imbalance in optimization training between different modalities. For example, in sentiment analysis tasks involving image and audio modalities, when one modality's training optimization is complete, another modality's training is not yet finished, at which point overall training stops, resulting in the underutilization of information from the other modality. To alleviate this training optimization imbalance problem and improve the model's accuracy and robustness, we adopt a balanced training scheme for image and audio modalities based on knowledge distillation. The specific implementation process is as follows:

[0057] In multimodal sentiment analysis tasks with image and audio modal data, the text modality is identified as the first source modality, and the image modality as the first target modality. There are no fixed requirements for the first source modality and the first target modality; they only need to be different modalities.

[0058] The text is encoded to obtain an intermediate vector, and the intermediate vector is then decoded to obtain the generated image.

[0059] The generated image is passed through an encoder to obtain a new intermediate vector, and the new intermediate vector is then passed through a decoder to obtain the generated text.

[0060] The reverse translation loss is calculated by comparing the generated text with the original text, and the forward translation loss is calculated by comparing the generated image with the original image.

[0061] The new intermediate vector is used to perform sentiment analysis to obtain predicted labels, which are then compared with the true labels to calculate the task loss.

[0062] The generated text and images are used to perform sentiment analysis, and their predicted labels are compared with the true labels to determine which modality has insufficient training optimization. The predicted labels of the modal with insufficient training optimization are distilled with the predicted labels from the previous step, and the distillation loss is calculated to drive further training optimization of this modality.

[0063] Calculate the total loss for the above tasks, including back translation loss, forward translation loss, task loss, and distillation loss, and continuously train and optimize the model by minimizing the total loss;

[0064] Repeat steps S1-S7 until the model reaches the preset value.

[0065] The final result is a multimodal sentiment analysis model with balanced training across modalities.

[0066] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for descriptive purposes only and is not intended to limit the claims.

Claims

1. A balanced multimodal learning method based on knowledge distillation, characterized in that, The steps include the following: Step S1: Acquire multimodal data, and determine a first source modality and a first target modality from the multimodal data; the first source modality is a text modality, and the first target modality is an image modality; Step S2: Encode the first source mode to obtain the second target mode; Step S3: Encode the second target mode to obtain the second source mode; The second target modality is the generated image, and the second source modality is the generated text; Step S4: Calculate the back translation loss between the first source mode and the second source mode, and calculate the forward translation loss between the first target mode and the second target mode; By comparing target modes With the second target mode The difference between them is used to calculate the forward translation loss. Forward translation loss measures the accuracy of the translation by comparing the difference between the generated first target mode and the actual target mode; it also measures the accuracy by comparing the first source mode. With the second source mode The difference between them is used to calculate the back translation loss. This ensures that the model learns an invertible mapping; Step S5: Obtain the task loss for completing the downstream task using the reverse intermediate vector; Step S6: Balance the training levels of the second target mode and the second source mode, and obtain the distillation loss by adaptively adjusting the mode training balancing. Step S7: Calculate the total loss and train the model by minimizing the total loss; Step S8: Repeat steps S1-S7 until the model reaches the preset value; Step S1 specifically includes the following steps: The first source mode is forward encoded to obtain the forward intermediate vector; The second target mode is obtained by forward decoding the positive intermediate vector; Step S2 specifically includes the following steps: The first target mode is reverse encoded to obtain the reverse intermediate vector; The second source mode is obtained by inverse decoding of the inverse intermediate vector; The distillation loss obtained by adaptively adjusting modal training balancing includes: The predicted labels generated when calculating the task loss are used as soft labels for the teacher model; The labels generated by the modal with lower training level in step S6 are used as the student labels for the student model. Knowledge distillation was performed using soft labels and student labels, and the distillation loss for both was calculated.

2. The balanced multimodal learning method based on knowledge distillation according to claim 1, characterized in that, The balancing of the training levels of the second target modality and the second source modality includes: Use the second objective modality to complete the downstream task and calculate the task effect; Use the second source mode to complete the downstream task and calculate the task effect; Comparing the overall performance of the two, if the generated target modality performs better in completing the task, it indicates that the encoder is trained better, meaning the source modality is trained to a higher degree. Conversely, if the decoder is trained worse, it indicates that the target modality is trained to a higher degree.

3. The balanced multimodal learning method based on knowledge distillation according to claim 1, characterized in that, The model is trained by minimizing the total loss, specifically by updating the model parameters by minimizing the total loss through backpropagation, in order to continuously improve the model's performance on various tasks and modalities.

Images