A multi-modal trained dynamic prediction adjustment strategy method

Abstract

The application realizes a dynamic prediction adjustment strategy method for a video and audio multi-modal recognition task through a method in the field of artificial intelligence. By calculating the imbalance ratio between different modalities in the feedforward stage, the difference degree of parameter optimization between different modalities is measured, and the advantage modal training data is discarded with a certain probability in the subsequent feedforward stage, so as to balance the training process. The method provided by the application can improve the imbalance phenomenon in the multi-modal model optimization process in the multi-modal task scene such as multi-modal behavior recognition, visual question answering and multi-modal emotion recognition, so that the training model suitable for specific multi-modal tasks can achieve more accurate and more robust effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a dynamic prediction adjustment strategy method. BACKGROUND

[0002] Some recent researches found that in the visual and audio multi-modal joint learning and perception tasks, there is a widespread problem: logically, multi-modal models are expected to be better than single-modal models, because they obtain data containing more information from multiple heterogeneous modalities. In most cases, it does achieve this purpose, but sometimes it is the opposite, multi-modal models may not be as good as single-modal models, and some researchers in recent researches believe that different modalities behave differently in the optimization process. For example, in the video recognition task, the audio modality often has a faster learning speed compared with the visual modality. Under the unified joint training optimization goal, it is difficult for the multi-modal model to learn all the modal representations well at the same time. Therefore, the potential of the multi-modal model may be limited by the difference in the optimization degree of different modalities, which further limits the overall multi-modal model effect.

[0003] For the inter-modal representation imbalance problem, some technical methods have been proposed previously, for example, some researchers proposed a method to increase the number of additional classifiers participating in training, to find the appropriate modality-specific coefficients by stage training and apply them to joint training; some researchers also choose to use more complex models, borrowing the idea of knowledge distillation to help the multi-modal model retain good representation ability for single-modal features. Although these methods can alleviate the problem of imbalance in inter-modal representation ability optimization to some extent, they all have the common shortcoming of increasing the additional training cost and cost (additional training modules or additional training process), and lack compatibility for various models and methods. Therefore, it is of great significance to explore a relatively effective and universal balanced multi-modal learning method.

[0004] In view of the problem of poor recognition accuracy caused by imbalance in the visual and audio multi-modal recognition and classification task, the above problem is caused by a dominant modality in the multi-modal perception task scenario. We propose a method that can adaptively adjust the training process according to the imbalance degree of the modality-specific module representation in training - the dynamic gradient modulation method (On-the-fly gradient modulation, referred to as OGM method). For multi-modal learning tasks such as equal visual and audio behavior recognition, for a given multi-modal training data and a deep neural network to be trained, the contribution of each modality to the overall discriminant model is measured, and the gradient optimization of each modality is adaptively controlled by monitoring the difference in their contribution to the learning goal. Using this method will be able to obtain a more balanced, more accurate and better comprehensive performance training model in multi-modal behavior recognition and other tasks.Figure 1 It is revealed that the representation ability of each single modality is worse than that of the independent single modality model in common multi-modal learning, which confirms that the phenomenon of sub-optimal multi-modal learning is real and will adversely affect the final model effect. Then we further think and explore, hoping to achieve similar regulation effect in the feedforward stage of model training. In view of the problem of low recognition accuracy caused by imbalance difference in audio-visual multi-modal recognition task scene, we originally propose an on-the-fly prediction modulation (OPM algorithm) to obtain a more accurate and efficient optimization model. And further improve the model for audio-visual multi-modal scene. SUMMARY

[0005] To this end, the application first proposes a dynamic prediction adjustment strategy method, establishes a dynamic prediction adjustment strategy algorithm for audio-visual multi-modal perception task, trains the model to be trained, and realizes more balanced and effective neural network training by adjusting only in the feedforward training stage. The input of the algorithm scheme is the neural network model to be trained and the training data of multiple modalities, such as video data containing pictures, sounds, etc.; the output is an efficient audio-visual multi-modal neural network model after optimization training, which can process corresponding to a specific multi-modal task.

[0006] The feedforward training stage is realized through five steps: step one, propose to estimate the single modality discrimination performance; step two, measure the discrimination difference ratio of a specific modality; step three, calculate the random probability of discarding the strong modality in the discrimination difference ratio in step two; step four, calculate the random probability of discarding the weak modality in the discrimination difference ratio in step two; step five, the discrimination difference ratio between modalities and the modality-specific discard probability value are calculated and updated in each batch processing stage, and the operation is performed according to the next batch processing operation.

[0007] The step one is to first perform the normal feedforward process when the model is trained in the audio-visual multi-modal mode, and the overall prediction result of the model is divided into two parts corresponding to the two parts, then The approximate prediction value of a specific modality (as above, u∈{a,v} represents two different modality inputs) is obtained i ,y i} i＝1,2...N , where the form of a single sample data is The visual information and the audio information are processed separately, and the model encoder parameters The model can learn parameters θ u , the model classifier parameters and b∈R M, where the multiple modalities involved in the operation are denoted by u e {a, v}. Note that this pre-test is specific to a particular sample, and further considering batch processing of data, a more representative modality-specific evaluation indicator for a batch of training samples can be obtained and As follows:

[0008]

[0009]

[0010] The step two is specifically measured by the following method to determine the discriminant difference ratio of a specific modality And it is used as the basis for subsequent training control strength, taking the ratio of modality a to v as an example:

[0011]

[0012] The specific calculation method of the random probability of the step three is:

[0013] Wherein And lambda is a hyperparameter adapted to a specific multimodal learning task, and z(·) is a monotonically increasing function with a value range of 0 to 1.

[0014] In the step four, for the case of , the model discards its input value In the feedforward stage of the next round of training with a probability of

[0015]

[0016] The technical effect to be achieved by the present application is to improve the mainstream tasks in the multimodal scene joint learning of audio-visual multimodal scenes, such as multimodal behavior recognition, visual question answering, and multimodal emotion recognition. By inputting the neural network model to be trained and the training data of multiple modalities including picture, sound and video data, an efficient multimodal neural network model optimized after training is output.

[0017] The method can improve the imbalance in the optimization process of the video-audio multimodal model to some extent, so that the training model suitable for a specific multimodal task can achieve more accurate and more robust results. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A curve graph of the respective representation capabilities of the two modalities under different training modes;

[0019] Figure 2A dynamic prediction adjustment OPM method framework flowchart;

[0020] Figure 3 A dynamic prediction adjustment OPM strategy algorithm flowchart; DETAILED DESCRIPTION

[0021] The following is a preferred embodiment of the present application and further describes the technical solutions of the present application in conjunction with the accompanying drawings, but the present application is not limited to this embodiment.

[0022] The present application proposes a dynamic prediction adjustment strategy method.

[0023] From the perspective of multi-modal learning optimization, the analysis of model representation sub-optimization is given. Taking a common deep learning multi-modal joint learning model as an example when processing video data, the relevant definitions are given: multi-modal data input D={x i ,y i} i＝1,2...N , wherein the single video sample data form is The visual information and the audio information therein are processed separately; two models to be trained encoder parameters are used to process and encode the original data of the two modalities respectively; the learnable parameters θ u of the model to be trained are the main parameters to be trained, and the model to be trained classifier parameters and b∈R M are used to realize the final video classification result output, and the multiple modalities participating in the operation are represented by u∈{a,v}, representing audio (audio) and visual (visual).

[0024] Therefore, in the multi-modal visual and audio perception task, the overall calculation of the model prediction value in the feedforward stage of the model training can be described as follows:

[0025]

[0026]

[0027] Similarly, the update of the two groups of main parameters W and θ in the feedback stage of the model training can be described as follows:

[0028]

[0029]

[0030] With the above feedforward and feedback forms, the training update process of the gradient in the model parameters can be combined and expressed as follows:

[0031]

[0032] According to the formula, it can be seen that in the gradient back propagation update stage, the size of its gradient value also depends on the numerical value of the two modal discriminant indicators. At this time, if the discriminant indicator of one of the modalities is relatively high, it may dominate the entire gradient update training process, even if the other relatively weak modality is still far from being optimized. Therefore, the modality with relatively low confidence in the correct class in a specific task can only obtain limited optimization, resulting in that the modality information is not fully utilized. Overall, the better performing modality dominates the optimization process, which will inevitably lead to the fact that when the multi-modal model approaches convergence, the less discriminative modality may still have a poor feature encoder optimization degree, which needs further training.

[0033] Based on the above problem analysis, in order to solve the under-optimization problem of the weak modality, the dynamic prediction adjustment strategy OPM method is designed to weaken the dependence of multi-modal prediction on the dominant modality. In the feedforward stage, the better performing modality feature is randomly discarded with a certain probability, thereby accelerating the training of the suppressed modality. The probability is determined by the discriminant difference ratio of the last iteration. In this way, the training speed and degree of the two modal corresponding model parameters can be effectively balanced, and the dynamic prediction adjustment strategy OPM method adaptively adjusts the drop probability of the modality with more discriminative information according to the discriminative difference degree between the modalities during the training process, so that the method can have better flexibility and universality.

[0034] The overall process is shown in Figure 2 The dynamic prediction adjustment strategy OPM method can be implemented in the following five steps:

[0035] Step one: the model video-audio multi-modal training is first carried out normally. Here, in order to monitor the discriminative difference between the modalities during the training process, it is first proposed to estimate the single modality discriminant performance. Inspired by deep Boltzmann machine, the model overall prediction result is divided into two parts respectively, then the following formula can be obtained is taken as the approximate prediction value of a certain modality (as above, u∈{a,v} represents two different modality inputs), and the multi-modal data input D={x i ,y i} i＝1,2...N , where the single sample data form is The visual information and the audio information are processed separately, and the model encoder parameters The model can learn the parameters θ u , the model classifier parameters and b∈R M, where the multiple modalities involved in the operation are denoted by u e {a, v}. Note that this pre-test is specific to the sample, so when we further consider batch processing of data, we can obtain a more representative modality-specific evaluation index for a batch of training samples and as follows:

[0036]

[0037]

[0038] Step two: since the modality with more discriminative information has a higher confidence in the correct class, i.e., the value of the above is higher. Then we propose to specifically measure the discriminative difference ratio for a specific modality and use it as a basis for subsequent training regulation. Taking the ratio of modality a to v as an example (similarly, the discriminative difference ratio of the other modality is the reciprocal):

[0039]

[0040] Step three: the discriminative difference ratio is often greater than 1 (considering a as the dominant modality), and in order to suppress the rapid convergence of the modality for calculation needs, we need to discard it with a higher random probability in the subsequent training round of the forward training phase. For the specific calculation of the discard probability , based on theoretical analysis and practical considerations, the following form is given, where and λ are hyperparameters adapted to the specific multi-modal learning task, and z(·) is a monotonically increasing function with a value range of 0 to 1:

[0041]

[0042] Step four: it can be noted that in the case of , the training data random discard probability value will be directly set to 0, because the core goal of this scheme is to weaken the dependence of multi-modal prediction on the dominant modality, so we want the weaker modalities to be trained more, i.e., in most cases, we do not discard their forward training data. For the dominant modality, the model discards its input value in the forward phase of the next round of training with a probability of , as shown in the following formula, so as to slow down its training convergence speed and make the model's representation ability for the two modalities tend to be balanced. We do not make any artificial adjustments to the feedback phase of the model training.

[0043]

[0044] Step five: It is worth mentioning that the dynamic prediction adjustment strategy OPM scheme is adaptive to the adjustment of the training process. The above steps are performed at each batch stage, that is, the inter-modal discriminative difference ratio and the modal-specific dropout probability value are calculated and updated, and the operation is performed according to the next batch operation. Therefore, the whole process still belongs to an end-to-end single-stage process, and does not require additional separate calculation stages and additional training costs. Finally, we will get a more efficient, robust and general multi-modal neural network model after training.

[0045] In summary, the dynamic prediction adjustment strategy OPM method alleviates the imbalance learning problem existing in the feedforward stage of the audio-visual multi-modal recognition task. Through the dynamic adjustment strategy proposed by us, the influence of the multi-modal prediction of the relatively weak modal is relatively weakened, so that the two are more balanced. The greater the discriminative index difference here, the stronger the modulation strength. According to the expected analysis, the influence of the other relatively suppressed modal on multi-modal prediction is improved, the representation ability is increased, which improves the learning ability of audio-visual multi-modal prediction as a whole.

Claims

1. A method of dynamically adjusting a prediction strategy, the method comprising: A dynamic prediction adjustment strategy algorithm is established for audio-visual multi-model perception tasks, and the model to be trained is trained, and the training of the model is realized through adjustment only in the feedforward training stage to realize more balanced and effective neural network training, the input is the neural network model to be trained and the training data of multiple modalities, that is, video data containing pictures and sound; the output is an efficient audio-visual multi-modal neural network model optimized after training, which can process a specific multi-modal task, The feedforward training stage is realized through five steps: step one, proposing to estimate the single-modal discrimination performance; step two, specifically measuring the discrimination difference ratio of a specific modality; step three, calculating the random probability of discarding the strong modality in the discrimination difference ratio in step two; step four, calculating the weak modality in the discrimination difference ratio in step two to discard with a certain random probability; step five, the discrimination difference ratio between modalities and the modal-specific discard probability value are calculated and updated at each batch processing stage, and the next batch processing operation is operated according to the calculation result; The step one as a model video and audio multi-modal training first normal feedforward process, the model overall prediction results are divided into corresponding two parts, then can get As the approximate prediction value of a specific modal, to Indicates two different modal input, multi-modal data input , Where the single sample data form is , Visual information and audio information are processed separately, model encoder parameters , Model learnable parameters , Model classifier parameters And , Multiple modalities involved in operation are represented by ; Note that the prediction is for a specific sample, and further consider the batch processing of data, then get more representative modal-specific evaluation indicators for a batch of training samples And As follows: The step two is specifically measured by the following method to determine the discriminant difference ratio for a specific mode , and it is used as the basis for subsequent training control strength, to the mode The ratio of is taken as an example: The specific calculation method of the random probability in step three is: wherein and are hyperparameters adapted to the specific multimodal learning task, is a monotonically increasing function taking values between 0 and 1 ; In step four, for the case , the model drops its input values with probability : 。

Images