A method and device for multi-modal audio and video identification with adaptive modal balance

Abstract

The present application provides a method and device for multimodal audio and video identification with adaptive modal balance, which relates to the field of speech identification technology. In the training process of the multimodal model, the identification performance of the audio model and the video modality in the multimodal model is determined; according to the identification performance of the audio model and the video model respectively, the gradient compensation coefficient is allocated to the audio model and the video model respectively; according to the gradient compensation coefficient, the audio model and the video model are updated respectively to obtain a multimodal audio and video identification model; the audio and video to be detected are input into the multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected. Through the above method, the model can use the information of the audio and visual modalities in a more balanced way, thereby improving the performance of false video identification, and the gradient compensation strategy ensures that the model of each modality has better adaptability and detection ability for false audio and video that rely on audio and video.

Description

Technical Field

[0001] The present application relates to the technical field of speech recognition, and in particular to a method and device for multi-modal audio and video recognition with adaptive modal balance. Background Art

[0002] With the development of deep learning, multimodal learning has shown superiority in many tasks, such as action recognition, audio and video speech recognition, and visual question answering tasks. Multimodal learning provides more comprehensive information parsing capabilities by integrating data from different sensors, such as visual and auditory information. Although multimodal learning has shown significant advantages in tasks such as action recognition, audio and video speech recognition, and visual question answering.

[0003] However, although multimodal learning can theoretically make full use of information from different senses, in the task of fake video detection, the optimization of one modality in traditional multimodal learning models often suppresses the other modality, resulting in the model being unable to fully utilize all available information. This imbalanced optimization not only limits the overall performance of the model in the task of fake video detection, but may also lead to insufficient adaptability of the model to new scenes.

[0004] Based on this, there is an urgent need for a method that can effectively balance and optimize the various modalities in multimodal learning to improve the accuracy and generalization ability of fake video detection tasks. Summary of the invention

[0005] The present application provides a method and device for multi-modal generated audio and video identification with adaptive modal balance to solve the problems of optimization imbalance in multi-modal learning and poor accuracy and generalization of false video identification.

[0006] In a first aspect of an embodiment of the present application, a method for multi-modal generated audio and video identification with adaptive modal balance is provided, the method comprising:

[0007] During training of the multimodal model, determining a discriminative performance of an audio model in the multimodal model, and determining a discriminative performance of a video model in the multimodal model;

[0008] According to the identification performance of the audio model and the video model, respectively, a gradient compensation coefficient is allocated to the audio model and the video model, wherein the better the identification performance is, the smaller the gradient compensation coefficient is allocated;

[0009] According to the gradient compensation coefficients respectively assigned to the audio model and the video model, the audio model and the video model are respectively updated to obtain a multimodal audio and video identification model;

[0010] The audio and video to be detected is input into the multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected.

[0011] In an optional embodiment of the present application, the method specifically includes:

[0012] The audio model in the multimodal model is used to extract the audio features of each sample speech in the sample speech set, and a corresponding audio identification result is generated according to the audio features of each sample speech;

[0013] The video model in the multimodal model is used to extract the video features of each sample video in the sample video set, and a corresponding video identification result is generated according to the video features of each sample video;

[0014] Generate a score of the audio modality according to the audio identification result of each sample speech in the sample speech set and the label of each sample speech, and generate a score of the video modality according to the video identification result of each sample video in the sample video set and the label of each sample video;

[0015] Based on the scores of the audio modality and the video modality, the contribution of the audio modality and the video modality to the audio and video authentication is calculated;

[0016] Based on the contribution of the audio mode and the video mode to the audio and video authentication, the audio gradient compensation coefficient and the video gradient compensation coefficient are calculated respectively;

[0017] Performing a gradient update on the audio model in the multimodal model based on the audio gradient compensation coefficient, and performing a gradient update on the video model in the multimodal model based on the video gradient compensation coefficient, to obtain the multimodal audio and video identification model;

[0018] The audio and video to be detected is input into the multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected.

[0019] In an optional embodiment of the present application, generating a score of an audio modality according to an audio identification result of each sample voice in the sample voice set and a label of each sample voice, and generating a score of a video modality according to a video identification result of each sample video in the sample video set and a label of each sample video, including:

[0020] Compare the audio identification result of each sample speech in the sample speech set with the label of each sample speech, and count a first number of sample speech having the same audio identification result as the corresponding label;

[0021] Determine a score of the audio mode according to the first counted quantity;

[0022] Comparing the video identification result of each sample video in the sample video set with the label of each sample video, and counting a second number of sample videos whose video identification result is the same as the corresponding label;

[0023] The score of the video modality is determined according to the second counted quantity.

[0024] In an optional embodiment of the present application, based on the scores of the audio modality and the video modality, the contribution of the audio modality and the video modality to the audio and video authentication is calculated, including:

[0025] Dividing the sample speech set and the sample video set into a plurality of batches of sample speech subsets and sample video subsets;

[0026] Calculate a first total score of each sample speech in the sample speech subset of the i-th batch and a second total score of each sample video in the sample video subset;

[0027] The contribution of the audio modality of the i-th batch to the audio and video authentication is: the ratio of the first total score to the first total score and the second total score;

[0028] The contribution of the video modality of the i-th batch to audio and video authentication is: the ratio of the second total score to the first total score and the second total score.

[0029] In an optional embodiment of the present application, based on the contribution of the audio modality and the video modality to the audio and video authentication, the audio gradient compensation coefficient and the video gradient compensation coefficient are calculated respectively, including:

[0030] Contribution of audio modality to audio and video authentication c a , determine the audio gradient compensation coefficient as

[0031] Contribution of video modality to audio and video authentication c v , determine the video gradient compensation coefficient as

[0032] In an optional embodiment of the present application, performing a gradient update on the audio model in the multimodal model based on the audio gradient compensation coefficient includes:

[0033] Determining a parameter update amount of the audio model according to a difference between an audio identification result of each sample speech in the sample speech set and a label of each sample speech;

[0034] Based on the audio gradient compensation coefficient and the parameter update amount of the audio model, gradient updating the parameters of the audio model;

[0035] Determining a parameter update amount of the video model according to a difference between a video identification result of each sample video in the sample video set and a label of each sample video;

[0036] Based on the video gradient compensation coefficient and the parameter update amount of the video model, the parameters of the video model are gradient updated.

[0037] In a second aspect of an embodiment of the present application, a multi-modal generated audio and video identification device with adaptive modal balance is provided, the device comprising:

[0038] A determination module, used to determine the identification performance of the audio model in the multimodal model and the identification performance of the video model in the multimodal model during the training of the multimodal model;

[0039] an allocating module, configured to allocate gradient compensation coefficients to the audio model and the video model respectively according to their respective identification performances, wherein the better the identification performance, the smaller the allocated gradient compensation coefficient;

[0040] An updating module, configured to update the audio model and the video model respectively according to the gradient compensation coefficients respectively allocated to the audio model and the video model, so as to obtain a multimodal audio and video identification model;

[0041] The first identification module is used to input the audio and video to be detected into the multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected.

[0042] In an optional embodiment of the present application, the device specifically includes:

[0043] An audio feature extraction and identification module, used to extract the audio features of each sample speech in the sample speech set by using the audio model in the multimodal model, and generate a corresponding audio identification result according to the audio features of each sample speech;

[0044] A video feature extraction and identification module, used to extract the video features of each sample video in the sample video set by using the video model in the multimodal model, and generate a corresponding video identification result according to the video features of each sample video;

[0045] A score generating module, configured to generate a score of an audio modality according to an audio identification result of each sample speech in the sample speech set and a label of each sample speech, and to generate a score of a video modality according to a video identification result of each sample video in the sample video set and a label of each sample video;

[0046] A contribution calculation module, used to calculate the contribution of the audio modality and the video modality to the audio and video authentication based on the scores of the audio modality and the video modality respectively;

[0047] A gradient compensation coefficient calculation module, used to calculate the audio gradient compensation coefficient and the video gradient compensation coefficient respectively based on the contribution of the audio mode and the video mode to the audio and video authentication;

[0048] A gradient updating module, configured to perform a gradient update on an audio model in the multimodal model based on the audio gradient compensation coefficient, and to perform a gradient update on a video model in the multimodal model based on the video gradient compensation coefficient, so as to obtain the multimodal audio and video identification model;

[0049] The second identification module is used to input the audio and video to be detected into the multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected.

[0050] In a third aspect of an embodiment of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the multi-modal generated audio and video identification method with adaptive modal balance as described in any one of the first aspects above.

[0051] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the multi-modal generated audio and video identification method with adaptive modal balance described in any one of the first aspects above is implemented.

[0052] The present application includes the following advantages: The embodiment of the present application provides a method and device for multi-modal audio and video identification with adaptive modal balance. During the training process of the multi-modal model, the identification performance of the audio model in the multi-modal model is determined, and the identification performance of the video model in the multi-modal model is determined; according to the identification performance of the audio model and the video model, the gradient compensation coefficient is respectively allocated to the audio model and the video model, and the better the identification performance, the smaller the allocated gradient compensation coefficient; according to the gradient compensation coefficients respectively allocated to the audio model and the video model, the audio model and the video model are respectively updated to obtain a multi-modal audio and video identification model; the audio and video to be detected are input into the multi-modal audio and video identification model to obtain the multi-modal audio and video identification result of the audio and video to be detected. Through the above method, the model can use the information of audio and visual modalities more balanced, thereby improving the performance of false video identification, and the gradient compensation strategy ensures that each modality can be properly trained, so that it has better adaptability and detection ability for false audio and video that rely on audio and video. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 It is a flowchart of the steps of a multi-modal generated audio and video identification method with adaptive modal balance provided in an embodiment of the present application;

[0055] Figure 2 It is a flowchart of a method for multi-modal generation of audio and video identification with adaptive modal balance provided in an embodiment of the present application;

[0056] Figure 3 It is an architecture diagram of a multi-modal generated audio and video identification device with adaptive modal balance provided in an embodiment of the present application;

[0057] Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] With the development of deep learning, multimodal learning has shown its superiority in many tasks, such as action recognition, audio-video speech recognition, and visual question-answering tasks. Multimodal learning provides more comprehensive information parsing capabilities by integrating data from different sensors, such as visual and auditory information. This learning method simulates the way humans use multiple senses to understand the world, such as observing with eyes, listening with ears, and touching with hands to perceive the surrounding environment. In tasks such as action recognition, audio-video speech recognition, and visual question-answering, multimodal learning has shown significant advantages. Although multimodal learning has shown significant advantages in tasks such as action recognition, audio-video speech recognition, and visual question-answering. However, although multimodal learning can theoretically make full use of information from different senses, in practical applications, especially in complex fake video detection tasks, multimodal learning faces the challenges of imbalanced optimization and insufficient generalization ability.

[0060] However, existing multimodal learning models suffer from an imbalanced optimization phenomenon, that is, the optimization process of the model is often dominated by the modality with better performance, resulting in the failure to fully explore the potential of the other modality. This phenomenon manifests itself in the multimodal forged audio and video identification as follows: in some data, audio is relatively easy to identify, which will cause the video identification model to be unable to fully learn the knowledge of the video, resulting in poor training of the video identification model and poor multimodal identification results. In addition, in some data, the audio identification model cannot be fully trained because video identification is relatively easy.

[0061] In the existing multimodal learning field, especially in the task of fake video detection, a core issue is how to effectively integrate and optimize information from different modalities. In traditional multimodal learning models, the optimization of one modality often suppresses another modality, resulting in the model being unable to fully utilize all available information. For example, in an audio-visual fusion task, if the audio modality performs better than the visual modality, the optimization process of the model may be overly biased towards audio and ignore the potential value of visual information. This imbalanced optimization not only limits the overall performance of the model in the task of fake video detection, but may also lead to insufficient adaptability of the model to new scenarios. Therefore, this application focuses on developing a method that can effectively balance and optimize the various modalities in multimodal learning to improve the accuracy and generalization ability of fake video detection tasks.

[0062] Based on this, in order to solve the optimization imbalance problem in multimodal learning and improve the accuracy and generalization ability of fake video detection, this application proposes a multimodal audio and video authentication method based on gradient amplitude compensation.

[0063] In a first aspect of the present application, a method for multi-modal generated audio and video identification with adaptive modal balance is provided. Figure 1 , Figure 1 1 is a flowchart of a method for multi-modal generation of audio and video identification with adaptive modal balance provided in an embodiment of the present application, the method comprising the following steps:

[0064] Step 101: during the training of a multimodal model, determining the identification performance of an audio model in the multimodal model, and determining the identification performance of a video model in the multimodal model;

[0065] Step 102: assigning gradient compensation coefficients to the audio model and the video model respectively according to their respective identification performances, wherein the better the identification performance, the smaller the assigned gradient compensation coefficient;

[0066] Step 103: updating the audio model and the video model respectively according to the gradient compensation coefficients respectively allocated to the audio model and the video model to obtain a multimodal audio and video identification model;

[0067] Step 104: input the audio and video to be detected into the multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected.

[0068] In order to clearly illustrate a multi-modal generated audio and video identification method with adaptive modal balance provided by an embodiment of the present application, the following is combined with Figure 2 For instructions, see Figure 2 , Figure 2 : This is a flow chart of a method for multi-modal generation of audio and video identification with adaptive modal balance provided in an embodiment of the present application, which is specifically described as follows:

[0069] In the present application, in the process of training the multimodal model, in order to optimize the balance between the audio modality and the video modality, a gradient compensation coefficient is introduced to optimize the audio modality and the video modality. Specifically, the identification performance of the audio model and the video model in the multimodal model is first determined, and the gradient compensation coefficients are respectively assigned to the audio model and the video model according to the identification performance of the audio model and the video model, and the audio model and the video model are respectively updated by the assigned gradient compensation coefficients to obtain a multimodal audio and video identification model. The multimodal audio and video identification model trained by the above method is used to identify the audio and video to be detected, and the identification results of the audio model and the video model are integrated to obtain the identification results of the audio and video to be detected. Among them, the identification results of the audio model and the video model are obtained by multiplying the output of the encoder in the corresponding mode by the classification weight and adding the bias term.

[0070] In an optional embodiment of the present application, the above-mentioned multi-modal generation audio and video identification method with adaptive modal balance specifically includes: using the audio model in the multi-modal model to extract the audio features of each sample speech in the sample speech set, and generating a corresponding audio identification result based on the audio features of each sample speech, specifically using the audio modality encoder in the audio model to extract the features of the speech; using the video model in the multi-modal model to extract the video features of each sample video in the sample video set, and generating a corresponding video identification result based on the video features of each sample video, specifically using the video modality encoder in the video model to extract the features of the video; generating an audio modality score based on the audio identification result of each sample speech in the sample speech set and the label of each sample speech, and, based on According to the video identification result of each sample video in the sample video set and the label of each sample video, the score of the video modality is generated; based on the respective scores of the audio modality and the video modality, the contribution of the audio modality and the video modality to the audio and video authentication is calculated; based on the respective contributions of the audio modality and the video modality to the audio and video authentication, the audio gradient compensation coefficient and the video gradient compensation coefficient are calculated respectively; based on the audio gradient compensation coefficient, the audio model in the multimodal model is gradient updated, and based on the video gradient compensation coefficient, the video model in the multimodal model is gradient updated to obtain a multimodal audio and video authentication model; the audio and video to be detected are input into the multimodal audio and video authentication model, the authentication results of the audio model and the video model are obtained and merged, and the authentication results of the audio and video to be detected are obtained.

[0071] In an optional embodiment of the present application, a score of an audio modality is generated based on an audio identification result of each sample speech in the sample speech set and a label of each sample speech, and a score of a video modality is generated based on a video identification result of each sample video in the sample video set and a label of each sample video, specifically comprising: comparing the audio identification result of each sample speech in the sample speech set with the label of each sample speech, and counting a first number of sample speech having the same audio identification result as the corresponding label; determining the score of the audio modality based on the first number counted; comparing the video identification result of each sample video in the sample video set with the label of each sample video, and counting a second number of sample videos having the same video identification result as the corresponding label; and determining the score of the video modality based on the second number counted.

[0072] Optionally, the input data and weights in the audio modality or video modality are processed by a softmax function, and the obtained prediction result is compared with the true label corresponding to the modality to calculate the corresponding score. For the score of each sample speech and each sample speech set in the modality, or for the score of each sample video and each sample video set in the modality, the score can be calculated using the following formula:

[0073]

[0074] Wherein, m represents the mode. When m=1, it represents the audio mode, and when m=2, it represents the video mode. k represents the identification result of the audio model or the video model. k＝y Indicates that when k is consistent with the true label y (the true label includes two cases: true or false), the value is 1, and when k is inconsistent with the true label, the value of k is 0; W m represents the weight of the audio modality encoder or the video modality encoder; x m represents the input data of the current batch; M represents the number of correct predictions (i.e., the first number or the second number) in the current modality (audio modality or video modality). m It indicates the correct score of prediction in the current mode (audio mode or video mode). The more correct predictions, the higher the score.

[0075] In an optional embodiment of the present application, based on the scores of the audio modality and the video modality, the contribution of the audio modality and the video modality to the audio and video authentication is calculated, specifically including: dividing the sample speech set and the sample video set into multiple batches of sample speech subsets and sample video subsets; first calculating the first total score of each sample speech in the i-batch sample speech subset and the second total score of each sample video in the sample video subset; then, the ratio of the first total score to the first total score and the second total score is used as the contribution of the audio modality of the i-th batch to the audio and video authentication, and the calculation formula can be expressed as follows:

[0076]

[0077] The ratio of the second total score to the first total score and the second total score is taken as the contribution of the video modality of the i-th batch to the audio and video authentication. The calculation formula can be expressed as follows:

[0078]

[0079] Among them, c a represents the contribution of the audio modality of the i-th batch to the audio and video authentication, c v represents the contribution of the video modality of the i-th batch to the audio and video authentication, B represents the batch, represents the first total score, Indicates the second total score.

[0080] In an optional embodiment of the present application, based on the contribution of the audio mode and the video mode to the audio and video authentication, the audio gradient compensation coefficient and the video gradient compensation coefficient are calculated respectively, specifically including: based on the contribution of the audio mode to the audio and video authentication, c a, determine the audio gradient compensation coefficient as It can be expressed as Contribution of video modality to audio and video authentication c v , determine the video gradient compensation coefficient as It can be expressed as

[0081] In an optional embodiment of the present application, the audio model in the multimodal model is gradient updated based on the audio gradient compensation coefficient, and the video model in the multimodal model is gradient updated based on the video gradient compensation coefficient, specifically including: firstly, determining the parameter update amount of the audio model according to the difference between the audio identification result of each sample voice in the sample voice set and the label of each sample voice; based on the audio gradient compensation coefficient and the parameter update amount of the audio model, gradient updating the parameters of the audio model, which can be expressed as follows:

[0082] W' a =W a +ΔW a ·k a ,

[0083] According to the difference between the video identification result of each sample video in the sample video set and the label of each sample video, the parameter update amount of the video model is determined; based on the video gradient compensation coefficient and the parameter update amount of the video model, the parameters of the video model are gradient updated, which can be expressed as follows:

[0084] W′ v =W v +ΔW v ·k v ,

[0085] Among them, W' a represents the parameters of the audio model after gradient update, W a represents the parameters of the updated audio model, ΔW a represents the minimized parameter obtained by calculating the parameters of the audio model using the stochastic gradient descent algorithm, k a represents the audio gradient compensation coefficient; W′ v represents the parameters of the video model after gradient update, W v represents the parameters of the updated video model, ΔW v represents the minimized parameter obtained by calculating the parameters of the video model using the stochastic gradient descent algorithm, k v Represents the video gradient compensation coefficient.

[0086] The audio model updated by the audio gradient compensation coefficient and the video model updated by the video gradient compensation coefficient are fused to obtain a multimodal model for comprehensive identification of false audio and video. By introducing an adaptive gradient compensation mechanism, adaptive modal balance is achieved, thereby improving the learning effect of the single-modal identification model, thereby greatly improving the accuracy of the multimodal model in identifying false audio and video. At the same time, the multimodal learning method solves the optimization imbalance problem in tasks such as false audio and video detection, and improves the performance and generalization ability of the multimodal model in the above tasks.

[0087] The embodiment of the present application provides a method for multi-modal audio and video identification with adaptive modal balance. During the training process of the multi-modal model, the identification performance of the audio model in the multi-modal model is determined, and the identification performance of the video model in the multi-modal model is determined; according to the identification performance of the audio model and the video model, the gradient compensation coefficient is respectively assigned to the audio model and the video model, and the better the identification performance, the smaller the assigned gradient compensation coefficient; according to the gradient compensation coefficients respectively assigned to the audio model and the video model, the audio model and the video model are respectively updated to obtain a multi-modal audio and video identification model; the audio and video to be detected are input into the multi-modal audio and video identification model to obtain the multi-modal audio and video identification result of the audio and video to be detected. Through the above method, the model can use the information of audio and visual modalities more balanced, thereby improving the performance of false video identification, and the gradient compensation strategy ensures that each modality can be properly trained, so that it has better adaptability and detection ability for false audio and video that rely on audio and video.

[0088] In a second aspect of the present application, a multi-modal generated audio and video identification device with adaptive modal balance is provided, referring to Figure 3 , Figure 3 : is an architecture diagram of a multi-modal generated audio and video identification device with adaptive modal balance provided in an embodiment of the present application, the device comprising:

[0089] A determination module 301 is used to determine the identification performance of the audio model in the multimodal model and the identification performance of the video model in the multimodal model during the training process of the multimodal model;

[0090] An allocating module 302, configured to allocate gradient compensation coefficients to the audio model and the video model respectively according to their respective identification performances, wherein the better the identification performance, the smaller the allocated gradient compensation coefficient;

[0091] An updating module 303, configured to update the audio model and the video model respectively according to the gradient compensation coefficients respectively allocated to the audio model and the video model, so as to obtain a multimodal audio and video identification model;

[0092] The first identification module 304 is used to input the audio and video to be detected into the multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected.

[0093] Wherein, the device specifically comprises:

[0094] An audio feature extraction and identification module, used to extract the audio features of each sample speech in the sample speech set by using the audio model in the multimodal model, and generate a corresponding audio identification result according to the audio features of each sample speech;

[0095] A video feature extraction and identification module, used to extract the video features of each sample video in the sample video set by using the video model in the multimodal model, and generate a corresponding video identification result according to the video features of each sample video;

[0096] A score generating module, configured to generate a score of an audio modality according to an audio identification result of each sample speech in the sample speech set and a label of each sample speech, and to generate a score of a video modality according to a video identification result of each sample video in the sample video set and a label of each sample video;

[0097] A contribution calculation module, used to calculate the contribution of the audio modality and the video modality to the audio and video authentication based on the scores of the audio modality and the video modality respectively;

[0098] A gradient compensation coefficient calculation module, used to calculate the audio gradient compensation coefficient and the video gradient compensation coefficient respectively based on the contribution of the audio mode and the video mode to the audio and video authentication;

[0099] A gradient updating module, configured to perform a gradient update on an audio model in the multimodal model based on the audio gradient compensation coefficient, and to perform a gradient update on a video model in the multimodal model based on the video gradient compensation coefficient, so as to obtain the multimodal audio and video identification model;

[0100] The second identification module is used to input the audio and video to be detected into the multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected.

[0101] Wherein, the score generation module includes:

[0102] A first quantity statistics submodule, used for comparing the audio identification result of each sample speech in the sample speech set with the label of each sample speech, and counting a first quantity of sample speech whose audio identification result is the same as the corresponding label;

[0103] An audio mode score determination submodule, configured to determine a score of the audio mode according to the first counted number;

[0104] A second quantity counting submodule, used for comparing the video identification result of each sample video in the sample video set with the label of each sample video, and counting a second number of sample videos whose video identification result is the same as the corresponding label;

[0105] The video modality score determination submodule is used to determine the score of the video modality according to the second quantity obtained by statistics.

[0106] Wherein, the contribution calculation module includes:

[0107] A batch division submodule, used for dividing the sample speech set and the sample video set into a plurality of batches of sample speech subsets and sample video subsets;

[0108] A total score calculation submodule, used to calculate a first total score of each sample speech in the sample speech subset of the i-th batch and a second total score of each sample video in the sample video subset;

[0109] The contribution calculation submodule of the audio modality is used for the contribution of the audio modality of the i-th batch to the audio and video authentication: the ratio of the first total score to the first total score and the second total score;

[0110] The contribution calculation submodule of the video modality is used for the contribution of the video modality of the i-th batch to the audio and video authentication: the ratio of the second total score to the first total score and the second total score.

[0111] Wherein, the gradient compensation coefficient calculation module includes:

[0112] The audio gradient compensation coefficient determination submodule is used to determine the contribution of the audio modality to the audio and video authentication. a , determine the audio gradient compensation coefficient as

[0113] The video gradient compensation coefficient determination submodule is used to determine the contribution of the video modality to audio and video authentication. v , determine the video gradient compensation coefficient as

[0114] Wherein, the gradient updating module comprises:

[0115] A submodule for determining the parameter update amount of the audio model, used to determine the parameter update amount of the audio model according to the difference between the audio identification result of each sample speech in the sample speech set and the label of each sample speech;

[0116] A submodule for performing gradient updating of parameters of the audio model, configured to perform gradient updating of the parameters of the audio model based on the audio gradient compensation coefficient and the parameter update amount of the audio model;

[0117] A submodule for determining the parameter update amount of the video model, used to determine the parameter update amount of the video model according to the difference between the video identification result of each sample video in the sample video set and the label of each sample video;

[0118] The submodule for performing gradient updating of parameters of the video model is used to perform gradient updating of the parameters of the video model based on the video gradient compensation coefficient and the parameter update amount of the video model.

[0119] Based on the same application concept, the embodiment of the present application discloses an electronic device in a third aspect. Figure 4 A schematic diagram of an electronic device disclosed in an embodiment of the present application is shown. Figure 4 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory of the electronic device is not less than 12G, the main frequency of the processor is not less than 2.4GHz, the memory 110 and the processor 120 are connected via a bus communication, and a computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement a multi-modal generated audio and video identification method with adaptive modal balance disclosed in an embodiment of the present application.

[0120] Based on the same application concept, the fourth aspect of the embodiment of the present application discloses a computer-readable storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, a multi-modal generated audio and video identification method with adaptive modal balance disclosed in the embodiment of the present application is implemented.

[0121] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0122] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0126] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0127] The above is a detailed introduction to the multi-modal generated audio and video identification method and device with adaptive modal balance provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multi-modal generated audio and video identification method with adaptive modal balance, It is characterized in that The method comprises: During the training of the multimodal audio and video identification model, determining the identification performance of the audio model in the multimodal audio and video identification model, and determining the identification performance of the video model in the multimodal audio and video identification model; According to the identification performance of the audio model and the video model, respectively, a gradient compensation coefficient is allocated to the audio model and the video model, wherein the better the identification performance is, the smaller the gradient compensation coefficient is allocated; According to the gradient compensation coefficients respectively assigned to the audio model and the video model, the audio model and the video model are respectively updated to obtain an updated multimodal audio and video identification model; Inputting the audio and video to be detected into the updated multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected; The method specifically comprises: The audio model in the multimodal audio and video identification model is used to extract the audio features of each sample speech in the sample speech set, and the corresponding audio identification result is generated according to the audio features of each sample speech; The video model in the multimodal audio and video identification model is used to extract the video features of each sample video in the sample video set, and a corresponding video identification result is generated according to the video features of each sample video; Generate a score of the audio modality according to the audio identification result of each sample speech in the sample speech set and the label of each sample speech, and generate a score of the video modality according to the video identification result of each sample video in the sample video set and the label of each sample video; Based on the scores of the audio modality and the video modality, the contribution of the audio modality and the video modality to the audio and video authentication is calculated; Based on the contribution of the audio mode and the video mode to the audio and video authentication, the audio gradient compensation coefficient and the video gradient compensation coefficient are calculated respectively; Performing a gradient update on an audio model in the multimodal audio and video identification model based on the audio gradient compensation coefficient, and performing a gradient update on a video model in the multimodal audio and video identification model based on the video gradient compensation coefficient, to obtain the updated multimodal audio and video identification model; The audio and video to be detected is input into the updated multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected.

2. The method for multi-modal generated audio and video identification with adaptive modal balance according to claim 1, It is characterized in that Generating a score of an audio modality according to an audio identification result of each sample speech in the sample speech set and a label of each sample speech, and generating a score of a video modality according to a video identification result of each sample video in the sample video set and a label of each sample video, including: Compare the audio identification result of each sample speech in the sample speech set with the label of each sample speech, and count a first number of sample speech having the same audio identification result as the corresponding label; Determine a score of the audio mode according to the first counted quantity; Comparing the video identification result of each sample video in the sample video set with the label of each sample video, and counting a second number of sample videos whose video identification result is the same as the corresponding label; The score of the video modality is determined according to the second counted quantity.

3. The multi-modal generated audio and video identification method with adaptive modal balance according to claim 1, It is characterized in that Based on the scores of the audio modality and the video modality, the contribution of the audio modality and the video modality to the audio and video authentication is calculated, including: Dividing the sample speech set and the sample video set into a plurality of batches of sample speech subsets and sample video subsets; Calculate a first total score of each sample speech in the sample speech subset of the i-th batch and a second total score of each sample video in the sample video subset; The contribution of the audio modality of the i-th batch to the audio and video authentication is: the ratio of the first total score to the first total score and the second total score; The contribution of the video modality of the i-th batch to audio and video authentication is: the ratio of the second total score to the first total score and the second total score.

4. The method for multi-modal generated audio and video identification with adaptive modal balance according to claim 1, It is characterized in that Based on the contribution of the audio mode and the video mode to the audio and video authentication, the audio gradient compensation coefficient and the video gradient compensation coefficient are calculated respectively, including: Contribution of audio modality to audio and video authentication , determine the audio gradient compensation coefficient as ; Contribution of video modality to audio and video authentication , determine the video gradient compensation coefficient as .

5. The method for multi-modal generated audio and video identification with adaptive modal balance according to claim 1, It is characterized in that The method includes: performing a gradient update on an audio model in the multimodal audio and video identification model based on the audio gradient compensation coefficient, and performing a gradient update on a video model in the multimodal audio and video identification model based on the video gradient compensation coefficient, including: Determining a parameter update amount of the audio model according to a difference between an audio identification result of each sample speech in the sample speech set and a label of each sample speech; Based on the audio gradient compensation coefficient and the parameter update amount of the audio model, gradient updating the parameters of the audio model; Determining a parameter update amount of the video model according to a difference between a video identification result of each sample video in the sample video set and a label of each sample video; Based on the video gradient compensation coefficient and the parameter update amount of the video model, the parameters of the video model are gradient updated.

6. A multi-modal generated audio and video identification device with adaptive modal balance, It is characterized in that The device comprises: A determination module, used to determine the identification performance of the audio model in the multimodal audio and video identification model during the training process of the multimodal audio and video identification model, and to determine the identification performance of the video model in the multimodal audio and video identification model; an allocating module, configured to allocate gradient compensation coefficients to the audio model and the video model respectively according to their respective identification performances, wherein the better the identification performance, the smaller the allocated gradient compensation coefficient; An updating module, configured to update the audio model and the video model respectively according to the gradient compensation coefficients respectively allocated to the audio model and the video model, to obtain an updated multimodal audio and video identification model; A first identification module, used for inputting the audio and video to be detected into the updated multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected; The device specifically comprises: An audio feature extraction and identification module, used to extract the audio features of each sample speech in the sample speech set using the audio model in the multimodal audio and video identification model, and generate a corresponding audio identification result according to the audio features of each sample speech; A video feature extraction and identification module, used to extract the video features of each sample video in the sample video set by using the video model in the multimodal audio and video identification model, and generate a corresponding video identification result according to the video features of each sample video; A score generating module, configured to generate a score of an audio modality according to an audio identification result of each sample speech in the sample speech set and a label of each sample speech, and to generate a score of a video modality according to a video identification result of each sample video in the sample video set and a label of each sample video; A contribution calculation module, used to calculate the contribution of the audio modality and the video modality to the audio and video authentication based on the scores of the audio modality and the video modality respectively; A gradient compensation coefficient calculation module, used to calculate the audio gradient compensation coefficient and the video gradient compensation coefficient respectively based on the contribution of the audio mode and the video mode to the audio and video authentication; A gradient updating module, configured to perform a gradient update on an audio model in the multimodal audio and video identification model based on the audio gradient compensation coefficient, and to perform a gradient update on a video model in the multimodal audio and video identification model based on the video gradient compensation coefficient, so as to obtain the updated multimodal audio and video identification model; The second identification module is used to input the audio and video to be detected into the updated multimodal audio and video identification model to obtain the multimodal audio and video identification result of the audio and video to be detected.

7. An electronic device comprising a memory, a processor and a computer program stored in the memory, It is characterized in that The processor executes the computer program to implement the multi-modal generated audio and video identification method with adaptive modal balance according to any one of claims 1 to 5.

8. A computer-readable storage medium, It is characterized in that A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the multi-modal generated audio and video identification method with adaptive modal balance described in any one of claims 1-5 is implemented.

Images