Audio multimodal classification method, system and computer device

Abstract

The present invention relates to the technical field of audio classification, and specifically to an audio multimodal classification method, system and computer equipment, comprising the following steps: obtaining audio data; using multiple classifiers to perform classification learning of multiple modal features on the audio data to obtain multiple audio modal classification models; using momentum gradient to perform multimodal fusion on the multiple audio modal classification models to obtain an audio multimodal classification model for improving audio classification performance. The present invention uses the multimodal features of audio data for classification, increases the feature quantity, makes the classification result depend on the diversified feature data, improves the accuracy of classification, and uses the momentum gradient descent method to fuse the classification models during the classification process, ensures the balance of the fitting ability and generalization ability of each classification model, and further improves the accuracy of multimodal classification.

Description

Technical Field

[0001] The present invention relates to the technical field of audio classification, and in particular to an audio multimodal classification method, system and computer equipment. Background Art

[0002] Audio can be classified into speech, music, noise, silence, ambient sound, etc. according to its nature. In order to distinguish audio categories, relevant technicians usually process the results of Bayesian maximum a posteriori classifier based on the posterior decision method of correlation to achieve audio classification. Alternatively, blind separation technology is used to separate the mixed audio and post-process them using different technologies.

[0003] However, the above existing technologies perform classification based on a single data feature of the audio data and the experience of experts, and the number of features is limited, which results in limited accuracy of classification based on a single feature. Summary of the invention

[0004] The purpose of the present invention is to provide an audio multimodal classification method, system and computer device to solve the technical problem in the prior art that the amount of classification features is limited, resulting in limited accuracy of classification relying on a single feature.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0006] In a first aspect of the present invention, the present invention provides an audio multimodal classification method, comprising the following steps:

[0007] Get audio data;

[0008] Using multiple classifiers, multiple modal features of audio data are classified and learned to obtain multiple audio modal classification models;

[0009] Momentum gradient is used to perform multimodal fusion on multiple audio modal classification models to obtain an audio multimodal classification model for improving audio classification performance.

[0010] As a preferred solution of the present invention, the modal features of the audio data include: channel signal features, audio sound wave features, Mel spectrum features, and MFCC features.

[0011] As a preferred solution of the present invention, the method for extracting modal features of audio data includes:

[0012] The audio data is separated into channel signals by using alsa, and encoded by using wav2vec2 to obtain the channel signal features;

[0013] The audio data is subjected to Mel spectrum extraction using Fourier transform and Mel filtering, and is encoded using AlexNet to obtain the Mel spectrum feature;

[0014] The audio data is subjected to MFCC extraction using Fourier transform, Mel filter and discrete cosine transform, and encoded using BiLSTM to obtain the MFCC features;

[0015] The audio data is encoded using wav2vec2 to obtain the audio sound wave features.

[0016] As a preferred solution of the present invention, the construction of multiple audio modality classification models includes:

[0017] Classify and label the audio data to obtain the category label of the audio data;

[0018] Using the first classifier, the channel signal features of the audio data and the category labels of the audio data are classified and learned to obtain a first audio modality classification model, wherein:

[0019] The first audio modality classification model is:

[0020] Label = SVM1(K1);

[0021] In the formula, Label is the category label of the audio data, K1 is the channel signal feature, and SVM1 is the first classifier;

[0022] The second classifier is used to classify and learn the audio sound wave features of the audio data and the category labels of the audio data to obtain a second audio modality classification model, wherein:

[0023] The second audio modality classification model is:

[0024] Label = SVM2(K2);

[0025] In the formula, Label is the category label of the audio data, K2 is the audio sound wave feature, and SVM2 is the second classifier;

[0026] The third classifier is used to classify and learn the Mel spectrum features of the audio data and the category labels of the audio data to obtain a third audio modality classification model, in which:

[0027] The third audio modality classification model is:

[0028] Label = SVM3(K3);

[0029] In the formula, Label is the category label of the audio data, K3 is the Mel spectrum feature, and SVM3 is the third classifier;

[0030] The fourth classifier is used to classify and learn the audio sound wave features of the audio data and the category labels of the audio data to obtain a fourth audio modality classification model, wherein:

[0031] The fourth audio modality classification model is:

[0032] Label = SVM4(K4);

[0033] In the formula, Label is the category label of the audio data, K4 is the MFCC feature, and SVM4 is the fourth classifier.

[0034] As a preferred solution of the present invention, the method for constructing the audio multimodal classification model includes:

[0035] The first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model are performance evaluated to obtain performance evaluation results of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model, and the performance evaluation formula is:

[0036]

[0037] Where P kt is the performance evaluation result of the kth audio modality classification model at training time sequence t, is the true value of the loss function of the kth audio modality classification model at training time sequence t, is the predicted value of the loss function of the kth audio modality classification model at training time sequence t, is the true value of the loss function of the kth audio modality classification model at the training time sequence t-Δt, is the predicted value of the loss function of the kth audio modality classification model at the training time sequence t-Δt;

[0038] is the fitting ability of the kth audio modality classification model at training time sequence t;

[0039] is the generalization ability of the kth audio modality classification model at training time sequence t;

[0040] According to the performance evaluation result, using a momentum gradient descent method to predict the weights of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model;

[0041] Using weights to perform weighted fusion on the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model to obtain an audio multimodal classification model;

[0042] The audio multimodal classification model is:

[0043]

[0044] In the formula, Label_max is the category label of the audio data, is the optimal value of the weight of the k-th audio modality classification model, SVMk∈[SVM1,SVM2,SVM3,SVM4], Kk∈[K1,K2,K3,K4], K1 is the channel signal feature, SVM1 is the first classifier, K2 is the audio sound wave feature, SVM2 is the second classifier, K3 is the Mel spectrum feature, SVM3 is the third classifier, K4 is the MFCC feature, and SVM4 is the fourth classifier.

[0045] As a preferred embodiment of the present invention, the weight prediction method includes:

[0046] Will Converted into the gradient form of the loss function, we get

[0047] Will Converted into the gradient form of the loss function, we get

[0048] Will and Enter the performance evaluation formula In the equation, we get the weight prediction function;

[0049] The weight prediction function is:

[0050]

[0051] In the formula, is the gradient of the loss function prediction value of the k-th audio modality classification model at the training time sequence t, is the gradient of the true value of the loss function of the kth audio modality classification model at training time sequence t, w kt is the weight of the kth audio modality classification model at training time t, v kt is the gradient descent rate of the loss function of the kth audio modality classification model at training time t, arg min is Get the minimum value of w kt The optimal value of w kt The optimal value of t∈[tstart, tend], tstart is the start training sequence, tend is the end training sequence, for and The angle between for and The angle between.

[0052] As a preferred embodiment of the present invention, in, The termination timing tend

[0053] As a preferred solution of the present invention, the first classifier, the second classifier, the third classifier and the fourth classifier have the same structure.

[0054] In a second aspect of the present invention, the present invention provides an audio multimodal classification system, comprising:

[0055] A data acquisition unit, used for acquiring audio data;

[0056] A classification learning unit, used to perform classification learning of multiple modal features on audio data using multiple classifiers to obtain multiple audio modal classification models;

[0057] The multimodal fusion unit is used to perform multimodal fusion on multiple audio modal classification models using momentum gradient to obtain an audio multimodal classification model for improving audio classification performance.

[0058] In a third aspect of the present invention, the present invention provides a computer device, comprising: at least one processor; and

[0059] a memory communicatively coupled to the at least one processor;

[0060] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the computer device to perform the audio multimodal classification method.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention utilizes the multimodal features of audio data for classification, increases the feature quantity, and makes the classification result depend on the diversified feature data, thereby improving the accuracy of classification. Moreover, the momentum gradient descent method is used to fuse the classification models during the classification process, thereby ensuring the balance of the fitting ability and generalization ability of each classification model, and further improving the accuracy of multimodal classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0064] Figure 1 A flow chart of an audio multimodal classification method provided by an embodiment of the present invention;

[0065] Figure 2 A block diagram of an audio multimodal classification system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] like Figure 1 As shown, in a first aspect of the present invention, the present invention provides an audio multimodal classification method, comprising the following steps:

[0068] Get audio data;

[0069] Using multiple classifiers, multiple modal features of audio data are classified and learned to obtain multiple audio modal classification models;

[0070] Momentum gradient is used to perform multimodal fusion on multiple audio modal classification models to obtain an audio multimodal classification model for improving audio classification performance.

[0071] The audio classification in the present invention can be divided into categories such as speech, music, noise, silence, and ambient sound.

[0072] In order to improve the accuracy of audio classification, the present invention classifies audio based on multimodal features, and uses multiple classifiers to classify audio according to the multimodal features, so that the classification result depends on diversified feature data, thereby improving the accuracy of classification.

[0073] The multimodal features in the present invention include channel signal features, audio sound wave features, Mel spectrum features, and MFCC features.

[0074] Correspondingly, the present invention constructs multiple audio modality classification models according to the channel signal characteristics, audio sound wave characteristics, Mel-spectrogram characteristics, and MFCC characteristics, namely, a first audio modality classification model, a second audio modality classification model, a third audio modality classification model, and a fourth audio modality classification model.

[0075] Furthermore, the present invention weights and fuses the first audio modality classification model, the second audio modality classification model, the third audio modality classification model and the fourth audio modality classification model to obtain an audio multimodal classification model, thereby realizing audio classification based on multimodal features, ensuring that classification is based on diversified feature data, and ultimately ensuring the accuracy of audio classification.

[0076] Moreover, in the process of fusing the first audio modality classification model, the second audio modality classification model, the third audio modality classification model and the fourth audio modality classification model, the momentum gradient descent method is used to balance and optimize the different fitting capabilities and generalization capabilities between the first audio modality classification model, the second audio modality classification model, the third audio modality classification model and the fourth audio modality classification model, thereby ensuring that the audio multimodal classification model is in the optimal fitting and generalization capabilities. The audio multimodal classification model with the best performance provides accuracy guarantee for audio classification.

[0077] Specifically, the present invention evaluates the different fitting capabilities and generalization capabilities among the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model. Combined with the momentum gradient, the objective function of equilibrium optimization is constructed The fusion weights of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model and the fourth audio modality classification model are obtained from the objective function, so that the audio multimodal classification model is constructed with the fusion weights, which can balance and optimize the fitting and generalization capabilities of each model.

[0078] The modal features of audio data include: channel signal features, audio sound wave features, Mel spectrum features, and MFCC features.

[0079] Methods for extracting modal features of audio data include:

[0080] The audio data is separated into channel signals using alsa, and encoded using wav2vec2 to obtain channel signal features;

[0081] The audio data is extracted from the Mel spectrum using Fourier transform and Mel filtering, and encoded using AlexNet to obtain the Mel spectrum features;

[0082] The audio data is extracted using Fourier transform, Mel filter and discrete cosine transform for MFCC, and encoded using BiLSTM to obtain MFCC features;

[0083] Encode the audio data using wav2vec2 to obtain audio sound wave features.

[0084] The present invention constructs multiple audio modality classification models according to the channel signal features, audio sound wave features, Mel spectrum features, and MFCC features, namely, a first audio modality classification model, a second audio modality classification model, a third audio modality classification model, and a fourth audio modality classification model, as follows:

[0085] The construction of multiple audio modality classification models includes:

[0086] Classify and label the audio data to obtain the category label of the audio data;

[0087] Using the first classifier, the channel signal features of the audio data and the category labels of the audio data are classified and learned to obtain a first audio modality classification model, wherein:

[0088] The first audio modality classification model is:

[0089] Label = SVM1(K1);

[0090] In the formula, Label is the category label of the audio data, K1 is the channel signal feature, and SVM1 is the first classifier;

[0091] The second classifier is used to classify and learn the audio sound wave features of the audio data and the category labels of the audio data to obtain a second audio modality classification model, wherein:

[0092] The second audio modality classification model is:

[0093] Label = SVM2(K2);

[0094] In the formula, Label is the category label of the audio data, K2 is the audio sound wave feature, and SVM2 is the second classifier;

[0095] The third classifier is used to classify and learn the Mel spectrum features of the audio data and the category labels of the audio data to obtain a third audio modality classification model, in which:

[0096] The third audio modality classification model is:

[0097] Label = SVM3(K3);

[0098] In the formula, Label is the category label of the audio data, K3 is the Mel spectrum feature, and SVM3 is the third classifier;

[0099] The fourth classifier is used to classify and learn the audio sound wave features of the audio data and the category labels of the audio data to obtain a fourth audio modality classification model, wherein:

[0100] The fourth audio modality classification model is:

[0101] Label = SVM4(K4);

[0102] In the formula, Label is the category label of the audio data, K4 is the MFCC feature, and SVM4 is the fourth classifier.

[0103] During the fusion process of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model and the fourth audio modality classification model, the momentum gradient descent method is used to balance and optimize the different fitting capabilities and generalization capabilities between the first audio modality classification model, the second audio modality classification model, the third audio modality classification model and the fourth audio modality classification model, thereby ensuring that the audio multimodal classification model is in the optimal fitting and generalization capabilities. The audio multimodal classification model with the best performance provides accuracy guarantee for audio classification.

[0104] The method for constructing an audio multimodal classification model includes:

[0105] The performance of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model is evaluated to obtain performance evaluation results of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model. The performance evaluation formula is:

[0106]

[0107] Where P kt is the performance evaluation result of the kth audio modality classification model at training time sequence t, is the true value of the loss function of the kth audio modality classification model at training time sequence t, is the predicted value of the loss function of the kth audio modality classification model at training time sequence t, is the true value of the loss function of the kth audio modality classification model at the training time sequence t-Δt, is the predicted value of the loss function of the kth audio modality classification model at the training time sequence t-Δt;

[0108] is the fitting ability of the kth audio modality classification model at training time sequence t;

[0109] is the generalization ability of the kth audio modality classification model at training time sequence t;

[0110] According to the performance evaluation results, using the momentum gradient descent method to predict the weights of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model;

[0111] Using weights to perform weighted fusion on the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model to obtain an audio multimodal classification model;

[0112] The audio multimodal classification model is:

[0113]

[0114] In the formula, Label_max is the category label of the audio data, is the optimal value of the weight of the k-th audio modality classification model, SVMk∈[SVM1,SVM2,SVM3,SVM4], Kk∈[K1,K2,K3,K4], K1 is the channel signal feature, SVM1 is the first classifier, K2 is the audio sound wave feature, SVM2 is the second classifier, K3 is the Mel spectrum feature, SVM3 is the third classifier, K4 is the MFCC feature, and SVM4 is the fourth classifier.

[0115] The weight prediction methods include:

[0116] Will Converted into the gradient form of the loss function, we get

[0117] Will Converted into the gradient form of the loss function, we get

[0118] Will and Enter the performance evaluation formula In the equation, we get the weight prediction function;

[0119] The weight prediction function is:

[0120]

[0121] In the formula, is the gradient of the loss function prediction value of the k-th audio modality classification model at the training time sequence t, is the gradient of the true value of the loss function of the kth audio modality classification model at training time sequence t, w kt is the weight of the kth audio modality classification model at training time t, v ktis the gradient descent rate of the loss function of the kth audio modality classification model at training time t, arg min is Get the minimum value of w kt The optimal value of w kt The optimal value of t∈[tstart, tend], tstart is the start training sequence, tend is the end training sequence, for and The angle between for and The angle between.

[0122] in, The termination timing tend

[0123] Specifically, the present invention evaluates the different fitting capabilities and generalization capabilities among the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model. Combined with the momentum gradient, the objective function of equilibrium optimization is constructed The fusion weights of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model and the fourth audio modality classification model are obtained from the objective function, so that the audio multimodal classification model is constructed with the fusion weights, which can balance and optimize the fitting and generalization capabilities of each model.

[0124] Therefore, the present invention uses momentum gradient to quantify the fitting ability and generalization ability, and associates the fusion weight with the fitting ability and generalization ability. When the optimal value of the fusion weight is obtained, the fitting ability and generalization ability are correspondingly improved, and the interpretability is strong.

[0125] The first classifier, the second classifier, the third classifier and the fourth classifier have the same structure.

[0126] like Figure 2 As shown, in a second aspect of the present invention, the present invention provides an audio multimodal classification system, comprising:

[0127] A data acquisition unit, used for acquiring audio data;

[0128] A classification learning unit, used to perform classification learning of multiple modal features on audio data using multiple classifiers to obtain multiple audio modal classification models;

[0129] The multimodal fusion unit is used to perform multimodal fusion on multiple audio modal classification models using momentum gradient to obtain an audio multimodal classification model for improving audio classification performance.

[0130] In a third aspect of the present invention, the present invention provides a computer device, comprising: at least one processor; and

[0131] a memory communicatively coupled to the at least one processor;

[0132] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the computer device to perform the audio multimodal classification method.

[0133] The present invention utilizes the multimodal features of audio data for classification, increases the feature quantity, and makes the classification result depend on the diversified feature data, thereby improving the accuracy of classification. Moreover, the momentum gradient descent method is used to fuse the classification models during the classification process, thereby ensuring the balance of the fitting ability and generalization ability of each classification model, and further improving the accuracy of multimodal classification.

[0134] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. An audio multimodal classification method, characterized in that: The following steps are involved: Get audio data; Using multiple classifiers, multiple modal features of audio data are classified and learned to obtain multiple audio modal classification models; Using momentum gradient to perform multimodal fusion on multiple audio modal classification models, an audio multimodal classification model for improving audio classification performance is obtained; The method for constructing the audio multimodal classification model includes: The performance of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model is evaluated to obtain performance evaluation results of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model. The performance evaluation formula is: Where P kt is the performance evaluation result of the kth audio modality classification model at training time sequence t, is the true value of the loss function of the kth audio modality classification model at training time sequence t, is the predicted value of the loss function of the kth audio modality classification model at training time sequence t, is the true value of the loss function of the kth audio modality classification model at the training time sequence t-Δt, is the predicted value of the loss function of the kth audio modality classification model at the training time sequence t-Δt; is the fitting ability of the kth audio modality classification model at training time sequence t; is the generalization ability of the kth audio modality classification model at training time sequence t; According to the performance evaluation results, using the momentum gradient descent method to predict the weights of the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model; Using weights to perform weighted fusion on the first audio modality classification model, the second audio modality classification model, the third audio modality classification model, and the fourth audio modality classification model to obtain an audio multimodal classification model; The audio multimodal classification model is: In the formula, Label_max is the category label of the audio data, is the optimal value of the weight of the k-th audio modality classification model, SVMk∈[SVM1,SVM2,SVM3,SVM4], Kk∈[K1,K2,K3,K4], K1 is the channel signal feature, SVM1 is the first classifier, K2 is the audio sound wave feature, SVM2 is the second classifier, K3 is the Mel spectrum feature, SVM3 is the third classifier, K4 is the MFCC feature, and SVM4 is the fourth classifier.

2. The method for multimodal audio classification according to claim 1, characterized in that: The modal features of the audio data include: channel signal features, audio sound wave features, Mel spectrum features, and MFCC features.

3. The method for multimodal audio classification according to claim 2, characterized in that: The method for extracting modal features of the audio data comprises: The audio data is separated into channel signals by using alsa, and encoded by using wav2vec2 to obtain the channel signal features; The audio data is subjected to Mel spectrum extraction using Fourier transform and Mel filtering, and is encoded using AlexNet to obtain the Mel spectrum feature; The audio data is subjected to MFCC extraction using Fourier transform, Mel filter and discrete cosine transform, and encoded using BiLSTM to obtain the MFCC features; The audio data is encoded using wav2vec2 to obtain the audio sound wave features.

4. The method for multimodal audio classification according to claim 3, characterized in that: The construction of multiple audio modality classification models includes: Classify and label the audio data to obtain the category label of the audio data; Using the first classifier, the channel signal features of the audio data and the category labels of the audio data are classified and learned to obtain a first audio modality classification model, wherein: The first audio modality classification model is: Label = SVM1(K1); In the formula, Label is the category label of the audio data, K1 is the channel signal feature, and SVM1 is the first classifier; The second classifier is used to classify and learn the audio sound wave features of the audio data and the category labels of the audio data to obtain a second audio modality classification model, wherein: The second audio modality classification model is: Label = SVM2(K2); In the formula, Label is the category label of the audio data, K2 is the audio sound wave feature, and SVM2 is the second classifier; The third classifier is used to classify and learn the Mel spectrum features of the audio data and the category labels of the audio data to obtain a third audio modality classification model, in which: The third audio modality classification model is: Label = SVM3(K3); In the formula, Label is the category label of the audio data, K3 is the Mel spectrum feature, and SVM3 is the third classifier; The fourth classifier is used to classify and learn the audio sound wave features of the audio data and the category labels of the audio data to obtain a fourth audio modality classification model, wherein: The fourth audio modality classification model is: Label = SVM4(K4); In the formula, Label is the category label of the audio data, K4 is the MFCC feature, and SVM4 is the fourth classifier.

5. The method for multimodal audio classification according to claim 4, characterized in that: The weight prediction method includes: Will Converted into the gradient form of the loss function, we get Will Converted into the gradient form of the loss function, we get Will and Enter the performance evaluation formula In the equation, we get the weight prediction function; The weight prediction function is: In the formula, is the gradient of the loss function prediction value of the k-th audio modality classification model at the training time sequence t, is the gradient of the true value of the loss function of the kth audio modality classification model at training time sequence t, w kt is the weight of the kth audio modality classification model at training time t, v kt is the gradient descent rate of the loss function of the kth audio modality classification model at training time t, arg min is Get the minimum value of w kt The optimal value of w kt The optimal value of t∈[tstart, tend], tstart is the start training sequence, tend is the end training sequence, for and The angle between for and The angle between.

6. The audio multimodal classification method according to claim 5, characterized in that: in, The termination timing is tend 7. The audio multimodal classification method according to claim 6, characterized in that: The first classifier, the second classifier, the third classifier and the fourth classifier have the same structure.

8. An audio multimodal classification system, characterized in that: An audio multimodal classification method according to any one of claims 1 to 7, the system comprising: A data acquisition unit, used for acquiring audio data; A classification learning unit, used to perform classification learning of multiple modal features on audio data using multiple classifiers to obtain multiple audio modal classification models; The multimodal fusion unit is used to perform multimodal fusion on multiple audio modal classification models using momentum gradient to obtain an audio multimodal classification model for improving audio classification performance.

9. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the computer device to execute the method according to any one of claims 1 to 7.

Images