Audio feature extraction method, system and terminal based on deep mutual learning framework

Abstract

The present invention discloses an audio feature extraction method, system and terminal based on a deep mutual learning framework, the method comprising: obtaining a training data set, preprocessing the training data set to obtain a preprocessing training set; performing deep mutual learning training and loss function adjustment operations on an initial Mel spectrum model and an initial MFCC model according to the preprocessing training set to obtain a final Mel spectrum model and a final MFCC model; inputting the to-be-processed audio data of a target object into the final Mel spectrum model and the final MFCC model respectively to obtain a Mel spectrum prediction value and an MFCC prediction value; performing average fusion processing on the Mel spectrum prediction value and the MFCC prediction value to obtain a final prediction feature. The present invention comprehensively considers the different perspectives and feature expressions of the two models through mutual training learning, generates a more comprehensive and accurate final prediction feature, effectively integrates the advantages of different audio features, and improves the performance and robustness of the model.

Description

Technical Field

[0001] The present invention relates to the field of audio detection technology, and in particular to an audio feature extraction method, system, terminal and computer-readable storage medium based on a deep mutual learning framework. Background Art

[0002] With the development of science and technology, human body sounds are often used to assist in providing reference data in medical treatment. During the assistance process, doctors can use neural networks to extract features of the target object's breathing sounds, coughing sounds or speaking sounds, and make references based on the results of the feature extraction.

[0003] However, existing methods usually only extract one of the Mel spectrum (Mel spectrum) or MFCC (Mel frequency cepstral coefficient) features corresponding to the audio data, and then perform subsequent feature extraction and prediction tasks, ignoring the fact that these two features are good at processing different types of audio signals, resulting in the final extracted prediction features being inaccurate; and, in the process of testing the same target object, it is necessary to input multiple types of voice inputs (breathing sounds, coughing sounds or speaking sounds, etc.) at the same time, which has high requirements on the input data and increases the difficulty of feature extraction.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide an audio feature extraction method, system, terminal and computer-readable storage medium based on a deep mutual learning framework, aiming to solve the technical problems in the prior art that multiple types of speech inputs are required and only one of the Mel spectrum or MFCC spectrum of audio data is used as input for subsequent feature extraction tasks, which makes feature extraction difficult and the final predicted features extracted are not accurate enough.

[0006] To achieve the above object, the present invention provides an audio feature extraction method based on a deep mutual learning framework, and the audio feature extraction method based on a deep mutual learning framework comprises the following steps:

[0007] Acquire a training data set, and perform screening operations, labeling operations, cutting operations, and downsampling operations on the training data set to obtain a preprocessed training set;

[0008] Performing deep mutual learning training on the initial Mel spectrum model and the initial MFCC model according to the preprocessed training set, calculating the total Mel spectrum loss and the total MFCC loss, and respectively training the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss to obtain a mutual learning Mel spectrum model and a mutual learning MFCC model;

[0009] performing an updating operation on the mutual learning Mel spectrum model and the mutual learning MFCC model according to the combined loss function to obtain a final Mel spectrum model and a final MFCC model;

[0010] Acquire the audio data to be processed of the target object, input the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively, and obtain the Mel spectrum prediction value and the MFCC prediction value respectively;

[0011] The Mel spectrum prediction value and the MFCC prediction value are averaged and fused to obtain the final prediction feature.

[0012] Optionally, the audio feature extraction method based on the deep mutual learning framework, wherein the step of obtaining a training data set, and performing a screening operation, a labeling operation, a cutting operation, and a downsampling operation on the training data set to obtain a preprocessed training set, specifically includes:

[0013] Acquire a training data set, perform a screening operation on the training data set, retain audio data that meets a preset state condition, and obtain a first data set;

[0014] Performing a labeling operation on the first data set to obtain a second data set, and performing a downsampling operation on the second data set to obtain a third data set;

[0015] Performing a cutting process on each audio data in the third data set according to a preset cutting step length to obtain a plurality of audio segments corresponding to each audio data, and deleting all audio segments whose length is less than the preset cutting step length to obtain a fourth data set;

[0016] The fourth data set is screened according to a preset amplitude threshold, and all audio segments in the fourth data set whose amplitudes do not meet the preset amplitude threshold are deleted to obtain a preprocessing training set.

[0017] Optionally, the audio feature extraction method based on the deep mutual learning framework, wherein the deep mutual learning training is performed on the initial Mel spectrum model and the initial MFCC model according to the preprocessed training set, the total Mel spectrum loss and the total MFCC loss are calculated, and the initial Mel spectrum model and the initial MFCC model are trained according to the total Mel spectrum loss and the total MFCC loss respectively to obtain the mutual learning Mel spectrum model and the mutual learning MFCC model, specifically including:

[0018] Inputting the preprocessed training set into the initial Mel spectrum model and the initial MFCC model respectively;

[0019] The initial Mel spectrum model performs prediction according to the preprocessed training set to obtain a plurality of Mel spectrum logical values;

[0020] The initial MFCC model performs prediction according to the preprocessed training set to obtain a plurality of MFCC logical values;

[0021] Calculate according to all the Mel spectrum logical values ​​and all the MFCC logical values ​​to obtain the Mel spectrum total loss and the MFCC total loss;

[0022] Training the initial Mel spectrum model according to the total Mel spectrum loss to obtain a mutual learning Mel spectrum model;

[0023] The initial MFCC model is trained according to the MFCC total loss to obtain a mutual learning MFCC model.

[0024] Optionally, the audio feature extraction method based on the deep mutual learning framework, wherein the calculation is performed according to all the Mel spectrum logical values ​​and all the MFCC logical values ​​to obtain the Mel spectrum total loss and the MFCC total loss, specifically including:

[0025] Calculating according to all the Mel spectrum logical values ​​to obtain a Mel spectrum prediction soft label corresponding to the initial Mel spectrum model, and obtaining a Mel spectrum output probability distribution corresponding to the initial Mel spectrum model according to the Mel spectrum prediction soft label;

[0026] Calculating according to all the MFCC logical values ​​to obtain the MFCC predicted soft labels corresponding to the initial MFCC model, and obtaining the MFCC output probability distribution corresponding to the initial MFCC model according to the MFCC predicted soft labels;

[0027] Calculate according to the Mel spectrum output probability distribution and the MFCC output probability distribution to obtain the Mel spectrum soft label loss and the MFCC soft label loss;

[0028] Calculate the cross entropy of the initial Mel spectrum model and the initial MFCC model respectively to obtain the Mel spectrum hard label loss and the MFCC hard label loss;

[0029] The total Mel spectrum loss is calculated according to the Mel spectrum soft label loss and the Mel spectrum hard label loss, and the total MFCC loss is calculated according to the MFCC soft label loss and the MFCC hard label loss;

[0030] Training the initial Mel spectrum model according to the total Mel spectrum loss to obtain a mutual learning Mel spectrum model;

[0031] The initial MFCC model is trained according to the MFCC total loss to obtain a mutual learning MFCC model.

[0032] Optionally, the audio feature extraction method based on the deep mutual learning framework, wherein the updating operation on the mutual learning Mel spectrum model and the mutual learning MFCC model according to the combined loss function to obtain the final Mel spectrum model and the final MFCC model, specifically includes:

[0033] Get the preset positive sample loss function, binary cross loss function and regularization loss function;

[0034] Obtaining a combined loss function according to the positive sample loss function, the binary cross loss function and the regularization loss function;

[0035] The mutual learning Mel spectrum model and the mutual learning MFCC model are respectively updated according to the combined loss function to obtain a final Mel spectrum model and a final MFCC model.

[0036] Optionally, the audio feature extraction method based on the deep mutual learning framework, wherein the step of obtaining the audio data to be processed of the target object, inputting the audio data to be processed into the final Mel spectrum model and the final MFCC model, respectively, and obtaining the Mel spectrum prediction value and the MFCC prediction value, respectively, specifically includes:

[0037] Acquire the audio data to be processed of the target object, and input the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively;

[0038] The final Mel spectrum model performs preliminary feature extraction processing on the audio data to be processed to obtain a Mel spectrum corresponding to the audio data to be processed, and performs prediction feature extraction processing based on the Mel spectrum to obtain a Mel spectrum prediction value;

[0039] The final MFCC model performs preliminary feature extraction processing on the audio data to be processed to obtain MFCC features corresponding to the audio data to be processed, and performs prediction feature extraction processing based on the MFCC features to obtain MFCC prediction values.

[0040] Optionally, in the audio feature extraction method based on a deep mutual learning framework, the final Mel spectrum model and the final MFCC model have the same structure, and the final Mel spectrum model includes an encoder and a classifier;

[0041] The encoder includes a TDNN block, three SE-Res2Block blocks, a convolution block, and an attention statistics pooling block;

[0042] The classifier includes a fully connected layer and a batch normalization layer.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also provides an audio feature extraction system based on a deep mutual learning framework, wherein the audio feature extraction system based on a deep mutual learning framework includes:

[0044] A training set preprocessing module is used to obtain a training data set, and perform screening operations, labeling operations, cutting operations, and downsampling operations on the training data set to obtain a preprocessed training set;

[0045] A mutual learning training module, used for performing deep mutual learning training on an initial Mel spectrum model and an initial MFCC model according to the preprocessed training set, calculating a total Mel spectrum loss and a total MFCC loss, and respectively training the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss to obtain a mutual learning Mel spectrum model and a mutual learning MFCC model;

[0046] A model updating module, used for updating the mutual learning Mel spectrum model and the mutual learning MFCC model according to a combined loss function to obtain a final Mel spectrum model and a final MFCC model;

[0047] A prediction value acquisition module, used to acquire the audio data to be processed of the target object, and input the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively, to obtain the Mel spectrum prediction value and the MFCC prediction value respectively;

[0048] The prediction value fusion module is used to average and fuse the Mel spectrum prediction value and the MFCC prediction value to obtain the final prediction feature.

[0049] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an audio feature extraction program based on a deep mutual learning framework stored in the memory and run on the processor, and when the audio feature extraction program based on the deep mutual learning framework is executed by the processor, the steps of the audio feature extraction method based on the deep mutual learning framework as described above are implemented.

[0050] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an audio feature extraction program based on a deep mutual learning framework, and when the audio feature extraction program based on the deep mutual learning framework is executed by a processor, the steps of the audio feature extraction method based on the deep mutual learning framework as described above are implemented.

[0051] In the present invention, a training data set is obtained, and a screening operation, a labeling operation, a cutting process and a downsampling operation are performed on the training data set to obtain a preprocessing training set; according to the preprocessing training set, a deep mutual learning training is performed on an initial Mel spectrum model and an initial MFCC model, a total Mel spectrum loss and a total MFCC loss are calculated, and the initial Mel spectrum model and the initial MFCC model are trained according to the total Mel spectrum loss and the total MFCC loss respectively to obtain a mutual learning Mel spectrum model and a mutual learning MFCC model; according to a combined loss function, an update operation is performed on the mutual learning Mel spectrum model and the mutual learning MFCC model to obtain a final Mel spectrum model and a final MFCC model; the audio data to be processed of the target object is obtained, and the audio data to be processed is respectively input into the final Mel spectrum model and the final MFCC model to obtain a Mel spectrum prediction value and a MFCC prediction value respectively; the Mel spectrum prediction value and the MFCC prediction value are averaged and fused to obtain a final prediction feature. Through mutual training and learning, the present invention comprehensively considers the different perspectives and feature expressions of the two models, generates more comprehensive and accurate final prediction features, effectively integrates the advantages of different audio features, and improves the performance and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of a preferred embodiment of the audio feature extraction method based on the deep mutual learning framework of the present invention;

[0053] Figure 2 It is a deep mutual learning training flow chart of a preferred embodiment of the audio feature extraction method based on the deep mutual learning framework of the present invention;

[0054] Figure 3 It is a detailed flow chart of deep mutual learning training of a preferred embodiment of the audio feature extraction method based on the deep mutual learning framework of the present invention;

[0055] Figure 4 It is a schematic diagram of the model structure of a preferred embodiment of the audio feature extraction method based on the deep mutual learning framework of the present invention;

[0056] Figure 5 It is a structural diagram of the SE-Res2Block block of a preferred embodiment of the audio feature extraction method based on the deep mutual learning framework of the present invention;

[0057] Figure 6 It is a schematic diagram of the overall stages of a preferred embodiment of the audio feature extraction method based on a deep mutual learning framework of the present invention;

[0058] Figure 7 It is a schematic diagram of the reasoning stage of a preferred embodiment of the audio feature extraction method based on the deep mutual learning framework of the present invention;

[0059] Figure 8 It is a schematic diagram of the principle of a preferred embodiment of the audio feature extraction system based on the deep mutual learning framework of the present invention;

[0060] Fig. 9 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0062] The audio feature extraction method based on the deep mutual learning framework described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the audio feature extraction method based on the deep mutual learning framework includes the following steps:

[0063] Step S10: Obtain a training data set, perform screening operations, labeling operations, cutting operations, and downsampling operations on the training data set to obtain a preprocessed training set.

[0064] Specifically, in a preferred embodiment of the present invention, the training data set used is a data set collected through a network platform. The training data set contains voice samples of volunteers from all over the world, covering patients with positive respiratory diseases and negative healthy people. The training data set divides all sound samples into eight categories according to the status of respiratory diseases, namely healthy, normal positive, mildly positive, nearly positive, with respiratory diseases, without respiratory diseases, recovering, and recovered; so after obtaining the training data set, the training data set needs to be preprocessed to obtain a preprocessed training set, so as to facilitate subsequent training of the model.

[0065] Furthermore, the obtaining of the training data set, performing screening operations, labeling operations, cutting operations and downsampling operations on the training data set to obtain a preprocessed training set specifically includes:

[0066] A training data set is obtained, and a screening operation is performed on the training data set to retain audio data that meets a preset state condition to obtain a first data set; a labeling operation is performed on the first data set to obtain a second data set, and a downsampling operation is performed on the second data set to obtain a third data set; each audio data in the third data set is cut according to a preset cutting step length to obtain a plurality of audio segments corresponding to each audio data, and all audio segments whose length is less than the preset cutting step length are deleted to obtain a fourth data set; a screening operation is performed on the fourth data set according to a preset amplitude threshold, and all audio segments whose amplitudes do not meet the preset amplitude threshold requirement are deleted in the fourth data set to obtain a preprocessed training set.

[0067] Specifically, after obtaining the training data set, the training data set is first screened to obtain the preset state conditions pre-set by the technician. In a preferred embodiment of the present invention, the preset state conditions are: healthy, normal positive, slightly positive and near positive. That is to say, only the audio data of the four states of healthy, normal positive, slightly positive and near positive in the training data set are retained to obtain the first data set.

[0068] The first data set is then labeled, and all audio data in the first data set are classified into two categories, positive and negative, to obtain a second data set; the sampling rate of the original audio data of the training data set is 44.1kHz, however, through spectrum analysis, it is found that some audio data lack high-frequency information, and the spectrum information of some audio data is even limited to the frequency band below 7-8kHz. Therefore, the present invention downsamples all audio data in the second data set, and downsamples all audio data to 16kHz to ensure that the spectrum information of all audio data is used fairly in the subsequent cutting process, and the task is mainly focused on the frequency band below 8kHz, thereby obtaining a third data set.

[0069] In order to make the best use of the training data set, the present invention cuts the duration of all audio data in the third data set, cuts each audio data in the second data set according to a preset cutting step (for example, 5s), thereby obtaining multiple audio segments corresponding to each audio data; after completing the cutting of all audio data, the audio segments whose length is less than the preset cutting step (for example, 5s) are discarded, thereby obtaining a fourth data set.

[0070] In order to remove nearly silent audio segments that may be generated during the cutting process, reduce invalid data, and improve the computational efficiency of the model, the present invention sets an effective amplitude threshold (preset amplitude threshold) for time domain sampling points. In each audio segment, the total number of sampling points with amplitudes greater than the preset amplitude threshold is counted. If the total number does not exceed 0.5f s(i.e., the number of sampleable points within 0.5 seconds), the audio segment can be considered as an invalid segment and deleted. After completing the screening operation on all audio segments in the fourth data set, the preprocessing training set is obtained.

[0071] Step S20, performing deep mutual learning training on the initial Mel spectrum model and the initial MFCC model according to the preprocessed training set, calculating the total Mel spectrum loss and the total MFCC loss, and respectively training the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss to obtain a mutual learning Mel spectrum model and a mutual learning MFCC model.

[0072] Specifically, in a preferred embodiment of the present invention, Figure 2 As shown in the figure, the Deep Mutual Learning Framework (DMLF) is used to jointly train the initial Mel spectrum model that uses Mel spectrum alone and the initial MFCC model that uses MFCC (Mel-frequency cepstral coefficient) alone. The core purpose of this interactive learning is to promote mutual learning between the two models so that they can absorb and integrate each other's advantages, thereby improving their respective performance. After completing the deep mutual learning training of the initial Mel spectrum model and the initial MFCC model, the mutual learning Mel spectrum model and the mutual learning MFCC model are obtained.

[0073] Further, the performing deep mutual learning training on the initial Mel spectrum model and the initial MFCC model according to the preprocessed training set, calculating the total Mel spectrum loss and the total MFCC loss, and respectively training the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss to obtain the mutual learning Mel spectrum model and the mutual learning MFCC model, specifically includes:

[0074] The preprocessed training set is respectively input into the initial Mel spectrum model and the initial MFCC model; the initial Mel spectrum model is predicted according to the preprocessed training set to obtain multiple Mel spectrum logical values, and the initial MFCC model is predicted according to the preprocessed training set to obtain multiple MFCC logical values; calculation is performed according to all the Mel spectrum logical values ​​and all the MFCC logical values ​​to obtain the total Mel spectrum loss and the total MFCC loss; the initial Mel spectrum model is trained according to the total Mel spectrum loss to obtain a mutual learning Mel spectrum model, and the initial MFCC model is trained according to the total MFCC loss to obtain a mutual learning MFCC model.

[0075] Specifically, Figure 3As shown, the preprocessed training set is first input into the initial Mel spectrum model and the initial MFCC model respectively, and the deep mutual learning (DML) training is started. DML can train two or more networks at the same time. In the framework of the present invention, the number of networks is 2. As an example, Figure 3 Only the process of the initial MFCC model learning from the initial Mel spectrum model is shown. For the initial MFCC model, the logical value predicted on the mth category is expressed as (MFCC logical value), where m=1, 2; similarly, the Mel spectrum prediction value predicted by the initial Mel spectrum model can be obtained.

[0076] Then, calculations can be performed based on all MFCC logical values ​​and all Mel spectrum logical values ​​to obtain the total Mel spectrum loss and the total MFCC loss. The initial Mel spectrum model is trained based on the total Mel spectrum loss, and the initial MFCC model is trained based on the total MFCC loss to obtain the mutual learning Mel spectrum model and the mutual learning MFCC model.

[0077] Further, the calculation is performed according to all the Mel spectrum logical values ​​and all the MFCC logical values ​​to obtain the total Mel spectrum loss and the total MFCC loss, which specifically includes:

[0078] Calculate according to all the Mel spectrum logical values ​​to obtain the Mel spectrum prediction soft label corresponding to the initial Mel spectrum model, and obtain the Mel spectrum output probability distribution corresponding to the initial Mel spectrum model according to the Mel spectrum prediction soft label; calculate according to all the MFCC logical values ​​to obtain the MFCC prediction soft label corresponding to the initial MFCC model, and obtain the MFCC output probability distribution corresponding to the initial MFCC model according to the MFCC prediction soft label; calculate according to the Mel spectrum output probability distribution and the MFCC output probability distribution to obtain the Mel spectrum soft label loss and the MFCC soft label loss; calculate the cross entropy of the initial Mel spectrum model and the initial MFCC model respectively to obtain the Mel spectrum hard label loss and the MFCC hard label loss; calculate the Mel spectrum total loss according to the Mel spectrum soft label loss and the Mel spectrum hard label loss, and calculate the MFCC total loss according to the MFCC soft label loss and the MFCC hard label loss.

[0079] Specifically, after calculating and obtaining a plurality of Mel spectrum prediction values ​​corresponding to the initial Mel spectrum model and a plurality of MFCC prediction values ​​corresponding to the initial MFCC model, the Mel spectrum prediction soft label corresponding to the initial Mel spectrum model is first calculated according to all the Mel spectrum prediction values, and then the MFCC prediction soft label corresponding to the initial MFCC model is calculated according to all the MFCC prediction values. Figure 3 For example, the specific calculation formula is: in, is the soft label predicted by MFCC; t is the temperature that controls the smoothness of the soft label. When t=1, the output is the prediction of the model; the calculation method of Mel prediction soft label is the same; it should be noted that in the process of calculating the hard label loss, the real label of the data in the preprocessing training set will be used.

[0080] Then, the MFCC output probability distribution is obtained based on the MFCC prediction soft label, and the Mel spectrum output probability distribution is obtained based on the Mel spectrum prediction soft label, which is specifically expressed as: in, The MFCC output probability distribution is the same as the Mel spectrum output probability distribution; the MFCC output probability distribution and the Mel spectrum output probability distribution will be passed to other networks as soft labels.

[0081] Then, according to the MFCC output probability distribution and the Mel spectrum output probability distribution, the MFCC soft label loss corresponding to the initial MFCC model and the Mel spectrum soft label loss corresponding to the initial Mel spectrum model are calculated respectively. Figure 3 For example, the MFCC soft label loss can be expressed as: Among them, KL represents the KL (Kullback-Leibler divergence) divergence calculation, which measures the difference between the two output probability distributions. Output probability distribution of Mel spectrum; the calculation method of Mel spectrum soft label loss is similar.

[0082] Finally, the cross entropy of the initial Mel spectrum model and the initial MFCC model is calculated respectively, so as to obtain the Mel spectrum hard label loss corresponding to the initial Mel spectrum model and the MFCC hard label loss corresponding to the initial MFCC model; the Mel spectrum total loss is obtained according to the Mel spectrum soft label loss and Mel spectrum hard label loss corresponding to the initial Mel spectrum model, and the MFCC total loss is obtained according to the MFCC soft label loss and MFCC hard label loss corresponding to the initial MFCC model. Figure 3 For example, the total MFCC loss is expressed as: in, is the total MFCC loss, is the MFCC hard label loss, λ dml is the weight of the soft label loss; the representation of the total Mel spectrum loss is the same as the representation of the total MFCC loss.

[0083] It should be noted that the preprocessed training set is divided into a training set and a test set. The training set is used for the deep mutual learning training stage and the subsequent combined loss function adjustment stage, and the test set is used to evaluate the performance of the final Mel spectrum model and the final MFCC model after training.

[0084] Step S30: updating the mutual learning Mel spectrum model and the mutual learning MFCC model according to the combined loss function to obtain a final Mel spectrum model and a final MFCC model.

[0085] Specifically, the updating operation of the mutual learning Mel spectrum model and the mutual learning MFCC model according to the combined loss function to obtain the final Mel spectrum model and the final MFCC model specifically includes:

[0086] Obtain a preset positive sample loss function, a binary cross loss function, and a regularization loss function; obtain a combined loss function according to the positive sample loss function, the binary cross loss function, and the regularization loss function; and update the mutual learning Mel spectrum model and the mutual learning MFCC model according to the combined loss function to obtain a final Mel spectrum model and a final MFCC model.

[0087] Specifically, there will be a sample imbalance problem in the training data set. Therefore, the present invention proposes a positive sample loss function to improve the model's prediction ability for positive samples, and weights it with the binary cross entropy loss and regularization loss to form a final combined loss function, wherein the positive sample loss function, the binary cross loss function and the regularization loss function can be set by technicians according to actual conditions, and can cope with different complex sample situations.

[0088] To solve the problem of imbalance in the number of positive and negative samples in the training data set, a positive sample loss function POS is proposed loss , the specific calculation formula is: Among them, M is the number of positive samples, y i is the sample label of the input audio data, is the predicted value of the model for the input audio data; the positive sample loss function is mainly used to ignore the existence of negative samples, shorten the distance between the model's predicted probability of positive samples and the positive sample label, which can effectively improve the model's prediction ability for positive samples.

[0089] The binary cross entropy loss function has good performance in classification tasks and is widely used in binary classification and some multi-classification tasks. The binary cross entropy loss function BCE loss It can be expressed as: Where N is the total number of samples, is the model prediction output, y i is the sample label.

[0090] In order to control the parameter complexity of the classifier and improve the generalization ability of the model, the present invention also introduces a regularized loss function R loss , which can be specifically expressed as: Where W is the weight of the classifier and T is the transpose.

[0091] Finally, the combined loss function BRP loss By POS loss and BCE loss Weighted composition, and add a smaller weight R loss , which can be specifically expressed as: BRP loss =(1-α)BCE loss +αPOS loss ++λR loss ; Among them, α and λ are used to control the weight of each loss proportion;

[0092] It should be noted that λ needs to be manually adjusted by technicians to control the positive sample loss function POS loss The present invention designs a manual adjustment scheme and a dynamic adjustment scheme, wherein the dynamic adjustment scheme can be divided into two types, as follows.

[0093] The first is linear dynamic adjustment, which can be expressed as: Among them, E and E cur-1 Respectively represent the total number of epochs (number of traversals) and the current epoch number of training. α is designed to increase as the number of epochs increases during training, thereby continuously increasing POS loss The size of the weight α. This allows the model to focus on the global data at the beginning of training, allowing the model to better fit all training data. As training progresses, POS loss The weight and reduce BCE loss , so that the model gradually focuses on positive samples with a smaller number of samples, thereby reducing the impact of unbalanced sample ratios.

[0094] The second is nonlinear dynamic adjustment, which can be expressed as: Where n is a constant, and in the preferred embodiment of the present invention, it takes a value of 2 or 3. The purpose of the nonlinear dynamic adjustment scheme is to slow down the growth rate of α in the early stage of training, and the larger the value of n, the slower the growth rate of α in the early stage of training, that is, the model can be more dependent on BCE in the early stage of training. loss , thus better fitting the global training data.

[0095] It should be noted that in the process of adjusting the model using the combined loss function, the mutual learning Mel spectrum model and the mutual learning MFCC model each have their own corresponding combined loss functions, and the adjustment process is also carried out separately, that is, the Mel spectrum combination loss and the MFCC combination loss are actually calculated separately, and then the mutual learning Mel spectrum model is updated and trained according to the Mel spectrum combination loss to obtain the final Mel spectrum model; the mutual learning MFCC model is updated and trained according to the MFCC combination loss to obtain the final MFCC model.

[0096] Furthermore, the final Mel spectrum model has the same structure as the final MFCC model, and the final Mel spectrum model includes an encoder and a classifier; the encoder includes a TDNN block, three SE-Res2Block blocks, a convolution block and an attention statistics pooling block; the classifier includes a fully connected layer and a batch normalization layer.

[0097] Specifically, the final Mel spectrum model and the final MFCC model have the same structure, including an encoder and a classifier.

[0098] In a preferred embodiment of the present invention, considering that the ECAPA-TDNN model (a time-delay neural network for speaker verification) has shown strong encoding capabilities in speaker recognition tasks, the present invention believes that applying the ECAPA-TDNN model as an encoder to the prediction of audio data is expected to improve system performance. Here, the present invention abandons the AAM-Softmax component (a component) in the original ECAPA-TDNN and only uses ECAPA-TDNN as the encoder of the model. The specific structure is as follows Figure 4 As shown in the figure, it includes a TDNN block (time-delay neural network) composed of a one-dimensional convolutional layer combined with a ReLU activation function and a batch normalization layer; three SE-Res2Block blocks (a neural network model that integrates the SE module in the Res2Net architecture). The specific structure is as follows Figure 5 As shown in the figure, through three SE-Res2Block blocks, multi-layer feature aggregation is realized to further enhance the representation ability of the model; a convolution block composed of a one-dimensional convolution layer combined with a ReLU activation function; an attention statistical pooling module composed of an attention statistical pooling layer and a ReLU activation function. The attention statistical pooling module is the core component of the encoder. By combining the attention mechanism and pooling operation, the model can reduce the data dimension while extracting important information, thereby enhancing the model's expression ability.

[0099] A multi-layer perceptron structure with fully connected layers and batch normalization layers is used in the classifier.

[0100] Furthermore, the convolution kernel size of the one-dimensional convolution layer in the TDNN block is k=5, and the dilation rate is d=1; Figure 5 As shown, each SE-Res2Block includes four parts, corresponding to Figure 5 A, B, C and D in the present invention, wherein D is an SE module, A, B and C all include a one-dimensional convolutional layer, the convolution kernel size of the one-dimensional convolutional layer of A is k=1, and the dilation rate is d=1, the one-dimensional convolutional layer in B is a one-dimensional dilated convolutional layer, and the convolution kernel size of the one-dimensional convolutional layer in C is k=1, and the dilation rate is d=1; it should be emphasized that the convolution kernel size and dilation rate of the A part, C part and D part of the three SE-Res2Blocks in the present invention are the same, the difference is that the convolution kernel size of the one-dimensional dilated convolution layer in the B part of the first (in the order of the arrows) SE-Res2Block is k=3, and the dilation rate is d=2; the convolution kernel size of the one-dimensional dilated convolution layer in the B part of the second SE-Res2Block is k=3, and the dilation rate is d=3; the convolution kernel size of the one-dimensional dilated convolution layer in the B part of the third SE-Res2Block is k=3, and the dilation rate is d=4.

[0101] The convolution kernel size of the one-dimensional convolution layer in the convolution block is k=1, and the dilation rate is d=1.

[0102] Step S40: Obtain the audio data to be processed of the target object, and input the audio data to be processed into the final Mel spectrum model and the final MFCC model, respectively, to obtain the Mel spectrum prediction value and the MFCC prediction value, respectively.

[0103] Specifically, in a preferred embodiment of the present invention, Figure 6 and Figure 7 As shown, the audio data to be processed of the target object (which may be coughing, breathing or speaking voice) is obtained, and the audio data is preprocessed and respectively input into the final Mel spectrum model and the final MFCC model, so as to obtain the Mel spectrum prediction value and the MFCC prediction value.

[0104] Further, the obtaining of the audio data to be processed of the target object, inputting the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively, and obtaining the Mel spectrum prediction value and the MFCC prediction value respectively, specifically includes:

[0105] The audio data to be processed of the target object is obtained, and the audio data to be processed is respectively input into the final Mel spectrum model and the final MFCC model; the final Mel spectrum model performs preliminary feature extraction processing on the audio data to be processed to obtain the Mel spectrum corresponding to the audio data to be processed, and performs predictive feature extraction processing based on the Mel spectrum to obtain the Mel spectrum prediction value; the final MFCC model performs preliminary feature extraction processing on the audio data to be processed to obtain the MFCC features corresponding to the audio data to be processed, and performs predictive feature extraction processing based on the MFCC features to obtain the MFCC prediction value.

[0106] Specifically, Figure 7 As shown in the figure, the audio data to be processed (cough / breathing / speech) are input into the final Mel spectrum model and the final MFCC model respectively. The final Mel spectrum model will perform preliminary feature extraction processing based on the audio data to be processed to obtain the Mel spectrum corresponding to the audio data to be processed. The final MFCC model will propose the MFCC features corresponding to the audio data to be processed; then the final Mel spectrum model will perform prediction feature extraction processing based on the Mel spectrum to obtain the Mel spectrum prediction value (equivalent to Figure 7 The Mel spectrum model prediction value in the final MFCC model will perform prediction feature extraction processing based on the MFCC features to obtain the MFCC prediction value (equivalent to Figure 7 MFCC model prediction value in ).

[0107] It should be noted that in the present invention, it is only necessary to input any type of audio data of the target object into the final Mel model and the final MFCC model to realize the prediction feature extraction of the target user, and there is no need to process multiple different types of audio data of the target object at the same time, thereby reducing the complexity of data collection and preprocessing, improving the overall work efficiency, and improving the robustness of the model.

[0108] Step S50: average and fuse the Mel spectrum prediction value and the MFCC prediction value to obtain a final prediction feature.

[0109] Specifically, in a preferred embodiment of the present invention, Figure 6 and Figure 7 As shown in Figure 2, the Mel spectrum prediction value and the MFCC prediction value are averaged and fused (i.e., the average operation is taken), which comprehensively considers the different perspectives and feature expressions of the two models and generates a more comprehensive and accurate final prediction feature (equivalent to Figure 6 The final predicted value and Figure 7 The predicted value in ), effectively integrating the advantages of different audio features and improving the performance and robustness of the model.

[0110] Furthermore, if Figure 8 As shown, based on the above-mentioned audio feature extraction method based on the deep mutual learning framework, the present invention also provides an audio feature extraction system based on the deep mutual learning framework, wherein the audio feature extraction system based on the deep mutual learning framework includes:

[0111] The training set preprocessing module 51 is used to obtain a training data set, and perform screening operations, labeling operations, cutting operations and downsampling operations on the training data set to obtain a preprocessed training set;

[0112] A mutual learning training module 52 is used to perform deep mutual learning training on the initial Mel spectrum model and the initial MFCC model according to the preprocessing training set, calculate the total Mel spectrum loss and the total MFCC loss, and train the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss, respectively, to obtain a mutual learning Mel spectrum model and a mutual learning MFCC model;

[0113] A model updating module 53, configured to update the mutual learning Mel spectrum model and the mutual learning MFCC model according to a combined loss function to obtain a final Mel spectrum model and a final MFCC model;

[0114] A prediction value acquisition module 54 is used to acquire the audio data to be processed of the target object, and input the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively to obtain the Mel spectrum prediction value and the MFCC prediction value respectively;

[0115] The prediction value fusion module 55 is used to average and fuse the Mel spectrum prediction value and the MFCC prediction value to obtain a final prediction feature.

[0116] Furthermore, if Fig. 9 As shown, based on the above-mentioned audio feature extraction method and system based on the deep mutual learning framework, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Fig. 9 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0117] The memory 20 may be an internal storage unit of the terminal in some embodiments, such as a hard disk or memory of the terminal. The memory 20 may also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an audio feature extraction program 40 based on a deep mutual learning framework is stored on the memory 20, and the audio feature extraction program 40 based on a deep mutual learning framework can be executed by the processor 10, thereby realizing the audio feature extraction method based on a deep mutual learning framework in the present application.

[0118] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the audio feature extraction method based on the deep mutual learning framework.

[0119] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0120] In one embodiment, when the processor 10 executes the audio feature extraction program 40 based on the deep mutual learning framework in the memory 20, the following steps are implemented:

[0121] Acquire a training data set, and perform screening operations, labeling operations, cutting operations, and downsampling operations on the training data set to obtain a preprocessed training set;

[0122] Performing deep mutual learning training on the initial Mel spectrum model and the initial MFCC model according to the preprocessed training set, calculating the total Mel spectrum loss and the total MFCC loss, and respectively training the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss to obtain a mutual learning Mel spectrum model and a mutual learning MFCC model;

[0123] performing an updating operation on the mutual learning Mel spectrum model and the mutual learning MFCC model according to the combined loss function to obtain a final Mel spectrum model and a final MFCC model;

[0124] Acquire the audio data to be processed of the target object, input the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively, and obtain the Mel spectrum prediction value and the MFCC prediction value respectively;

[0125] The Mel spectrum prediction value and the MFCC prediction value are averaged and fused to obtain the final prediction feature.

[0126] The step of obtaining a training data set and performing screening, labeling, cutting and downsampling operations on the training data set to obtain a preprocessed training set specifically includes:

[0127] Acquire a training data set, perform a screening operation on the training data set, retain audio data that meets a preset state condition, and obtain a first data set;

[0128] Performing a labeling operation on the first data set to obtain a second data set, and performing a downsampling operation on the second data set to obtain a third data set;

[0129] Performing a cutting process on each audio data in the third data set according to a preset cutting step length to obtain a plurality of audio segments corresponding to each audio data, and deleting all audio segments whose length is less than the preset cutting step length to obtain a fourth data set;

[0130] The fourth data set is screened according to a preset amplitude threshold, and all audio segments in the fourth data set whose amplitudes do not meet the preset amplitude threshold are deleted to obtain a preprocessing training set.

[0131] The method of performing deep mutual learning training on the initial Mel spectrum model and the initial MFCC model according to the preprocessed training set, calculating the total Mel spectrum loss and the total MFCC loss, and respectively training the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss to obtain the mutual learning Mel spectrum model and the mutual learning MFCC model specifically includes:

[0132] Inputting the preprocessed training set into the initial Mel spectrum model and the initial MFCC model respectively;

[0133] The initial Mel spectrum model performs prediction according to the preprocessed training set to obtain a plurality of Mel spectrum logical values;

[0134] The initial MFCC model performs prediction according to the preprocessed training set to obtain a plurality of MFCC logical values;

[0135] Calculate according to all the Mel spectrum logical values ​​and all the MFCC logical values ​​to obtain the Mel spectrum total loss and the MFCC total loss;

[0136] Training the initial Mel spectrum model according to the total Mel spectrum loss to obtain a mutual learning Mel spectrum model;

[0137] The initial MFCC model is trained according to the MFCC total loss to obtain a mutual learning MFCC model.

[0138] The calculation is performed according to all the Mel spectrum logical values ​​and all the MFCC logical values ​​to obtain the total Mel spectrum loss and the total MFCC loss, specifically including:

[0139] Calculating according to all the Mel spectrum logical values ​​to obtain a Mel spectrum prediction soft label corresponding to the initial Mel spectrum model, and obtaining a Mel spectrum output probability distribution corresponding to the initial Mel spectrum model according to the Mel spectrum prediction soft label;

[0140] Calculating according to all the MFCC logical values ​​to obtain the MFCC predicted soft labels corresponding to the initial MFCC model, and obtaining the MFCC output probability distribution corresponding to the initial MFCC model according to the MFCC predicted soft labels;

[0141] Calculate according to the Mel spectrum output probability distribution and the MFCC output probability distribution to obtain the Mel spectrum soft label loss and the MFCC soft label loss;

[0142] Calculate the cross entropy of the initial Mel spectrum model and the initial MFCC model respectively to obtain the Mel spectrum hard label loss and the MFCC hard label loss;

[0143] The total Mel spectrum loss is calculated according to the Mel spectrum soft label loss and the Mel spectrum hard label loss, and the total MFCC loss is calculated according to the MFCC soft label loss and the MFCC hard label loss.

[0144] The updating operation of the mutual learning Mel spectrum model and the mutual learning MFCC model according to the combined loss function to obtain the final Mel spectrum model and the final MFCC model specifically includes:

[0145] Get the preset positive sample loss function, binary cross loss function and regularization loss function;

[0146] Obtaining a combined loss function according to the positive sample loss function, the binary cross loss function and the regularization loss function;

[0147] The mutual learning Mel spectrum model and the mutual learning MFCC model are respectively updated according to the combined loss function to obtain a final Mel spectrum model and a final MFCC model.

[0148] The step of obtaining the audio data to be processed of the target object and inputting the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively to obtain the Mel spectrum prediction value and the MFCC prediction value respectively includes:

[0149] Acquire the audio data to be processed of the target object, and input the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively;

[0150] The final Mel spectrum model performs preliminary feature extraction processing on the audio data to be processed to obtain a Mel spectrum corresponding to the audio data to be processed, and performs prediction feature extraction processing based on the Mel spectrum to obtain a Mel spectrum prediction value;

[0151] The final MFCC model performs preliminary feature extraction processing on the audio data to be processed to obtain MFCC features corresponding to the audio data to be processed, and performs prediction feature extraction processing based on the MFCC features to obtain MFCC prediction values.

[0152] The final Mel spectrum model has the same structure as the final MFCC model, and the final Mel spectrum model includes an encoder and a classifier;

[0153] The encoder includes a TDNN block, three SE-Res2Block blocks, a convolution block, and an attention statistics pooling block;

[0154] The classifier includes a fully connected layer and a batch normalization layer.

[0155] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an audio feature extraction program based on a deep mutual learning framework, and when the audio feature extraction program based on the deep mutual learning framework is executed by a processor, the steps of the audio feature extraction method based on the deep mutual learning framework as described above are implemented.

[0156] In summary, the present invention provides an audio feature extraction method, system and terminal based on a deep mutual learning framework, the method comprising: obtaining a training data set, performing screening operations, labeling operations, cutting processing and downsampling operations on the training data set to obtain a preprocessing training set; performing deep mutual learning training on an initial Mel spectrum model and an initial MFCC model according to the preprocessing training set, calculating a total Mel spectrum loss and a total MFCC loss, and respectively training the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss to obtain a mutual learning Mel spectrum model and a mutual learning MFCC model; updating the mutual learning Mel spectrum model and the mutual learning MFCC model according to a combined loss function to obtain a final Mel spectrum model and a final MFCC model; obtaining audio data to be processed of a target object, inputting the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively, and obtaining a Mel spectrum prediction value and an MFCC prediction value respectively; performing average fusion processing on the Mel spectrum prediction value and the MFCC prediction value to obtain a final prediction feature. Through mutual training and learning, the present invention comprehensively considers the different perspectives and feature expressions of the two models, generates more comprehensive and accurate final prediction features, effectively integrates the advantages of different audio features, and improves the performance and robustness of the model.

[0157] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0158] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0159] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An audio feature extraction method based on a deep mutual learning framework, characterized in that: The audio feature extraction method based on the deep mutual learning framework includes: Acquire a training data set, and perform screening operations, labeling operations, cutting operations, and downsampling operations on the training data set to obtain a preprocessed training set; Performing deep mutual learning training on the initial Mel spectrum model and the initial MFCC model according to the preprocessed training set, calculating the total Mel spectrum loss and the total MFCC loss, and respectively training the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss to obtain a mutual learning Mel spectrum model and a mutual learning MFCC model; performing an updating operation on the mutual learning Mel spectrum model and the mutual learning MFCC model according to the combined loss function to obtain a final Mel spectrum model and a final MFCC model; Acquire the audio data to be processed of the target object, input the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively, and obtain the Mel spectrum prediction value and the MFCC prediction value respectively; The Mel spectrum prediction value and the MFCC prediction value are averaged and fused to obtain the final prediction feature.

2. The audio feature extraction method based on the deep mutual learning framework according to claim 1 is characterized in that: The obtaining of the training data set, and performing screening, labeling, cutting and downsampling operations on the training data set to obtain a preprocessed training set specifically includes: Acquire a training data set, perform a screening operation on the training data set, retain audio data that meets a preset state condition, and obtain a first data set; Performing a labeling operation on the first data set to obtain a second data set, and performing a downsampling operation on the second data set to obtain a third data set; Performing a cutting process on each audio data in the third data set according to a preset cutting step length to obtain a plurality of audio segments corresponding to each audio data, and deleting all audio segments whose length is less than the preset cutting step length to obtain a fourth data set; The fourth data set is screened according to a preset amplitude threshold, and all audio segments in the fourth data set whose amplitudes do not meet the preset amplitude threshold are deleted to obtain a preprocessing training set.

3. The audio feature extraction method based on the deep mutual learning framework according to claim 1 is characterized in that: The method of performing deep mutual learning training on the initial Mel spectrum model and the initial MFCC model according to the preprocessed training set, calculating the total Mel spectrum loss and the total MFCC loss, and respectively training the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss to obtain the mutual learning Mel spectrum model and the mutual learning MFCC model, specifically includes: Inputting the preprocessed training set into the initial Mel spectrum model and the initial MFCC model respectively; The initial Mel spectrum model performs prediction according to the preprocessed training set to obtain a plurality of Mel spectrum logical values; The initial MFCC model performs prediction according to the preprocessed training set to obtain a plurality of MFCC logical values; Calculate according to all the Mel spectrum logical values ​​and all the MFCC logical values ​​to obtain the Mel spectrum total loss and the MFCC total loss; Training the initial Mel spectrum model according to the total Mel spectrum loss to obtain a mutual learning Mel spectrum model; The initial MFCC model is trained according to the MFCC total loss to obtain a mutual learning MFCC model.

4. The audio feature extraction method based on the deep mutual learning framework according to claim 3 is characterized in that: The calculation is performed according to all the Mel spectrum logical values ​​and all the MFCC logical values ​​to obtain the Mel spectrum total loss and the MFCC total loss, specifically including: Calculating according to all the Mel spectrum logical values ​​to obtain a Mel spectrum prediction soft label corresponding to the initial Mel spectrum model, and obtaining a Mel spectrum output probability distribution corresponding to the initial Mel spectrum model according to the Mel spectrum prediction soft label; Calculating according to all the MFCC logical values ​​to obtain the MFCC predicted soft labels corresponding to the initial MFCC model, and obtaining the MFCC output probability distribution corresponding to the initial MFCC model according to the MFCC predicted soft labels; Calculate according to the Mel spectrum output probability distribution and the MFCC output probability distribution to obtain the Mel spectrum soft label loss and the MFCC soft label loss; Calculate the cross entropy of the initial Mel spectrum model and the initial MFCC model respectively to obtain the Mel spectrum hard label loss and the MFCC hard label loss; The total Mel spectrum loss is calculated according to the Mel spectrum soft label loss and the Mel spectrum hard label loss, and the total MFCC loss is calculated according to the MFCC soft label loss and the MFCC hard label loss.

5. The audio feature extraction method based on the deep mutual learning framework according to claim 1 is characterized in that: The updating operation of the mutual learning Mel spectrum model and the mutual learning MFCC model according to the combined loss function to obtain a final Mel spectrum model and a final MFCC model specifically includes: Get the preset positive sample loss function, binary cross loss function and regularization loss function; Obtaining a combined loss function according to the positive sample loss function, the binary cross loss function and the regularization loss function; The mutual learning Mel spectrum model and the mutual learning MFCC model are respectively updated according to the combined loss function to obtain a final Mel spectrum model and a final MFCC model.

6. The audio feature extraction method based on the deep mutual learning framework according to claim 1 is characterized in that: The step of obtaining the audio data to be processed of the target object and inputting the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively to obtain the Mel spectrum prediction value and the MFCC prediction value respectively includes: Acquire the audio data to be processed of the target object, and input the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively; The final Mel spectrum model performs preliminary feature extraction processing on the audio data to be processed to obtain a Mel spectrum corresponding to the audio data to be processed, and performs prediction feature extraction processing based on the Mel spectrum to obtain a Mel spectrum prediction value; The final MFCC model performs preliminary feature extraction processing on the audio data to be processed to obtain MFCC features corresponding to the audio data to be processed, and performs prediction feature extraction processing based on the MFCC features to obtain MFCC prediction values.

7. The audio feature extraction method based on the deep mutual learning framework according to claim 1 is characterized in that: The final Mel spectrum model has the same structure as the final MFCC model, and the final Mel spectrum model includes an encoder and a classifier; The encoder includes a TDNN block, three SE-Res2Block blocks, a convolution block, and an attention statistics pooling block; The classifier includes a fully connected layer and a batch normalization layer.

8. An audio feature extraction system based on a deep mutual learning framework, characterized in that: The audio feature extraction system based on the deep mutual learning framework includes: A training set preprocessing module is used to obtain a training data set, and perform screening operations, labeling operations, cutting operations, and downsampling operations on the training data set to obtain a preprocessed training set; A mutual learning training module, used for performing deep mutual learning training on an initial Mel spectrum model and an initial MFCC model according to the preprocessed training set, calculating a total Mel spectrum loss and a total MFCC loss, and respectively training the initial Mel spectrum model and the initial MFCC model according to the total Mel spectrum loss and the total MFCC loss to obtain a mutual learning Mel spectrum model and a mutual learning MFCC model; A model updating module, used for updating the mutual learning Mel spectrum model and the mutual learning MFCC model according to a combined loss function to obtain a final Mel spectrum model and a final MFCC model; A prediction value acquisition module, used to acquire the audio data to be processed of the target object, and input the audio data to be processed into the final Mel spectrum model and the final MFCC model respectively, to obtain the Mel spectrum prediction value and the MFCC prediction value respectively; The prediction value fusion module is used to average and fuse the Mel spectrum prediction value and the MFCC prediction value to obtain the final prediction feature.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and an audio feature extraction program based on a deep mutual learning framework stored in the memory and executable on the processor. When the audio feature extraction program based on the deep mutual learning framework is executed by the processor, the steps of the audio feature extraction method based on the deep mutual learning framework as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an audio feature extraction program based on a deep mutual learning framework. When the audio feature extraction program based on the deep mutual learning framework is executed by a processor, the steps of the audio feature extraction method based on the deep mutual learning framework as described in any one of claims 1 to 7 are implemented.

Images