An audio recognition method, device, apparatus and storage medium

Abstract

The application discloses an audio recognition method, device and equipment and a storage medium. The method comprises the following steps: obtaining to-be-recognized audio; performing mel-frequency cepstrum transformation on the to-be-recognized audio to obtain a mel-frequency cepstrum matrix; inputting the mel-frequency cepstrum matrix into a recognition model to obtain a sound event result, wherein the recognition model is obtained by iteratively training a neural network model by using a target sample set, and the target sample set comprises mel-frequency cepstrum matrix samples and sound event labels corresponding to the mel-frequency cepstrum matrix samples. According to the technical scheme, the model size can be effectively reduced and the recognition efficiency can be improved under the condition of ensuring the recognition accuracy.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and particularly relate to an audio recognition method, device, equipment and storage medium. BACKGROUND

[0002] Sound carries a lot of information about the daily environment and the physical events occurring therein, and sound event detection can perceive the sound scene, identify each sound source category, and develop signal processing methods to automatically detect sound events, which has great value in many fields. Therefore, in recent years, sound event recognition and detection technology has become a research hotspot and has attracted the attention of many scholars.

[0003] Early sound event recognition is mainly based on traditional signal processing methods and machine learning methods, such as PCA, SVM, etc. However, due to the relatively simple recognition model, the recognition rate is usually low. In recent years, with the rapid development of artificial neural network technology, more and more scholars use neural network models to train audio data sets to improve recognition accuracy, such as CNN, LSTM, GRU, etc. Through the training of a large number of audio data sets, the neural network model greatly improves the recognition accuracy, but the size of the neural network model and the required computing amount increase greatly. The current research pays little attention to the balance between recognition efficiency and recognition accuracy, and a large number of studies only focus on the improvement of recognition accuracy, and pay little attention to the model size and recognition time. SUMMARY

[0004] Embodiments of the present application provide an audio recognition method, device, equipment and storage medium, which can effectively reduce the model size while ensuring the recognition accuracy and improve the recognition efficiency.

[0005] In a first aspect, embodiments of the present application provide an audio recognition method, comprising:

[0006] obtaining to-be-recognized audio;

[0007] performing mel-frequency cepstrum transform on the to-be-recognized audio to obtain a mel-frequency cepstrum matrix;

[0008] inputting the mel-frequency cepstrum matrix into a recognition model to obtain a sound event result, the recognition model being obtained by iteratively training a neural network model with a target sample set, the target sample set comprising: a mel-frequency cepstrum matrix sample and a sound event label corresponding to the mel-frequency cepstrum matrix sample.

[0009] In a second aspect, embodiments of the present application also provide an audio recognition device, which comprises:

[0010] an obtaining module configured to obtain to-be-recognized audio;

[0011] a transformation module, configured to perform a mel-frequency cepstrum transform on the audio to be recognized to obtain a mel-frequency cepstrum matrix;

[0012] an identification module, configured to input the mel-frequency cepstrum matrix into an identification model to obtain a sound event result, the identification model being obtained by iteratively training a neural network model using a target sample set, the target sample set including mel-frequency cepstrum matrix samples and sound event labels corresponding to the mel-frequency cepstrum matrix samples.

[0013] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to any of the embodiments of the present application when executing the program.

[0014] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program executable on a processor, and the processor implements the method according to any of the embodiments of the present application when executing the program.

[0015] The embodiments of the present application obtain an audio to be recognized, perform a mel-frequency cepstrum transform on the audio to be recognized to obtain a mel-frequency cepstrum matrix, input the mel-frequency cepstrum matrix into an identification model to obtain a sound event result, and the identification model is obtained by iteratively training a neural network model using a target sample set, the target sample set including mel-frequency cepstrum matrix samples and sound event labels corresponding to the mel-frequency cepstrum matrix samples, which can effectively reduce the model size and improve the recognition efficiency while ensuring the recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0017] Figure 1 is a flowchart of an audio recognition method in the embodiments of the present application;

[0018] Figure 2 is an architecture diagram of a sound event recognition system in the embodiments of the present application;

[0019] Figure 3 is a mutual information dimension reduction framework diagram in the embodiments of the present application;

[0020] Figure 4 is a dynamic frame reservation diagram in the embodiments of the present application;

[0021] Figure 5 is a schematic diagram of a neural path network model in the embodiments of the present application;

[0022] Figure 6 is a classification identification model schematic diagram of a fusion attention mechanism in an embodiment of the present application;

[0023] Figure 7 is an identification accuracy comparison diagram of an ablation experiment in an embodiment of the present application;

[0024] Figure 8 is an identification accuracy and parameter scale comparison diagram in an embodiment of the present application;

[0025] Figure 9 is a structural schematic diagram of an audio identification device in an embodiment of the present application;

[0026] Figure 10 is a structural schematic diagram of an electronic device in an embodiment of the present application;

[0027] Figure 11 is a structural schematic diagram of a computer readable storage medium containing a computer program in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, and not to limit the present application. In addition, it should be noted that, for ease of description, only the parts related to the present application are shown in the drawings, not all structures. Furthermore, the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0029] Before discussing the example embodiments in more detail, it is to be noted that some example embodiments are described as processes or methods depicted as flow diagrams. Although the process is described in a sequential order, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations can be re-arranged. The process can be terminated when its operations are completed, but can also have additional steps not included in the figure. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. Furthermore, the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0030] The term "comprising" and its derivations used in the present application are open-ended, that is "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment".

[0031] It should be noted that similar reference numerals and letters refer to like items throughout the accompanying drawings, and once an item is defined in one drawing, it is not necessary to further define and explain it in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0032] Figure 1 A flowchart of an audio recognition method provided for an embodiment of the present application. The embodiment can be applicable to the case of audio recognition. The method can be executed by an audio recognition device in the embodiment of the present application. The device can be realized in the form of software and / or hardware. As shown in the figure, the method specifically includes the following steps: Figure 1

[0033] S110, obtaining an audio to be recognized.

[0034] Among them, the audio to be recognized can be any one of an air conditioning sound corresponding audio, a car siren sound corresponding audio, a child playing sound corresponding audio, a dog barking sound corresponding audio, a drilling sound corresponding audio, an engine idle sound corresponding audio, a gun sound corresponding audio, a hand drill corresponding audio, a police whistle sound corresponding audio, and a street music sound corresponding audio.

[0035] Specifically, the way to obtain the audio to be recognized can be to obtain the audio to be recognized through an audio acquisition device.

[0036] S120, performing Mel-frequency cepstrum transform on the audio to be recognized to obtain a Mel-frequency cepstrum matrix.

[0037] Specifically, the Mel-frequency cepstrum transform is performed on the audio to be recognized to obtain the Mel-frequency cepstrum matrix. For example, the audio to be recognized can be preprocessed first, the length of the input vector is controlled by zero padding, and then MFCC calculation is performed on the audio to be recognized.

[0038] S130, inputting the Mel-frequency cepstrum matrix into a recognition model to obtain a sound event result, the recognition model being obtained by iteratively training a neural network model with a target sample set, the target sample set including a Mel-frequency cepstrum matrix sample and a sound event label corresponding to the Mel-frequency cepstrum matrix sample.

[0039] ​The manner of iteratively training the neural network model by using the target sample set can be: inputting the mel-cepstrum matrix sample in the target sample set into the neural network model to obtain a predicted sound event; training parameters of the neural network according to a target function formed by the predicted sound event and a sound event label image corresponding to the mel-cepstrum matrix sample; returning to perform the operation of inputting the mel-cepstrum matrix sample in the target sample set into the neural network model to obtain the predicted sound event until an identification model is obtained. The manner of iteratively training the neural network model by using the target sample set can also be: obtaining a mutual information value of each frame in the mel-cepstrum matrix sample in the target sample set; performing dimension reduction on the mel-cepstrum matrix sample according to the mutual information value of each frame in the mel-cepstrum matrix sample to obtain a reduced mel-cepstrum matrix sample; inputting the reduced mel-cepstrum matrix sample into the neural network model to obtain a predicted sound event; training parameters of the neural network according to a target function formed by the predicted sound event and a sound event label image corresponding to the mel-cepstrum matrix sample; returning to perform the operation of inputting the mel-cepstrum matrix sample in the target sample set into the neural network model to obtain the predicted sound event until an identification model is obtained. The manner of iteratively training the neural network model by using the target sample set can also be: inputting the mel-cepstrum matrix sample in the target sample set into the neural network model to obtain a predicted sound event; adjusting the first ratio threshold according to the predicted sound event and a sound event label corresponding to the mel-cepstrum matrix sample to obtain a target ratio threshold; training parameters of the neural network according to a target function formed by the predicted sound event and a sound event label image corresponding to the mel-cepstrum matrix sample; returning to perform the operation of inputting the mel-cepstrum matrix sample in the target sample set into the neural network model to obtain the predicted sound event until an identification model is obtained.

[0040] The mel-cepstrum matrix sample can be obtained by: obtaining an audio sample, and performing mel-cepstrum transformation on the audio sample to obtain the mel-cepstrum matrix sample.

[0041] Optionally, the manner of iteratively training the neural network model by using the target sample set comprises:

[0042] inputting the mel-cepstrum matrix sample in the target sample set into the neural network model to obtain a predicted sound event;

[0043] training parameters of the neural network according to a target function formed by the predicted sound event and a sound event label image corresponding to the mel-cepstrum matrix sample;

[0044] returning to perform the operation of inputting the mel-cepstrum matrix sample in the target sample set into the neural network model to obtain the predicted sound event until an identification model is obtained.

[0045] Specifically, the manner of inputting the mel-spectrogram matrix sample in the target sample set into the neural network model to obtain the predicted sound event can be: performing dimension reduction processing on the mel-spectrogram matrix sample in the target sample set to obtain a dimension-reduced mel-spectrogram matrix sample; inputting the dimension-reduced mel-spectrogram matrix sample into the neural network model to obtain the predicted sound event. The manner of inputting the mel-spectrogram matrix sample in the target sample set into the neural network model to obtain the predicted sound event can also be: obtaining mutual information values of each frame in the mel-spectrogram matrix sample in the target sample set; performing dimension reduction on the mel-spectrogram matrix sample according to the mutual information values of each frame in the mel-spectrogram matrix sample to obtain a dimension-reduced mel-spectrogram matrix sample; inputting the dimension-reduced mel-spectrogram matrix sample into the neural network model to obtain the predicted sound event.

[0046] Optionally, the manner of inputting the mel-spectrogram matrix sample in the target sample set into the neural network model to obtain the predicted sound event comprises:

[0047] obtaining mutual information values of each frame in the mel-spectrogram matrix sample in the target sample set;

[0048] performing dimension reduction on the mel-spectrogram matrix sample according to the mutual information values of each frame in the mel-spectrogram matrix sample to obtain a dimension-reduced mel-spectrogram matrix sample;

[0049] inputting the dimension-reduced mel-spectrogram matrix sample into the neural network model to obtain the predicted sound event.

[0050] Specifically, the manner of obtaining the mutual information values of each frame in the mel-spectrogram matrix sample in the target sample set can be, for example, obtaining a label of an original frame according to its belonging, and respectively calculating mutual information values with input frames.

[0051] Specifically, the manner of dimension reduction of the mel-cepstrum matrix sample according to the mutual information value of each frame in the mel-cepstrum matrix sample can be: sorting the frames in the mel-cepstrum matrix sample in descending order according to the mutual information value of each frame to obtain a first matrix; if a ratio of cumulative mutual information values of the first preset number of frames in the first matrix to cumulative mutual information values of all frames in the mel-cepstrum matrix sample is greater than or equal to a first ratio threshold, generating the dimension-reduced mel-cepstrum matrix sample according to the first preset number of frames in the first matrix. The first ratio threshold can be pre-set. The manner of dimension reduction of the mel-cepstrum matrix sample according to the mutual information value of each frame in the mel-cepstrum matrix sample can also be: sorting the frames in the mel-cepstrum matrix sample in descending order according to the mutual information value of each frame to obtain a first matrix; determining a first ratio threshold according to the scene label corresponding to the mel-cepstrum matrix sample, and if a ratio of cumulative mutual information values of the first preset number of frames in the first matrix to cumulative mutual information values of all frames in the mel-cepstrum matrix sample is greater than or equal to the first ratio threshold, generating the dimension-reduced mel-cepstrum matrix sample according to the first preset number of frames in the first matrix.

[0052] Optionally, the dimension reduction of the mel-cepstrum matrix sample according to the mutual information value of each frame in the mel-cepstrum matrix sample to obtain the dimension-reduced mel-cepstrum matrix sample comprises:

[0053] sorting the frames in the mel-cepstrum matrix sample in descending order according to the mutual information value of each frame to obtain a first matrix;

[0054] if a ratio of cumulative mutual information values of the first preset number of frames in the first matrix to cumulative mutual information values of all frames in the mel-cepstrum matrix sample is greater than or equal to a first ratio threshold, generating the dimension-reduced mel-cepstrum matrix sample according to the first preset number of frames in the first matrix.

[0055] The first ratio threshold can be pre-set or determined according to the scene label corresponding to the mel-cepstrum matrix sample, and embodiments of the present application do not limit this.

[0056] Specifically, if a ratio of cumulative mutual information values of the first preset number of frames in the first matrix to cumulative mutual information values of all frames in the mel-cepstrum matrix sample is greater than or equal to a first ratio threshold, the dimension-reduced mel-cepstrum matrix sample is generated according to the first preset number of frames in the first matrix, for example, the mel-cepstrum matrix sample can be dimension-reduced based on the following formula:

[0057]

[0058] wherein ζ is a threshold value, MI(m) is the cumulative mutual information value of all frames before dimension reduction, MI(m) is the cumulative mutual information value of all frames after dimension reduction, m and k are the frame numbers before and after dimension reduction respectively.

[0059] Optionally, the target sample set further comprises a scene label corresponding to the mel-cepstrum matrix sample.

[0060] Correspondingly, if the ratio of the cumulative mutual information values of the first preset number of frames in the first matrix to the cumulative mutual information values of all frames in the mel-cepstrum matrix sample is greater than or equal to a first ratio threshold value, before generating the mel-cepstrum matrix sample after dimension reduction according to the first preset number of frames in the first matrix, the method further comprises:

[0061] Determining the first ratio threshold value according to the scene label corresponding to the mel-cepstrum matrix sample.

[0062] Specifically, the way of determining the first ratio threshold value according to the scene label corresponding to the mel-cepstrum matrix sample can be: pre-establishing a database, the database storing a corresponding relationship between scene labels and ratio threshold values, querying the database according to the scene label corresponding to the mel-cepstrum matrix sample to obtain the ratio threshold value corresponding to the scene label corresponding to the mel-cepstrum matrix sample.

[0063] Optionally, after inputting the mel-cepstrum matrix sample after dimension reduction into the neural network model to obtain the predicted sound event, the method further comprises:

[0064] Adjusting the first ratio threshold value according to the predicted sound event and the sound event label corresponding to the mel-cepstrum matrix sample to obtain a target ratio threshold value.

[0065] Specifically, the manner of adjusting the first ratio threshold according to the predicted sound event and the sound event label corresponding to the mel-spectrogram sample to obtain the target ratio threshold can be: in each training process, the ratio threshold is adjusted according to the predicted sound event and the sound event label corresponding to the mel-spectrogram sample. For example, the target sample set includes: a mel-spectrogram sample A and a corresponding sound event label X and a scene label W, and a mel-spectrogram sample B and a corresponding sound event label Y and a scene label W. The frames in the mel-spectrogram sample A are sorted in descending order according to the mutual information value of each frame in the mel-spectrogram sample A to obtain a first matrix. The first ratio threshold L is determined according to the scene label W corresponding to the mel-spectrogram sample A. If the ratio of the cumulative mutual information value of the first preset number of frames in the first matrix to the cumulative mutual information value of all frames in the mel-spectrogram sample is greater than or equal to the first ratio threshold L, the reduced mel-spectrogram sample is generated according to the first preset number of frames in the first matrix. The reduced mel-spectrogram sample A is input into the neural network model to obtain the predicted sound event I. The recognition accuracy r is determined according to the predicted sound event I and the sound event label X. The recognition accuracy r and the first ratio threshold L are recorded. The frames in the mel-spectrogram sample B are sorted in descending order according to the mutual information value of each frame in the mel-spectrogram sample B to obtain a first matrix. The first ratio threshold K is determined according to the scene label Y corresponding to the mel-spectrogram sample B. If the ratio of the cumulative mutual information value of the first preset number of frames in the first matrix to the cumulative mutual information value of all frames in the mel-spectrogram sample is greater than or equal to the first ratio threshold K, the reduced mel-spectrogram sample is generated according to the first preset number of frames in the first matrix. The reduced mel-spectrogram sample B is input into the neural network model to obtain the predicted sound event o. The recognition accuracy s is determined according to the predicted sound event o and the sound event label Y. The recognition accuracy s and the first ratio threshold K are recorded. The relationship graph of the recognition accuracy and the first ratio threshold is generated according to the recognition accuracy and the first ratio threshold. The first ratio threshold is adjusted according to the relationship graph to obtain the target ratio threshold.

[0066] In one specific example, as shown in FIG. 2, the specific process is as follows: Figure 2

[0067] a. Perform mel-spectral transformation on the audio to be identified, the main steps are as follows: fast Fourier transform, triangular band-pass filter filtering, logarithmic operation, and cepstrum calculation. In the triangular band-pass filter filtering, in order to ensure the continuity between frames, the windows of adjacent filters overlap by 1 / 3 to 1 / 2.

[0068] ​b. Build a dimension reduction architecture based on mutual information. Extract the frames at the same position of each mel-cepstrum matrix sample in the target sample set to construct several new feature frame matrices. Obtain the labels of the original frames according to their attribution, and calculate the mutual information values with the input frames, respectively. Then, perform processing through the proposed dynamic frame reservation mechanism to realize the dimension reduction processing of the data frames.

[0069] c. The dynamic frame reservation mechanism proposed in step b, that is, first sort the frames in the mel-cepstrum matrix sample in descending order according to the mutual information values of each frame in the mel-cepstrum matrix sample, and then determine a percentage threshold according to the scene label corresponding to the mel-cepstrum matrix sample. Calculate the cumulative percentage of the mutual information values of all frames in the mel-cepstrum matrix sample according to the percentage threshold. Reserve the data frames within the percentage threshold.

[0070] d. The value of the cumulative percentage threshold of the mutual information value in step c is determined by the data set pre-training link, that is, in the specific scene data set pre-training, the cumulative percentage of the mutual information value is dynamically adjusted. When the model sound event recognition reaches a stable high accuracy and the time spent in recognition remains at a relatively low level, the current cumulative percentage of the mutual information value can be used as the threshold in the current scene.

[0071]

[0072] wherein ζ is the threshold, MI(m) is the cumulative mutual information value of all frames before dimension reduction, MI(k) is the cumulative mutual information value of all frames after dimension reduction, and m and k are the frame numbers before and after dimension reduction, respectively.

[0073] e. Build an artificial neural network model. Input the audio frame data x t after dimension reduction into the LSTM to obtain the output frame h t .

[0074] f. Construct a classification recognition model with fusion attention mechanism. Calculate the energy of the output frame h t of the artificial neural network model, and obtain the attention weight α it through the single-layer perception machine SLP and the softmax function. Input h t into the single-layer perception machine SLP and the softmax function to obtain the data frame s i . With the iteration, the frame with larger energy obtains larger weight, and the frame with smaller energy obtains smaller weight, thereby realizing the attention mechanism.

[0075] g. In step f, input the data frame s i processed by the weight into the multi-layer perception machine MLP and the softmax function for classification and recognition processing, and finally obtain the recognition result.

[0076] In another specific example, a fusion attention and mutual information sound event recognition method of the present application is described in detail below through a specific sound event detection identification instance:

[0077] 1. The UrbanSound8K dataset is selected for identification model verification. The dataset contains 10 types of urban sound events: air conditioner sound (ac), car horn sound (ch), children playing sound (cp), dog barking sound (db), drilling sound (dr), engine idling sound (ei), gunshot sound (gs), jackhammer (jh), siren sound (si), and street music sound (sm), covering most of the noise on urban streets. It contains 8732 audio files, each lasting less than 4s, with a sampling rate ranging from 16kHz to 44.1kHz.

[0078] 2. The model is trained using the Tensorflow-GPU framework. The dataset is randomly divided into training and testing sets in a 4:1 ratio. After experimental verification, the learning rate is set to 5e-3, and the Batch size is set to 80. The above framework includes an Adam optimizer and a dropout setting of 0.5. Random errors generated during initialization are avoided by fixing the random seed. In the UrbanSound8K dataset, the original audio is converted into an MFCC matrix with a size of 174*40.

[0079] 3. Mutual information values are used to reduce the dimensionality of the audio frames. As shown in Figure 3 each frame in the same position of each mel-frequency cepstral coefficient matrix sample in the target sample set is extracted to construct several new feature frame matrices. The labels of the original frames are obtained, and the mutual information values are calculated with the input frames. The final reserved frames are selected through a dynamic frame reservation mechanism, achieving the effect of dimensionality reduction.

[0080] 4. Dynamic frame reservation mechanism, i.e., first sort the frames in the mel-frequency cepstral coefficient matrix sample in descending order according to the mutual information values of each frame in the mel-frequency cepstral coefficient matrix sample, then determine a percentage threshold according to the scene label corresponding to the mel-frequency cepstral coefficient matrix sample; calculate the cumulative percentage of the mutual information values of all frames in the mel-frequency cepstral coefficient matrix sample according to the percentage threshold; reserve the data frames within the percentage threshold.

[0081]

[0082] where ζ is the threshold, MI(m) is the cumulative mutual information value of all frames before dimensionality reduction, and MI(k) is the cumulative mutual information value of all frames after dimensionality reduction. m and k are the number of frames before and after dimensionality reduction, respectively.

[0083] 5. The threshold value for the cumulative percentage of mutual information is determined during the dataset pre-training process. That is, the cumulative percentage of mutual information is dynamically adjusted during pre-training on a specific scenario dataset. When the model achieves a stable high accuracy in sound event recognition and the recognition time remains relatively low, the current cumulative percentage of mutual information can be used as the threshold for the current scenario. For example... Figure 4 The figure shows the relationship between the cumulative mutual information percentage threshold and recognition accuracy and recognition time in the US8K dataset. Figure 4 As can be seen from the data, when the threshold ζ is 80%, the recognition accuracy remains at a high level, while the recognition time is reduced to a certain extent. Therefore, 80% can be selected as the frame retention threshold in this example.

[0084] 6. Select the LSTM model as the artificial neural network model. Input the dimensionality-reduced retained frames into the LSTM model for training / recognition. Each audio frame can be represented as x. it ,t∈[0,T], let x it h is the input to the model. it The output of the model, i.e., h it =LSTM(x it ), t∈[1,k]. The LSTM model framework is as follows: Figure 5 As shown.

[0085] 7. An attention model is embedded into the LSTM model to improve recognition performance. To avoid the bias inherent in the sigmoid function, which makes it unsuitable for using the LSTM output layer as a latent representation of feature frames, this paper assigns weights to the frames of audio samples based on their energy levels to achieve the purpose of attention. The attention mechanism architecture is as follows: Figure 6 As shown.

[0086] h it As input to the SLP (Single-Layer Perceptron), and its hidden layer u is extracted. it The weights and biases of the SLP are W, respectively. w and b w .

[0087] u it =tanh(W w h it +b w (1)

[0088] To obtain the attention weights for each frame, an initial weight vector u is introduced. w Attention weights α are obtained through the softmax function. it As the iteration proceeds, the initialization vector u... wThe weights will be continuously updated until the weights of frames with higher energy are increased and the weights of frames with lower energy are decreased. In summary, this method allows more resources to be allocated to high-energy frames.

[0089]

[0090] The initial attention weight α can be obtained from equation (2). it And obtain the input vector of the classifier through equation (3).

[0091] s i =∑ t α it h it (3)

[0092] Equation (3) multiplies the weights by the initial frame to obtain the attention level of each frame in the model.

[0093] 8. Each frame in the Mel-frequency cepstral matrix sample is used as the input to the LSTM unit, and the output h of the LSTM unit is... it By splicing and integrating the data and inputting an attention mechanism to obtain weights, a classifier composed of a multilayer sensor-machine interface (MLP) and a softmax function is finally used to classify and determine the sound events.

[0094] 9. Conduct ablation experiments to determine the improvement in model training performance caused by the attention mechanism and mutual information dimensionality reduction method. Compare the recognition rate and time consumption of four structures: ML (mutual information + LSTM), MLA (mutual information + LSTM + attention), MG (mutual information + GRU), MGA (mutual information + GRU + attention), GA (GRU + attention), LA (LSTM + attention), G (GRU), and L (LSTM). Figure 7 It can be seen that, regardless of the presence or absence of an attention mechanism, the recognition accuracy decreases monotonically with the decrease in the number of frames. However, with the addition of an attention model, the rate of decline in recognition accuracy is slowed down. The recognition accuracy of MLA and MGA is higher than that of ML and MG, indicating that attention significantly improves the model's recognition accuracy. The recognition accuracy of MLA and ML is higher than that of MGA and MG, respectively, indicating that the sound event recognition performance of the LSTM model is superior to that of GRU.

[0095] 10. The proposed MLA, LA, GA, and ML architectures are compared with existing audio classification architectures on the UrbanSound8K dataset. The results are as follows: Figure 8 As shown. By Figure 8 It is evident that the combination of mutual information methods and attention mechanisms can improve training efficiency, reduce the parameter size of the recognition model, and reduce computer resource consumption while ensuring a high recognition rate, placing it at a high level among similar tasks.

[0096] The above merely describes preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

[0097] The technical scheme of the embodiment obtains to-be-recognized audio, performs mel-frequency cepstrum transformation on the to-be-recognized audio to obtain a mel-frequency cepstrum matrix, and inputs the mel-frequency cepstrum matrix into a recognition model to obtain a sound event result, wherein the recognition model is obtained by iteratively training a neural network model using a target sample set, and the target sample set includes mel-frequency cepstrum matrix samples and sound event labels corresponding to the mel-frequency cepstrum matrix samples, so that the model size can be effectively reduced and the recognition efficiency can be improved while ensuring recognition accuracy.

[0098] Figure 9 A structural schematic diagram of an audio recognition device according to an embodiment of the present application is provided. The embodiment can be applied to the case of audio recognition. The device can be implemented in the form of software and / or hardware, and can be integrated into any device that provides an audio recognition function, such as a computer, a mobile phone, a tablet computer, a smart speaker, etc. Figure 9 As shown in the figure, the audio recognition device specifically includes an acquisition module 210, a transformation module 220, and a recognition module 230.

[0099] The acquisition module is configured to acquire to-be-recognized audio.

[0100] The transformation module is configured to perform mel-frequency cepstrum transformation on the to-be-recognized audio to obtain a mel-frequency cepstrum matrix.

[0101] The recognition module is configured to input the mel-frequency cepstrum matrix into a recognition model to obtain a sound event result, wherein the recognition model is obtained by iteratively training a neural network model using a target sample set, and the target sample set includes mel-frequency cepstrum matrix samples and sound event labels corresponding to the mel-frequency cepstrum matrix samples.

[0102] Optionally, the recognition module is specifically configured to:

[0103] input the mel-frequency cepstrum matrix samples in the target sample set into the neural network model to obtain predicted sound events;

[0104] train parameters of the neural network according to a target function formed by the predicted sound events and the sound event labels corresponding to the mel-frequency cepstrum matrix samples;

[0105] return to perform the operation of inputting the mel-frequency cepstrum matrix samples in the target sample set into the neural network model to obtain predicted sound events until a recognition model is obtained.

[0106] The product described above can execute the method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0107] The technical scheme of the embodiment, by acquiring the to-be-recognized audio; performing mel-frequency cepstrum transform on the to-be-recognized audio to obtain a mel-frequency cepstrum matrix; inputting the mel-frequency cepstrum matrix into a recognition model to obtain a sound event result, the recognition model being obtained by iteratively training a neural network model based on a target sample set, the target sample set including: a mel-frequency cepstrum matrix sample and a sound event label corresponding to the mel-frequency cepstrum matrix sample, can effectively reduce the model size and improve the recognition efficiency while ensuring the recognition accuracy.

[0108] Figure 10 A structural schematic diagram of an electronic device is provided for the embodiment of the application. Figure 10 A block diagram of an electronic device 312 suitable for implementing embodiments of the application is shown. Figure 10 The electronic device 312 shown is merely one example and should not be construed as limiting the functionality or use of embodiments of the application. The device 312 is a typical computing device with trajectory fitting functionality.

[0109] As shown in Figure 10 The electronic device 312 is shown in the form of a general-purpose computing device. The components of the electronic device 312 can include, but are not limited to, one or more processors 316, storage 328, and a bus 318 that connects the different system components, including the storage 328 and the processor 316.

[0110] The bus 318 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures. Examples of such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0111] The electronic device 312 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 312 and includes both volatile and nonvolatile media, removable and non-removable media.

[0112] Storage 328 can include a computer system readable media in the form of volatile memory, such as random access memory (RAM) 330 and / or cache memory 332. Electronic device 312 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 334 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Figure 10 Although not shown, a magnetic disk drive can also be utilized in some embodiments to read from and write to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be utilized in some embodiments to read from and write to a removable, non-volatile optical disk (e.g., a Figure 10 Compact Disc-Read Only Memory (CD-ROM), a Digital Video Disc-Read Only Memory (DVD-ROM) or other optical media). In these instances, each drive can be connected to the bus 318 by one or more data media interfaces. The storage 328 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0113] The program 336 having a set (at least one) of program modules 326 can be stored in, for example, storage 328, including an operating system, one or more application programs, other program modules, and program data, and have been described above and which implemented the

[0114] The electronic device 312 can also communicate with one or more external devices 314 such as a keyboard or pointing device, a camera, a Display 324, etc.; one or more devices that enable a user to interact with the electronic device 312; and / or one or more devices (e.g., a network card, a modem, etc.) that enable the electronic device 312 to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 322. Still yet, such communication can occur electronically over a network 326 such as a Local Area Network (LAN) and / or a Wide Area Network (WAN). Such network interface 320 can be comprised of any of a wide variety of well-known devices such as a modem, a network adapter, a Bluetooth® device, a phone card, a cable, and the like. Where implementations are implemented using a computer-based system, various processes and functions described can each be implemented with one or more computer programs tangibly embodied in a computer-readable storage medium, such as a magnetic or optical disk, a magnetic tape, a semiconductor memory device, etc. The program code can be implemented in any of a variety of programming languages, such as C, C++, Java, and the like. The program code can be stored in any type of now known or later developed computer-readable storage medium, such as a magnetic or optical disk, a magnetic tape, a semiconductor memory device, etc. Further, any or all of the program code can be downloaded from an external source via a suitable communication interface, such as a modem, a network adapter, a phone card, a cable, and the like.

[0115] The processor 316 performs various functions through running programs stored in the storage 328, such as implementing the audio recognition method provided by the above-mentioned embodiments of the present application:

[0116] obtaining to-be-recognized audio;

[0117] performing a mel-frequency cepstrum transform on the to-be-recognized audio to obtain a mel-frequency cepstrum matrix;

[0118] inputting the mel-frequency cepstrum matrix into a recognition model to obtain a sound event result, the recognition model being obtained by iteratively training a neural network model with a target sample set, the target sample set comprising: a mel-frequency cepstrum matrix sample and a sound event label corresponding to the mel-frequency cepstrum matrix sample.

[0119] Figure 11 A structural diagram of a computer-readable storage medium containing a computer program in an embodiment of the present application. The embodiment of the present application provides a computer-readable storage medium 61, which stores a computer program 610, the program being executed by one or more processors to implement the audio recognition method provided by all the embodiments of the present application:

[0120] obtaining to-be-recognized audio;

[0121] performing a mel-frequency cepstrum transform on the to-be-recognized audio to obtain a mel-frequency cepstrum matrix;

[0122] The mel-spectrogram matrix is input into a recognition model to obtain a sound event result, the recognition model being obtained by iteratively training a neural network model using a target sample set, the target sample set including mel-spectrogram matrix samples and sound event labels corresponding to the mel-spectrogram matrix samples.

[0123] Any combination of one or more computer-readable medium can be utilized. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. Computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the document, the computer-readable storage medium can be any tangible medium that contains, or stores a program for use by or in connection with an instruction execution system, apparatus, or device.

[0124] A computer-readable signal medium can include a computer-readable program code carried in a baseband or as a part of a carrier wave, in which the computer-readable program code is carried. Such a propagated signal can take a wide variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. Computer-readable signal medium can also be any computer-readable medium that is not a computer-readable storage medium, which can be used to carry or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0125] The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0126] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0127] The computer readable medium described above can be included in the electronic device described above; alternatively, the computer readable medium can exist as a separate entity, which is not incorporated in the electronic device.

[0128] Computer program code for carrying out operations of the present application can be written in any of one or more programming languages or combinations of languages including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages such as C or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0129] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0130] The units described in the embodiments of the present disclosure can be implemented by hardware, software, or a combination thereof. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0131] The functions described above in the detailed description can be performed by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0132] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0133] Note that the above merely describes preferred embodiments of the present application and the applied technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, reconfigurations and substitutions can be made by those skilled in the art without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. An audio recognition method, characterized by, The method comprises the following steps: obtaining to-be-recognized audio; performing a mel-frequency cepstrum transform on the to-be-recognized audio to obtain a mel-frequency cepstrum matrix; inputting the mel-frequency cepstrum matrix into a recognition model to obtain a sound event result, wherein the recognition model is obtained by iteratively training a neural network model based on a target sample set, and the target sample set comprises mel-frequency cepstrum matrix samples and sound event labels corresponding to the mel-frequency cepstrum matrix samples; wherein the iteratively training the neural network model based on the target sample set comprises: obtaining mutual information values of each frame in the mel-frequency cepstrum matrix samples in the target sample set; performing descending order sorting on the frames in the mel-frequency cepstrum matrix samples according to the mutual information values of each frame to obtain a first matrix; if a ratio of cumulative mutual information values of a preset number of frames in the first matrix to cumulative mutual information values of all frames in the mel-frequency cepstrum matrix samples is greater than or equal to a first ratio threshold, generating dimension-reduced mel-frequency cepstrum matrix samples according to the preset number of frames in the first matrix; inputting the dimension-reduced mel-frequency cepstrum matrix samples into the neural network model to obtain predicted sound events; training parameters of the neural network according to a target function formed by the predicted sound events and the sound event labels corresponding to the mel-frequency cepstrum matrix samples; returning to perform the operation of inputting the mel-frequency cepstrum matrix samples in the target sample set into the neural network model to obtain predicted sound events until a recognition model is obtained.

2. The method of claim 1, wherein, The target sample set further comprises scene labels corresponding to the mel-frequency cepstrum matrix samples. Correspondingly, before the operation of generating dimension-reduced mel-frequency cepstrum matrix samples according to the preset number of frames in the first matrix if the ratio of the cumulative mutual information values of the preset number of frames in the first matrix to the cumulative mutual information values of all frames in the mel-frequency cepstrum matrix samples is greater than or equal to the first ratio threshold, the method further comprises: determining the first ratio threshold according to the scene labels corresponding to the mel-frequency cepstrum matrix samples.

3. The method of claim 1, wherein, After the operation of inputting the dimension-reduced mel-frequency cepstrum matrix samples into the neural network model to obtain predicted sound events, the method further comprises: adjusting the first ratio threshold according to the predicted sound events and the sound event labels corresponding to the mel-frequency cepstrum matrix samples to obtain a target ratio threshold.

4. An audio recognition apparatus, characterized by comprising: The method comprises the following steps: an obtaining module, configured to obtain to-be-recognized audio; a transforming module, configured to perform a mel-frequency cepstrum transform on the to-be-recognized audio to obtain a mel-frequency cepstrum matrix; an identifying module, configured to input the mel-frequency cepstrum matrix into a recognition model to obtain a sound event result, wherein the recognition model is obtained by iteratively training a neural network model based on a target sample set, and the target sample set comprises mel-frequency cepstrum matrix samples and sound event labels corresponding to the mel-frequency cepstrum matrix samples; The identification module is specifically configured to: acquire mutual information values of each frame in a mel-cepstrum matrix sample in the target sample set; sort the frames in the mel-cepstrum matrix sample in descending order according to the mutual information values of each frame in the mel-cepstrum matrix sample to obtain a first matrix; if a ratio of cumulative mutual information values of a preset number of frames in the first matrix to cumulative mutual information values of all frames in the mel-cepstrum matrix sample is greater than or equal to a first ratio threshold, generate a reduced-dimension mel-cepstrum matrix sample according to the preset number of frames in the first matrix; input the reduced-dimension mel-cepstrum matrix sample into a neural network model to obtain a predicted sound event; train parameters of the neural network according to a target function formed by the predicted sound event and a sound event label image corresponding to the mel-cepstrum matrix sample; return to perform an operation of inputting the mel-cepstrum matrix sample in the target sample set into the neural network model to obtain the predicted sound event until an identification model is obtained.

5. An electronic device, comprising: comprising: one or more processors; memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the programs cause the processors to implement the method according to any one of claims 1-3.

6. A computer-readable storage medium including a computer program, having stored thereon a computer program, characterized by The program is executed by one or more processors to implement the method according to any one of claims 1-3.

Images