Audio classification method, system and device based on deep learning and medium

Abstract

The invention discloses an audio classification method, system and device based on deep learning and a medium, and the method comprises the steps: obtaining an audio sequence, carrying out the preprocessing of the audio sequence, and obtaining an audio data set; segmenting an audio frame according to the audio data set, and extracting a frequency spectrum sequence, a wavelet phase diagram, a wavelet intensity diagram and a Mel spectrum of the audio data set; respectively extracting feature vectors corresponding to the audio frames, the spectrum sequences, the wavelet phase diagrams, the wavelet intensity diagrams and the Mel spectrograms by adopting a plurality of groups of independent multi-scale feature extraction modules to obtain feature vectors extracted by each group of multi-scale feature extraction modules; performing feature fusion on the feature vectors extracted by each group of multi-scale feature extraction modules to obtain a feature fusion vector; and performing audio classification according to the feature fusion vector to obtain an audio classification result. According to the invention, the accuracy and anti-interference performance of audio classification can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio classification technology, and in particular to an audio classification method, system, device and medium based on deep learning. Background Art

[0002] Traditional audio classification methods are mainly based on spectrogram extraction methods, which distinguish different types of sound signals by observing and analyzing audio features such as Mel-frequency cepstral coefficients, spectral centroid, and spectral flux. Traditional methods have good classification effects when there is less environmental noise and obvious audio features, but they still have limitations such as poor anti-interference and difficulty in handling complex and changing scenes.

[0003] With the continuous improvement of machine computing power, deep neural networks have been rapidly developed. Algorithms based on deep learning have been widely used in the field of audio classification and have achieved excellent results. However, existing deep learning algorithms, such as models based on convolutional neural networks, usually have difficulty accurately extracting the core features of data when processing time series. Although models based on recurrent neural networks can capture the time dependency of sequence data, they are prone to gradient vanishing or exploding when processing long time series, resulting in unstable training. Although models based on long short-term memory networks improve the stability of training, their complex structure makes the training time longer, and the high hardware configuration requirements make it difficult to achieve real-time processing on large-scale data sets.

[0004] Therefore, most related audio classification methods have relatively poor classification capabilities and relatively low audio classification accuracy. Summary of the invention

[0005] This application aims to propose an audio classification method, system, device and medium based on deep learning, which can improve the accuracy and anti-interference ability of audio classification.

[0006] In a first aspect, an embodiment of the present application provides an audio classification method based on deep learning, the method comprising:

[0007] Acquire an audio sequence, and preprocess the audio sequence to obtain an audio data set;

[0008] Segmenting audio frames according to the audio data set, and extracting a spectrum sequence, a wavelet phase map, a wavelet intensity map, and a Mel-spectrogram of the audio data set;

[0009] Using multiple groups of independent multi-scale feature extraction modules to respectively extract feature vectors corresponding to the audio frame, the spectrum sequence, the wavelet phase map, the wavelet intensity map, and the Mel-spectrogram, to obtain feature vectors extracted by each group of the multi-scale feature extraction modules;

[0010] Performing feature fusion on the feature vectors extracted by each group of the multi-scale feature extraction modules to obtain a feature fusion vector;

[0011] Audio classification is performed according to the feature fusion vector to obtain an audio classification result.

[0012] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0013] The method obtains an audio sequence and preprocesses the audio sequence to obtain an audio data set. By preprocessing the obtained audio sequence, a good data basis can be provided for later audio classification. Then, the audio frame is segmented according to the audio data set, and the spectrum sequence, wavelet phase map, wavelet intensity map and Mel spectrum map of the audio data set are extracted. By extracting the core information of different levels of the original signal, a good data basis can be provided for later audio classification. Then, multiple groups of independent multi-scale feature extraction modules are used to extract feature vectors corresponding to the audio frame, the spectrum sequence, the wavelet phase map, the wavelet intensity map and the Mel spectrum map, respectively, to obtain the feature vector extracted by each group of multi-scale feature extraction modules, and the feature vector extracted by each group of multi-scale feature extraction modules is feature fused to obtain a feature fusion vector, which can avoid different features from influencing each other in the early stage of training, thereby retaining the uniqueness of the features themselves obtained by different extraction methods, and at the same time, the deep-level important information of different features is obtained through multi-scale feature extraction. Feature fusion improves the correlation of features and retains the uniqueness of different features, which can improve the anti-interference ability of audio classification. Finally, audio classification is performed according to the feature fusion vector to obtain an audio classification result, which can improve the accuracy and anti-interference of audio classification.

[0014] In some implementations, segmenting the audio frames according to the audio data set and extracting a spectrum sequence, a wavelet phase map, a wavelet intensity map, and a Mel-spectrogram of the audio data set includes:

[0015] Performing a fast Fourier transform on the audio sequence in the audio data set to obtain a spectrum sequence;

[0016] Performing frame segmentation on the audio sequence in the audio data set to obtain multiple audio frames;

[0017] Performing Morlet wavelet transform on each of the audio frames to obtain wavelet coefficients;

[0018] Based on the wavelet coefficients, calculating a wavelet phase map and a wavelet intensity map;

[0019] Each of the audio frames is filtered using short-time Fourier transform and a Mel filter bank to obtain a Mel spectrogram.

[0020] In some implementations, the multiple groups of independent multiscale feature extraction modules include a first multiscale feature extraction module group, a second multiscale feature extraction module group, a third multiscale feature extraction module group, a fourth multiscale feature extraction module group, and a fifth multiscale feature extraction module group. The multiple groups of independent multiscale feature extraction modules are used to respectively extract feature vectors corresponding to the audio frame, the spectrum sequence, the wavelet phase map, the wavelet intensity map, and the Mel-spectrogram, respectively, to obtain feature vectors extracted by each group of the multiscale feature extraction modules, including:

[0021] Inputting the audio frame into the first multi-scale feature extraction module group for feature extraction to obtain a first feature vector, and inputting the spectrum sequence into the second multi-scale feature extraction module group for feature extraction to obtain a second feature vector, wherein the first multi-scale feature extraction module group and the second multi-scale feature extraction module group have the same structure;

[0022] The wavelet phase map is input into the third multi-scale feature extraction module group for feature extraction to obtain a third feature vector, the wavelet intensity map is input into the fourth multi-scale feature extraction module group for feature extraction to obtain a fourth feature vector, and the Mel spectrum map is input into the fifth multi-scale feature extraction module group for feature extraction to obtain a fifth feature vector. The third multi-scale feature extraction module group, the fourth multi-scale feature extraction module group and the fifth multi-scale feature extraction module group have the same structure.

[0023] In some implementations, the first multi-scale feature extraction module group includes a one-dimensional feature extraction sub-module group and a spatial attention sub-module, and the step of inputting the audio frame into the first multi-scale feature extraction module group for feature extraction to obtain a first feature vector includes:

[0024] Inputting the audio frame into the one-dimensional feature extraction submodule group for feature extraction to obtain a one-dimensional feature extraction vector;

[0025] Expanding the one-dimensional feature extraction vector into a high-dimensional feature vector with the same dimension as the output of the third multi-scale feature extraction module group, the fourth multi-scale feature extraction module group, and the fifth multi-scale feature extraction module;

[0026] The high-dimensional feature vector is input into the spatial attention submodule to obtain a first feature vector.

[0027] In some implementations, the step of inputting the audio frame into the one-dimensional feature extraction submodule group for feature extraction to obtain a one-dimensional feature extraction vector includes:

[0028]

[0029] M c (f i j )=Relu(BN(g m (f i j )))+Relu(BN(g n (f i j )))

[0030] M s (f i j+1 )=σ(AvgPool(f i j+1 )+MaxPool(f i j+1 ))

[0031] Among them, BN means batch normalization, g m represents a one-dimensional convolution operation with a filter size of m, g n represents a one-dimensional convolution operation with a filter size of n, f j j It represents the input of the i-th one-dimensional feature extraction vector to the j-th one-dimensional feature extraction submodule, Relu represents the activation function, + represents concatenation along the channel dimension, represents element-by-element addition, f i j+1 represents the output feature vector, M s (f i j+1 ) represents the input vector of the j+1th one-dimensional feature extraction submodule, σ represents the sigmoid activation function, AvgPool represents average pooling, and MaxPool represents maximum pooling.

[0032] In some implementations, the step of inputting the wavelet phase map into a third multi-scale feature extraction module group for feature extraction to obtain a third feature vector includes:

[0033]

[0034] M c (f i j )=Relu(BN(g 1*s (f i j )))+Relu(BN(g s*1 (f i j )))+Relu(BN(g s*s (f i j)))

[0035] Among them, BN means batch normalization, g s*s represents a two-dimensional convolution operation with a filter size of s*s, g 1*s represents a two-dimensional convolution operation with a filter size of 1*s, g s*1 represents a two-dimensional convolution operation with a filter size of s*1, f i j It represents the input of the third eigenvector of the i-th multi-scale feature extraction module of the j-th multi-scale feature extraction module, Relu represents the activation function, + represents concatenation along the channel dimension, represents element-by-element addition, f i j+1 The output feature vector is also used as the input vector of the j+1th third multi-scale feature extraction module.

[0036] In some implementations, performing audio classification according to the feature fusion vector to obtain an audio classification result includes:

[0037] Inputting the feature fusion vector into the MobileNet feature extraction layer to obtain a deep feature vector;

[0038] Expanding the deep feature vector by global average pooling to obtain a feature vector of fixed length;

[0039] The fixed-length feature vector is input into Kolmogorov-Arnold Networks for classification to obtain an audio classification result.

[0040] In a second aspect, an embodiment of the present application further provides an audio classification system based on deep learning, the system comprising:

[0041] An audio data acquisition unit, used to acquire an audio sequence and preprocess the audio sequence to obtain an audio data set;

[0042] A data feature extraction unit, used to segment audio frames according to the audio data set, and extract a spectrum sequence, a wavelet phase map, a wavelet intensity map, and a Mel-spectrogram of the audio data set;

[0043] A feature vector extraction unit, configured to respectively extract feature vectors corresponding to the audio frame, the spectrum sequence, the wavelet phase map, the wavelet intensity map, and the Mel-spectrogram using multiple groups of independent multi-scale feature extraction modules, and obtain feature vectors extracted by each group of the multi-scale feature extraction modules;

[0044] A feature vector fusion unit, used for fusing the feature vectors extracted by each group of the multi-scale feature extraction modules to obtain a feature fusion vector;

[0045] The audio classification unit is used to perform audio classification according to the feature fusion vector to obtain an audio classification result.

[0046] In a third aspect, an embodiment of the present application also provides an electronic device, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a deep learning-based audio classification method as described above.

[0047] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the deep learning-based audio classification method as described above.

[0048] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0050] Figure 1 is a flowchart of an embodiment of an audio classification method based on deep learning provided by the present application;

[0051] Figure 2 It is a schematic diagram of the overall process of the best embodiment of the audio classification method based on deep learning provided by the present application;

[0052] Figure 3 It is a structural schematic diagram of a multi-scale feature extraction module group in the best embodiment of the audio classification method based on deep learning provided by the present application;

[0053] Figure 4 It is a structural schematic diagram of multi-scale feature fusion in the best embodiment of the audio classification method based on deep learning provided by the present application;

[0054] Figure 5 is a structural diagram of an embodiment of an audio classification system based on deep learning provided by the present application;

[0055] Figure 6 It is a schematic structural diagram of an embodiment of an electronic device provided by the present application. DETAILED DESCRIPTION

[0056] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0057] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0058] It should also be understood that the references to "one embodiment" or "some embodiments" described in the specification of the embodiments of the present application mean that one or more embodiments of the embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0059] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0060] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0061] First, some nouns involved in this application are analyzed:

[0062] Upsampling: refers to increasing the sampling rate or resolution of a signal or image, thereby making it larger. It increases the number of sample points by interpolating or filling in new data points.

[0063] Downsampling: refers to reducing the sampling rate or resolution of a signal or image, thereby making it smaller. It reduces the number of sample points by skipping or merging data points.

[0064] Residual connection: A type of skip connection that adds input to the middle layer or output of the network. It can solve the vanishing gradient problem in neural networks, and it can also make it easier for gradients to flow through very deep neural networks.

[0065] Multi-head attention module: used to capture the correlation between inputs from different heads.

[0066] Spatial attention module: used to capture important information at different locations in the image.

[0067] Dimensionality increase: refers to increasing the number of channels of the feature vector so that it can represent more types of feature information. It increases the number of channels by convolving with more convolution kernels of size 1.

[0068] Dimensionality reduction: refers to reducing the number of channels of the feature vector, thereby reducing the useless feature information it represents. It reduces the number of channels by convolving with fewer convolution kernels of size 1.

[0069] Traditional audio classification methods are mainly based on spectrogram extraction methods, which distinguish different types of sound signals by observing and analyzing audio features such as Mel-frequency cepstral coefficients, spectral centroid, and spectral flux. Traditional methods have good classification effects when there is less environmental noise and obvious audio features, but they still have limitations such as poor anti-interference and difficulty in handling complex and changing scenes.

[0070] With the continuous improvement of machine computing power, deep neural networks have been rapidly developed. Algorithms based on deep learning have been widely used in the field of audio classification and have achieved excellent results. However, existing deep learning algorithms, such as models based on convolutional neural networks, usually have difficulty accurately extracting the core features of data when processing time series. Although models based on recurrent neural networks can capture the time dependency of sequence data, they are prone to gradient vanishing or exploding when processing long time series, resulting in unstable training. Although models based on long short-term memory networks improve the stability of training, their complex structure makes the training time longer, and the high hardware configuration requirements make it difficult to achieve real-time processing on large-scale data sets.

[0071] Therefore, most related audio classification methods have relatively poor classification capabilities and relatively low audio classification accuracy.

[0072] In order to solve the above-mentioned problems of poor audio classification ability and low classification accuracy, the present application proposes an audio classification method, system, device and medium based on deep learning.

[0073] Reference Figure 1 The present invention provides an audio classification method based on deep learning, which comprises the following steps:

[0074] Step S100: obtaining an audio sequence and preprocessing the audio sequence to obtain an audio data set;

[0075] Step S200, segmenting the audio frames according to the audio data set, and extracting the spectrum sequence, wavelet phase map, wavelet intensity map and Mel-spectrogram of the audio data set;

[0076] Step S300, using multiple groups of independent multi-scale feature extraction modules to respectively extract feature vectors corresponding to the audio frame, the spectrum sequence, the wavelet phase map, the wavelet intensity map and the Mel-spectrogram, and obtain feature vectors extracted by each group of multi-scale feature extraction modules;

[0077] Step S400, performing feature fusion on the feature vectors extracted by each group of multi-scale feature extraction modules to obtain a feature fusion vector;

[0078] Step S500: Perform audio classification according to the feature fusion vector to obtain an audio classification result.

[0079] In this embodiment, an audio data set is obtained by acquiring an audio sequence and preprocessing the audio sequence. By preprocessing the acquired audio sequence, a good data basis can be provided for later audio classification. Then, the audio frame is segmented according to the audio data set, and a spectrum sequence, a wavelet phase map, a wavelet intensity map and a Mel-spectrogram of the audio data set are extracted. By extracting the core information of different levels of the original signal, a good data basis can be provided for later audio classification. Then, multiple groups of independent multi-scale feature extraction modules are used to respectively extract feature vectors corresponding to the audio frame, the spectrum sequence, the wavelet phase map, the wavelet intensity map and the Mel-spectrogram, and obtain The feature vectors extracted by each group of multi-scale feature extraction modules are subjected to feature fusion to obtain feature fusion vectors, which can avoid different features from influencing each other in the early stage of training, thereby retaining the uniqueness of the features themselves obtained by different extraction methods, and at the same time obtaining deep important information of different features through multi-scale feature extraction. Feature fusion improves the correlation of features and retains the uniqueness of different features, which can improve the anti-interference ability of audio classification; finally, audio classification is performed according to the feature fusion vector to obtain the audio classification result, which can improve the accuracy and anti-interference of audio classification.

[0080] The above-mentioned acquired audio sequence may be an audio sequence acquired from different musical instruments and objects of different materials, or may be an audio sequence acquired from the environment or voice, etc., and this embodiment does not make any specific limitation.

[0081] The above-mentioned preprocessing of the audio sequence to obtain the audio data set may be to use a Gaussian filter to denoise the audio sequence, and construct the denoised audio sequence as the audio data set.

[0082] The above-mentioned extraction of audio frames, spectrum sequences, wavelet phase maps, wavelet intensity maps and Mel-spectrograms of the audio data set can be performed by using methods such as Fourier transform, wavelet transform and Mel filter bank to extract audio frames, spectrum sequences, wavelet phase maps, wavelet intensity maps and Mel-spectrograms of the audio data set.

[0083] The above-mentioned use of multiple groups of independent multi-scale feature extraction modules to respectively extract feature vectors corresponding to audio frames, spectrum sequences, wavelet phase maps, wavelet intensity maps and Mel-spectrograms can be a group of multi-scale feature extraction modules to extract feature vectors corresponding to audio frames, a group of multi-scale feature extraction modules to extract feature vectors corresponding to spectrum sequences, a group of multi-scale feature extraction modules to extract feature vectors corresponding to wavelet phase maps, a group of multi-scale feature extraction modules to extract feature vectors corresponding to wavelet intensity maps, and a group of multi-scale feature extraction modules to extract feature vectors corresponding to Mel-spectrograms. The multiple groups of independent multi-scale feature extraction modules can be multi-scale feature extraction modules with the same structure or multi-scale feature extraction modules with different structures, and this embodiment does not make any specific restrictions.

[0084] The feature vectors extracted by each group of multi-scale feature extraction modules are subjected to feature fusion to obtain the feature fusion vectors. The feature vectors extracted by each group of multi-scale feature extraction modules are subjected to feature fusion by adopting a multi-head attention mechanism to obtain the feature fusion vectors. Technical solutions known to those skilled in the art can also be used to perform feature fusion on the feature vectors extracted by each group of multi-scale feature extraction modules, which are not specifically described or limited in this embodiment.

[0085] The audio classification is performed according to the feature fusion vector to obtain the audio classification result. The feature fusion vector can be input into a combination network of MobileNet and Kolmogorov-Arnold Networks to perform audio classification to obtain the audio classification result.

[0086] The above multi-head attention mechanism can be to divide the input sequence into multiple subspaces, each subspace corresponds to a "head". Each head performs attention calculation independently and generates an output vector. These output vectors are then concatenated, linearly transformed, and other operations to finally generate the output of the model.

[0087] The above MobileNet can be a set of mobile-first computer vision models. Through TensorFlowMobile, these models can run efficiently on mobile devices offline.

[0088] For a detailed introduction to the above-mentioned Kolmogorov-Arnold Networks, please refer to the content at the website “https: / / www.cnblogs.com / lsgxeva / p / 18187522”, which will not be described in detail in this embodiment.

[0089] In some implementations, segmenting audio frames according to an audio data set and extracting a spectrum sequence, a wavelet phase map, a wavelet intensity map, and a Mel-spectrogram of the audio data set includes:

[0090] The audio sequence in the audio data set is subjected to fast Fourier transform to obtain a spectrum sequence;

[0091] Perform frame segmentation on the audio sequence in the audio data set to obtain multiple audio frames;

[0092] Perform Morlet wavelet transform on each audio frame to obtain wavelet coefficients;

[0093] Based on the wavelet coefficients, the wavelet phase map and the wavelet intensity map are calculated;

[0094] Each audio frame is filtered using short-time Fourier transform and Mel filter bank to obtain a Mel spectrogram.

[0095] In this embodiment, the audio sequence in the audio data set is subjected to fast Fourier transform to obtain a spectrum sequence; the audio sequence in the audio data set is frame segmented to obtain multiple audio frames; each audio frame is subjected to Morlet wavelet transform to obtain wavelet coefficients; based on the wavelet coefficients, the wavelet phase map and wavelet intensity map are calculated; each audio frame is filtered using short-time Fourier transform and Mel filter bank to obtain a Mel spectrogram. In this way, through various extraction methods such as fast Fourier transform, Morlet wavelet transform, short-time Fourier transform and Mel filter bank, the core information of different levels of the original signal in the audio data set can be quickly extracted, thereby providing a good data foundation for the subsequent audio classification.

[0096] In some embodiments, multiple groups of independent multiscale feature extraction modules, including a first multiscale feature extraction module group, a second multiscale feature extraction module group, a third multiscale feature extraction module group, a fourth multiscale feature extraction module group, and a fifth multiscale feature extraction module group, use multiple groups of independent multiscale feature extraction modules to respectively extract feature vectors corresponding to audio frames, spectrum sequences, wavelet phase maps, wavelet intensity maps, and mel-spectrograms, respectively, to obtain feature vectors extracted by each group of multiscale feature extraction modules, including:

[0097] Inputting the audio frame into a first multi-scale feature extraction module group for feature extraction to obtain a first feature vector, and inputting the spectrum sequence into a second multi-scale feature extraction module group for feature extraction to obtain a second feature vector, wherein the first multi-scale feature extraction module group and the second multi-scale feature extraction module group have the same structure;

[0098] The wavelet phase map is input into the third multi-scale feature extraction module group for feature extraction to obtain a third eigenvector, the wavelet intensity map is input into the fourth multi-scale feature extraction module group for feature extraction to obtain a fourth eigenvector, and the mel spectrum map is input into the fifth multi-scale feature extraction module group for feature extraction to obtain a fifth eigenvector. The structures of the third multi-scale feature extraction module group, the fourth multi-scale feature extraction module group and the fifth multi-scale feature extraction module group are the same.

[0099] In this embodiment, multiple groups of independent multi-scale feature extraction modules are used to extract feature vectors corresponding to audio frames, spectrum sequences, wavelet phase maps, wavelet intensity maps, and mel-spectrograms, respectively, to obtain feature vectors extracted by each group of multi-scale feature extraction modules. By inputting different spectrograms and sequence features into independent multi-scale deep multi-scale feature extraction modules, different features are prevented from influencing each other in the early stage of training, thereby retaining the uniqueness of the features obtained by different extraction methods, and at the same time, the deep-level important information of different features is obtained through the multi-scale feature extraction module, and then the deep-level important information of different features is used for the later audio classification, which can improve the accuracy of audio classification and improve the anti-interference ability of audio classification.

[0100] In some implementations, the first multi-scale feature extraction module group includes a one-dimensional feature extraction sub-module group and a spatial attention sub-module, and the audio frame is input into the first multi-scale feature extraction module group for feature extraction to obtain a first feature vector, including:

[0101] Input the audio frame into the one-dimensional feature extraction submodule group for feature extraction to obtain a one-dimensional feature extraction vector;

[0102] Expanding the one-dimensional feature extraction vector into a high-dimensional feature vector with the same dimension as the output of the third multi-scale feature extraction module group, the fourth multi-scale feature extraction module group, and the fifth multi-scale feature extraction module;

[0103] The high-dimensional feature vector is input into the spatial attention submodule to obtain the first feature vector.

[0104] In this embodiment, the audio frame is input into the one-dimensional feature extraction submodule group for feature extraction to obtain a one-dimensional feature extraction vector; the one-dimensional feature extraction vector is expanded into a high-dimensional feature vector of the same dimension as the output of the third multi-scale feature extraction module group, the fourth multi-scale feature extraction module group, and the fifth multi-scale feature extraction module; the high-dimensional feature vector is input into the spatial attention submodule to obtain a first feature vector. In this way, by expanding the one-dimensional feature extraction vector into a high-dimensional feature vector, the high-dimensional feature vector has the same dimension as the output of the third multi-scale feature extraction module group, the fourth multi-scale feature extraction module group, and the fifth multi-scale feature extraction module, so that subsequent feature fusion can be performed, providing a good data foundation for later audio classification.

[0105] In some implementations, the audio frame is input into a one-dimensional feature extraction submodule group for feature extraction to obtain a one-dimensional feature extraction vector, including:

[0106]

[0107] M c (f i j )=Relu(BN(g m (f i j )))+Relu(BN(g n (f i j )))

[0108] M s (f i j+1 )=σ(AvgPool(f i j+1 )+MaxPool(f i j+1 ))

[0109] Among them, BN means batch normalization, g m represents a one-dimensional convolution operation with a filter size of m, g n represents a one-dimensional convolution operation with a filter size of n, f j j It represents the input of the i-th one-dimensional feature extraction vector to the j-th one-dimensional feature extraction submodule, Relu represents the activation function, + represents concatenation along the channel dimension, represents element-by-element addition, f i j+1 represents the output feature vector, M s (f i j+1) represents the input vector of the j+1th one-dimensional feature extraction submodule, σ represents the sigmoid activation function, AvgPool represents average pooling, and MaxPool represents maximum pooling.

[0110] In some implementations, the wavelet phase map is input into a third multi-scale feature extraction module group for feature extraction to obtain a third feature vector, including:

[0111]

[0112] M c (f i j )=Relu(BN(g 1*s (f i j )))+Relu(BN(g s*1 (f i j )))+Relu(BN(g s*s (f i j )))

[0113] Among them, BN means batch normalization, g s*s represents a two-dimensional convolution operation with a filter size of s*s, g 1*s represents a two-dimensional convolution operation with a filter size of 1*s, g s*1 represents a two-dimensional convolution operation with a filter size of s*1, f i j It represents the input of the third eigenvector of the i-th multi-scale feature extraction module of the j-th multi-scale feature extraction module, Relu represents the activation function, + represents concatenation along the channel dimension, represents element-by-element addition, f i j+1 The output feature vector is also used as the input vector of the j+1th third multi-scale feature extraction module.

[0114] In some implementations, audio classification is performed according to the feature fusion vector quantity to obtain an audio classification result, including:

[0115] Input the feature fusion vector into the MobileNet feature extraction layer to obtain the deep feature vector;

[0116] The deep feature vector is expanded through global average pooling to obtain a feature vector of fixed length;

[0117] The fixed-length feature vector is input into Kolmogorov-Arnold Networks for classification to obtain the audio classification result.

[0118] In this embodiment, the feature fusion vector is input into the MobileNet feature extraction layer to obtain a deep feature vector, which is then expanded by global average pooling to obtain a feature vector of fixed length. Finally, the feature vector of fixed length is input into the Kolmogorov-Arnold Networks for classification. Kolmogorov-Arnold Networks can more flexibly capture nonlinear relationships in multi-dimensional feature space by introducing B-spline functions, thereby improving the network's ability to fit complex audio features, thereby improving the accuracy and anti-interference ability of audio classification.

[0119] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0120] Traditional audio classification methods are mainly based on spectrogram extraction methods, which distinguish different types of sound signals by observing and analyzing audio features such as Mel-frequency cepstral coefficients, spectral centroids, and spectral flux. Traditional methods have good classification effects when there is less ambient noise and obvious audio features, but they still have limitations such as poor anti-interference and difficulty in handling complex and changing scenes.

[0121] With the continuous improvement of machine computing power, deep neural networks have developed rapidly. Algorithms based on deep learning have been widely used in the field of audio classification and have achieved excellent results. Compared with traditional audio classification methods, on the one hand, the interference of human factors can be reduced by using spectrograms such as Mel-spectrograms to replace traditional digital features. On the other hand, adaptive learning can be performed through the back-propagation mechanism of the model, so that the model can extract richer features in complex and dynamic environments. Therefore, deep neural networks that combine spectrograms and feature extraction have become the mainstream research direction.

[0122] Existing audio classification methods based on deep learning, for example, models based on convolutional neural networks usually have difficulty accurately extracting the core features of data when processing time series. Models based on recurrent neural networks, although they can capture the time dependency of sequence data, are prone to gradient vanishing or exploding when processing long time series, resulting in unstable training. Although models based on long short-term memory networks improve the stability of training, their complex structure makes the training time longer, and the high hardware configuration requirements make it difficult to achieve real-time processing on large-scale data sets. Therefore, the classification capabilities of most related technologies are not ideal.

[0123] In order to solve the problem that the classification ability of the above-mentioned existing related technologies is not ideal, refer to Figure 2This embodiment designs a method for combining traditional spectrum and time spectrum into a deep neural network classification model. This method selects a variety of different signal features and independently inputs them into a multi-scale feature extraction module so that different signals will not affect each other during feature extraction, thereby ensuring that feature depth information will not be confused. The fused features are then input into a combined network of MobileNet and Kolmogorov-Arnold Networks, thereby improving the network's ability to fit complex audio features and improving the accuracy and anti-interference of audio classification. The technical solution of this embodiment specifically includes the following steps:

[0124] Step 1: Obtain audio sequences from different musical instruments and objects of different materials, filter the collected audio sequences using a Gaussian filter, and obtain an audio data set.

[0125] Specifically, the collected audio sequence is first resampled to the same frequency of 40 khz and then truncated into an audio sequence of the same length L.

[0126] Then, a Gaussian filter is applied to the audio sequence, and a Gaussian kernel of size σ is used for smoothing to obtain a denoised audio sequence, and an audio dataset is constructed through the denoised audio sequence. The size of j is 2000 sampling points.

[0127] Step 2: Segment the audio sequence after filtering and denoising to obtain multiple audio frames (i.e. Figure 2 The audio sequence after filtering and denoising is subjected to Fourier transform and wavelet transform to extract the spectrum sequence, wavelet phase map, wavelet intensity map and Mel-spectrogram of the audio data set as preprocessing.

[0128] Specifically, the audio sequence of length L is subjected to fast Fourier transform to obtain a frequency domain sequence (i.e., spectrum sequence) of length L, namely:

[0129]

[0130] Where x[n] represents the nth sample of the time domain signal, X[K] represents the kth component of the frequency domain sequence, and j represents the imaginary unit. Represents basis functions.

[0131] Then, the audio sequence of length L is divided into frames, and the frame length is set to 2048 and the frame shift is set to 1024.

[0132] The wavelet coefficients of each frame are calculated using the Morlet wavelet with a wavelet scale of 1 to 100 and then the average is taken, that is:

[0133]

[0134] Where x(t) represents an audio frame, which is time series data, a represents the scale, b represents the time displacement, t represents the time, and W(a,b) represents the transformation coefficient at the scale a and the time displacement b.

[0135] The scale is then converted to frequency f, namely:

[0136]

[0137] Among them, fc represents the center frequency of the wavelet, a represents the scale, and sr represents the sampling rate.

[0138] Then calculate the wavelet intensity and wavelet phase, namely:

[0139]

[0140] Where W(a,b) represents the wavelet coefficient, Re represents the real part, Im represents the imaginary part, |W(a,b)| represents the wavelet intensity, represents the wavelet phase.

[0141] The wavelet intensity map and wavelet phase map are scaled to w*h, i.e., 224*224, respectively, by bilinear interpolation.

[0142] Then, the above method is used to perform framing, and a Mel filter bank with a frequency of 100 is used. After short-time Fourier transform, each frame is filtered by applying the Mel filter bank, that is:

[0143]

[0144] Among them, M m [n] represents the output value of the mth Mel filter in the nth frame, H m (f k ) represents the mth Mel filter for frequency f k The response, |X[k,n]| 2 represents the spectral power of each frame, and M(f) represents the Mel frequency.

[0145] According to the output of the above Mel filter bank and the Mel frequency, a Mel spectrogram can be obtained. The obtained Mel spectrogram is logarithmized and scaled to w*h, i.e., 224*224, by bilinear interpolation.

[0146] Step 3: Input the preprocessed features in step 2 (i.e., time series, spectrum series, wavelet phase map, wavelet intensity map, and Mel-spectrogram) into multiple independent multi-scale deep feature extraction modules to obtain their respective feature vectors.

[0147] Specifically, the preprocessed features are input into multiple groups of independent multi-scale deep feature extraction modules (i.e., multi-scale feature extraction modules) to obtain their respective feature vectors. Figure 3 ,in Figure 3 (a) is a schematic diagram of the structure of the 1DConv convolution module. Figure 3 (b) is a schematic diagram of the structure of the 2DConv convolution module. Figure 3 (c) is a schematic diagram of the structure of the C1DMSF module. Figure 3 (d) is a schematic diagram of the structure of the C2DMSF module. Figure 3 (e) is a schematic diagram of the structure of the multi-scale feature extraction module group.

[0148] The time series I1 and frequency domain sequence (i.e., spectrum sequence) I2 obtained in step 2 are respectively input into two sets of independent one-dimensional multi-scale feature extraction modules (C1DMSF) (the two sets of independent one-dimensional multi-scale feature extraction modules include a first multi-scale feature extraction module group and a second multi-scale feature extraction module group). I1 and I2 are transformed as follows after each C1DMSF of each set of multi-scale feature extraction modules, namely:

[0149]

[0150] M c (f i j )=Relu(BN(g m (f i j )))+Relu(BN(g n (f i j )))

[0151] M s (f i j+1 )=σ(AvgPool(f i j+1 )+MaxPool(f i j+1 ))

[0152] Among them, BN means batch normalization, g m represents a one-dimensional convolution operation with a filter size of m, g n represents a one-dimensional convolution operation with a filter size of n, f i jIt represents the input of the i-th one-dimensional feature extraction vector to the j-th one-dimensional multi-scale feature extraction module. When the j-th one-dimensional multi-scale feature extraction module is the initial one-dimensional multi-scale feature extraction module, the input is a time series or a spectrum series. The input of the next one-dimensional multi-scale feature extraction module is the output of the previous one-dimensional multi-scale feature extraction module. Relu represents the activation function, and + represents concatenation along the channel dimension. represents element-by-element addition, f i j+1 represents the output vector, M s (f i j+1 ) is used as the input vector of the j+1th module, σ represents the sigmoid activation function, AvgPool represents average pooling, and MaxPool represents maximum pooling.

[0153] The wavelet intensity map I3, wavelet phase map I4 and Mel spectrum map I5 obtained in step 2 are respectively input into three groups of independent two-dimensional multi-scale feature extraction modules (C2DMSF) (the three groups of independent two-dimensional multi-scale feature extraction modules include the third multi-scale feature extraction module group, the fourth multi-scale feature extraction module group and the fifth multi-scale feature extraction module group). I3, I4, I5 are transformed as follows after each layer of C2DMSF in each group, namely:

[0154]

[0155] M c (f i j )=Relu(BN(g 1*s (f i j )))+Relu(BN(g s*1 (f i j )))+Relu(BN(g s*s (f i j )))

[0156] Among them, BN means batch normalization, g s*s represents a two-dimensional convolution operation with a filter size of s*s, g 1*s represents a two-dimensional convolution operation with a filter size of 1*s, g s*1 represents a two-dimensional convolution operation with a filter size of s*1, f i jRepresents the input of the i-th two-dimensional feature tensor (i.e., the third eigenvector, the fourth eigenvector, or the fifth eigenvector) to the j-th two-dimensional multi-scale feature extraction module. When the j-th two-dimensional multi-scale feature extraction module is the initial two-dimensional multi-scale feature extraction module, the input is a wavelet phase map, a wavelet intensity map, or a Mel spectrum map. The input of the next two-dimensional multi-scale feature extraction module is the output of the previous two-dimensional multi-scale feature extraction module. Relu represents the activation function, and + represents concatenation along the channel dimension. represents element-by-element addition, f i j+1 Indicates that the output tensor is also the input tensor of the j+1th module.

[0157] Each group of multi-scale feature extraction modules includes 4 feature extraction modules connected in series. The input of the one-dimensional sequence is fixed to 80000*1, so the feature vector size of each module in the multi-scale feature extraction module group is fixed to [20000*16, 5000*16, 1250*32, 312*32]. The size of the feature map is fixed to 224*224*1, so the feature vector size of each module in the multi-scale feature extraction module group is fixed to [224*224*16, 224*224*16, 224*224*32, 224*224*32]. Therefore, the feature vectors A1, A2, P3, P4, P5 corresponding to I1, I2, I3, I4, I5 are obtained.

[0158] Use interpolation to transform the size of A1 and A2 to 224*32, then expand the new dimension to 224*32*1, then transform it to 224*32*224 by dimensionality increase, and finally rearrange the two-dimensional tensor to get 224*224*32.

[0159] The rearranged two-dimensional tensor is input into the spatial attention module and residually connected to obtain the final two-dimensional tensors P1 and P2.

[0160]

[0161] M s (f i )=σ(g s*s (AvgPool(f i )+MaxPool(f i )))

[0162] Among them, AvgPool represents average pooling, MaxPool represents maximum pooling, and g s*s represents the convolution operation with a filter size of s*s, σ represents the sigmoid activation function, and f i represents the input vector, M srepresents the spatial attention module, ⊙ represents element-wise multiplication, and f i ' represents the output result, + represents concatenation along the channel dimension, Represents element-wise addition.

[0163] Step 4: After the feature vector extracted in step 3 is fused through the multi-head attention mechanism, a multi-channel fusion feature (i.e., feature fusion vector) is obtained.

[0164] Specifically, P1, P2, P3, P4, and P5 are used as the five heads of the multi-head attention module for input. Figure 4 As shown, for each P i Perform linear projection and get Q i K i V i Vector, that is:

[0165] Q i =P i W Q

[0166] K i =P i W K

[0167] V i =P i W V

[0168] Among them, W Q represents the query projection matrix, W K represents the projection matrix of the key, W V The projection matrix representing the value, Q i represents the query vector, K i represents the key vector, V i Represents a vector of values.

[0169] For each query Q i , calculate its value with all other keys K j The dot product of , we get the attention score:

[0170]

[0171] Among them, score ij Represents the query Q i With key K j The similarity score, d k Indicates the dimension of the key vector.

[0172] The attention scores are normalized by softmax to obtain the attention weight of each query, that is:

[0173]

[0174] Among them, α ij Represents the query Q i Key K j The attention weight.

[0175] According to the attention weights, the value vector V j Weighted summation gives the representation vector Z i ,Right now:

[0176]

[0177] Among them, Z i Represents the vector calculated by the attention mechanism, with a shape of 224*224*16.

[0178] The above Z1, Z2, Z3, Z4, and Z5 are connected along the channel dimension and then reduced in dimension to obtain a feature fusion vector X with a shape of 224*224*32.

[0179] Step 5: Input the processed fusion features into the combined network of MobileNet and Kolmogorov-Arnold Networks to classify different types of sounds.

[0180] The feature fusion vector X obtained in step 4 is input into the MobileNet V2 model, where the first layer of the convolution kernel of the model needs to be modified to accept 32 channels. After the feature depth is extracted, the feature is pooled by global average pooling. Figure 1 Dimensionally expanded to obtain a fixed-length feature vector. After the global average pooling layer, the Kolmogorov-Arnold Networks are input for classification, where the number of neurons in the two hidden layers of the Kolmogorov-Arnold Networks is 128 and 64 respectively.

[0181] In this embodiment, the model parameters are determined by training. A dataset collected in the laboratory is used, from which 50 categories of audio sequences are selected for training. There are about 200 audio sequences of each category, totaling about 10,000 audio sequences. The training set and the test set are divided into 8:2. The training samples are about 8,000 groups of data and the test samples are 2,000 groups of data. The pytorch framework is used for implementation, and the configuration environment includes cuda10.2, python3.9, and torch1.11.

[0182] It should be noted that the model training parameters of this embodiment can be specifically set according to actual construction requirements and are not specifically limited in this embodiment.

[0183] In this embodiment, compared with the prior art, the advantage of the technical solution of this embodiment is that after extracting a variety of time domain, frequency domain and time-frequency spectrum features through traditional methods, the multi-scale deep features of each are independently extracted and then fused. Since the fusion of different features in the early stage of training may lose the independent characteristics of each feature, mutual interference will reduce the overall recognition rate. Therefore, this embodiment uses multiple groups of independent multi-scale feature extraction modules to extract features, so that the deep information of different features can be correctly extracted. Then, after the multi-head attention fusion module is used for feature fusion, the MobileNet V2 model is used for feature extraction, and then the Kolmogorov-ArnoldNetworks model is used for classification. Since the model introduces the B-spline function, it can capture nonlinear relationships more flexibly in the multi-dimensional feature space, improve the network's ability to fit complex audio features, and thus improve the accuracy and anti-interference of audio classification.

[0184] Reference Figure 5 The embodiment of the present application further provides an audio classification system based on deep learning, the system comprising an audio data acquisition unit 100, a data feature extraction unit 200, a feature vector extraction unit 300, a feature vector fusion unit 400 and an audio classification unit 500, wherein:

[0185] The audio data acquisition unit 100 is used to acquire an audio sequence and pre-process the audio sequence to obtain an audio data set;

[0186] The data feature extraction unit 200 is used to segment the audio frames according to the audio data set, and extract the spectrum sequence, wavelet phase map, wavelet intensity map and Mel-spectrogram of the audio data set;

[0187] The feature vector extraction unit 300 is used to use multiple groups of independent multi-scale feature extraction modules to respectively extract feature vectors corresponding to the audio frame, the spectrum sequence, the wavelet phase map, the wavelet intensity map and the Mel-spectrogram, and obtain feature vectors extracted by each group of multi-scale feature extraction modules;

[0188] A feature vector fusion unit 400 is used to perform feature fusion on the feature vectors extracted by each group of multi-scale feature extraction modules to obtain a feature fusion vector;

[0189] The audio classification unit 500 is used to perform audio classification according to the feature fusion vector to obtain an audio classification result.

[0190] It should be noted that since the deep learning-based audio classification system in this embodiment and the above-mentioned deep learning-based audio classification method are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0191] Reference Figure 6 The embodiment of the present application further provides an electronic device, the electronic device comprising:

[0192] at least one memory;

[0193] at least one processor;

[0194] at least one program;

[0195] The program is stored in the memory, and the processor executes at least one program to implement the above-mentioned deep learning-based audio classification method implemented in the present disclosure.

[0196] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0197] The electronic device according to the embodiment of the present application is described in detail below.

[0198] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0199] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the audio classification method based on deep learning in the embodiment of the present disclosure.

[0200] Input / output interface 1800, used to implement information input and output;

[0201] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0202] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0203] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0204] The embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned audio classification method based on deep learning.

[0205] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0206] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0207] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present disclosure and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0208] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0209] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0210] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0211] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0212] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0213] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0215] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs. The above is a detailed description of the embodiments of the present application in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Within the scope of knowledge possessed by ordinary technicians in the relevant technical field, various changes can be made without departing from the purpose of the present application.

[0216] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A method for audio classification based on deep learning, characterized in that: The method comprises: Acquire an audio sequence, and preprocess the audio sequence to obtain an audio data set; Segmenting audio frames according to the audio data set, and extracting a spectrum sequence, a wavelet phase map, a wavelet intensity map, and a Mel-spectrogram of the audio data set; Using multiple groups of independent multi-scale feature extraction modules to respectively extract feature vectors corresponding to the audio frame, the spectrum sequence, the wavelet phase map, the wavelet intensity map, and the Mel-spectrogram, to obtain feature vectors extracted by each group of the multi-scale feature extraction modules; Performing feature fusion on the feature vectors extracted by each group of the multi-scale feature extraction modules to obtain a feature fusion vector; Audio classification is performed according to the feature fusion vector to obtain an audio classification result.

2. The audio classification method based on deep learning according to claim 1, characterized in that: The step of segmenting the audio frames according to the audio data set and extracting the spectrum sequence, wavelet phase map, wavelet intensity map and Mel-spectrogram of the audio data set includes: Performing a fast Fourier transform on the audio sequence in the audio data set to obtain a spectrum sequence; Performing frame segmentation on the audio sequence in the audio data set to obtain multiple audio frames; Performing Morlet wavelet transform on each of the audio frames to obtain wavelet coefficients; Based on the wavelet coefficients, calculating a wavelet phase map and a wavelet intensity map; Each of the audio frames is filtered using short-time Fourier transform and a Mel filter bank to obtain a Mel spectrogram.

3. The audio classification method based on deep learning according to claim 1, characterized in that: The multiple groups of independent multiscale feature extraction modules include a first multiscale feature extraction module group, a second multiscale feature extraction module group, a third multiscale feature extraction module group, a fourth multiscale feature extraction module group and a fifth multiscale feature extraction module group. The multiple groups of independent multiscale feature extraction modules are used to respectively extract the feature vectors corresponding to the audio frame, the spectrum sequence, the wavelet phase map, the wavelet intensity map and the Mel-spectrogram, and obtain the feature vectors extracted by each group of the multiscale feature extraction modules, including: Inputting the audio frame into the first multi-scale feature extraction module group for feature extraction to obtain a first feature vector, and inputting the spectrum sequence into the second multi-scale feature extraction module group for feature extraction to obtain a second feature vector, wherein the first multi-scale feature extraction module group and the second multi-scale feature extraction module group have the same structure; The wavelet phase map is input into the third multi-scale feature extraction module group for feature extraction to obtain a third feature vector, the wavelet intensity map is input into the fourth multi-scale feature extraction module group for feature extraction to obtain a fourth feature vector, and the Mel spectrum map is input into the fifth multi-scale feature extraction module group for feature extraction to obtain a fifth feature vector. The third multi-scale feature extraction module group, the fourth multi-scale feature extraction module group and the fifth multi-scale feature extraction module group have the same structure.

4. The audio classification method based on deep learning according to claim 3, characterized in that: The first multi-scale feature extraction module group includes a one-dimensional feature extraction sub-module group and a spatial attention sub-module, and the audio frame is input into the first multi-scale feature extraction module group for feature extraction to obtain a first feature vector, including: Inputting the audio frame into the one-dimensional feature extraction submodule group for feature extraction to obtain a one-dimensional feature extraction vector; Expanding the one-dimensional feature extraction vector into a high-dimensional feature vector with the same dimension as the output of the third multi-scale feature extraction module group, the fourth multi-scale feature extraction module group, and the fifth multi-scale feature extraction module; The high-dimensional feature vector is input into the spatial attention submodule to obtain a first feature vector.

5. The audio classification method based on deep learning according to claim 4, characterized in that: The step of inputting the audio frame into the one-dimensional feature extraction submodule group for feature extraction to obtain a one-dimensional feature extraction vector includes: M c (f i j )=Relu(BN(g m (f i j )))+Relu(BN(g n (f i j ))) M s (f i j+1 )=σ(AvgPool(f i j+1 )+MaxPool(f i j+1 )) Among them, BN means batch normalization, g m represents a one-dimensional convolution operation with a filter size of m, g n represents a one-dimensional convolution operation with a filter size of n, f i j It represents the input of the i-th one-dimensional feature extraction vector to the j-th one-dimensional feature extraction submodule, Relu represents the activation function, + represents concatenation along the channel dimension, represents element-by-element addition, f i j+1 represents the output feature vector, M s (f i j+1 ) represents the input vector of the j+1th one-dimensional feature extraction submodule, σ represents the sigmoid activation function, AvgPool represents average pooling, and MaxPool represents maximum pooling.

6. The audio classification method based on deep learning according to claim 3, characterized in that: The step of inputting the wavelet phase map into a third multi-scale feature extraction module group for feature extraction to obtain a third feature vector comprises: M c (f i j )=Relu(BN(g 1*s (f i j )))+Relu(BN(g s*1 (f i j )))+Relu(BN(g s*s (f i j ))) Among them, BN means batch normalization, g s*s represents a two-dimensional convolution operation with a filter size of s*s, g 1*s represents a two-dimensional convolution operation with a filter size of 1*s, g s*1 represents a two-dimensional convolution operation with a filter size of s*1, f i j It represents the input of the third eigenvector of the i-th multi-scale feature extraction module of the j-th multi-scale feature extraction module, Relu represents the activation function, + represents concatenation along the channel dimension, represents element-by-element addition, f i j+1 The output feature vector is also used as the input vector of the j+1th third multi-scale feature extraction module.

7. The audio classification method based on deep learning according to claim 1, characterized in that: The performing audio classification according to the feature fusion vector to obtain an audio classification result includes: Inputting the feature fusion vector into the MobileNet feature extraction layer to obtain a deep feature vector; Expanding the deep feature vector by global average pooling to obtain a feature vector of fixed length; The fixed-length feature vector is input into Kolmogorov-Arnold Networks for classification to obtain an audio classification result.

8. An audio classification system based on deep learning, characterized in that: The system comprises: An audio data acquisition unit, used to acquire an audio sequence and preprocess the audio sequence to obtain an audio data set; A data feature extraction unit, used to segment audio frames according to the audio data set, and extract a spectrum sequence, a wavelet phase map, a wavelet intensity map, and a Mel-spectrogram of the audio data set; A feature vector extraction unit, configured to respectively extract feature vectors corresponding to the audio frame, the spectrum sequence, the wavelet phase map, the wavelet intensity map, and the Mel-spectrogram using multiple groups of independent multi-scale feature extraction modules, and obtain feature vectors extracted by each group of the multi-scale feature extraction modules; A feature vector fusion unit, used for fusing the feature vectors extracted by each group of the multi-scale feature extraction modules to obtain a feature fusion vector; The audio classification unit is used to perform audio classification according to the feature fusion vector to obtain an audio classification result.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the deep learning-based audio classification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the deep learning-based audio classification method as described in any one of claims 1 to 7.

Images