An unknown audio event recognition algorithm based on spiking neural network

Abstract

The present application relates to an unknown audio event recognition algorithm based on a pulse neural network, which includes the following steps: constructing an audio data set and splitting it into a training set, a validation set, and a test set; preprocessing each audio data segment in the audio data set to generate a 3D log‑mel spectrogram; constructing a pulse neural network model and performing classification training; using cross entropy loss and contrast loss to jointly train the pulse neural network model; using audio data of known categories in the validation set to input into the pulse neural network and the autoencoder to obtain a threshold for distinguishing known categories from unknown audio categories; using the trained pulse neural network model to recognize the collected audio data. The present invention can effectively identify and distinguish unknown sound events without relying on pre-labeled unknown category information, improve the overall recognition accuracy of the system, and provide support for subsequent unknown sound event analysis and processing.

Description

Technical Field

[0001] The present application relates to the technical field of audio data processing, and in particular to an unknown audio event recognition algorithm based on a pulse neural network. Background Art

[0002] Sound event classification is an application of speech information retrieval, which can automatically identify and annotate specific sound events in audio signals, such as speech, music, animal calls, etc. At present, speech information retrieval technology has made significant progress in the indexing and query of audio data, and is widely used in speech recognition, music retrieval, and sound event detection. Existing audio recognition methods usually rely on deep learning models, especially convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which can effectively identify and classify known categories of sound events by training on a large amount of labeled data. However, the effectiveness of these methods depends on the richness and diversity of training data. When the algorithm encounters a sound event that does not appear in the training set, it usually misclassifies it as the closest known category. This misclassification not only affects the accuracy of recognition, but also may cause the system to become less robust when processing audio events. For example, in practical applications, the system may misidentify the abnormal sound of the engine as background noise, thereby ignoring potential fault warnings. Summary of the invention

[0003] The purpose of the present invention is to provide an unknown audio event recognition algorithm based on a pulse neural network, which can effectively identify and distinguish unknown sound events without relying on pre-labeled unknown category information, improve the overall recognition accuracy of the system, and provide support for subsequent unknown sound event analysis and processing.

[0004] The technical solution adopted by the present invention is: an unknown audio event recognition algorithm based on a pulse neural network, comprising the following steps:

[0005] S1: construct an audio dataset, and split the audio dataset into a training set, a validation set, and a test set;

[0006] S2: Preprocess each audio data segment in the audio dataset to generate a 3D log-mel spectrogram;

[0007] S3: construct a spiking neural network model, and input the 3D log-mel spectrogram corresponding to the test set into the spiking neural network model for classification training; the spiking neural network model includes a convolutional layer, multiple spiking neural units, a multi-layer perceptron MLP, a reshaping layer, and a long short-term memory network LSTM; the spiking neural unit includes two spiking units and residual convolution, and the spiking unit is composed of a convolutional layer, a batch normalization layer, and at least two small spiking neurons;

[0008] S4: Use cross entropy loss and contrast loss to jointly train the spiking neural network model;

[0009] S5: Use the known audio data in the validation set to input the pulse neural network and autoencoder to obtain the mean square error average loss obtained by autoencoder inference ,set up A threshold for distinguishing known categories from unknown audio categories; the autoencoder includes an encoder and a decoder, and the autoencoder includes an input layer, at least two hidden layers and an output layer;

[0010] S6: Use the trained pulse neural network model to identify the collected audio data, and input the probability value output by the pulse neural network model into the autoencoder, and use the autoencoder to determine whether the input data belongs to a known category or an unknown category. If the loss obtained by the autoencoder reasoning is higher than the threshold, it is determined to be an unknown category. Otherwise, the probability value output by the pulse neural network model is used to determine which known category the audio belongs to.

[0011] Furthermore, the specific steps of step S2 are:

[0012] S201: The audio data is transformed into a normal distribution through z-standardization, so that the audio data with different features have the same dimension and distribution, which is convenient for the pulse neural network model to learn. The specific formula is:

[0013] ;

[0014] in, x Represents the original audio data, represents the mean of the original audio data, represents the standard deviation of the original audio data, and z represents the original audio data after z-standardization;

[0015] S202: using a Mel filter bank to generate a log-mel spectrogram corresponding to the z-normalized original audio data, and using the log-mel spectrum feature values ​​in the log-mel spectrogram as the original static features of the audio data;

[0016] S203: For each frame of log-mel spectrum, the difference between the previous frame and the next frame is calculated to obtain the corresponding first-order time derivative, that is, the first-order Delta difference feature, and the first-order Delta difference feature is used to capture the change of the feature over time;

[0017] The specific process of solving the first-order Delta difference feature is:

[0018] Select the window size used to calculate the first-order delta difference feature N 1. For each momentt , respectively calculate the log-mel spectrum eigenvalue of each frame before and after 1~ N 1 frame weighted difference, and for the front and back n The weighted difference of the frames is As a normalization factor, the weighted difference result is normalized. The specific formula is:

[0019] ;

[0020] in, represents the first-order Delta difference feature, C t+n Indicates time frame t After n The log-Mel spectrum eigenvalue of the frame, C t-n Indicates time frame t Before n The log-Mel spectrum eigenvalue of the frame, n =1,2,……, N 1;

[0021] S204: Calculate the difference between the previous frame and the next frame for the first-order Delta differential feature to obtain the corresponding second-order Delta differential feature, and use the second-order Delta differential feature to capture the acceleration information of the feature changing over time to describe the dynamic change characteristics of the signal;

[0022] The specific process of solving the second-order Delta difference feature is:

[0023] Select the window size used to calculate the second-order delta difference feature N 2. Calculate the first-order Delta difference feature of each frame before and after 1~ N The weighted difference of 2 frames, and for the front and back n The weighted difference of the frames is As a normalization factor, the weighted difference result is normalized. The specific formula is:

[0024] ;

[0025] in, Indicates t The second-order Delta difference feature of the frame, Indicates t + n The first-order Delta difference feature of the frame, Indicates t - n The first-order Delta difference feature of the frame, n =1,2,……, N 2;

[0026] S205: The log-mel spectrum feature value obtained in step S202, the first-order Delta difference feature obtained in step S203, and the second-order Delta difference feature obtained in step S204 are stacked according to the feature dimension to obtain a 3D log-mel spectrum signal.

[0027] Furthermore, the specific steps of step S202 are:

[0028] S2021: Split the z-normalized raw audio data into overlapping frames, each containing N 3 samples, use the Hamming window to reduce spectrum leakage, and obtain the windowed signal. The specific formula is:

[0029] ;

[0030] ;

[0031] in, w [ n ] represents the Hamming window, x w [ n ] represents the signal obtained after windowing, x [ n ] represents the signal of each frame, 0≤ n ≤ N 3-1;

[0032] S2022: Use Fourier transform to discretize the windowed signal of each frame, and calculate the power spectrum of the discrete signal of each frame. The specific formula is:

[0033] ;

[0034] ;

[0035] in, X [ k ] represents the spectrum, P [ k ] represents the power spectrum of each frame of discrete signal, k represents the frequency index, j represents an imaginary unit;

[0036] S2023: Convert the frequency scale and power spectrum of each frame signal to a Mel spectrum scale, and calculate the logarithm of the Mel spectrum scale to obtain a log-Mel spectrum graph. The specific formula is:

[0037] ;

[0038] ;

[0039] ;

[0040] in, H m [ k ] indicates the filter response of the Mel filter bank, f ( m ) indicates the m The center frequency of the Mel filter, S m Represents the Mel spectrum energy calculated by the Mel filter bank.

[0041] Furthermore, the specific steps of step S3 are:

[0042] S301: Input the 3D log-mel spectrum signal into the convolution layer and reshape it according to the batch size and the number of channels;

[0043] S302: Processing the spectrum signal output by the convolution layer through the pulse neural unit, using the pulse unit to provide the previous and next time series information, the specific steps are:

[0044] S3021: Process the spectrum signal through small pulse neurons to capture the time characteristics of the spectrum signal. The specific process is as follows:

[0045] ;

[0046] ;

[0047] ;

[0048] in, U [ t ]express t The membrane potential before the moment reset, S [ t ]express t The output spike at the moment is equal to 1 when there is a spike, otherwise it is 0. T represents the time constant, which affects the decay rate of the membrane potential. V [ t -1] indicates t -1 The membrane potential after the spike is triggered, I [ t ] indicates that at time t The input current at the moment, represents the Heaviside step function, represents the threshold of membrane potential, when U [ t] When this value is exceeded, the neuron fires a spike, V [ t ]express t The membrane potential after the moment of triggering the spike, V reset represents the value to which the membrane potential is reset after a spike is triggered;

[0049] S3022: Use the convolutional layer in the pulse unit to extract the scale-invariant information of the spectrum signal, and input each frame I , set the convolution kernel size to K , perform convolution operation, the specific formula is:

[0050] ;

[0051] in, S ( u , v ) represents the convolution result matrix u Line v The value of the column, U Represents convolution K Height, V Represents convolution K The width of I ( u + a , v + b ) represents the input matrix I Middle u + a Line v + b The value of K ( a , b ) represents the convolution kernel K Middle a Line b The value of the column;

[0052] S3023: Use batch normalization layers to normalize the output of the convolutional layer in the spike unit to accelerate the training of the spike neural network model and improve its stability. The specific formula is:

[0053] ;

[0054] ;

[0055] ;

[0056] in, y i represents the output after batch normalization, represents the first learnable parameter, used to scale the normalized value, x i Indicates the batch i The input feature value of samples, represents the mean value of feature solution for each batch, represents the characteristic variance of each batch, is a constant, represents the second learnable parameter, which is used to translate the normalized value, Q Indicates the number of samples in the batch;

[0057] S3024: After the pulse unit extracts the specific features in the input spectrum signal, it fuses it with the input spectrum signal through residual convolution to improve the transmission efficiency of the information flow. The output expression of the pulse neural unit is:

[0058] Output = F ( w ) + Conv ( w );

[0059] in, Output represents the output of the spiking neural unit, w Represents the features of the input ，F Represents the output of two pulse units ,Conv Represents the convolution operation;

[0060] S303: obtaining scale-invariant features of the spectrum signal through the spiking neural unit, and processing the output feature map spiking neural unit using spatial average pooling to obtain a feature vector;

[0061] S304: The feature vector is sequentially input into a multi-layer perceptron MLP, a reshaping layer and a long short-term memory network LSTM for processing. The long short-term memory network LSTM includes an input gate , Forget Gate and output gate ,in, sig Represents the activation function sigmoid, which is used to map the input value to between 0 and 1. W i represents the weight matrix of the input gate, W f represents the weight matrix of the forget gate, W o represents the weight matrix of the output gate, h t-1 represents the hidden state at the previous moment, x t represents the input vector at the current moment, b i represents the bias vector of the input gate, b f represents the bias vector of the forget gate, b oRepresents the bias vector of the output gate; candidate states generated based on the current input and the previous hidden state , where tanh represents the hyperbolic tangent function, which is used to map the input value to between -1 and 1. W c represents the weight matrix of candidate states, b c The bias vector representing the candidate state is combined with the previous unit state through the regulation of the forget gate and the input gate c t-1 And the candidate state is updated to get the unit state ;

[0062] S305: Input the features output by the long short-term memory network LSTM into the reshaping layer and the multi-layer perceptron MLP for processing. x 1 After hidden layer calculation ,in, represents the feature vector of the hidden layer output, W 1 represents the weight matrix from the input layer to the hidden layer, b 1 represents the bias vector of the hidden layer; the hidden feature vector output by the hidden layer is sent to the output layer for operation, and the output feature vector ,in, W 2 represents the weight matrix from the hidden layer to the output layer, b 2 represents the bias vector of the output layer.

[0063] Furthermore, the calculation formulas of the cross entropy loss and contrast loss are:

[0064] ;

[0065] ;

[0066] in, L ce represents the cross entropy loss, BS Indicates the size of the batch data, x b Indicates b samples, y b Indicates b The labels of samples, f ( x b , y b ) indicates that the model is x b On the real label y b The output on D represents the total number of categories, dRepresents the category index, x b,d Indicates b The sample in d Input on the class, f ( x b,d ) represents the first d The output value of the class, represents the contrast loss, P 1 represents the set of positive samples in the batch data, x 1 represents the current sample, x 1+ represents positive samples similar to the current sample, that is, samples belonging to the same category as the current sample. represents the temperature parameter, sim represents the cosine similarity, x 1- Represents negative samples that are dissimilar to the current sample, that is, samples that belong to different categories from the current sample. N neg Represents the set of negative samples in the batch data.

[0067] Furthermore, the number of nodes in the encoder input layer is obtained by averaging the feature map output by the convolution layer of the pulse unit through pooling; when training the autoencoder, the mean square error is used to train the autoencoder, and the specific formula is:

[0068] ;

[0069] in, MSE represents the mean square error, S represents the number of samples, y ae represents the true predicted value, Represents the autoencoder prediction value.

[0070] The beneficial effects of the present invention are:

[0071] The present invention is different from the traditional method of using 2D-mel spectrogram when preprocessing data. Instead, it fuses dynamic and original static features to form three-dimensional log-mel features to capture more detailed low-frequency speech signals so as to better understand the characteristics of sound events. The present invention combines pulse neurons with residual neural networks to construct a pulse neural network model, which can effectively process time-related features in audio while extracting feature information. When training the model, cross entropy loss is used for classification, and contrast loss is combined. Contrast loss can reduce the feature distance of samples of the same category and expand the feature distance of samples of different categories, so that the pulse neural network model learns more compact features and improves the recognition accuracy of unknown categories and known categories. In the process of distinguishing unknown categories from known categories, unlike the traditional method of using the maximum probability value logit in the output layer of the neural network to determine the distribution, the present invention uses an autoencoder to set a threshold according to the reconstruction loss of a known category to avoid performance loss when the model is poorly calibrated. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A schematic diagram of the structure of a pulse neural network model in an embodiment of the present invention;

[0074] Figure 2 Schematic diagram of the structure of the autoencoder in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the limitations of the specific embodiments disclosed below.

[0076] The embodiment of the present invention proposes an unknown audio event recognition algorithm based on a pulse neural network, comprising the following steps:

[0077] S1: Construct an audio dataset and split the audio dataset into a training set, a validation set, and a test set. In an embodiment of the present invention, the DCASE2019 Subtask C Open set Acoustic Scene Classification dataset is selected as the audio dataset, which is selected from the document "A multi-device dataset for urban acoustic scene classification. In Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018)" published by Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen in 2018.

[0078] S2: Preprocess each audio data segment in the audio dataset to generate a 3D log-mel spectrogram. The specific steps are as follows:

[0079] S201: The audio data is transformed into a normal distribution through z-standardization, so that the audio data with different features have the same dimension and distribution, which is convenient for the pulse neural network model to learn. The specific formula is:

[0080] ;

[0081] in, x Represents the original audio data, represents the mean of the original audio data, represents the standard deviation of the original audio data, and z represents the original audio data after z-standardization;

[0082] S202: Use the Mel filter bank to generate a log-mel spectrogram corresponding to the z-normalized original audio data, and use the log-mel spectrum feature values ​​in the log-mel spectrogram as the original static features of the audio data. In this embodiment of the present invention, the number of Mel filter banks is set to 128, and a 1×128×320 log-mel spectrogram is generated. The specific steps are:

[0083] S2021: Split the z-normalized raw audio data into overlapping frames, each containing N 3 samples, use the Hamming window to reduce spectrum leakage, and obtain the windowed signal. The specific formula is:

[0084] ;

[0085] ;

[0086] in, w [ n ] represents the Hamming window, represents the signal obtained after windowing, x [ n ] represents the signal of each frame, 0≤ n ≤ N 3-1.

[0087] S2022: Use Fourier transform to discretize the windowed signal of each frame, and calculate the power spectrum of the discrete signal of each frame. The specific formula is:

[0088] ;

[0089] ;

[0090] in, X [ k ] represents the spectrum, P [ k ] represents the power spectrum of each frame of discrete signal, k represents the frequency index, j Represents an imaginary unit.

[0091] S2023: Convert the frequency scale and power spectrum of each frame signal to a Mel spectrum scale, and calculate the logarithm of the Mel spectrum scale to obtain a log-Mel spectrum graph. The specific formula is:

[0092] ;

[0093] ;

[0094] ;

[0095] in, H m [ k ] indicates the filter response of the Mel filter bank, f ( m ) indicates the m The center frequency of the Mel filter, S m Represents the Mel spectrum energy calculated by the Mel filter bank.

[0096] S203: For each frame of log-mel spectrum, the difference between the previous frame and the next frame is calculated to obtain the corresponding first-order time derivative, that is, the first-order Delta difference feature, and the first-order Delta difference feature is used to capture the change of the feature over time;

[0097] The specific process of solving the first-order Delta difference feature is:

[0098] Select the window size used to calculate the first-order delta difference feature N 1. For each moment t , respectively calculate the log-mel spectrum eigenvalue of each frame before and after 1~ N 1 frame weighted difference, and for the front and back n The weighted difference of the frames is As a normalization factor, the weighted difference result is normalized. The specific formula is:

[0099] ;

[0100] in, represents the first-order Delta difference feature, C t+n Indicates time frame t After n The log-Mel spectrum eigenvalue of the frame, C t-n Indicates time frame t Before n The log-Mel spectrum eigenvalue of the frame, n =1,2,……, N 1. In an embodiment of the present invention, N The value of 1 is 2.

[0101] S204: Calculate the difference between the previous frame and the next frame for the first-order Delta differential feature to obtain the corresponding second-order Delta differential feature, and use the second-order Delta differential feature to capture the acceleration information of the feature changing over time to describe the dynamic change characteristics of the signal.

[0102] The specific process of solving the second-order Delta difference feature is:

[0103] Select the window size used to calculate the second-order delta difference feature N 2. Calculate the first-order Delta difference feature of each frame before and after 1~ N The weighted difference of 2 frames, and for the front and back n The weighted difference of the frames is As a normalization factor, the weighted difference result is normalized. The specific formula is:

[0104] ;

[0105] in, Indicates t The second-order Delta difference feature of the frame, Indicatest + n The first-order Delta difference feature of the frame, Indicates t - n The first-order Delta difference feature of the frame, n =1,2,……, N 2. In the embodiment of the present invention, N The value of 2 is 2.

[0106] S205: The log-mel spectrum feature value obtained in step S202, the first-order Delta difference feature obtained in step S203, and the second-order Delta difference feature obtained in step S204 are stacked according to the feature dimension to obtain a 3D log-mel spectrum signal of 3×1×128×320.

[0107] S3: Construct a spiking neural network model, and input the 3D log-mel spectrum graph corresponding to the test set into the spiking neural network model for classification training; the spiking neural network model is as follows: Figure 1 As shown, it includes a convolutional layer, multiple spiking neural units, a multi-layer perceptron MLP, a reshaping layer and a long short-term memory network LSTM; the spiking neural unit includes two spiking units and residual convolution, and the spiking unit is composed of a convolutional layer, a batch normalization layer and at least two spiking neurons. Since the audio signal is a continuous time series, the spiking neural network can encode time information through the time interval and order of the pulses, and fully capture the timing features in the audio, such as rhythm and pitch changes. The specific training process of the spiking neural network model is:

[0108] S301: Input the 3D log-mel spectrum signal into a convolutional layer and reshape it according to the batch size and the number of channels.

[0109] S302: Processing the spectrum signal output by the convolution layer through the pulse neural unit, using the pulse unit to provide the previous and next time series information, the specific steps are:

[0110] S3021: Process the spectrum signal through small pulse neurons to capture the time characteristics of the spectrum signal. The specific process is as follows:

[0111] ;

[0112] ;

[0113] ;

[0114] in, U [ t ]express tThe membrane potential before the moment reset, S [ t ]express t The output spike at the moment is equal to 1 when there is a spike, otherwise it is 0. T represents the time constant, which affects the decay rate of the membrane potential. V [ t -1] indicates t -1 The membrane potential after the spike is triggered, I [ t ] indicates that at time t The input current at the moment, represents the Heaviside step function, represents the threshold of membrane potential, when U [ t ] When this value is exceeded, the neuron fires a spike, V [ t ]express t The membrane potential after the moment of triggering the spike, V reset Indicates the value at which the membrane potential is reset after a spike is triggered. Spiking neurons are designed from the perspective of biological plasticity. Since they transmit sparse spikes rather than continuous representations, they can ensure low energy consumption and high robustness while capturing more temporal features. In this embodiment of the present invention, a "hard reset" method is used to reset V [ t ], ensuring that after the spike is triggered S [ t ]=1, membrane potential V [ t ] will return to V reset = 0. This feature can effectively capture frequency-varying feature information such as pitch. Compared with the traditional classification network based on resnet18, the spiking neural network used in the embodiment of the present invention can effectively improve the recognition of unknown categories by 2%. The spiking neuron is designed from the perspective of biological plasticity. Since it transmits sparse spikes instead of continuous representation, it can ensure low energy consumption and high robustness while capturing more time features. In the embodiment of the present invention, the "hard reset" method is used to reset V [ t ], ensuring that after the spike is triggered S [ t ]=1, membrane potential V [ t ] will return to V reset= 0. This feature can effectively capture characteristic information with frequency changes such as pitch. Compared with the traditional classification network based on resnet18, the pulse neural network used in the embodiment of the present invention can effectively improve the recognition rate of unknown categories by 2%.

[0115] S3022: Use the convolutional layer in the pulse unit to extract the scale-invariant information of the spectrum signal, and input each frame I , set the convolution kernel size to K , perform convolution operation, the specific formula is:

[0116] ;

[0117] in, S ( u , v ) represents the convolution result matrix u Line v The value of the column, U Represents convolution K Height, V Represents convolution K The width of I ( u + a , v + b ) represents the input matrix I Middle u + a Line v + b The value of K ( a , b ) represents the convolution kernel K Middle a Line b The value of the column.

[0118] S3023: Use batch normalization layers to normalize the output of the convolutional layer in the spike unit to accelerate the training of the spike neural network model and improve its stability. The specific formula is:

[0119] ;

[0120] ;

[0121] ;

[0122] in, y i represents the output after batch normalization, represents the first learnable parameter, used to scale the normalized value, x i Indicates the batch i The input feature value of samples, represents the mean value of feature solution for each batch, represents the characteristic variance of each batch, is a constant, represents the second learnable parameter, which is used to translate the normalized value, Q Indicates the number of samples in the batch.

[0123] S3024: Figure 1 As shown, in an embodiment of the present invention, a pulse neural unit includes two pulse units and a convolution residual. After the pulse unit extracts the specific features in the input spectrum signal, it is fused with the input spectrum signal through residual convolution to improve the transmission efficiency of the information flow. The output expression of the pulse neural unit is:

[0124] Output = F ( w ) + Conv ( w );

[0125] in, Output represents the output of the spiking neural unit, w Represents the features of the input ，F Represents the output of two pulse units ,Conv Represents a convolution operation.

[0126] S303: Obtain the scale-invariant features of the spectrum signal through the pulse neural unit, and process the output feature map pulse neural unit using spatial average pooling to obtain a feature vector of size 512.

[0127] S304: The feature vector is sequentially input into a multi-layer perceptron MLP, a reshaping layer and a long short-term memory network LSTM for processing. The long short-term memory network LSTM includes an input gate , Forget Gate and output gate ,in, sig Represents the activation function sigmoid, which is used to map the input value to between 0 and 1. W i represents the weight matrix of the input gate, W f represents the weight matrix of the forget gate, W o represents the weight matrix of the output gate, h t-1 represents the hidden state at the previous moment, x t represents the input vector at the current moment, b i represents the bias vector of the input gate, b f represents the bias vector of the forget gate,b o Represents the bias vector of the output gate; candidate states generated based on the current input and the previous hidden state , where tanh represents the hyperbolic tangent function, which is used to map the input value to between -1 and 1. W c represents the weight matrix of candidate states, b c The bias vector representing the candidate state is combined with the previous unit state through the regulation of the forget gate and the input gate c t-1 And the candidate state is updated to get the unit state

[0128] S305: Input the features output by the long short-term memory network LSTM into the reshaping layer and the multi-layer perceptron MLP for processing. x 1 After hidden layer calculation ,in, represents the feature vector of the hidden layer output, W 1 represents the weight matrix from the input layer to the hidden layer, b 1 represents the bias vector of the hidden layer; the hidden feature vector output by the hidden layer is sent to the output layer for operation, and the output feature vector ,in, W 2 represents the weight matrix from the hidden layer to the output layer, b 2 represents the bias vector of the output layer. In the embodiment of the present invention, sig It is a Sigmoid activation function, and the number of output feature vectors is 10, representing the probabilities of 10 categories.

[0129] S4: Use cross entropy loss and contrast loss to jointly train the spiking neural network model; the calculation formulas of the cross entropy loss and contrast loss are:

[0130] ;

[0131] ;

[0132] in, L ce represents the cross entropy loss, BS Indicates the size of the batch data, x b Indicates b samples, y b Indicates b The labels of samples, f ( x b , yb ) indicates that the model is x b On the real label y b The output on D represents the total number of categories, d Represents the category index, x b,d Indicates b The sample in d Input on the class, f ( x b,d ) represents the first d The output value of the class, represents the contrast loss, P 1 represents the set of positive samples in the batch data, x 1 represents the current sample, x 1+ represents positive samples similar to the current sample, that is, samples belonging to the same category as the current sample. represents the temperature parameter, sim represents the cosine similarity, x 1- Represents negative samples that are dissimilar to the current sample, that is, samples that belong to different categories from the current sample. N neg Represents the set of negative samples in the batch data.

[0133] S5: Use the known audio data in the validation set to input the pulse neural network and autoencoder to obtain the mean square error average loss obtained by autoencoder inference ,set up A threshold for distinguishing known categories from unknown audio categories; the autoencoder includes an encoder and a decoder, and the autoencoder includes an input layer, at least two hidden layers and an output layer.

[0134] In an embodiment of the present invention, the number of nodes in the input layer of the encoder is obtained by averaging the feature map output by the convolution layer of the pulse unit. In an embodiment of the present invention, both the encoder and the decoder are provided with three hidden layers. After experimental testing, the number of neurons in the input layer, three hidden layers and output layer of the encoder is preferably 256, [128, 128, 64] and 8, and the number of neurons in the decoder is consistent with that of the encoder, that is, the number of neurons in the input layer, three hidden layers and output layer is 256, [128, 128, 64] and 8. When training the autoencoder, the mean square error is used to train the autoencoder, and the specific formula is:

[0135] ;

[0136] in,MSE represents the mean square error, S represents the number of samples, y ae represents the true predicted value, Represents the autoencoder prediction value.

[0137] S6: Use the trained pulse neural network model to identify the collected audio data, and input the probability value output by the pulse neural network model into the autoencoder, and use the autoencoder to determine whether the input data belongs to a known category or an unknown category. If the loss obtained by the autoencoder reasoning is higher than the threshold, it is determined to be an unknown category. Otherwise, the probability value output by the pulse neural network model is used to determine which known category the audio belongs to.

[0138] When preprocessing data, the embodiment of the present invention is different from the traditional use of 2D-mel spectrogram. Instead, it fuses dynamic and original static features to form a three-dimensional log-mel feature to capture more detailed low-frequency speech signals and the temporal context and dynamic changes in speech signals, so as to better understand the characteristics of sound events and extract the dependencies between information such as human voice and high-frequency dynamic sound range; by constructing a pulse neural network model through pulse neurons and residual neural networks, it is possible to effectively process the time correlation characteristics in audio while extracting feature information. The activity of pulse neural networks is usually sparse, which means that at any point in time, only a small number of neurons are active. This sparsity not only reduces the computational burden, but also ensures the ability to extract important features. When training the model, cross entropy loss is used for classification and contrast loss is combined. Contrastive loss can reduce the feature distance of samples of the same category and expand the feature distance of samples of different categories, so that the pulse neural network model can learn more compact features and improve the recognition accuracy of unknown and known categories. In the process of distinguishing unknown and known categories, the embodiment of the present invention is different from the traditional method of using the maximum probability value logit in the output layer of the neural network to determine the distribution. Instead, the threshold is set according to the reconstruction loss of the known category through the autoencoder to avoid performance loss when the model is poorly calibrated.

[0139] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An unknown audio event recognition algorithm based on pulse neural network, characterized in that: The steps include: S1: construct an audio dataset, and split the audio dataset into a training set, a validation set, and a test set; S2: Preprocess each audio data segment in the audio dataset to generate a 3D log-mel spectrogram; S3: construct a spiking neural network model, and input the 3D log-mel spectrogram corresponding to the training set into the spiking neural network model for classification training; the spiking neural network model includes a convolutional layer, multiple spiking neural units, a multi-layer perceptron MLP, a reshaping layer, and a long short-term memory network LSTM; the spiking neural unit includes two spiking units and residual convolution, and the spiking unit is composed of a convolutional layer, a batch normalization layer, and at least two small spiking neurons; S4: Use cross entropy loss and contrast loss to jointly train the spiking neural network model; S5: Use the known audio data in the validation set to input the pulse neural network and autoencoder to obtain the mean square error average loss obtained by autoencoder inference ,set up A threshold for distinguishing known categories from unknown audio categories; the autoencoder comprises an encoder and a decoder, the encoder comprises an input layer, at least two hidden layers and an output layer, and the decoder comprises an input layer, at least two hidden layers and an output layer; S6: Use the trained pulse neural network model to identify the collected audio data, and input the probability value output by the pulse neural network model into the autoencoder, and use the autoencoder to determine whether the input data belongs to a known category or an unknown category. If the loss obtained by the autoencoder reasoning is higher than the threshold, it is determined to be an unknown category. Otherwise, the probability value output by the pulse neural network model is used to determine which known category the audio belongs to.

2. The unknown audio event recognition algorithm based on pulse neural network according to claim 1 is characterized in that: The specific steps of step S2 are: S201: The audio data is transformed into a normal distribution through z-standardization, so that the audio data with different features have the same dimension and distribution, which is convenient for the pulse neural network model to learn. The specific formula is: ； in, x Represents the original audio data, represents the mean of the original audio data, represents the standard deviation of the original audio data, and z represents the original audio data after z-standardization; S202: using a Mel filter bank to generate a log-mel spectrogram corresponding to the z-normalized original audio data, and using the log-mel spectrum feature values ​​in the log-mel spectrogram as the original static features of the audio data; S203: For each frame of log-mel spectrum, the difference between the previous frame and the next frame is calculated to obtain the corresponding first-order time derivative, that is, the first-order Delta difference feature, and the first-order Delta difference feature is used to capture the change of the feature over time; The specific process of solving the first-order Delta difference feature is: Select the window size used to calculate the first-order delta difference feature N 1. For each moment t , respectively calculate the log-mel spectrum eigenvalue of each frame before and after 1~ N 1 frame weighted difference, and for the front and back n The weighted difference of the frames is As a normalization factor, the weighted difference result is normalized. The specific formula is: ； in, represents the first-order Delta difference feature, C t+n Indicates time frame t After n The log-Mel spectrum eigenvalue of the frame, C t-n Indicates time frame t Before n The log-Mel spectrum eigenvalue of the frame, n =1,2,……, N 1; S204: Calculate the difference between the previous frame and the next frame for the first-order Delta differential feature to obtain the corresponding second-order Delta differential feature, and use the second-order Delta differential feature to capture the acceleration information of the feature changing over time to describe the dynamic change characteristics of the signal; The specific process of solving the second-order Delta difference feature is: Select the window size used to calculate the second-order delta difference feature N 2. Calculate the first-order Delta difference feature of each frame before and after 1~ N The weighted difference of 2 frames, and for the front and back n The weighted difference of the frames is As a normalization factor, the weighted difference result is normalized. The specific formula is: ； in, Indicates t The second-order Delta difference feature of the frame, Indicates t + n The first-order Delta difference feature of the frame, Indicates t - n The first-order Delta difference feature of the frame, n =1,2,……, N 2; S205: stacking the log-mel spectrum feature value obtained in step S202, the first-order Delta difference feature obtained in step S203, and the second-order Delta difference feature obtained in step S204 according to the feature dimension to obtain a 3D log-mel spectrum signal.

3. The unknown audio event recognition algorithm based on pulse neural network according to claim 2 is characterized in that: The specific steps of step S202 are: S2021: Split the z-normalized raw audio data into overlapping frames, each containing N 3 samples, use the Hamming window to reduce spectrum leakage, and obtain the windowed signal. The specific formula is: ； ； in, w [ n ] represents the Hamming window, x w [ n ] represents the signal obtained after windowing, x [ n ] represents the signal of each frame, 0 ≤ n ≤ N 3 -1; S2022: Use Fourier transform to discretize the windowed signal of each frame, and calculate the power spectrum of the discrete signal of each frame. The specific formula is: ； ； in, X [ k ] represents the spectrum, P [ k ] represents the power spectrum of each frame of discrete signal, k represents the frequency index, j represents an imaginary unit; S2023: Convert the frequency scale and power spectrum of each frame signal to a Mel spectrum scale, and calculate the logarithm of the Mel spectrum scale to obtain a log-Mel spectrum graph. The specific formula is: ； ； ； in, H m [ k ] indicates the filter response of the Mel filter bank, f ( m ) indicates the m The center frequency of the Mel filter, S m Represents the Mel spectrum energy calculated by the Mel filter bank.

4. The unknown audio event recognition algorithm based on pulse neural network according to claim 3 is characterized in that: The specific steps of step S3 are: S301: Input the 3D log-mel spectrum signal into the convolution layer and reshape it according to the batch size and the number of channels; S302: Processing the spectrum signal output by the convolution layer through the pulse neural unit, using the pulse unit to provide the previous and next time series information, the specific steps are: S3021: Process the spectrum signal through small pulse neurons to capture the time characteristics of the spectrum signal. The specific process is as follows: ； ； ； in, U [ t ]express t The membrane potential before the moment reset, S [ t ]express t The output spike at the moment is equal to 1 when there is a spike, otherwise it is 0. T represents the time constant, which affects the decay rate of the membrane potential. V [ t -1] indicates t -1 The membrane potential after the spike is triggered, I [ t ] indicates that at time t The input current at the moment, represents the Heaviside step function, represents the threshold of membrane potential, when U [ t ] When this value is exceeded, the neuron fires a spike, V [ t ]express t The membrane potential after the moment of triggering the spike, V reset represents the value to which the membrane potential is reset after a spike is triggered; S3022: Use the convolutional layer in the pulse unit to extract the scale-invariant information of the spectrum signal, and input each frame I , set the convolution kernel size to K , perform convolution operation, the specific formula is: ； in, S ( u , v ) represents the convolution result matrix u Line v The value of the column, U Represents convolution K Height, V Represents convolution K The width of I ( u+a , v+b ) represents the input matrix I Middle u+a Line v+b The value of K ( a , b ) represents the convolution kernel K Middle a Line b The value of the column; S3023: Use batch normalization layers to normalize the output of the convolutional layer in the spike unit to accelerate the training of the spike neural network model and improve its stability. The specific formula is: ； ； ； in, y i represents the output after batch normalization, represents the first learnable parameter, used to scale the normalized value, x i Indicates the batch i The input feature value of samples, represents the mean value of feature solution for each batch, represents the characteristic variance of each batch, is a constant, represents the second learnable parameter, which is used to translate the normalized value, Q Indicates the number of samples in the batch; S3024: After the pulse unit extracts the specific features in the input spectrum signal, it fuses it with the input spectrum signal through residual convolution to improve the transmission efficiency of the information flow. The output expression of the pulse neural unit is: Output = F ( w ) + Conv ( w )； in, Output represents the output of the spiking neural unit, w Represents the features of the input ，F Represents the output of two pulse units , Conv Represents the convolution operation; S303: obtaining scale-invariant features of the spectrum signal through the spiking neural unit, and processing the output feature map spiking neural unit using spatial average pooling to obtain a feature vector; S304: The feature vector is sequentially input into a multi-layer perceptron MLP, a reshaping layer and a long short-term memory network LSTM for processing. The long short-term memory network LSTM includes an input gate , Forget Gate and output gate ,in, sig Represents the activation function sigmoid, which is used to map the input value to between 0 and 1. W i represents the weight matrix of the input gate, W f represents the weight matrix of the forget gate, W o represents the weight matrix of the output gate, h t-1 represents the hidden state at the previous moment, x t represents the input vector at the current moment, b i represents the bias vector of the input gate, b f represents the bias vector of the forget gate, b o Represents the bias vector of the output gate; candidate states generated based on the current input and the previous hidden state , where tanh represents the hyperbolic tangent function, which is used to map the input value to between -1 and 1. W c represents the weight matrix of candidate states, b c The bias vector representing the candidate state is combined with the previous unit state through the regulation of the forget gate and the input gate c t-1 And the candidate state is updated to get the unit state ; S305: Input the features output by the long short-term memory network LSTM into the reshaping layer and the multi-layer perceptron MLP for processing. x 1 After hidden layer calculation ,in, represents the feature vector of the hidden layer output, W 1 represents the weight matrix from the input layer to the hidden layer, b 1 represents the bias vector of the hidden layer; the hidden feature vector output by the hidden layer is sent to the output layer for operation, and the output feature vector ,in, W 2 represents the weight matrix from the hidden layer to the output layer, b 2 represents the bias vector of the output layer.

5. The unknown audio event recognition algorithm based on pulse neural network according to claim 4 is characterized in that: The calculation formulas of the cross entropy loss and contrast loss are: ； ； in, L ce represents the cross entropy loss, BS Indicates the size of the batch data, x b Indicates b samples, y b Indicates b The labels of samples, f ( x b , y b ) indicates that the model is x b On the real label y b The output on D represents the total number of categories, d Represents the category index, x b,d Indicates b The sample in d Input on the class, f ( x b,d ) represents the first d The output value of the class, represents the contrast loss, P 1 represents the set of positive samples in the batch data, x 1 Represents the current sample, x 1+ represents positive samples similar to the current sample, that is, samples belonging to the same category as the current sample. represents the temperature parameter, sim represents the cosine similarity, x 1- Represents negative samples that are dissimilar to the current sample, that is, samples that belong to different categories from the current sample. N neg Represents the set of negative samples in the batch data.

6. The unknown audio event recognition algorithm based on pulse neural network according to claim 5 is characterized in that: The number of nodes in the encoder input layer is obtained by averaging the feature map output by the convolution layer of the pulse unit; when training the autoencoder, the mean square error is used to train the autoencoder, and the specific formula is: ； in, MSE represents the mean square error, S represents the number of samples, y ae represents the true predicted value, Represents the autoencoder prediction value.

Images