A low-computing method for automatic sound scene classification of cochlear implants

Abstract

This invention discloses a low-computing-power cochlear implant automatic sound scene classification method, which includes simulation and DSP implementation. The simulation includes database construction, sample MFCC feature vector extraction, feature value normalization, model training, model prediction, and result output; the DSP implementation includes DSP scene prediction, model prediction and derivation, and comprehensive decision-making and output. The present invention facilitates feature extraction, has a low computational load, and is easy to implement on low-computing computing devices such as cochlear implants, with good results.

Description

Technical Field

[0001] The present invention belongs to the field of signal processing, and in particular relates to a low-computing-power cochlear implant automatic sound scene classification method. Background Art

[0002] Scene sound information contains rich acoustic information, which can be used for scene classification, event determination and target positioning. Acoustic Scene Classification (ASC) extracts acoustic features from the sound information of the environment, and determines which scene classification the environment of the sound source belongs to based on these features. ASC technology plays an important role in human-computer interaction, smart home, security monitoring, robotics and other fields, and has therefore become a hot research field. Generally speaking, the ASC system mainly consists of two parts - a sound feature extraction module and a classification model determination module. The former extracts sound features from the sound signal. This process can be automatically extracted through artificial feature engineering or artificial neural networks. The latter inputs the sound features into the classification model for scene determination. The different automatic sound scene recognition solutions currently available have more or less differences in the implementation of these two modules. Sound features extracted through artificial feature engineering primarily include time-domain features such as short-time zero-crossing rate, short-time average amplitude, and short-time average energy; frequency-domain features such as subband energy fraction, spectral entropy, and spectral flux; time-frequency domain features such as spectrograms and mel-spectrograms; and cepstral domain features such as mel-frequency cepstral coefficients (MFCCs). These features can be selected and combined as needed. Currently, the most popular features used in ASC are generally based on mel-frequency metrics, such as mel-spectrograms and log-mel-spectrograms. This is primarily because these features have lower feature dimensionality than raw spectrograms, and can produce approximate spectral feature representations and sufficient inter-class variance for discrimination by various classification models. Traditional classification models include support vector machines (SVMs), random forests, decision trees, Gaussian mixture models, and K-nearest neighbor classifiers. Newer technologies utilize artificial neural network models, including multilayer perceptrons (MLPs), convolutional neural networks (CNNs), recurrent neural networks (GRUs, RNNs, LSTMs), and residual networks (ResNets). Currently, the most commonly used classification model in ASC is a CNN-based deep learning approach. This approach typically combines time-frequency features such as spectrograms, mel-spectrograms, or constant Q-variation, achieving good prediction results. However, it requires a high level of computation and is relatively complex.

[0003] Cochlear implants convert sound signals into electrical impulses that directly stimulate the patient's residual auditory nerve fibers, restoring hearing. They are suitable for patients with severe or profound sensorineural hearing loss. Currently, most cochlear implants enable users to hear clearly, understand, and communicate in quiet environments. However, maintaining this performance in the complex and changing soundscapes of everyday life remains a challenge, and has become a key indicator of the effectiveness and practicality of cochlear implants. Excellent hearing performance relies on excellent sound processing algorithms, but no single algorithm is universally applicable, making it crucial to select the appropriate processing strategy for each scenario. Before the advent of intelligent sound management systems, the common practice was to program several sound programs within the external device's processor, such as "Daily" and "Noise," each with a specific sound processing strategy. Users would then manually select a program using a remote assistant or buttons on the external device. However, for most users, manually selecting a program is cumbersome, difficult, or even impossible. Even experienced users often struggle to select the most appropriate program for the given situation. The emergence of sound scene classification technology has solved this problem. Combined with automated sound processing strategy configuration, it can actively select the appropriate listening program based on the identified sound scene, allowing users to obtain the best listening effect that can be achieved technically.

[0004] Sound scene classification is fundamental to intelligent hearing assistance systems. It directly influences the configuration of sound processing strategies and, in turn, impacts the user's hearing experience. Identification errors can potentially impair speech intelligibility or listening comfort, making it crucial. It's worth noting that the real-time and low-power requirements of cochlear implants limit the application of more complex sound scene classification algorithms. For example, the commonly used ASC method, extracting spectrogram or mel-spectrogram features and using deep CNN models to classify scenes, is computationally expensive. While potentially effective, it's difficult to implement in cochlear implant processors. In cochlear implants, ASC technology only occupies a small portion of DSP computing resources. Its purpose is to enable the subsequent intelligent configuration of various sound processing strategies (such as single-microphone noise reduction, dual-microphone noise reduction, and wind noise identification and removal) for scene adaptation. Therefore, the number of operations required by the classification algorithm should be limited. Furthermore, to mitigate the negative impact of frequent program switching on the user's listening experience, a comprehensive decision-making module is typically added after the model determines the scene classification, ensuring smooth transitions between scenes. The purpose of the present invention is to realize an automatic sound scene classification method with low computational complexity but relatively high prediction accuracy, which can be implemented in low-computing-power terminal devices such as cochlear implants. Based on this purpose, we have done a lot of research on feature extraction and screening as well as classification model selection in the early stage. According to our research experience, under the same model training data set, the importance of feature extraction and screening is higher than the importance of model screening, that is, if a suitable feature combination is screened out, the choice of classifier will not make much difference in terms of the best prediction accuracy that can be obtained in the end. The final choice of classification model is mainly based on the computational complexity and the difficulty of implementation on DSP. Our research results show that simply using MFCC features combined with a dual hidden layer fully connected network model can achieve a relatively optimal state in terms of the balance between accuracy and computational complexity. The MFCC based on Mel frequency cepstral coefficients is widely used in the processing of speech and music audio. Its analysis is based on the human auditory mechanism, in order to simulate the human auditory processing process and obtain better acoustic features. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a low-computing-power cochlear implant automatic sound scene classification method, which has low computational complexity and relatively high prediction accuracy and can be implemented in low-computing-power terminal devices such as cochlear implants.

[0006] To achieve the above object, the present invention provides a low-computing-power cochlear implant automatic sound scene classification method, comprising the following steps:

[0007] S10, simulation;

[0008] S20, DSP implementation;

[0009] Wherein, S10 includes the following steps:

[0010] S101, database construction: All audio downloaded from the Internet, collected in the field, and synthesized later are processed into a mono .wav format with a 16kHz sampling rate. Each sample is cut into 1s duration. The samples in the database are classified into at least speech, noise, noisy speech, and music. 80% of each category is used as a training set and 20% as a test set.

[0011] S102, sample MFCC feature vector extraction;

[0012] S103, eigenvalue normalization, normalizing the MFCC feature vectors of all samples in the training set to obtain a normalized sample feature set for model training. At the same time, the normalized scale is saved for normalization of the feature vectors of the test set samples;

[0013] S104, model training. The model used is a fully connected neural network with two hidden layers. 20% of the training set is selected as a validation set to adjust model parameters and evaluate whether overfitting occurs during model training and requires early stopping of training. The model with the highest prediction accuracy on the validation set is retained as the preferred model for subsequent prediction performance evaluation on the test set.

[0014] S105, using the preferred model to make predictions on the test set, and collecting the prediction results as an evaluation of the prediction performance of the model;

[0015] S106, the performance of the model was evaluated by 10 five-fold cross validations;

[0016] S20 includes the following steps:

[0017] S201, DSP scene prediction implementation, sampling frequency is 16kHz, frame length is 16ms, frame shift is 16ms, the number of Mel filters is set to 10, and the scene recognition result is output every 1s;

[0018] S202, prediction and derivation of the model, including the output of hidden layer 1: out1 = relu (input * W1 + B1);

[0019] Output of hidden layer 2: out2 = relu(out1*W2+B2);

[0020] Output layer: out3 = softmax(out2*W3+B3);

[0021] Classification determination: class = argmax(out3);

[0022] Where * represents the multiplication of corresponding elements; W1, W2, W3 represent the weights of each layer; B1, B2, B3 represent the bias of each layer; out1, out2, out3 represent the output of each layer after the activation function; class is the scene classification result determined by the model; relu is the linear rectification function; softmax is the normalized exponential function;

[0023] S203, making a comprehensive decision and outputting the scene classification prediction result of the smoothing model.

[0024] Preferably, the S102, extracting the sample MFCC feature vector, specifically includes the following steps:

[0025] S1021, pre-emphasize the sample signal to compensate for the loss of high-frequency components, and pass the time domain signal through a first-order high-pass filter H(z) = 1-μz -1 , μ is a constant of 0.9-1;

[0026] S1022, dividing the signal into frames. Since the speech signal is a quasi-steady-state signal, its short-term invariance is utilized to divide the sample signal into frames for processing;

[0027] S1023, windowing the framed signal, that is, multiplying each frame signal by a window function such as a Hamming window or a Henning window;

[0028] S1024, perform Fourier transform on the windowed time domain signal to convert it into the frequency domain. The transformation formula is as follows:

[0029]

[0030] Where x(n) is the time domain signal, X(k) is the frequency domain signal, n represents the sampling point in the discrete signal, and N represents the number of FFT points;

[0031] S1025, calculating the spectral line energy of the frequency domain signal, and calculating the energy spectrum or amplitude spectrum;

[0032] S1026, passing the spectral line energy through the Mel triangular filter bank to obtain the energy output of the Mel filter, which is equivalent to multiplying the energy spectrum E(i,k) of each frame with the frequency domain response function Hm(k) of the Mel filter and adding them together to obtain the energy output of the mth filter;

[0033] S1027, calculating the logarithm log[S(i,m)] of the output energy of each filter in the filter bank;

[0034] S1028, performing discrete cosine transform to obtain MFCC coefficients for each frame;

[0035] S1029. Calculate the mean and variance of the MFCC coefficients for all frames of the sample to obtain the MFCC feature vector of the sample for input to the model.

[0036] Preferably, in S1022, the frame length is set to 16 ms and the frame shift is set to 16 ms. A 1-second sample is divided into 62 frames for processing.

[0037] Preferably, in S1025, energy spectrum calculation is used. The modulus value of the signal after discrete Fourier transform is squared to obtain the energy spectrum |X(k)| 2 .

[0038] Preferably, the formula in S1026 includes:

[0039]

[0040] where i represents the i-th frame, k represents the k-th spectral line in the frequency domain, m represents the m-th Mel filter, and N is the number of FFT points;

[0041] The frequency response of a triangular filter bank with M band-pass filters is:

[0042]

[0043] 0 ≤ m < M. In the Mel frequency domain, these filters have equal bandwidth; f(m) is the center frequency of the filter, defined as:

[0044]

[0045] where f l and f h are the lowest and highest frequencies within the filter frequency range respectively; N is the number of FFT points; Fs is the sampling frequency; is the inverse function of F mel , and the formula is:

[0046]

[0047] where b is the perceived frequency in Mel units.

[0048] Preferably, in S1028, discrete cosine transform is performed to obtain the MFCC coefficients for each frame. The specific formula is:

[0049]

[0050] where i represents the i-th frame; n represents the n-th spectral line; m is the m-th filter, with a total of M.

[0051] Preferably, in S1029, the mean and variance of the MFCC coefficients of all frames of the sample are calculated to obtain the MFCC feature vector of the sample, which is used as the input of the model.

[0052] Features=[m(1),std(1),m(2),std(2),…,m(i),std(i),…,m(10),std(10)],

[0053] Among them, m(i) represents the mean of the i-th MFCC coefficients of all frames of the sample, and std(i) represents the standard deviation or mean absolute deviation of the i-th MFCC coefficients of all frames of the sample.

[0054] Preferably, the normalization in S103 adopts min-max normalization or z-score normalization.

[0055] Preferably, the step S201 specifically includes the following steps:

[0056] S2011, extract MFCC coefficients for every 16ms of audio input, output feature vector size of 1*10, and put the result into feature matrix F;

[0057] S2012, repeat S2011, and put the MFCC coefficient values extracted from each subsequent 16ms audio frame into F until the size of F is 10*62;

[0058] S2013, calculate the mean and variance of each MFCC coefficient in F, and output the MFCC sample feature vector with a size of 1*20;

[0059] S2014, using the normalized scale generated in the training set, normalize the MFCC sample feature vectors in S2013 and clear the feature matrix F;

[0060] S2015: Input the normalized feature vector in S2014 into the trained optimal model, and let the model perform scene category discrimination.

[0061] Preferably, the step S203 specifically includes the following steps:

[0062] S2031: Set a 1*4 integral matrix, initialized to 0, to represent the cumulative integral of each scene classification, namely, speech, noise, noisy speech, and music;

[0063] S2032: Assuming the model discrimination result is noise, the score of the noise class is increased by 1. If the current model prediction classification result is different from the previous model prediction result, the scores of the remaining classes are reduced by 1; if the current model prediction classification result is the same as the previous model prediction result, the scores of the remaining classes are reduced by 2; if the score of a class exceeds 15, the score of the class is set to 15; if the score of a class is less than 0, the score of the class is set to 0;

[0064] S2033, if the integral value corresponding to a category in the integral matrix reaches 10, the output result after comprehensive decision-making is that category;

[0065] S2034: To avoid unstable scene predictions that result in the cumulative integral value failing to reach the set threshold and no scene result being output for more than 15 seconds, if the cumulative threshold for scene judgment is not reached 15 seconds after the system is turned on, the scene that appears most frequently in the last 10 seconds will be used as the result output.

[0066] The beneficial effects of the present invention are: solving the problems in the prior art of inconvenient feature extraction, large amount of calculation, and difficulty in practical application in cochlear implant processors. The present invention simply uses MFCC coefficients (only 10 filters are used) as input for model judgment, and there is no need to extract other time domain, frequency domain and other features, which greatly simplifies the feature extraction process and reduces the amount of calculation required for feature extraction in DSP; the selected fully connected network model only contains two hidden layers of 10 nodes, the model architecture is simple, the number of parameters is small, and the amount of derivation calculation is low; in summary, both the feature extraction module and the model recognition module are easy and suitable for implementation on the cochlear implant DSP, and under such low computing power, the classification accuracy is still relatively good. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention provides the following drawings for illustration:

[0068] Figure 1 This is a flowchart of a method for automatic sound scene classification for a cochlear implant with low computing power according to an embodiment of the present invention;

[0069] Figure 2 This is a flowchart of step S102 of a method for automatic sound scene classification for a cochlear implant with low computing power according to an embodiment of the present invention;

[0070] Figure 3 This is a Mel filter response diagram in S102 of a low-computing-power cochlear implant automatic sound scene classification method according to an embodiment of the present invention;

[0071] Figure 4 Schematic diagram of the network architecture in S104 of a low-computing-power cochlear implant automatic sound scene classification method according to an embodiment of the present invention;

[0072] Figure 5 This is a flowchart of step S20 of a low-computing-power cochlear implant automatic sound scene classification method according to an embodiment of the present invention;

[0073] Figure 6 This is a schematic diagram of a confusion matrix in S106 of a method for automatic sound scene classification for a cochlear implant with low computing power according to an embodiment of the present invention;

[0074] Figure 7 Schematic diagram of the S20 experimental environment of a low-computing-power cochlear implant automatic sound scene classification method according to an embodiment of the present invention;

[0075] Figure 8 Schematic diagram of scene prediction accuracy and switching times of a low-computing-power cochlear implant automatic sound scene classification method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0077] See also Figure 1 , which shows a low-computing-power cochlear implant automatic sound scene classification method according to an embodiment of the present invention, comprising the following steps:

[0078] S10, simulation;

[0079] S20, DSP implementation;

[0080] Wherein, S10 includes the following steps:

[0081] S101. Database Construction: The audio samples used for sound scene recognition model training primarily come from online downloads and field collection. The former involves downloading the required sound samples from publicly available sound sample libraries online, while the latter involves recording audio from real-world scenes using recording equipment. The collected sound samples are then manually classified and screened. Noisy speech samples come from field collection and post-synthesis (with a signal-to-noise ratio of -10 to 15 dB). Due to the varying sources, the audio samples are stored in different formats and sampling rates. All samples are preprocessed into a mono 16kHz .wav format. Each sample is cropped to a 1s duration, and samples with low sound pressure levels are deleted using an appropriate threshold. Table 1 shows the number of samples per category in the database. To balance the number of categories during model training, 39,000 samples (10.8 hours) from each category are randomly selected to construct the modeling dataset, with 80% of each category used as the training set and 20% used as the test set.

[0082] Table 1 Classification of sample libraries and corresponding sample numbers

[0083]

[0084] Note: 39,000 samples were randomly selected for each classification for subsequent model building.

[0085] S102, sample MFCC feature vector extraction process:

[0086] See also Figure 2 , specifically including the following steps:

[0087] S1021, pre-emphasize the sample signal to compensate for the loss of high-frequency components, which is equivalent to passing the time domain signal through a first-order high-pass filter: H(z) = 1-μz -1 , μ is a constant, and its value is generally in the range of 0.9-1. In the specific embodiment of the present invention, it is 0.98;

[0088] S1022: Frame the signal. Since the speech signal is a quasi-steady-state signal, its short-term invariance is utilized to divide the sample signal into frames for processing. In the present invention, the frame length is set to 16ms and the frame shift is set to 16ms. Therefore, a 1s sample can be divided into 62 frames for processing.

[0089] S1023, windowing the framed signal, that is, multiplying each frame signal by a window function such as a Hamming window or a Henning window to reduce spectrum leakage. In a specific embodiment of the present invention, a Henning window is used;

[0090] S1024, perform Fourier transform on the windowed time domain signal to convert it into the frequency domain. The transformation formula is as follows:

[0091]

[0092] Wherein, x(n) is a time domain signal, X(k) is a frequency domain signal, n represents a sampling point in a discrete signal, and N represents the number of FFT points. In the specific embodiment of the present invention, N=256;

[0093] S1025, calculate the spectral line energy of the frequency domain signal, which can be an energy spectrum or an amplitude spectrum. In the specific embodiment of the present invention, energy spectrum calculation is adopted. Specifically, the modulus value of the signal after discrete Fourier transformation is obtained and then squared to obtain the energy spectrum |X(k)| 2 ;

[0094] S1026, the spectral line energy is passed through the Mel triangular filter bank to obtain the energy output of the Mel filter. This is equivalent to multiplying the energy spectrum E(i,k) of each frame with the frequency domain response function Hm(k) of the Mel filter and adding them together to obtain the energy output of the mth filter. The specific formula is as follows:

[0095]

[0096] Where, \(i\) represents the \(i\)-th frame, \(k\) represents the \(k\)-th spectral line in the frequency domain, \(m\) represents the \(m\)-th Mel filter, and \(N\) is the number of FFT points.

[0097] The frequency response of a triangular filter bank with \(M\) band-pass filters is as follows:

[0098]

[0099] For \(0\leq m < M\), in the Mel frequency domain, these filters have equal bandwidths.

[0100] The center frequency of the filter, denoted as \(f(m)\), is defined as follows:

[0101]

[0102] Where, \(f\) l and \(f\) h are the lowest and highest frequencies within the filter frequency range respectively; \(N\) is the number of FFT points; \(F_s\) is the sampling frequency; is the inverse function of \(F\) mel with the formula as follows

[0103]

[0104] Where, \(b\) is the perceived frequency in Mel units.

[0105] In a specific embodiment of the present invention, to minimize the computational amount as much as possible, \(M\) is taken as \(10\). The responses of each Mel filter are shown in Figure 3 ;

[0106] S1027, calculate the logarithm \(\log[S(i,m)]\) of the output energy of each filter in the filter bank;

[0107] S1028, perform a discrete cosine transform (DCT) to obtain the MFCC coefficients for each frame. The specific formula is as follows:

[0108]

[0109] Where, \(i\) represents the \(i\)-th frame, \(n\) represents the \(n\)-th spectral line, and \(m\) is the \(m\)-th filter (a total of \(M\) filters);

[0110] S1029, calculate the mean and variance of the MFCC coefficients for all frames of the sample to obtain the MFCC feature vector of the sample, which is used as the input of the model. The specific implementation in the present invention is as follows:

[0111] Features=[m(1),std(1),m(2),std(2),…,m(i),std(i),…,m(10),std(10)], where m(i) represents the mean of the i-th MFCC coefficient of all frames of the sample, and std(i) represents the standard deviation of the i-th MFCC coefficient of all frames of the sample. In the present invention, the number of Mel filters is set to 10, and the length of the output feature vector is 20. The standard deviation here can also be replaced by the mean absolute deviation. The order of arrangement of the sample feature values can be changed, such as putting the mean of each coefficient first and then the standard deviation of each coefficient.

[0112] S103, eigenvalue normalization: normalize the MFCC feature vectors of all samples in the training set to obtain a normalized sample feature set for model training. At the same time, save the normalized scale for normalization of the sample feature vectors in the test set. The normalization method can adopt min-max normalization, z-score normalization, etc. The present invention adopts z-score normalization. The normalized scale includes the mean vector MN and variance vector STD of each feature in the sample feature set, wherein MN = [M(1), M(2), ..., M(i), ... M(20)], M(i) represents the mean of the sample training feature set of the i-th MFCC coefficient; STD = [STD(1), STD(2), ..., STD(i), ... STD(20)], STD(i) represents the standard deviation of the sample training feature set of the i-th MFCC coefficient. If the number of Mel filters set is not equal to 10, or the arrangement of the MFCC feature vectors of the sample is different from that of the present invention, the corresponding normalization scale also needs to be adjusted.

[0113] S104, model training: The model used in the present invention is a fully connected neural network with two hidden layers. The network architecture is shown in Figure 4 The input layer contains 20 nodes, corresponding to 20 normalized input features. The two hidden layers each contain 10 nodes, and the activation function is relu. The output layer contains 4 nodes, corresponding to 4 classification scenarios (speech, noise, noisy speech, and music). The activation function softmax is used to map the output to the range of 0-1. The label corresponding to the node with the largest probability value is the scene classification result determined by the model. Of course, the network model architecture can be changed or even replaced with other classifiers such as SVM and random forest. Under the premise of extracting the same features, different classifiers are selected for training, and the overall prediction results are not much different. During model training, 20% of the training set is selected as the validation set to adjust the model parameters and evaluate whether overfitting occurs during the model training process and whether training needs to be stopped early. The model with the highest prediction accuracy on the validation set is retained as the preferred model for subsequent prediction performance evaluation on the test set.

[0114] Prediction performance evaluation of the preferred model: The preferred model is used to make predictions on the test set, and the prediction results are counted as an evaluation of the model's prediction performance. S105: The sample input features of the test set need to be normalized according to the normalization scale (MN and STD vector) described above. The specific method in the present invention is as follows:

[0115] Norm_features = (features - MN) / STD, where features is the original MFCC feature vector of the test set samples, and Norm_features is the normalized feature vector. If other normalization methods such as min-max are used, make corresponding adjustments.

[0116] S106, the performance of the model is evaluated by 10 five-fold cross-validation methods. On the larger data set described in Table 1, the data set is divided into 5 parts. 4 of them are used as training sets and the remaining 1 is used as a test set. The recognition results on the test set are recorded. The above results are repeated 10 times, and the average of the 50 prediction results is taken as the evaluation of the model prediction performance. Figure 6 This is an example of a confusion matrix for a particular test result of the model on the test set. The overall recognition accuracy (mean ± standard deviation) of the statistical model is 0.90 ± 0.005, with speech at 0.98 ± 0.003, noise at 0.92 ± 0.008, music at 0.81 ± 0.01, and noisy speech at 0.88 ± 0.02. This shows that in the implementation of the present invention, the model has the highest prediction accuracy for speech, followed by noise, with both predictions exceeding 90%, followed by noisy speech, and finally music.

[0117] See also Figure 5 ,S20,DSP implementation phase includes the following steps:

[0118] S201, DSP scene prediction system implementation: The implementation in DSP takes the implementation of the present invention as an example, with a sampling frequency of 16 kHz, a frame length of 16 ms, a frame shift of 16 ms, the number of Mel filters set to 10, and a scene recognition result output every 1 second. Specifically, the following steps are included:

[0119] S2011, extract MFCC coefficients for every 16ms of audio input, output feature vector size of 1*10, and put the result into feature matrix F;

[0120] S2012, repeat S2011, and put the MFCC coefficient values extracted from each subsequent 16ms audio frame into F until the size of F is 10*62;

[0121] S2013, calculate the mean and variance of each MFCC coefficient in F, and output the MFCC sample feature vector with a size of 1*20;

[0122] S2014, using the normalized scale generated in the training set, normalize the MFCC sample feature vectors in S2013 and clear the feature matrix F;

[0123] In step S2015, the normalized feature vector from step S2014 is input into the trained optimal model, and the model is used to identify the scene category.

[0124] S202, the prediction derivation process of the model includes:

[0125] Output of hidden layer 1: out1 = relu(input*W1+B1);

[0126] Output of hidden layer 2: out2 = relu(out1*W2+B2);

[0127] Output layer: out3 = softmax(out2*W3+B3);

[0128] Classification determination: class = argmax(out3);

[0129] In the above formula, * represents the multiplication of corresponding elements; W1, W2, W3 represent the weights of each layer; B1, B2, B3 represent the bias of each layer; out1, out2, out3 represent the output of each layer after the activation function; class is the scene classification result determined by the model; relu is the linear rectification function, and the function expression is relu(z i )=max(0,z i ); softmax is a normalized exponential function, and its expression is

[0130]

[0131] Where C is the number of scene classification categories, i is the i-th output node; argmax is the subscript position of the maximum value in the input;

[0132] Then input the result in S2015 into the comprehensive decision-making module, combine the historical judgment results of the historical model and the manual rule setting to obtain the final scene judgment result.

[0133] S203, comprehensive decision: To reduce the adverse listening experience caused by frequent scene switching, a scene comprehensive decision module is added after the model prediction. This module exists to smooth the scene classification prediction results of the model and can be implemented in different ways.

[0134] The present invention is specifically implemented as follows:

[0135] S2031: Set a 1*4 integral matrix, initialized to 0, to represent the cumulative integral of each scene classification (speech, noisy speech, noise, music);

[0136] S2032: Assuming the model discrimination result is noise, the score of the noise class is increased by 1, and the scores of the other classes are reduced by 1 (if the current model prediction classification result is different from the previous model prediction result) or 2 (if the current model prediction classification result is the same as the previous model prediction result). If the score of a class exceeds 15, the score of the class is set to 15. If the score of a class is less than 0, the score of the class is set to 0.

[0137] S2033, if the integral value corresponding to a category in the integral matrix reaches 10, the output result after comprehensive decision-making is that category;

[0138] S2034: To avoid the situation where unstable scene prediction causes the cumulative integral value to fail to reach the set threshold and the scene result is not output for a long time (more than 15 seconds), 15 seconds after the real-time scene recognition system is turned on, if the cumulative threshold for scene judgment is not reached, the scene that appears most frequently in the last 10 seconds is used as the result output.

[0139] Real-time prediction performance evaluation of the DSP scene classification system of the present invention (experimental environment demonstration) Figure 7 ) After the entire classification method including comprehensive decision-making is implemented on the cochlear implant's external speech processor DSP, the real-time scene recognition results are displayed using the mobile phone app interface. In order to evaluate the prediction accuracy and stability, we randomly selected some test sound samples that were not in the training library, spliced them into a 5-minute file, and there were 7-10 test files for each scene. Use a large speaker (Edifier R1600TIII) to play the test file, use the cochlear implant microphone to receive the audio (the cochlear implant is 0.5m away from the large speaker), adjust the volume knob of the large speaker so that the sound pressure level at the cochlear implant microphone reaches 75dB (when playing white noise), and manually record the audio playback time and the scene switched to each time the mobile phone changes during the entire file test. Analyze the scene recognition results, count the number of scene switches and prediction accuracy for each file, and estimate the prediction accuracy by the proportion of time the interface displays the corresponding test file scene type. The results show that the recognition accuracy in each scenario is 0.99±0.04 (n=9) for speech, 0.96±0.09 (n=8) for noise, 0.94±0.12 (n=8) for noisy speech, 0.94±0.15 (n=7) for pure music, and 0.92±0.13 (n=6) for music with human voice ( Figure 8, left); the number of scene switching within 5 minutes was 1.1±0.3 for speech, 1.4±0.7 for noise, 1.3±0.5 for speech with noise, 1.4±0.8 for pure music, and 1.3±0.5 for music with vocals ( Figure 8 , right). It can be seen that the sound scene classification system of the present invention has high scene recognition accuracy in practical applications, and the scene switching is also relatively stable.

[0140] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A low-computing-power cochlear implant automatic sound scene classification method, characterized in that: The following steps are involved: S10, simulation; S20, DSP implementation; Wherein, S10 includes the following steps: S101, database construction: All audio downloaded from the Internet, collected in the field, and synthesized later are processed into a mono .wav format with a 16kHz sampling rate. Each sample is cut into 1s duration. The samples in the database are classified into at least speech, noise, noisy speech, and music. 80% of each category is used as a training set and 20% as a test set. S102, sample MFCC feature vector extraction; S103, eigenvalue normalization, normalizing the MFCC feature vectors of all samples in the training set to obtain a normalized sample feature set for model training. At the same time, the normalized scale is saved for normalization of the feature vectors of the test set samples; S104, model training. The model used is a fully connected neural network with two hidden layers. 20% of the training set is selected as a validation set to adjust model parameters and evaluate whether overfitting occurs during model training and requires early stopping of training. The model with the highest prediction accuracy on the validation set is retained as the preferred model for subsequent prediction performance evaluation on the test set. S105, using the preferred model to make predictions on the test set, and collecting the prediction results as an evaluation of the prediction performance of the model; S106, the performance of the model was evaluated by 10 five-fold cross validations; S20 includes the following steps: S201, DSP scene prediction implementation, sampling frequency is 16kHz, frame length is 16ms, frame shift is 16ms, the number of Mel filters is set to 10, the mean and variance of each Mel filter output are counted every 1s as the input feature vector of the model, and the scene recognition result is output once; S202, prediction and derivation of the model, including the output of hidden layer 1: out1 = relu (input * W1 + B1); Output of hidden layer 2: out2 = relu(out1*W2+B2); Output layer: out3 = softmax(out2*W3+B3); Classification determination: class = argmax(out3); Where * represents the multiplication of corresponding elements; W1, W2, W3 represent the weights of each layer; B1, B2, B3 represent the bias of each layer; out1, out2, out3 represent the output of each layer after the activation function; class is the scene classification result determined by the model; relu is the linear rectification function; softmax is the normalized exponential function; S203, making comprehensive decisions and outputting the scene classification prediction results of the smoothing model; The S203 specifically includes the following steps: S2031: Set a 1*4 integral matrix, initialized to 0, to represent the cumulative integral of each scene classification, namely, speech, noise, noisy speech, and music; S2032. Assume that the model discrimination result is noise. Then, increment the integral of the noise class by 1. If the current model's predicted classification result is different from the previous model's predicted result, decrement the integrals of the other classes by 1. If the current model's predicted classification result is the same as the previous model's predicted result, decrement the integrals of the other classes by 2. If the integral of a certain classification exceeds 15, set the integral of that class equal to 15. If the integral of a certain classification is less than 0, set the integral of that class to 0. S2033. If the integral value corresponding to a certain classification in the integral matrix reaches 10, the output result after comprehensive decision-making is that class. S2034. To avoid the situation where the cumulative integral value fails to reach the set threshold due to unstable scene prediction and no scene result is output for more than 15s, after 15s of startup, if the cumulative threshold for scene determination is not reached, use the scene that appears most frequently in the recent 10s as the result output.

2. The low-computing-power cochlear implant automatic sound scene classification method according to claim 1, characterized in that: Regarding S102 mentioned above, the extraction of sample MFCC feature vectors specifically includes the following steps: S1021, pre-emphasize the sample signal to compensate for the loss of high-frequency components, and pass the time domain signal through a first-order high-pass filter H(z) = 1-μz -1 , μ is a constant of 0.9-1; S1022. Frame the signal. Since the speech signal is a quasi-steady signal, utilize its short-time invariance and process the sample signal frame by frame. S1023. Window the framed signal, that is, multiply each frame of the signal by a window function such as a Hamming window or a Hanning window. S1024. Perform Fourier transform on the windowed time-domain signal to convert it to the frequency domain. The transformation formula is as follows: where x(n) is the time-domain signal, X(k) is the frequency-domain signal, n represents the sampling points in the discrete signal, and N represents the number of FFT points. S1025. Calculate the spectral line energy of the frequency-domain signal, and calculate the energy spectrum or amplitude spectrum. S1026. Pass the spectral line energy through the Mel triangular filter bank to obtain the energy output after passing through the Mel filter, which is equivalent to multiplying and adding the energy spectrum E(i,k) of each frame with the frequency-domain response function Hm(k) of the Mel filter to obtain the energy output of the mth filter. S1027. Calculate the logarithm log[S(i,m)] of the energy output of each filter in the filter bank. S1028. Perform discrete cosine transform to obtain the MFCC coefficients of each frame. S1029. Calculate the mean and variance of the MFCC coefficients of all frames of the sample to obtain the MFCC feature vector of the sample for input to the model.

3. The low-computing-power cochlear implant automatic sound scene classification method according to claim 2, characterized in that: In S1022, the frame length is set to 16ms, the frame shift is set to 16ms, and 1s of the sample is divided into 62 frames for processing.

4. The low-computing-power cochlear implant automatic sound scene classification method according to claim 2, characterized in that: In the S1025, energy spectrum calculation is performed to obtain the modulus of the discrete Fourier transformed signal and then square it to obtain the energy spectrum |X(k)| 2 .

5. The low-computing-power cochlear implant automatic sound scene classification method according to claim 2, characterized in that: The formula in S1026 includes: where i represents the ith frame, k represents the kth spectral line in the frequency domain, m represents the mth Mel filter, and N is the number of FFT points. The frequency response of a triangular filter bank with M band-pass filters is: 0 ≤ m < M. In the Mel frequency domain, these filters have equal bandwidth; f(m) is the center frequency of the filter, defined as: Among them, f l and f h are the lowest and highest frequencies in the filter frequency range respectively; N is the number of FFT points; Fs is the sampling frequency; It's F mel The inverse function of is: where b is the perceptual frequency in Mel units.

6. The low-computing-power cochlear implant automatic sound scene classification method according to claim 2, characterized in that: Regarding S1028, perform discrete cosine transform to obtain the MFCC coefficients of each frame. The specific formula is: where i represents the ith frame; n represents the nth spectral line; m is the mth filter, with a total of M.

7. The low-computing-power cochlear implant automatic sound scene classification method according to claim 2, characterized in that: The S1029 calculates the mean and variance of the MFCC coefficients of all frames of the sample to obtain the MFCC feature vector of the sample, which is used as the input of the model. Features=[m(1),std(1),m(2),std(2),…,m(i),std(i),…,m(10),std(10)], Among them, m(i) represents the mean of the i-th MFCC coefficients of all frames of the sample, and std(i) represents the standard deviation or mean absolute deviation of the i-th MFCC coefficients of all frames of the sample.

8. The low-computing-power cochlear implant automatic sound scene classification method according to claim 1, characterized in that: The normalization in S103 adopts min-max standardization or z-score standardization.

9. The low-computing-power cochlear implant automatic sound scene classification method according to claim 1, characterized in that: The S201 specifically includes the following steps: S2011, extract MFCC coefficients for every 16ms of audio input, output feature vector size of 1*10, and put the result into feature matrix F; S2012, repeat S2011, and put the MFCC coefficient values extracted from each subsequent 16ms audio frame into F until the size of F is 10*62; S2013, calculate the mean and variance of each MFCC coefficient in F, and output the MFCC sample feature vector with a size of 1*20; S2014, using the normalized scale generated in the training set, normalize the MFCC sample feature vectors in S2013 and clear the feature matrix F; S2015: Input the normalized feature vector in S2014 into the trained optimal model, and let the model perform scene category discrimination.

Images