An Automatic Classification Method for Indoor Ambient Sound Based on a Lightweight ECAPA-TDNN Neural Network

Abstract

This invention discloses an automatic classification method for indoor ambient sound based on a lightweight ECAPA-TDNN neural network, belonging to the field of ambient sound classification technology. The method includes the following steps: First, augmenting the initial ambient sound data through time masking, frequency masking, and audio data shifting. Second, extracting Mel spectrogram features from the indoor ambient sound data through pre-emphasis, short-time Fourier transform, and Mel filtering; dividing the obtained ambient sound Mel spectrogram features into training and testing sets. Third, constructing an ECAPA-TDNN network model, optimizing the neuron parameters of the ECAPA-TDNN network using the training set; and then using the trained neural network for classifying the ambient sound test set. Compared to ambient sound classification methods using traditional training classification frameworks, the method proposed in this invention has higher accuracy, wider applicability, and consumes fewer computational resources.

Description

Technical Field

[0001] This invention relates to the field of environmental sound classification technology, specifically to an accurate classification technology for indoor complex scene environmental sounds based on a lightweight time-delay neural network. Background Technology

[0002] Sound, as an important information carrier, is often used to assist in environmental perception and information decision-making due to its advantages such as ease of collection and lack of limitations imposed by light and spatial angles. It is widely applied in the field of smart home security detection. Smart devices receive indoor environmental audio signals and, through environmental sound recognition technology, can detect and determine living events, such as the cries of a baby, the sound of an elderly person falling, and knocking on a door. They can also quickly perceive changes in the environment, such as footsteps approaching from a distance, and make timely and appropriate responses and decisions. Therefore, developing high-precision environmental sound recognition technology is crucial. Currently, deep learning-based environmental sound recognition methods are a mainstream approach, but they still face two major challenges: First, the types of environmental sounds are complex, the original features of different environmental sound signals have low discriminative power, and the data volume is small, making subsequent classification and labeling tasks difficult; second, most neural network parameters related to environmental sound classification are large in scale and computationally complex, making them unsuitable for terminal deployment. Based on the above analysis, this invention proposes an automatic indoor environmental sound classification method based on a lightweight ECAPA-TDNN neural network to address the problems mentioned above. Summary of the Invention

[0003] The present invention aims to provide an automatic classification method for indoor ambient sound based on a lightweight ECAPA-TDNN neural network, so as to solve any of the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: an automatic classification method for indoor ambient sound based on a lightweight ECAPA-TDNN neural network, comprising the following steps:

[0005] S1. Audio preprocessing: Convert the multi-channel ambient sound signal into a fixed number of channels and perform sampling and normalized length processing;

[0006] S2, Data Augmentation: Time and frequency masking is applied to the ambient sound signal in S1, and the audio is shifted to augment the ambient sound data;

[0007] S3. Feature Extraction: Based on augmented ambient sound data, the data is pre-emphasized and subjected to short-time Fourier transform (STFT), followed by Mel filtering to output Mel spectrum feature vectors; cepstral average subtraction is used to perform secondary random normalization on the Mel spectrum feature vectors to establish and load the feature dataset;

[0008] S4. Construct the ECAPA-TDNN classifier: Input the ambient sound feature dataset extracted in S3 into the convolutional layer (increase dimensionality), and use Kaiming normalization and zeroing of bias to increase the non-linear relationship between the layers of the network; after batch normalization, it enters the convolutional layer for iteration. When the number of channels in the convolutional layer reaches the threshold, it outputs to the next layer.

[0009] S5. Construct the compression-activation (SE) module and pooling and linear layer module of the ECAPA-TDNN classifier: perform average pooling, convolution and non-linear transformation on the output of S4;

[0010] S6. Training phase: Input the labels and extracted environmental audio Mel spectrogram feature data into the network to optimize the structure and parameters of the ECAPA-TDNN network model. After repeated iterations to reach the maximum number of empirical settings, the training of the network is completed.

[0011] S7. Testing Phase: The trained ECAPA-TDNN classifier is used to classify the environmental sound features in the test sample dataset to obtain the test classification results.

[0012] Preferably, the SpecAugment method is applied to the log Mel spectrum obtained in step S3. This strategy randomly masks some frames in the time domain and randomly masks some channels in the frequency domain.

[0013] Preferably, the features extracted in step S3 are Mel spectrograms, and the feature vectors of the Mel spectrograms are subjected to secondary random normalization using cepstral average subtraction.

[0014] Preferably, the number of times the ECAPA-TDNN network classification model is trained in step S6 is 450-500.

[0015] Preferably, when the ECAPA-TDNN network classification model is trained 450-500 times, the loss rate and accuracy gradually converge, and the loss function is defined as follows:

[0016]

[0017] Compared with the prior art, the beneficial effects of the present invention are:

[0018] This invention extracts Mel spectrogram features from different types of environmental sound signals (e.g., common environmental sounds such as television sounds, cooking sounds, running water sounds, alarm sounds, children crying sounds, curtain drawing sounds, etc.); it then builds and loads training and datasets based on the extracted Mel spectrogram features; it constructs an ECAPA-TDNN network model, trains the ECAPA-TDNN network classification model, and completes classification and recognition. By extracting Mel spectrogram features from environmental audio and using them to train the ECAPA-TDNN classification network for classification, this invention achieves higher accuracy and consumes fewer computational resources compared to environmental sound classification methods that use traditional training frameworks. Attached Figure Description

[0019] Figure 1 This is a system flowchart of an embodiment of the automatic classification method for indoor ambient sound based on a lightweight ECAPA-TDNN neural network of the present invention;

[0020] Figure 2 This is a time-domain waveform diagram of an ambient sound signal from an embodiment of the present invention, which is an automatic classification method for indoor ambient sound based on a lightweight ECAPA-TDNN neural network.

[0021] Figure 3 This is a time-domain waveform diagram of a preprocessed ambient sound signal from an embodiment of the present invention, which is based on a lightweight ECAPA-TDNN neural network for automatic classification of indoor ambient sound.

[0022] Figure 4 This is a spectrum diagram of an ambient sound signal with negative decibels removed, representing an embodiment of an automatic classification method for indoor ambient sound based on a lightweight ECAPA-TDNN neural network according to the present invention.

[0023] Figure 5 The Mel spectrogram of the ambient sound signal is shown in Embodiment 1 of the present invention, which is an automatic classification method for indoor ambient sound based on a lightweight ECAPA-TDNN neural network.

[0024] Figure 6 This is a diagram illustrating the Mel spectrogram feature extraction process in an embodiment of an automatic indoor ambient sound classification method based on a lightweight ECAPA-TDNN neural network according to the present invention.

[0025] Figure 7 This is an example of the ECAPA-TDNN framework diagram of an automatic indoor ambient sound classification method based on a lightweight ECAPA-TDNN neural network according to the present invention.

[0026] Figure 8 This is a loss function value change / convergence graph of an embodiment of the automatic indoor ambient sound classification method based on a lightweight ECAPA-TDNN neural network of the present invention.

[0027] Figure 9 This is a graph showing the F1 score variation of an indoor ambient sound classification and recognition method based on a lightweight ECAPA-TDNN neural network, according to the present invention.

[0028] Figure 10 This is a graph showing the variation in the recognition accuracy of various types of ambient sounds in an automatic indoor ambient sound classification method based on a lightweight ECAPA-TDNN neural network, as presented in this invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Please see Figure 1-10 This invention provides an automatic classification method for indoor ambient sound based on a lightweight ECAPA-TDNN neural network, comprising the following steps:

[0031] S1. Audio Preprocessing:

[0032] S11. Convert the multi-channel original ambient sound signal into a dual-channel signal.

[0033] S12, Set the audio sampling rate to 22050Hz.

[0034] S13. Standardize audio length, extract all audio signals longer than 10 seconds, and pad the excess parts.

[0035] S2, Data Augmentation:

[0036] S21. Calculate the average frequency aver_Fre. For data with the number of channels within the range, randomly mask it by adding horizontal bars. For other data, directly return to generate a Mel chart.

[0037] S22. Translate audio data, providing a random number r, a translation limit s, and the translation amount is defined as r × s × audio length.

[0038] S23. Generate Mel graph, set Mel filter bank to 64, short-time Fourier transform window length to 1024, and no frame shift.

[0039] S24. Convert the power scale to a decibel scale and set the minimum negative cutoff value to 80.

[0040] S25. Labeling: Based on the 2022 iFLYTEK AI Developer Competition Low Complexity Home Ambient Sound Challenge dataset used in the experiment, ambient sounds can be divided into six categories: television sound, cooking sound, running water sound, alarm sound, child crying sound, and curtain drawing sound. After feature extraction, Mel-spectral features of the ambient sounds are obtained, and the ambient sound data in the training set is labeled. The target ambient sound type is set into different groups as the classification setting. This experiment classifies ambient sounds by type, with six categories: television sound (label value 0); cooking sound (label value 1); running water sound (label value 2, 3); alarm sound (label value 4); and child crying sound (label value 5).

[0041] S3. The cepstral average subtraction method is used to perform a second random normalization on the feature vectors of the Mel spectrum to construct training and test sets.

[0042] S4. Construct the classifier module for ECAPA-TDNN:

[0043] The environmental sound signal features extracted by S41 and S2 first enter the input layer of ECAPA-TDNN, and then enter the convolutional layer (increased dimension). The convolutional kernel size is set to (3,3), the padding size is (1,1), and the stride is (2,2). Kaiming normalization and bias zeroing are used to increase the nonlinear relationship between the network layers. After batch normalization, it enters the input layer for iteration. At this time, the number of channels is doubled. If the number of channels does not reach 256 after the above changes, the output is used as input again and enters the input layer. When the number of channels reaches 256, it is output to the next module.

[0044] S5. Construct the compression and activation (SE) module of ECAPA-TDNN, and perform average pooling, convolution, and non-linear transformation on the output of s4:

[0045] S6. Training the ECAPA-TDNN network classification model: Input the processed Mel spectrogram features of various environmental sound signals into the network to train the ECAPA-TDNN network model with labels. Use the Adam optimizer and set the initial learning rate to 0.001. Use the linear annealing strategy to dynamically change the learning rate.

[0046] S7. Classification Test: Using the trained ECAPA-TDNN network structure and parameters, the environmental sound features in the test sample dataset are classified to obtain the classification results and identify various types of environmental sounds.

[0047] In step S2, frequency and time masking are applied to the ambient sound signal from S1 to enhance its characteristics.

[0048] The ECAPA-TDNN network model in step S5 includes a compression and activation module, which updates the number of audio feature channels from 256 to 16, and then restores them through convolutional layers, thereby improving the model's sensitivity to channel features.

[0049] In step S6, the ECAPA-TDNN network classification model is trained 450-500 times, with experimental parameters continuously adjusted and optimized to obtain the most suitable parameters. During the experiment, it was found that the model's loss rate and accuracy gradually converged around 400 iterations; therefore, the number of training iterations was adjusted to 450. Each training iteration uses the model parameters obtained from the previous epoch for the next training iteration, retraining the framework. The difference in loss rate and accuracy remained small, hence the choice of 450 iterations.

[0050] When the training times of the ECAPA-TDNN network classification model are 450-500 times, the loss rate and accuracy gradually tend to converge. The loss function is defined as: only 0 or 1, and when it is 0, the result is 0, and only when it is 1 will there be a classification result and the environmental sound features will be tested and classified.

[0051] Step 1, Audio Preprocessing Stage: The ambient sound data downloaded from the 2022 iFLYTEK AI Developer Competition Low Complexity Home Ambient Sound Challenge dataset will be processed by channel conversion, sampling, and truncation to construct the ambient sound data heard in a real indoor ambient sound scenario;

[0052] Step 2, Data Augmentation and Labeling Stage: Frequency masking and time masking are applied to the preprocessed ambient sound data to enhance the characteristics of the ambient sound signal. Strategies such as audio data shifting are used to augment the amount of ambient sound data. Mel spectrogram features are extracted from the processed ambient sound signal, and labels are applied to construct the dataset.

[0053] Step 3, Ambient Sound Classification Process: Input the training set from the dataset obtained in Step 2 into the ECAPA-TDNN classification network for classification, and finally classify six different ambient sounds, thus training a classification framework.

[0054] Example 1:

[0055] The following section demonstrates the feasibility and superiority of the algorithm by classifying different ambient sounds using the designed ambient sound classification model based on this strategy.

[0056] First, we processed the data. The dataset used was from the 2022 iFLYTEK AI Developer Competition Low Complexity Home Ambient Sound Challenge. The ambient sounds were categorized into six types: TV sound, cooking sound, running water sound, alarm sound, child crying sound, and curtain drawing sound. We first converted the multi-channel raw ambient sound signal into a dual-channel signal, setting the audio sampling rate to 22050Hz. Then, we normalized the audio length. For audio data of different durations, we extracted all audio signals longer than 10 seconds, padded out the excess portions, and then continued the experiment.

[0057] This experiment continues to use six processed ambient sounds for data augmentation. The main focus of this experiment is on audio collected in indoor home settings, effectively classified, and then used for ambient sound judgment. Because the amount of ambient sound data under these conditions is limited, affecting the subsequent network training effect, data augmentation and label processing are required for the original ambient sound dataset. First, the average frequency (aver_Fre) of this type of ambient sound is calculated. Data with channels within the specified range are randomly masked by adding horizontal bars; other data are directly returned to generate Mel spectrograms. Next, the audio data is shifted, providing a random number and a shift limit defined as where is the audio length. The power scale is converted to a decibel scale, and a minimum negative cutoff value of 80 is set. The augmented dataset is then created and loaded.

[0058] Taking television audio as an example, an ECAPA-TDNN classifier module is constructed. The extracted ambient sound signal features of the television audio first enter the input layer of the ECAPA-TDNN, and then enter the convolutional layer (increased dimensionality). The convolutional kernel size is set to (3,3), the padding size is (1,1), and the stride is (2,2). Kaiming normalization and bias zeroing are used to increase the nonlinear relationship between the layers of the network. After batch normalization, it enters the input layer for iteration. At this time, the number of channels is doubled. If the number of channels does not reach 256 after the above changes, the output is used as input again and enters the input layer. When the number of channels reaches 256, it is output to the next module.

[0059] Let be the output tensor of the Mel spectrogram features of each TV audio environment after passing through the classifier module, and let be the tensor after compression and activation modules. First, it enters the average pooling layer of this module to be transformed into a feature tensor of size , where H and W are the length and width of the TV audio features, respectively. Then it enters convolutional layer A, with a kernel size of 1, padding of 0, and stride of 0. Based on experience, the reduction ratio is set to 16, resulting in 16 channels. After passing through the ReLU activation function, it is input into convolutional layer B (activation module), where the number of channels is converted to 256 (reduction operation).

[0060] Based on the Mel spectrogram features of the TV ambient sound, the output tensor x after passing through the classifier module, and the tensor of x after passing through the compression and excitation modules, let , the updated x be input into the average pooling layer to update this audio tensor to its size, and then enter the linear layer. In the linear layer, a dropout operation is performed. It passes through 5 linear layers, and the dropout probability increases from 0.1 to 0.5 each time. The output is the average output of the sum of the 5 linear layers.

[0061] Training the ECAPA-TDNN network classification model: The processed Mel spectrogram features of the television ambient sound signal are input into the network for label training. In each iteration, the input is defined as the input minus its mean and divided by its standard deviation. After passing through the model's classification layer, the loss function value is calculated (using the cross-entropy loss function). The Adam optimizer is used with an initial learning rate of 0.001 to accelerate gradient descent. The scheduler uses a linear annealing strategy to dynamically change the learning rate. When the number of iterations reaches 500, the network is considered to have completed the training process for the television ambient sound.

[0062] Any content not described in detail in this specification is prior art known to those skilled in the art.

[0063] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An automatic classification method for indoor ambient sound based on a lightweight ECAPA-TDNN neural network, characterized in that, Includes the following steps: S1. Audio preprocessing: Convert the multi-channel ambient sound signal into a fixed number of channels and perform sampling and normalized length processing; S2, Data Augmentation: Time and frequency masking are applied to the ambient sound signal in S1, and the audio is shifted to augment the ambient sound data; S3. Feature Extraction: Based on augmented ambient sound data, the data is pre-emphasized and subjected to short-time Fourier transform, followed by Mel filtering to output Mel spectrum feature vectors; cepstral average subtraction is used to perform secondary random normalization on the Mel spectrum feature vectors to establish and load the feature dataset; S4. Input the ambient sound feature dataset extracted in S3 into the convolutional layer, and use Kaiming normalization and zeroing of bias to increase the non-linear relationship between the layers of the network; after batch normalization, it enters the convolutional layer for iteration. When the number of channels in the convolutional layer reaches the threshold, it outputs to the next layer. S5. Construct the compression-activation module and pooling and linear layer module of the ECAPA-TDNN classifier: Perform average pooling, convolution and non-linear transformation on the output of S4. S6. Training phase: Combine the output of S5 with the label information for training. After the maximum number of iterations, complete the optimization training of the ECAPA-TDNN classifier structure and parameters. S7. Testing Phase: The trained ECAPA-TDNN classifier is used to classify the environmental sound features in the test sample dataset to obtain the test classification results.

2. The automatic indoor ambient sound classification method based on a lightweight ECAPA-TDNN neural network according to claim 1, characterized in that: The SpecAugment method is applied to the log Mel spectrum obtained in step S3. This strategy randomly masks some frames in the time domain and randomly masks some channels in the frequency domain.

3. The automatic indoor ambient sound classification method based on a lightweight ECAPA-TDNN neural network according to claim 1, characterized in that: The features extracted in step S3 are Mel spectrograms, and the feature vectors of the Mel spectrograms are subjected to secondary random normalization using cepstral average subtraction.

4. The automatic indoor ambient sound classification method based on a lightweight ECAPA-TDNN neural network according to claim 1, characterized in that: The training iterations for optimizing the ECAPA-TDNN classifier structure and parameters in step S6 are between 450 and 500.

5. The automatic indoor ambient sound classification method based on a lightweight ECAPA-TDNN neural network according to claim 4, characterized in that: The loss function for training the ECAPA-TDNN classifier is defined as follows: ， When the number of training iterations is between 450 and 500, the loss rate and accuracy gradually converge.

Images