Audio classification method under voice interference

Abstract

The invention discloses an audio classification method in a voice interference environment, and aims to solve the problem that the audio classification accuracy is reduced in a complex acoustic scene. According to the method, a self-supervised audio spectrogram Transform (SSAST) is adopted as a feature extraction backbone network, firstly, large-scale unlabeled audio data is utilized to carry out pre-training on an SSAST model, and time-frequency spectrogram feature representation with high robustness is learned; in a model fine tuning stage, a labeled training audio sample is input into a pre-trained SSAST network, and depth feature representation of an intermediate layer is extracted. In order to fully excavate discrimination information of time sequence features, depth features are input into Bi-LSTM and Bi-GRU in parallel for fine tuning, and feature splicing is performed on output results according to time dimensions to obtain fusion features. And the fusion features are compressed into single vectors through time pooling, and category probability distribution is obtained through a full connection layer. And finally, by combining sample category labels and taking minimization of cross entropy loss as a target, optimizing full-connection layer classifier parameters based on feature fusion to obtain an optimal classifier. In the test stage, audio samples to be classified are input into the trained system, category probability distribution is output, and the classification performance is verified through the real labels of the test set.

Description

Technical Field

[0001] The present invention belongs to the field of audio classification, and in particular relates to an audio classification method under speech interference. Background Art

[0002] As an important branch of artificial intelligence, audio classification technology has been widely used in fields such as environmental sound recognition, voice assistants, intelligent monitoring, and industrial detection. For example, in smart home scenarios, the system needs to accurately identify doorbells, crying babies, or smoke alarms to trigger corresponding actions; in the field of public safety, audio classification can detect abnormal sounds (such as breaking glass, cries for help) in real time to assist in security decision-making. However, actual acoustic scenes often have complex speech interference, such as background human voices, equipment noise, or multi-source mixed signals. These interferences can cause the accuracy of audio classification to drop significantly. Taking the environmental sound classification dataset (such as ESC-50) as an example, it covers multiple categories such as animal sounds, natural sounds, and human non-speech sounds. However, in real applications, the target sound is often masked by background speech, and traditional methods have difficulty in effectively distinguishing the target from the interference signal.

[0003] In recent years, although deep learning methods (such as convolutional neural networks (CNN) and recurrent neural networks (RNN)) have made progress in automatic feature extraction, significant problems still exist. For example, CNN is good at local feature extraction, but its ability to model global time series is insufficient; although RNN can process sequence data, it suffers from gradient vanishing and low computational efficiency. In addition, existing deep models rely heavily on large-scale labeled data, while the cost of labeling is high in actual scenarios, and the model's generalization ability is insufficient under speech interference. Although self-supervised learning can alleviate the problem of data dependence, its application in the audio field is still immature. In particular, how to improve feature robustness through effective strategies (such as occlusion masks) in the pre-training stage still needs further exploration. Summary of the Invention

[0004] In response to the problems in the existing technology that traditional manual feature extraction and shallow models have poor recognition effects in complex acoustic environments, and the problem that deep learning methods have a strong dependence on large-scale labeled data and high computational overhead, the present invention proposes an audio classification method in a speech interference environment.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A method for audio classification in a speech-interference environment first establishes a mixed audio database containing several speech samples, each with a corresponding category label. The mixed audio database is then divided into a training set consisting of samples of known categories and a test set consisting of samples of unknown categories. Each sample has a known and unique category label. The method includes the following steps:

[0007] Step 1: Each paragraph sample in the training sample set is processed separately, and the input original audio waveform signal x∈R L Converted into a time-spectrum graph S∈R by short-time Fourier transform (STFT) F×T , where F is the number of frequency dimensions, which is 512, and T is the number of time frames. Divide S into several fixed-size spectrogram blocks S∈R f×t ,total blocks, where f is the frequency dimension of each spectrogram block, the value is 16, and t is the number of time frames of each spectrogram block, the value is 16

[0008] Step 2: Randomly generate R square occlusion mask areas for each spectrum block in the training sample set to form a mask set M = {M j}, whose mask area is controlled by the dynamic clustering factor α. The total mask area satisfies:

[0009]

[0010] Among them, Area(M j ) is the jth occlusion mask area M j The area of ​​the input spectrum block S is 1, Area(S) is the area of the entire input spectrum block S, and its value is 256. α∈(0,1) is the dynamic clustering factor, and R satisfies

[0011] The masked spectrogram is fed into the SSAST model and the following loss function is jointly optimized:

[0012]

[0013] in, It is a discriminative contrast loss that uses contrastive learning to construct positive and negative pairs to measure feature discrimination; is the generative reconstruction loss, which measures the quality of occlusion restoration; λ1=1, λ2=10 are weighting coefficients.

[0014] Step 3: After SSAST completes pre-training, extract its Transformer encoder Lth s The intermediate feature representation H∈R T×D , where T = 256, D = 768, T is the time step, and D is the feature dimension.

[0015] The feature sequence H is input into Bi-LSTM and Bi-GRU in parallel to obtain

[0016] H BiLSTM =BiLSTM(H),H BiGRU =BiGRU(H)

[0017] The features of the two are spliced according to the time dimension to obtain the fusion feature representation:

[0018] H fusion =(H BiLSTM ||H BiGRU )∈R T×2D (3)

[0019] The operator “||” represents vector concatenation, which means expanding and concatenating two column vectors into a single column vector according to their dimensions.

[0020] Step 4: Fusion feature H fusion Compressed into a single vector via temporal pooling:

[0021] h∈Pool(H fusion ) (4)

[0022] Then the category probability distribution is obtained through the fully connected layer:

[0023]

[0024] Where W h ∈R C×d is the weight matrix of the fully connected layer, b∈R C is the bias vector, C is the number of categories, and d is the dimension of the feature vector after pooling, which is 768. Through the training sample set of known categories, the cross entropy loss function is used to optimize the classifier parameters, and the optimal virtual classifier is obtained through iterative training.

[0025] Step 5: Extract the spectral features of the test sample set in the same way as steps 1 to 3, input them into the trained optimal virtual classifier, obtain the category probability distribution, and evaluate the classification performance in combination with the real labels.

[0026] Furthermore, the feature sequence processing process in step 3 is as follows:

[0027] The feature tensor X∈R output by the 6th layer of the Transformer encoder T×D After the time series is reorganized, the frequency domain dimension is merged to obtain the reorganized feature tensor X′∈R B×P×D′ , where B = 32 is the batch size, P = 32 is the number of patches, and D′ = 6144 is the feature dimension, which are input into Bi-LSTM and Bi-GRU respectively;

[0028] Set up parallel Bi-LSTM and Bi-GRU network modules to reorganize the feature tensor X′∈R B×P×D′ Input into the module and get its bidirectional hidden state output respectively:

[0029]

[0030] Where H is the hidden layer dimension, which is 512. Global average pooling is performed on the above two time series outputs along the time dimension:

[0031]

[0032] To extract its global semantic features, and then obtain a global semantic vector representation of fixed dimension:

[0033]

[0034] Concatenate the two vectors above to get the fused representation:

[0035]

[0036] Furthermore, the feature sequence in step 4 can be pooled for subsequent classification in the following ways:

[0037] h∈Pool(H fusion )→FC (13)

[0038] Among them, FC is the fully connected classification layer (Fully Connected Layer, FC).

[0039] Furthermore, the process of obtaining the optimal virtual classifier in step 4 is:

[0040] The cross entropy loss function is used to optimize the model parameters, specifically:

[0041]

[0042] Where C is the number of classification categories, y i is the one-hot encoding of the true label, is the predicted probability distribution of the model output; by minimizing the above cross-entropy loss function, all parameters including the sixth layer Transformer, Bi-LSTM, Bi-GRU and fully connected classification layer in the SSAST encoder layer are jointly optimized to improve the overall performance of the audio classification model under speech interference conditions and obtain the unknown category judgment result of the test sample.

[0043] Beneficial effects: The present invention proposes an audio classification method in a speech interference environment, which integrates the self-supervised Transformer model and the dual-channel recurrent neural network structure. It can effectively extract audio time series features in a complex acoustic background and improve the recognition ability of unknown semantic events. Specifically, the present invention introduces the SSAST model. In the pre-training stage, the occlusion mask mechanism and the joint loss function are used to learn a more robust spectrogram representation. In the fine-tuning stage, the Transformer encoding results of the middle layer are input into the Bi-LSTM and Bi-GRU structures to capture long-term dependencies and short-term dynamic features, respectively. Subsequently, time pooling is performed on the Bi-LSTM and Bi-GRU outputs to obtain a fixed-length semantic feature vector, and a three-path output mechanism is further constructed: the LSTM path, the GRU path and its splicing path are used for feature expression and classification decisions.

[0044] The advantages of this architecture are: on the one hand, the bidirectional LSTM has a significant advantage in modeling long-term context, capable of extracting temporal semantic information with high contextual stability; on the other hand, the GRU structure is computationally efficient and has fewer parameters, making it suitable for modeling short-term dynamic features. The parallel use of the two enables complementary modeling of temporal dependencies at different scales. Furthermore, by concatenating the pooled output vectors of the LSTM and GRU, a more integrated global feature representation is constructed, thereby improving the model's discriminative and generalization capabilities without significantly increasing the computational burden.

[0045] Traditional audio classification models are often only applicable to scenarios with known categories, and their accuracy drops significantly when faced with interference factors such as speech masking and environmental noise. The multi-path fusion structure proposed in this paper can exploit redundant semantics in different channels, significantly enhancing system robustness. Experiments show that this method achieves good classification results on multiple speech interference test sets, with particularly strong recognition capabilities in scenarios with unknown samples and unbalanced data distribution, demonstrating the practicality and effectiveness of this invention for complex speech recognition tasks in real applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of an audio classification method under speech interference according to the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1As shown, the method provided by the present invention first performs interference signal mixing processing on the original audio samples to construct an extended data set containing background noise. Subsequently, the mixed samples are feature extracted and input into the time series modeling network to learn their category characteristics. In the training phase, the model parameters are optimized through the labeled samples, so that the system can accurately capture the time series structure and category distribution characteristics in the audio signal; in the testing phase, the audio to be classified is input into the trained model, the corresponding features are extracted and the classification judgment is performed, thereby achieving accurate identification of the audio sample category.

[0049] The ESC-50 dataset used in the experiment is a public environmental sound classification dataset, which is widely used in tasks such as environmental audio recognition and sound event detection. The audio in this dataset is all mono, sampled at 44.1kHz, and evenly distributed across 50 environmental sound categories, including: 1. Animal sounds: Dog, Rooster, Pig, Cow, Frog, Cat, Hen, Insects, Sheep, Crow; 2. Natural soundscapes & water sounds: Rain, Seawaves, Crackling fire, Crickets, Chirping birds, Water drops, Wind, Pouring water, Toilet flush, Thunderstorm; 3. Human non-speech sounds: Crying baby, Sneezing, Clapping, Breathing, Coughing, Footsteps, Laughing, Brushing teeth, Snoring, Drinking sipping; 4. Interior / domestic sounds: Door knock, Mouse click, Keyboard typing, Door creaks, Can opening, Washing machine, Vacuum cleaner, Clock Alarm, Clock tick, Glass breaking; 5. Traffic and noise (Exterior / urban noise): Helicopter, Chainsaw, Siren, Carhom, Engine, Train, Church bells, Airplane, Fireworks, Hand saw. Each category contains 40 samples, for a total of 2000 samples.

[0050] This paper proposes a method for constructing an extended ESC-50 dataset with speech interference (hereinafter referred to as ESC-50-mix). The extended audio dataset used in the experiment is WSJ0 (Wall Street Journal 0), a standard English reading speech corpus recorded by multiple speakers. The speech content is derived from news texts in the Wall Street Journal. The sampling rate is 16kHz and the dataset includes over 100 speakers of different genders.

[0051] The experiment extracted audio from different speakers from the WSJ0 speech dataset. Non-overlapping speaker groups were constructed based on speaker ID prefixes. Within these groups, three speech samples corresponding to different speakers were randomly selected to form speech mixture triplets. The three speech samples in each triplet were subjected to positive and negative gain processing based on a random signal-to-noise ratio (SNR).

[0052] In the experiment, after constructing the speech triples, each audio sample in the ESC-50 dataset was matched one-to-one with a set of speech triples. These samples were then weighted and mixed using the audio processing tool sox. The mixing process adhered to the signal-to-noise ratio parameters specified in the triples and maintained the consistency of the mixed samples' duration with the original ESC-50 samples. The mixed samples retained the original ambient sound labels for subsequent classification tasks. The resulting ESC-50-mix contained 2,000 audio samples with background speech interference, covering all 50 ambient sound categories in the original ESC-50 dataset, with 40 samples per category.

[0053] This paper proposes an audio classification method based on a multi-path recurrent neural network. By optimizing the learning rate and network architecture, this method achieves enhanced extraction of audio temporal features. The method first uses a self-supervised audio spectrogram transformer (SSAST) to extract audio features at multiple scales, followed by temporal modeling using a parallel recurrent neural network architecture.

[0054] In terms of network architecture design, the present invention provides multiple comparative examples.

[0055] The basic solution uses the original SSAST model as the feature extractor and integrates the token-level output through the average pooling mechanism. During the fine-tuning phase, the optimal performance is achieved by systematically adjusting the learning rate, specifically setting the learning rate to 10 -3 , 10 -4 and 10 -5 The comparative experiments were conducted at three levels, and the test set accuracy results were 14.5%, 72.7%, and 55.8% respectively. The output results show that different learning rate settings have a significant impact on the model convergence speed and final performance. The learning rate is set to 10 -4can achieve better accuracy.

[0056] To enhance time series modeling capabilities, this paper further proposes three improved network architectures. The first is an architecture integrating bidirectional LSTM (AST-LSTM). This architecture introduces a bidirectional LSTM layer after the SSAST feature extractor, with a hidden dimension set to 512 and the number of layers set to 1 to 2. LSTM effectively alleviates the vanishing gradient problem through a gating mechanism consisting of input, forget, and output gates, enabling it to capture long-range dependencies in audio signals. To improve computational efficiency, the system performs memory optimization on LSTM parameters using the flatten_parameters method.

[0057] The second comparison (AST-GRU) uses a bidirectional GRU instead of LSTM. GRU reduces the number of parameters while maintaining the time series modeling capability by simplifying the design of the update gate and reset gate, thereby improving training efficiency. The bidirectional GRU also sets a 512-dimensional hidden layer and uses the same dropout strategy to prevent overfitting. Experiments show that when the learning rate is set to 10 -4 When , the GRU architecture converges faster than LSTM on some tasks.

[0058] The third embodiment of the present invention (AST-FUSION) integrates LSTM and GRU in parallel. This architecture first reorganizes the output features of SSAST into a tensor form of (B, 32, 8, 768), and then inputs them into the LSTM and GRU networks in parallel. For the output of each network, the system separates the forward and backward hidden states and performs average pooling to generate a fixed-dimensional feature representation. To optimize feature fusion, a dimensionality reduction layer is introduced to compress the outputs of LSTM and GRU into a more compact representation space, and finally a comprehensive feature vector is formed through splicing operations.

[0059] Furthermore, by setting the learning rate to 10 -4 The experimental results comparing the performance of four architectures under different settings show the significant advantages of recurrent neural networks in modeling audio temporal features. The specific performance comparison is shown in Table 2.

[0060] Table 2

[0061] Model Architecture Learning rate setting Test set accuracy Performance improvements Raw AST <![CDATA[10 -4 ]]> 72.7％ Benchmark AST-LSTM <![CDATA[10 -4 ]]> 74.3％ +1.6％ AST-GRU <![CDATA[10 -4 ]]> 74.7％ +2.0％ AST-FUSION <![CDATA[10 -4 ]]> 74.9％ +2.2％

[0062] As shown in Table 2, the three improved architectures all achieved performance improvements compared to the baseline model. In particular, the AST-FUSION hybrid architecture achieved the best classification performance by integrating the complementary advantages of LSTM and GRU in parallel.

[0063] In terms of loss functions and optimization strategies, the system supports both binary cross-entropy loss (BCE with Logits Loss) and multi-class cross-entropy loss (Cross-Entropy Loss), which can be flexibly selected based on the task type. The Adam optimizer is used during training, supporting learning rate warmup and adaptive adjustment mechanisms. The learning rate scheduler uses the StepLR strategy, which adjusts the learning rate according to a set decay factor after a specified number of training rounds to ensure stable model convergence.

[0064] In summary, the present invention achieves fine-grained control of model performance through a systematic learning rate optimization strategy; provides a flexible time series modeling approach through the design of multiple recurrent neural network architectures; and fully leverages the complementary advantages of different network structures through a parallel fusion mechanism, providing an efficient and reliable solution for audio classification tasks.

[0065] The above description is only a preferred embodiment of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for audio classification in a speech interference environment, characterized in that: The Self-Supervised Audio Spectrogram Transformer (SSAST) is used to pre-train labeled training audio samples and extract feature representations of the spectrogram. The features extracted from the SSAST intermediate layer are input into a parallel bidirectional long short-term memory network (Bi-LSTM) and a bidirectional gated recurrent unit (Bi-GRU). Combined with the training samples and their category labels in a speech interference environment, the parameters of the SSAST encoder, Bi-LSTM, Bi-GRU, and fully connected classification layer in the model are jointly optimized by minimizing cross-entropy loss optimization to obtain the optimal classifier. The test audio sample features are input into the optimal classifier, the category probability distribution is output, and the classification performance is verified in combination with the true labels of the test set. The method specifically includes the following steps: Step 1: Audio preprocessing and spectrum generation: input the original audio waveform signal x∈R L (where L is the audio sampling length) is converted into a time-spectrum graph S∈R by short-time Fourier transform (STFT) F×T , where F is the number of frequency dimensions, which is 512, and T is the number of time frames. Divide S into several fixed-size spectrogram blocks S∈R f×t ,total blocks, where f is the frequency dimension of each spectrogram block, which is 16, and t is the number of time frames of each spectrogram block, which is 16. Step 2: Occlusion mask generation and self-supervised pre-training. For each spectral block, randomly generate R square occlusion mask areas to form a mask set M = {M j }, whose mask area is controlled by the dynamic clustering factor. The total mask area satisfies: Among them, Area(M j ) is the jth occlusion mask area M j The area of ​​the input spectrum block S is 1, Area(S) is the area of the entire input spectrum block S, and its value is 256. α∈(0,1) is the dynamic clustering factor, and R satisfies The masked spectrogram is fed into the SSAST model and the following loss function is jointly optimized: in, It is a discriminative contrast loss that uses contrastive learning to construct positive and negative pairs to measure feature discrimination; is the generative reconstruction loss, which measures the quality of occlusion restoration; λ1=1 and λ2=10 are weighting coefficients. Step 3: Feature extraction and backbone network construction. After SSAST completes pre-training, extract its Transformer encoder Lth s The intermediate feature representation H∈R T×D , where T = 256, D = 768, T is the time step, and D is the feature dimension. The feature sequence H is input into Bi-LSTM and Bi-GRU in parallel to obtain H BiLSTM =BiLSTM(H),H BiGRU =BiGRU(H) The features of the two are spliced according to the time dimension to obtain the fusion feature representation: H fusion =(H BiLSTM ||H BiGRU )∈R T×2D (3) The operator "||" represents vector concatenation, which means expanding and concatenating two column vectors into a single column vector according to their dimensions. Step 4: Classifier fine-tuning and training optimization, fusion feature H fusion Compressed into a single vector via temporal pooling: h∈Pool(H fusion ) (4) Then the category probability distribution is obtained through the fully connected layer: Where W h ∈R C×d is the weight matrix of the fully connected layer, b∈R C is the bias vector, C is the number of categories, and d is the dimension of the feature vector after pooling, which is 768. Through labeled training samples, the classifier parameters are optimized using the cross entropy loss function, and the optimal virtual classifier is obtained through iterative training. Step 5: Test sample classification prediction: Extract the spectrogram features of each test audio sample in the same way as steps 1 to 3, input them into the trained optimal virtual classifier, obtain the category probability distribution, and combine it with the real label to evaluate the classification performance.

2. The method for audio classification under speech interference according to claim 1, characterized in that: The feature tensor X∈R of the 6th layer output of the Transformer encoder obtained in step 3 T×D After the time series is reorganized, the frequency domain dimension is merged to obtain the reorganized feature tensor X′∈R B×P×D′ , where B = 32 is the batch size, P = 32 is the number of patches, and D′ = 6144 is the feature dimension. They are input into Bi-LSTM and Bi-GRU respectively to further perform time series modeling on the feature sequence.

3. The method for audio classification under speech interference according to claim 1, characterized in that: After the SSAST output, parallel Bi-LSTM and Bi-GRU network modules are set up, and their bidirectional sequence outputs are extracted at the same time for multi-path fusion. Assume that the output features of the input audio after passing through the SSAST module are: X∈R B×P×D (6) Where B = 32 is the batch size, P = 32 is the number of patches, and D = 6144 is the feature dimension. This sequence is simultaneously input to the Bi-LSTM and Bi-GRU modules to obtain their bidirectional hidden state outputs: Among them, H = 512 is the hidden layer dimension. Global average pooling is performed on the above two time series outputs along the time dimension to extract their global semantic features and obtain the global semantic vector: Then we get a fixed-dimensional vector representation The present invention further uses the above two pooling results and their splicing forms for subsequent classification, that is, constructing the following path output: Splicing Get the fusion representation:

4. The method according to claim 1, wherein During the fine-tuning phase, the cross-entropy loss function is used to optimize the parameters of the sixth Transformer layer, Bi-LSTM layer, Bi-GRU layer, and fully connected classification layer in the SSAST encoder layer. Specifically: Where C is the number of classification categories; y i is the one-hot encoding of the true label; is the predicted probability distribution output by the model. By minimizing the above cross-entropy loss function, the parameters are jointly optimized to improve the overall performance of the audio classification model under speech interference conditions.

Images