Bird chirp sound classification and recognition method and device
By combining convolutional neural networks and Transformer models for bird call classification, and using short-time Fourier transform and Mel frequency cepstral coefficients for feature extraction, the problems of single feature extraction and noise interference in bird call recognition are solved, achieving higher recognition accuracy and discrimination.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF INFORMATION SCI & TECH
- Filing Date
- 2022-10-31
- Publication Date
- 2026-05-29
Smart Images
Figure CN115762533B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method and apparatus for classifying and recognizing bird calls, belonging to the field of bird call recognition technology. Background Technology
[0002] Birds, as an important component of the ecosystem, are widely distributed and sensitive to environmental changes. Most scholars consider birds as indicator species for monitoring environmental changes, making the monitoring, identification, and classification of bird species of great significance. Traditional bird monitoring primarily relies on analyzing morphological differences to determine the distribution of different bird species. However, in practice, bird morphological monitoring suffers from high costs, limited scope, and low efficiency, making bird song monitoring the mainstream approach for identifying bird species.
[0003] The main process of bird call recognition includes three aspects: preprocessing, feature extraction, and classification methods. Previously, scholars both domestically and internationally have conducted extensive research on bird call-based bird recognition using methods such as manual feature extraction and machine learning, but the improvement in recognition performance has been relatively limited. The main reason is that existing methods mostly rely on relatively simple feature extraction methods, and the recognition performance is easily affected by external natural noise. To address this issue, a bird call recognition method based on a combined convolutional neural network and Transformer model is proposed. Summary of the Invention
[0004] To address the problems existing in the prior art, this invention proposes a method and apparatus for classifying and recognizing bird calls based on a combined convolutional neural network and Transformer model. Feature extraction is performed based on short-time Fourier transform and Mel frequency cepstral coefficients, multi-feature fusion increases the difference of features, and feature processing is performed based on a combined convolutional neural network and Transformer model to reduce the interference of environmental noise and improve the accuracy of bird call classification and recognition.
[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical means:
[0006] In a first aspect, the present invention proposes a method for classifying and recognizing bird calls, comprising the following steps:
[0007] Acquire audio data of birdsong;
[0008] The bird song audio data is preprocessed to obtain preprocessed audio data.
[0009] Fourier transform is performed on the preprocessed audio data to obtain the spectrogram of the birdsong;
[0010] The MFCC hybrid feature vector of the preprocessed audio data is obtained based on Mel frequency cepstral coefficients and difference operations;
[0011] A CNN network is used to process spectrograms, and local fine-grained spectral features are obtained after training.
[0012] The Transformer encoder network is used to process the MFCC mixed feature vector, and after training, global sequence features that take into account the context are obtained.
[0013] The recognition and classification results of bird calls are obtained by splicing and fusing local fine-grained spectral features with global sequence features and then using a Softmax classifier.
[0014] In conjunction with the first aspect, further preprocessing of the bird song audio data includes: pre-emphasis and frame-by-frame windowing.
[0015] In conjunction with the first aspect, a further method for obtaining the MFCC hybrid feature vector of the preprocessed audio data based on Mel-frequency cepstral coefficients and differential operations is as follows:
[0016] The power energy spectrum is obtained by performing a fast Fourier transform on the preprocessed audio data.
[0017] The power energy spectrum is input into the Mel filter bank to obtain the Mel energy value;
[0018] The static components of the Mel frequency cepstral coefficients are obtained by taking the logarithm of the Mel energy value and performing a discrete cosine transform.
[0019] The static components of the Mel frequency cepstral coefficients are calculated using finite difference to obtain the first-order difference vector and the second-order difference vector;
[0020] The static components of the Mel frequency cepstral coefficients, the first-order difference vector, and the second-order difference vector are vertically concatenated and fused to obtain the MFCC hybrid feature vector.
[0021] In conjunction with the first aspect, the backbone of the CNN network is a ResNet50 network, and the network structure includes a first convolutional layer, four sets of residual modules, and fully connected layers. The size of the convolutional kernels is set to 3×3, and the size of the pooling layers is set to 2×2×2.
[0022] In conjunction with the first aspect, the Transformer encoder network further includes an input layer, a multi-head attention module, a multilayer perceptron module, and an output layer.
[0023] Secondly, the present invention proposes a bird call classification and recognition device, comprising:
[0024] The audio reading module is used to acquire audio data of birdsong.
[0025] The preprocessing module is used to preprocess the bird song audio data to obtain preprocessed audio data;
[0026] The STFT module is used to perform Fourier transform on the preprocessed audio data to obtain the spectrogram of the birdsong.
[0027] The MFCC module is used to obtain the MFCC hybrid feature vector of the preprocessed audio data based on Mel frequency cepstral coefficients and differential operations;
[0028] The network model training module is used to process spectrograms using a CNN network, and obtain local fine-grained spectral features after training; it uses a Transformer encoder network to process MFCC mixed feature vectors, and obtains global sequence features that take into account the context after training.
[0029] The classification and recognition module is used to concatenate and fuse local fine-grained spectral features with global sequence features and then use a Softmax classifier to obtain the recognition and classification results of bird calls.
[0030] In conjunction with the second aspect, the MFCC module operates as follows:
[0031] The power energy spectrum is obtained by performing a fast Fourier transform on the preprocessed audio data.
[0032] The power energy spectrum is input into the Mel filter bank to obtain the Mel energy value;
[0033] The static components of the Mel frequency cepstral coefficients are obtained by taking the logarithm of the Mel energy value and performing a discrete cosine transform.
[0034] The static components of the Mel frequency cepstral coefficients are calculated using finite difference to obtain the first-order difference vector and the second-order difference vector;
[0035] The static components of the Mel frequency cepstral coefficients, the first-order difference vector, and the second-order difference vector are vertically concatenated and fused to obtain the MFCC hybrid feature vector.
[0036] In conjunction with the second aspect, further, in the network model training module, the backbone of the CNN network is a ResNet50 network, and the network structure includes a first convolutional layer, four sets of residual modules, and fully connected layers. The size of the convolutional kernels is set to 3×3, and the size of the pooling layers is set to 2×2×2. The Transformer encoder network includes an input layer, a multi-head attention module, a multi-layer perceptron module, and an output layer.
[0037] The following advantages can be obtained by adopting the above technical means:
[0038] This invention proposes a method and apparatus for classifying and recognizing bird calls. It extracts multiple features from the audio data of the bird calls to be identified. Firstly, it uses Short-Time Fourier Transform (STFT) to construct a three-dimensional spectrogram of STFT energy as the first input feature. Secondly, it fuses the static features of Mel-frequency cepstral coefficients (MFCC), first-order differences, and second-order differences to form an MFCC hybrid feature vector as the second input feature. This enhances the utilization of the contextual continuity of temporal features and reduces the interference of environmental noise in the bird call audio. This invention uses multi-feature fusion instead of the single feature in existing technologies, expanding the differences between similar bird calls and improving the distinguishability of different bird calls. This invention utilizes a network model combining CNN and Transformer for feature extraction. The CNN network can extract local frequency domain information from the STFT spectrogram, while the Transformer network can extract global temporal domain information from the MFCC hybrid feature vector, taking into account contextual features. After fusing the two, classification can fully leverage the advantages of STFT spectrogram features and MFCC hybrid features, further reducing interference from external natural noise and effectively improving the accuracy of network model recognition, thereby effectively improving the classification and recognition accuracy of bird song audio. Attached Figure Description
[0039] Figure 1 This is a flowchart illustrating the steps of a bird call classification and recognition method according to the present invention.
[0040] Figure 2 This is a structural block diagram of the Transformer encoder in an embodiment of the present invention;
[0041] Figure 3 The figure shows the accuracy change curves of the CNN-Transformer network model after training for 100 epochs on the Birdsdata and xeno-canto datasets in this embodiment of the invention.
[0042] Figure 4 This is the loss optimization convergence curve of the CNN-Transformer network model in this embodiment of the invention after training for 100 epochs on the Birdsdata and xeno-canto datasets;
[0043] Figure 5 This is a schematic diagram of the structure of a bird call classification and recognition device according to the present invention. Detailed Implementation
[0044] The technical solution of the present invention will be further described below with reference to the accompanying drawings:
[0045] This invention proposes a method for classifying and recognizing bird calls, such as... Figure 1 As shown, the specific steps include the following:
[0046] Step A: Read the original bird sound audio at a frequency of 44.1kHz to obtain bird sound audio data.
[0047] Step B: Preprocess the bird song audio data to obtain preprocessed audio data. In this embodiment of the invention, the preprocessing includes amplitude normalization, pre-emphasis, and frame-by-frame windowing: Amplitude normalization uses maximum value normalization, that is, the amplitude parameter of each frame of the obtained audio is divided by the maximum absolute value of all elements in the audio segment, so that the amplitude is completely distributed between [-1, 1]; pre-emphasis uses a first-order FIR high-pass digital filter for differential calculation, and the pre-emphasis coefficient is set to 0.935 to compensate for the loss of high-frequency parts in the bird song audio; frame-by-frame windowing uses a Hamming window as the window function, and the frame length is set to 23ms and the frame shift is set to 11ms. The main purpose is to analyze the dynamic parameter characteristics as statically as possible, and at the same time obtain more stable spectral parameters. Through the above preprocessing steps, each segment of bird song audio data can be divided into 173 frames of audio data for subsequent operations.
[0048] Step C: Perform a Fourier transform (STFT) on the preprocessed audio data to obtain the spectrogram of the bird calls, and save the STFT spectrogram as a three-dimensional image of [256,256,3].
[0049] Step D: Obtain the MFCC hybrid feature vector of the preprocessed audio data based on Mel frequency cepstral coefficients and differential operations.
[0050] Step D01: Perform a Fast Fourier Transform on the preprocessed audio data to obtain the power energy spectrum. Then, input the power energy spectrum into the Mel filter bank to calculate the specific Mel energy value. Finally, take the logarithm of the Mel energy value and perform a Discrete Cosine Transform to obtain the Mel frequency cepstral coefficient static component. Take 12 groups of data from the 1st to the 13th group of the Mel frequency cepstral coefficient static component to obtain a two-dimensional array of size [12, 173].
[0051] Step D02: Perform differential calculation on the static components of the Mel frequency cepstral coefficients to obtain a first-order difference vector and a second-order difference vector. Both the first-order difference vector and the second-order difference vector are two-dimensional arrays of [12, 173].
[0052] Step D03: Vertically concatenate and fuse the static components of the Mel frequency cepstral coefficients, the first-order difference vector, and the second-order difference vector to obtain the MFCC hybrid feature vector (a two-dimensional array of size [36, 173]).
[0053] To improve the accuracy of bird call recognition, this invention constructs a CNN-Transformer network model. The STFT spectrogram is used as the first branch input feature of the overall network and input into the CNN network with ResNet50 as the backbone. The MFCC hybrid feature vector is used as the second branch input feature of the overall network and input into the Transformer encoder network. The two network models are used to extract features from the two branch input features respectively.
[0054] In this embodiment of the invention, the CNN network structure includes a first convolutional layer, four sets of residual modules, and a fully connected layer. The convolutional kernel size is set to 3×3, and the pooling layer size is set to 2×2×2. The Transformer encoder network is as follows: Figure 2 As shown, it includes an input layer, a multi-head attention module, a multilayer perceptron module, and an output layer.
[0055] Step E: Process the spectrogram using a CNN network to obtain local fine-grained spectral features after training. The input size of the CNN network is 256×256. The spectrogram is input into the CNN network, and after passing through the first convolutional layer, the output size is 128×128. Then, it passes through four sets of residual modules, and finally through a fully connected layer to obtain two-dimensional output features, namely local fine-grained spectral features.
[0056] Step F: Process the MFCC hybrid feature vector using a Transformer encoder network to obtain global sequence features that take into account the context after training. Input the 36×173 MFCC hybrid feature vector into the Transformer encoder network. The Transformer input layer first performs positional encoding on the feature vector, resulting in a new 36×173 input feature. Next, the Transformer's second layer multi-head attention module, consisting of eight attention computation heads, averages the 173 frames of input audio data, calculates their respective QKV attention weight matrices, and exchanges global information. The calculated parameter matrices are then reassembled to obtain a new weight matrix. Finally, this matrix is fed into a multilayer perceptron, where two linear activation layers are used to obtain the final decision array, which is an N×173 output matrix, representing the global sequence features containing contextual information.
[0057] Step G: After splicing and fusing the local fine-grained spectral features with the global sequence features, the bird call recognition and classification results are obtained through the Softmax classifier.
[0058] To verify the effectiveness of the method of the present invention, the following experiments are provided in the embodiments of the present invention:
[0059] This invention uses bird song audio from the Birdsdata and Xeno-Canto bird sound datasets as experimental sample data. Birdsdata is a manually annotated natural sound standard dataset released by Beijing BaiNiao Data Technology Co., Ltd. This dataset publicly collects 14,311 WAV audio files, each 2 seconds long, representing 20 categories of common bird calls in China. Xeno-Canto bird sound data comes from a global wild bird sound database, containing audio from 44 common bird species in Eurasia, all recorded in natural environments, totaling 34,703 WAX audio files, ranging in length from 30 seconds to 5 minutes, and including ambient noise. The sampling frequency of both datasets is 44.1 kHz.
[0060] The bird song audio data was preprocessed with a pre-emphasis factor of 0.935. Hamming windowing was used for frame segmentation and windowing with a frame length of 23ms and a frame shift of 11ms.
[0061] The hardware operating system used in this invention experiment was Ubuntu 20.04, and the network model was built entirely using the PyTorch 1.8.0 deep learning framework. During the overall training process, the number of iterations (epochs) was set to 100. In the specific experiment, the entire dataset was divided into training and test sets in an 8:2 ratio, and then five experiments were conducted using five-fold cross-validation. The experiment used accuracy and F1-score as evaluation metrics and consisted of two parts: 1. The bird song classification and recognition method of this invention was subjected to five-fold cross-validation on both datasets, and the mean and standard deviation were calculated to observe the training performance; 2. The bird song classification and recognition method of this invention was compared with other newer methods to verify the test accuracy of the method of this invention.
[0062] Figure 3 , 4 The figures show the accuracy change curves and loss optimization convergence curves of the CNN-Transformer network model proposed in this invention after training on the Birdsdata and Xeno-Canto datasets for 100 epochs. Figure 3 , 4 As can be seen, the network model proposed in this invention can initially converge in about 40 epochs, with a relatively fast overall convergence speed and good performance when finally converging.
[0063] In the experiment of this invention, the dataset (Birdsdata dataset or xeno-canto dataset) is divided into five equal parts: A, B, C, D, and E. Each time, four parts are taken as the training set and one part is taken as the test set for the experiment.
[0064] The evaluation results of the method of this invention on the Birdsdata dataset are shown in Table 1:
[0065] Table 1
[0066]
[0067] The evaluation results of the method of this invention on the xeno-canto dataset are shown in Table 2:
[0068] Table 2
[0069]
[0070] Analysis of the data in Tables 1 and 2 shows that the network model of this invention can achieve good recognition results on two datasets with different characteristics, demonstrating good generalization ability.
[0071] In the experiments of this invention, VGGNet, CRNN, CNN-LSTM, CNN (ResNet50), and Transformer were used as comparison methods, and together with the method of this invention, bird call recognition was performed on the Birdsdata dataset and the xeno-canto dataset. The recognition results are shown in Table 3.
[0072] Table 3
[0073]
[0074] As shown in Table 3, the accuracy of our method differs significantly from other methods, with our method exhibiting higher accuracy. VGGNet and CRNN both use only a single feature input; while CNN-LSTM's accuracy on the Xeno-Canto dataset is only 1.5% lower, its parameter count is far greater than our method. Our method not only fully demonstrates the advantages of multi-feature fusion but also effectively improves recognition accuracy.
[0075] This invention also proposes a bird call classification and recognition device, such as... Figure 5 As shown, it mainly includes an audio reading module, a preprocessing module, an STFT module, an MFCC module, a network model training module, and a classification and recognition module.
[0076] The audio reading module is used to read the original bird sound audio at a frequency of 44.1kHz to obtain bird song audio data.
[0077] The preprocessing module is mainly used to preprocess the bird song audio data to obtain preprocessed audio data. Preprocessing includes operations such as pre-emphasis and frame windowing. Through preprocessing, each segment of bird song audio data can be divided into 173 frames of audio data.
[0078] The STFT module is mainly used to perform Fourier transform on the preprocessed audio data to obtain the spectrogram of bird calls.
[0079] The MFCC module is primarily used to obtain the MFCC hybrid feature vector of preprocessed audio data based on Mel frequency cepstral coefficients and differential operations. The specific operations of the MFCC module are as follows: A Fast Fourier Transform is performed on the preprocessed audio data to obtain the power energy spectrum. This power energy spectrum is then input into the Mel filter bank to calculate the specific Mel energy value. Finally, the logarithm of the Mel energy value is taken, and a Discrete Cosine Transform is performed to obtain the Mel frequency cepstral coefficient static component. Differential calculations are then performed on the Mel frequency cepstral coefficient static component to obtain the first-order and second-order difference vectors. The Mel frequency cepstral coefficient static component, the first-order difference vector, and the second-order difference vector are vertically concatenated and fused to obtain the MFCC hybrid feature vector.
[0080] The network model training module includes a CNN-Transformer network model, which uses a CNN network to process spectrograms and obtains local fine-grained spectral features after training; and a Transformer encoder network to process MFCC mixed feature vectors and obtain global sequence features that take into account the context after training.
[0081] In the network model training module, the backbone of the CNN network is the ResNet50 network. The network structure includes a first convolutional layer, four sets of residual modules, and fully connected layers. The size of the convolutional kernels is set to 3×3, and the size of the pooling layers is set to 2×2×2. The Transformer encoder network includes an input layer, a multi-head attention module, a multi-layer perceptron module, and an output layer.
[0082] The classification and recognition module is used to concatenate and fuse local fine-grained spectral features with global sequence features, and then use a Softmax classifier to obtain the recognition and classification results of bird calls.
[0083] Compared with existing technologies, this invention utilizes a three-dimensional spectrogram of STFT energy drawn using Short Time Fourier Transform (STFT) as the first input feature, and fuses the static features of Mel Frequency Cepstral Coefficients (MFCC), first-order differences, and second-order differences to form an MFCC hybrid feature vector as the second input feature. This solves the problem of relatively simple feature extraction in existing bird sound recognition methods, enhances the utilization of temporal feature context continuity, expands the differences between similar bird sounds, and helps improve the distinguishability of different bird sounds, while reducing the interference of environmental noise in bird sound audio. This invention uses a network model combining CNN and Transformer for feature extraction. The CNN network can extract local frequency domain information from the STFT spectrogram, while the Transformer network can extract global temporal information from the MFCC hybrid feature vector that takes into account context features. After fusing the two, classification can fully utilize the advantages of STFT spectrogram features and MFCC hybrid features, further reduce the interference of external natural noise, effectively improve the accuracy of network model recognition, and thus effectively improve the classification accuracy of bird sound audio.
[0084] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for classifying and recognizing bird calls, characterized in that, Includes the following steps: Acquire audio data of birdsong; The bird song audio data is preprocessed to obtain preprocessed audio data. Fourier transform is performed on the preprocessed audio data to obtain the spectrogram of the birdsong; The MFCC hybrid feature vector of the preprocessed audio data is obtained based on Mel frequency cepstral coefficients and difference operations; A CNN network is used to process spectrograms, and local fine-grained spectral features are obtained after training. The Transformer encoder network is used to process the MFCC mixed feature vector, and after training, global sequence features that take into account the context are obtained. The recognition and classification results of bird calls are obtained by splicing and fusing local fine-grained spectral features with global sequence features and then using a Softmax classifier. The spectrogram is used as the input feature of the first branch and input into a CNN network with ResNet50 as the backbone. The MFCC hybrid feature vector is used as the input feature of the second branch and input into a Transformer encoder network. The two branches of input features are extracted by the CNN network and the Transformer encoder network respectively.
2. The bird song classification and recognition method according to claim 1, characterized in that, Preprocessing of bird song audio data includes: pre-emphasis and frame-by-frame windowing.
3. The bird song classification and recognition method according to claim 1, characterized in that, The method for obtaining the MFCC hybrid feature vector of preprocessed audio data based on Mel-frequency cepstral coefficients and difference operations is as follows: The power energy spectrum is obtained by performing a fast Fourier transform on the preprocessed audio data. The power energy spectrum is input into the Mel filter bank to obtain the Mel energy value; The static components of the Mel frequency cepstral coefficients are obtained by taking the logarithm of the Mel energy value and performing a discrete cosine transform. The static components of the Mel frequency cepstral coefficients are calculated using finite difference to obtain the first-order difference vector and the second-order difference vector; The static components of the Mel frequency cepstral coefficients, the first-order difference vector, and the second-order difference vector are vertically concatenated and fused to obtain the MFCC hybrid feature vector.
4. The bird song classification and recognition method according to claim 1, characterized in that, The backbone of the CNN network is a ResNet50 network. The network structure includes a first convolutional layer, four sets of residual modules, and fully connected layers. The size of the convolutional kernels is set to 3×3, and the size of the pooling layers is set to 2×2×2.
5. The bird song classification and recognition method according to claim 1, characterized in that, The Transformer encoder network consists of an input layer, a multi-head attention module, a multilayer perceptron module, and an output layer.
6. A bird call classification and recognition device, characterized in that, include: The audio reading module is used to acquire audio data of birdsong. The preprocessing module is used to preprocess the bird song audio data to obtain preprocessed audio data; The STFT module is used to perform Fourier transform on the preprocessed audio data to obtain the spectrogram of the birdsong. The MFCC module is used to obtain the MFCC hybrid feature vector of the preprocessed audio data based on Mel frequency cepstral coefficients and differential operations; The network model training module is used to process spectrograms using a CNN network, and obtain local fine-grained spectral features after training; it uses a Transformer encoder network to process MFCC mixed feature vectors, and obtains global sequence features that take into account the context after training. The classification and recognition module is used to concatenate and fuse local fine-grained spectral features with global sequence features and then use a Softmax classifier to obtain the recognition and classification results of bird calls; The network model training module takes the spectrogram as the first branch input feature and inputs it into a CNN network with ResNet50 as the backbone. It takes the MFCC hybrid feature vector as the second branch input feature and inputs it into a Transformer encoder network. The CNN network and the Transformer encoder network are used to extract features from the two branch input features respectively.
7. A bird call classification and recognition device according to claim 6, characterized in that, The MFCC module operates as follows: The power energy spectrum is obtained by performing a fast Fourier transform on the preprocessed audio data. The power energy spectrum is input into the Mel filter bank to obtain the Mel energy value; The static components of the Mel frequency cepstral coefficients are obtained by taking the logarithm of the Mel energy value and performing a discrete cosine transform. The static components of the Mel frequency cepstral coefficients are calculated using finite difference to obtain the first-order difference vector and the second-order difference vector; The static components of the Mel frequency cepstral coefficients, the first-order difference vector, and the second-order difference vector are vertically concatenated and fused to obtain the MFCC hybrid feature vector.
8. A bird call classification and recognition device according to claim 6, characterized in that, In the network model training module, the backbone of the CNN network is a ResNet50 network. The network structure includes a first convolutional layer, four sets of residual modules, and fully connected layers. The size of the convolutional kernels is set to 3×3, and the size of the pooling layers is set to 2×2×2. The Transformer encoder network includes an input layer, a multi-head attention module, a multi-layer perceptron module, and an output layer.