A large-scale bird sound recognition method based on deep learning technology

Abstract

The application relates to the cross field of artificial intelligence and bird ecology, and specifically discloses a large-scale bird sound recognition method based on a deep learning technology, which collects bird sound audio through a project area recording device, trains a bird sound event detection model by using a Bird audio detection challenge 2018 data set, and separates effective bird sound and background noise by using field recording data for category prediction; meanwhile, a bird sound recognition data set is constructed by combining a bird species directory of a Chinese bird watching recording center and an audio file of Xeno-Canto, a bird sound recognition model is trained by using the background noise and the bird sound recognition data set, and verification is carried out; finally, effective bird sound data is input into the model for label prediction to recognize bird sound. The application aims to save data processing cost and improve the recognition accuracy in a complex scene by carrying out extensive bird sound recognition in a project area through the deep learning technology.

Description

Technical Field

[0001] This application relates to the intersection of artificial intelligence and bird ecology, and more specifically, to a method for large-scale bird sound recognition based on deep learning technology. Background Technology

[0002] In the fields of avian ecology and environmental monitoring, experts have been exploring more effective ways to monitor and study birds. Active song monitoring is a traditional method that involves manually recording bird calls, but this method is costly and may interfere with birds' natural behavior. With technological advancements, passive sound monitoring technology has become popular. It automatically records sounds by installing sensors, reducing human bias, but requires significant manual post-analysis and verification.

[0003] Many researchers have introduced artificial intelligence (AI) into passive monitoring, especially deep learning algorithms, bringing revolutionary changes to this field. AI can process large amounts of data, improve monitoring efficiency, and reduce the need for manpower. For example, an AI model based on the DenseNet121 network structure achieved a high accuracy rate of 96.9% in bird call recognition.

[0004] However, most current algorithms can only identify a limited number of bird sounds in a given dataset, and their comprehensive recognition ability in noisy natural environments including multiple bird species is not ideal. This is not only due to the lack of robustness of the algorithms themselves, but also because there is a lack of sufficiently diverse and reliable bird sound recordings to train AI models.

[0005] Furthermore, current AI research largely focuses on bird classification, with limited studies on the relationship between bird vocalizations and the ecological environment. In practical applications, AI's insufficient recognition capabilities in complex acoustic scenarios limit its application in long-term, large-scale biodiversity research.

[0006] Therefore, a large-scale bird sound recognition method based on deep learning technology is provided. Summary of the Invention

[0007] To address the aforementioned technical problems, this application is proposed. This application provides a large-scale bird sound recognition method based on deep learning technology. This method aims to conduct extensive bird sound monitoring in areas under construction using deep learning technology. It constructs a complete and innovative bird sound recognition framework based on deep learning technology, using spectrum segmentation technology and the ECAPA-TDNN model.

[0008] Specifically, according to one aspect of this application, a large-scale bird sound recognition method based on deep learning technology is provided, comprising:

[0009] S1. Use recording equipment in the project area to collect bird sounds near the project area to obtain field recording data;

[0010] S2. Three datasets obtained from the Bird audio detection challenge 2018-DCASETask 3 will be used as bird sound event detection datasets to train the bird sound event detection model.

[0011] S3. Input the field recording data into the trained bird sound event detection model for category prediction to obtain effective bird sound data and background noise data;

[0012] S4. Obtain the bird species catalog recorded in the project area from the China Birdwatching Recording Center, and download the corresponding bird species audio files in Xeno-Canto. Perform data preprocessing on the audio files to obtain a bird sound recognition dataset.

[0013] S5. Use the background noise data and the bird sound recognition dataset to train the bird sound recognition model;

[0014] S6. Validate the trained bird sound recognition model;

[0015] S7. Input the valid bird sound data into the bird sound recognition model for label prediction to obtain the recognition result;

[0016] In the construction of the bird sound event detection model and the bird sound recognition model, the spectrograms from the bird sound event detection dataset and the bird sound recognition dataset are used as inputs, respectively, and the ECAPA-TDNN model is selected as the basic model architecture. The specific model construction process is as follows:

[0017] The spectrograms in the bird sound event detection dataset and the bird sound recognition dataset are preprocessed by frequency band segmentation to generate sub-bands Z with different frequency band characteristics;

[0018] The segmented frequency band features are subjected to L2 normalization and connected to a fully connected layer to increase the complex correlation between features;

[0019] An acceleration strategy based on sparse matrices is adopted, using the COO (Coordinate Format) storage format to perform matrix multiplication operations on the non-zero elements in the sparse matrix, thereby improving computational efficiency and resource utilization.

[0020] The ECAPA-TDNN model is used to capture speech features from spectrograms, which combines dilated convolution, Res2Net structure and SENet module;

[0021] A fully connected layer is added after the output of the ECAPA-TDNN model to map the feature space to the category space;

[0022] The fully connected layer of the bird sound event detection model outputs two nodes, and the output values, after passing through the softmax function, represent the probabilities of bird sound samples and noise samples, respectively. The fully connected layer of the bird sound recognition model outputs 179 nodes, representing the probabilities of 179 bird species, respectively.

[0023] Optionally, S4 includes: S41, manually checking the spectrogram and waveform of the audio file and filtering out low-quality and incorrectly labeled audio; S42, cutting long audio every 30 seconds, retaining the last audio segment if it is less than 30 seconds but greater than 15 seconds, and discarding it if it is less than 15 seconds; S43, deleting original audio segments with a length of less than 2 seconds.

[0024] Optionally, step S5 includes: S51, unifying the sampling rate of the acoustic signals in the bird sound recognition dataset to 32,000 kHz, and uniformly converting the audio format to a lossless WAV format to obtain preprocessed audio data; S52, randomly slicing the preprocessed audio data into segments with a uniform segment length of 10 seconds; S53, normalizing the amplitude of the sliced ​​audio data; S54, mixing the background noise data into the normalized audio data with a 50% probability to obtain audio sample data; S55, performing spectral enhancement on the log-Mel spectrum of the audio sample data, including randomly masking 0 to 5 frames in the time domain and randomly masking 0 to 10 channels in the frequency domain; S56, using techniques such as speech rate perturbation, volume enhancement, and adding Gaussian noise to train the model.

[0025] Specifically, in process S56, a stepwise fusion strategy is adopted to integrate the angle-interval-based loss function into the cross-entropy loss function. The specific process is as follows: in the early stage of training, the cross-entropy loss function L is used as the basis for the loss function. CE To optimize this, β(t)≈0, and the formula is:

[0026]

[0027] As the model gradually converges, β(t) begins to increase, and a loss function L based on the angle interval is gradually introduced. AAM The angular interval in the middle is given by the formula:

[0028] L fuse (t)=L CE +β(t)·L AAM ;

[0029] In the later stages of training, β(t) grows to a relatively high value close to 1, at which point the loss function L based on the angular interval... AAM As the cross-entropy loss becomes dominant, its influence gradually decreases, and the formula is:

[0030]

[0031] Where N is the number of samples, C is the number of categories, and x i W is the feature of the i-th input sample. j It is the weight of the classification layer, y i It is the correct category, m is the increased angular interval, and s is an adjustable scale factor. The angle between the i-th sample and its correct class weight is represented by t, where t is the training step size, and β(t) is the weight coefficient that gradually changes with the training step size t.

[0032] Specifically, in process S56, an adaptive learning rate adjustment strategy is adopted to improve the training efficiency and generalization ability of the model. The specific operations are as follows: in the early stage of model training, the learning rate is gradually increased through a warmup strategy, and the Adam optimizer is used to accelerate the convergence of the model; the batch size is set to 64; during the model training process, the system dynamically adjusts the number of training epochs according to the changes in the loss function; in the later stage of training, the learning rate is gradually reduced through cosine annealing for fine-tuning.

[0033] Optionally, in step S6, the following evaluation metrics are used to compare and evaluate the performance of each model in this method:

[0034]

[0035] In this context, TP represents a true positive sample, TN represents a true negative sample, FP represents a false positive sample, and FN represents a false negative sample.

[0036] Optionally, in step S7, two strategies are used to determine the correct recognition result: The first strategy uses a bird sound recognition model output probability of 0.7 as the threshold for correct recognition, meaning audio samples with an output probability greater than or equal to 0.7 are considered correctly recognized samples. The second strategy, based on the first strategy, further employs a "continuous enhancement recognition strategy" to post-process the model recognition result. Specifically, the second strategy includes: when a bird is recognized, if the same bird sound is recognized within 30 seconds before and after, but its model output probability is less than 0.7 and greater than 0.5, then this is also output as a correctly recognized sample.

[0037] Specifically, according to another aspect of this application, a large-scale bird sound recognition system based on deep learning technology is provided, comprising:

[0038] The bird sound audio acquisition module is used to collect bird sound audio near the project area using the project area recording equipment to obtain field recording data.

[0039] The bird sound event detection model training module is used to train the bird sound event detection model by using three datasets obtained from the Bird audiodetection challenge 2018-DCASE Task 3, a bird sound event detection competition.

[0040] The bird call category prediction module is used to input the field recording data into the trained bird call event detection model for category prediction, so as to obtain effective bird call data and background noise data.

[0041] The bird sound recognition dataset acquisition module is used to obtain the bird species catalog recorded in the project area from the China Bird Watching Recording Center, download the corresponding bird species audio files in Xeno-Canto, and perform data preprocessing operations on the audio files to obtain the bird sound recognition dataset.

[0042] A bird sound recognition model training module is used to train a bird sound recognition model using the background noise data and the bird sound recognition dataset;

[0043] The model validation module is used to validate the trained bird sound recognition model;

[0044] The label prediction module is used to input the valid bird sound data into the bird sound recognition model to perform label prediction in order to obtain the recognition result.

[0045] Compared with existing technologies, this application provides a large-scale bird call recognition method based on deep learning technology. This method employs a sparse matrix-based acceleration strategy to reduce the storage requirements of the high-dimensional feature matrix generated after frequency band segmentation of the spectrogram, and uses an adaptive learning rate adjustment strategy during model training. This improves computational speed, significantly reduces memory usage, and thus saves data processing costs. Furthermore, this method selects the ECAPA-TDNN model as its basic architecture. The ECAPA-TDNN model utilizes dilated convolution and Res2Net to obtain multi-scale contextual information. The SE-Res2Block model within the ECAPA-TDNN model combines Res2Net and SENet; this structure facilitates the processing of multi-scale features, reduces the number of model parameters, and improves the accuracy of bird call recognition in complex scenes. It is also applied to bird activity and behavior analysis to assess the impact of engineering projects on the ecological environment. Attached Figure Description

[0046] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0047] Figure 1 The illustration shows a visual flowchart of a large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application.

[0048] Figure 2 The illustration shows a flowchart of a large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application.

[0049] Figure 3 The illustration shows a flowchart of step S4 in the large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application.

[0050] Figure 4 The illustration shows a flowchart of step S5 in the large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application.

[0051] Figure 5 The illustration shows a schematic diagram of the mid-frequency band segmentation module in a large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application.

[0052] Figure 6 The diagram illustrates the network structures of SE-Res2Block and ECAPA-TDNN in a large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application.

[0053] Figure 7 The figure shows an overall schematic diagram of the bird sound recognition model in the large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application.

[0054] Figure 8 The illustration shows a schematic diagram of a large-scale bird sound recognition system based on deep learning technology according to an embodiment of this application. Detailed Implementation

[0055] The embodiments according to this application will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0056] Example 1:

[0057] Figure 1The illustration shows a visual flowchart of a large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application. Figure 1 This document visually illustrates the entire process of the large-scale bird call recognition method based on deep learning technology, from data acquisition to the final recognition result. This graphical approach helps users quickly understand the entire bird call recognition process, including steps such as data acquisition, processing, model training, and recognition.

[0058] Figure 2 The illustration shows a flowchart of a large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application. Figure 1 As shown, the large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application includes: S1, collecting bird sound audio near the project area using a recording device in the project area to obtain field recording data; S2, using three datasets obtained from the Bird audio detection challenge 2018-DCASE Task 3 as bird sound event detection datasets to train a bird sound event detection model; S3, inputting the field recording data into the trained bird sound event detection model for category prediction to obtain valid bird sound data and background noise data; S4, obtaining a catalog of bird species recorded in the project area from the China Birdwatching Recording Center, and simultaneously downloading audio files of the corresponding bird species from Xeno-Canto, and performing data preprocessing operations on the audio files to obtain a bird sound recognition dataset; S5, using the background noise data and the bird sound recognition dataset to train a bird sound recognition model; S6, validating the trained bird sound recognition model; S7, inputting the valid bird sound data into the bird sound recognition model for label prediction to obtain recognition results.

[0059] In this embodiment, step S1 involves using a project area recording device to collect bird sound audio near the project area to obtain field recording data. It should be understood that bird sound data in the natural environment is highly diverse, including different bird species, different calls, and interference from various environmental noises. Furthermore, considering that field recording can capture the authentic calls of birds in their natural environment, including sounds against various natural backgrounds, this is crucial for training and testing AI models to identify different bird species. Therefore, to provide more realistic and diverse data for training a more accurate and robust AI model, before identifying and predicting bird sounds, the project area recording device is first used to collect bird sound audio near the project area to obtain field recording data. This helps the model better adapt to the complexities of the real world and improves its robustness in practical applications.

[0060] In this embodiment, step S2 involves using three datasets obtained from the Bird audiodetection challenge 2018-DCASE Task 3, a bird sound event detection competition, as the bird sound event detection datasets to train the bird sound event detection model. It should be understood that the bird sound event detection model can automatically identify and distinguish bird sound events in audio (such as valid bird sounds, invalid background noise, etc.), which is a key step in automating bird sound monitoring. Considering that competition datasets typically contain a variety of different bird sounds and background noises, this diversity helps train a model with strong robustness to noise and interference. Therefore, the three datasets obtained from the Bird audiodetection challenge 2018-DCASE Task 3 are used as the training datasets for the bird sound event detection model. In this way, by training with data collected in various environments, the model can better handle various noises and interferences that may be encountered in practical applications.

[0061] In this embodiment, step S3 involves inputting the field recording data into a trained bird call event detection model for category prediction to obtain valid bird call data and background noise data. It should be understood that category prediction distinguishes between valid bird calls and background noise in the field recording data, and these two types of data play important roles in this embodiment. Valid bird call data includes actual bird call events and can be used for further training of the bird call recognition model to identify and classify different bird species. This data is of significant value for studying bird behavior, population dynamics, and biodiversity. While background noise data does not contain bird calls, it is equally important for model training, helping the model learn to distinguish between the audio features of bird calls and non-bird calls, improving the model's recognition ability in complex environments. Therefore, further category prediction of the field recording data using the trained bird call event detection model accurately filters out valid bird call data.

[0062] In this embodiment, step S4 involves obtaining a catalog of bird species recorded in the project area from the China Birdwatching Recording Center, downloading the corresponding audio files for each bird species from Xeno-Canto, and performing data preprocessing on the audio files to obtain a bird sound recognition dataset. It should be understood that the bird species catalog provided by the China Birdwatching Recording Center has high scientific validity and accuracy, while Xeno-Canto is an international platform with bird sound samples from around the world, providing extensive diversity and reliability for the dataset. Therefore, this data is used as the training dataset for the bird sound recognition model. Furthermore, considering that audio annotations are added by the uploader and may be inaccurate, verification is needed to minimize the impact of incorrect labels on the results. Also, most audio files are between 2 seconds and 5 minutes in length; longer audio files may contain a large amount of noise and silent sections, which occupy a large portion of the audio duration, while the actual bird calls constitute a smaller proportion. Manual editing of long audio files is necessary to remove these long, useless segments. Therefore, further preprocessing of these datasets is required.

[0063] Specifically, Figure 3 The illustration shows a flowchart of step S4 in a large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application. Figure 3 As shown, step S4 includes: S41, manually checking the spectrogram and waveform of the audio file and filtering out low-quality and incorrectly labeled audio; S42, cutting long audio every 30 seconds, retaining the last audio segment if it is less than 30 seconds but greater than 15 seconds, and discarding it if it is less than 15 seconds; S43, deleting original audio segments with a length of less than 2 seconds.

[0064] In particular, the method for constructing this dataset is highly versatile and scalable. It can be used to construct bird sound datasets for various complex regions, and has broad application prospects.

[0065] In this embodiment, step S5 uses the background noise data and the bird sound recognition dataset to train the bird sound recognition model. It should be understood that the background noise data is interference data from the real environment obtained from field recordings, which helps the model learn to distinguish between bird sounds and environmental noise. The bird sound recognition dataset contains a catalog of bird species recorded in the project area, providing the features needed for the model to learn to recognize different bird sounds. Using the combined data of both to train the bird sound recognition model can improve the model's accuracy in recognizing bird sounds and better adapt to complex acoustic scenarios. Therefore, the background noise data and the bird sound recognition dataset are used to train the bird sound recognition model. However, since the annotations on the audio files are added by the uploader, there may be errors. Furthermore, differences in the signal-to-noise ratio and sound acquisition environment (equipment / sampling rate / temperature / weather, etc.) between the training and on-site recordings can easily lead to spatial domain mismatch. Therefore, before model training, all audio samples must undergo format unification, random slicing, normalization, and data augmentation.

[0066] Specifically, Figure 4 The illustration shows a flowchart of step S5 in a large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application. Figure 4 As shown, step S5 includes: S51, unifying the sampling rate of the acoustic signals in the bird sound recognition dataset to 32,000kHz, and converting the audio format to a lossless WAV format to obtain preprocessed audio data; S52, randomly slicing the preprocessed audio data into segments with a uniform segment length of 10s; S53, normalizing the amplitude of the sliced ​​audio data; S54, mixing the background noise data into the normalized audio data with a 50% probability to obtain audio sample data; S55, performing spectral enhancement on the log-Mel spectrum of the audio sample data, including randomly masking 0 to 5 frames in the time domain and randomly masking 0 to 10 channels in the frequency domain; S56, training the model using techniques such as speech rate perturbation, volume enhancement, and adding Gaussian noise.

[0067] Specifically, it should be understood that the loss function is used to evaluate the degree of error between the model's prediction and the actual result. In multi-class speech classification tasks, the cross-entropy loss function is the most commonly used optimization objective. It trains the classifier by minimizing the difference between the predicted probability of the true class and the target distribution. However, it also has some obvious limitations: (1) Lack of sufficient inter-class discriminability: Cross-entropy loss mainly focuses on improving the accuracy of prediction, but does not explicitly encourage features of different classes to maintain a sufficiently large distance in the feature space. Therefore, for highly similar classes, the model may produce blurring at the inter-class boundaries and cannot distinguish features of different classes well. (2) Insufficient aggregation of intra-class features: Cross-entropy loss does not impose sufficient constraints on the feature vectors of samples of the same class, resulting in samples of the same class being relatively scattered in the feature space, affecting the accuracy of classification.

[0068] To address the aforementioned shortcomings of cross-entropy loss, several angle-margin-based loss functions have been proposed in recent years to improve the model's discriminative ability. Among them, AAM-Softmax (Additive Angular Margin Softmax) loss explicitly enhances inter-class distance by introducing an angle margin m, forcing a greater angular separation between different classes, while simultaneously promoting tighter aggregation of intra-class samples.

[0069] However, directly combining AAM-Softmax with the cross-entropy loss function may introduce instability in the early stages of model training. Therefore, to ensure the stability of the model in the early stages of training while gradually improving inter-class discrimination ability, a gradual fusion strategy is adopted, integrating AAM-Softmax into the cross-entropy loss function progressively. This gradual fusion method avoids the model being overly affected by angular intervals in the early stages, thus maintaining the smoothness of training.

[0070] Specifically, in process S56, a stepwise fusion strategy is adopted to integrate the angle-interval-based loss function into the cross-entropy loss function. The specific process is as follows: in the early stage of training, the cross-entropy loss function L is used as the basis for the loss function. CE To optimize this, β(t)≈0, and the formula is:

[0071]

[0072] As the model gradually converges, β(t) begins to increase, and a loss function L based on the angle interval is gradually introduced. AAM The angular interval in the middle is given by the formula:

[0073] L fuse (t)=L CE +β(t)·L AAM ;

[0074] In the later stages of training, β(t) grows to a relatively high value close to 1, at which point the loss function L based on the angular interval... AAM As the cross-entropy loss becomes dominant, its influence gradually decreases, and the formula is:

[0075]

[0076] Where N is the number of samples, C is the number of categories, and x i W is the feature of the i-th input sample. j It is the weight of the classification layer, y i is the correct class, m is the increased angular margin, which aims to force a larger distance between different classes; s is an adjustable scaling factor, which aims to enhance the stability of the values ​​during training. The angle between the i-th sample and its correct class weight is represented by t, where t is the training step size, and β(t) is the weight coefficient that gradually changes with the training step size t.

[0077] Furthermore, to improve the training efficiency and generalization ability of the model, during the training process, the ADMA optimizer is used to gradually fuse the loss function and adjust the learning rate using a cosine annealing learning rate reduction strategy combined with warmup.

[0078] Specifically, in process S56, an adaptive learning rate adjustment strategy is adopted to improve the training efficiency and generalization ability of the model. The specific operations are as follows: in the early stage of model training, the learning rate is gradually increased through a warmup strategy, and the Adam optimizer is used to accelerate the convergence of the model; the batch size is set to 64; during the model training process, the system dynamically adjusts the number of training epochs according to the changes in the loss function; in the later stage of training, the learning rate is gradually reduced through cosine annealing for fine-tuning.

[0079] In this embodiment, step S6 verifies the trained bird sound recognition model. It should be understood that model verification is a crucial step in ensuring the effectiveness, reliability, and impartiality of the AI ​​model in practical applications. This step improves the model's generalization ability, ensures its performance in new environments, and enhances the trust and acceptance of the model among all stakeholders.

[0080] Specifically, in step S6, the following evaluation metrics are used to compare and evaluate the performance of each model in this method:

[0081]

[0082]

[0083] In this context, TP represents a true positive sample, TN represents a true negative sample, FP represents a false positive sample, and FN represents a false negative sample.

[0084] In this embodiment, step S7 involves inputting the valid bird call data into the bird call recognition model for label prediction to obtain the recognition result. It should be understood that by inputting valid bird call data into a trained model, the label of the bird call can be accurately predicted. This automated bird call recognition model can significantly improve the efficiency and accuracy of bird monitoring, reduce the workload of manual sound identification, and make bird ecology and environmental monitoring more efficient.

[0085] Specifically, step S7 uses two strategies to determine the correct recognition result: The first strategy uses a bird sound recognition model output probability of 0.7 as the threshold for correct recognition; that is, audio samples with an output probability greater than or equal to 0.7 are considered correctly recognized samples. The second strategy, based on the first strategy, further employs a "continuous enhancement recognition strategy" to post-process the model recognition result. Specifically, the second strategy includes: when a bird is recognized, if the same bird sound is recognized within 30 seconds before and after, but its model output probability is less than 0.7 and greater than 0.5, then this is also output as a correctly recognized sample.

[0086] Specifically, in this embodiment, constructing a bird sound recognition framework (i.e., the bird sound event detection model and the bird sound recognition model) is also crucial. During the construction of the bird sound event detection model and the bird sound recognition model, the spectrograms from the bird sound event detection dataset and the bird sound recognition dataset are used as inputs, respectively, and the ECAPA-TDNN model is selected as the model's basic architecture. The specific process is as follows:

[0087] First, considering that bird sound audio data is continuous, directly using the ECAPA-TDNN model for multi-scale feature extraction may result in the model failing to fully capture subtle features across different frequency bands, thus affecting recognition accuracy. Frequency band segmentation, however, divides the spectrogram into a set of non-overlapping frequency bands, which, after entering a fully connected layer, generate a series of sub-bands with different frequency band characteristics. Therefore, before inputting the ECAPA-TDNN model, frequency band segmentation preprocessing is performed on the spectrograms from both the bird sound event detection dataset and the bird sound recognition dataset. By dividing the input spectrogram into multiple sub-bands and processing the features of each sub-band individually, this method can more accurately capture detailed information within each frequency band, thereby improving the model's performance in recognizing bird sounds. This approach helps overcome potential shortcomings of the original ECAPA-TDNN model in processing frequency information.

[0088] Specifically, Figure 5 The illustration shows a schematic diagram of the mid-frequency band segmentation module in a large-scale bird call recognition method based on deep learning technology according to an embodiment of this application. Figure 5 As shown, the specific segmentation scheme is as follows: the spectrogram is divided into a set of non-overlapping frequency bands, which are then processed through fully connected layers to generate a series of sub-bands Z with different frequency band characteristics, including 20 low-frequency sub-bands with a bandwidth of 200Hz, 8 sub-bands with a bandwidth of 500Hz, and 4 high-frequency sub-bands with a bandwidth of 2kHz. Then, the merged sub-band features are subjected to L2 normalization, and finally, two fully connected layers are connected for feature mapping to increase the complex correlation between features.

[0089] However, in the process of frequency band feature extraction, the spectrogram generates high-dimensional feature matrices after frequency band segmentation. These matrices contain a large number of invalid or near-zero elements, especially in the high-frequency band, where the effective information of the features is highly sparsity. To address this issue, an acceleration strategy based on sparse matrices is proposed to significantly improve the computational efficiency and resource utilization of the system.

[0090] Specifically, in the feature matrix after frequency band segmentation, sparse matrices can be used to represent the non-zero feature elements. During matrix multiplication, only these non-zero elements are calculated, avoiding unnecessary operations and thus speeding up processing. Furthermore, using COO (Coordinate Format) to store the sparse matrix significantly reduces the storage space required for the feature matrix, especially when processing large amounts of bird sound data, resulting in a substantial reduction in memory usage.

[0091] Next, the ECAPA-TDNN model is used to capture speech features from the spectrogram. Specifically, Figure 6 The diagram illustrates the network structures of SE-Res2Block and ECAPA-TDNN in a large-scale bird call recognition method based on deep learning technology according to an embodiment of this application. Figure 6As shown, this embodiment employs the SE-Res2Block and ECAPA-TDNN network models. The ECAPA-TDNN model is a Time Delay Neural Network (TDNN) based neural network, one of the best single-unit speaker recognition models currently available. Building upon TDNN, it captures contextual information at different scales by using dilated convolutions and incorporating a Res2Net structure. Furthermore, it integrates an SENet module, enabling the network to automatically learn and adjust the weights of each feature channel, thereby extracting more refined speech features. This gives ECAPA-TDNN a significant advantage over other networks such as ResNet and MobileNet in recognizing subtle differences in various bird calls.

[0092] SE-Res2Block is a key component of the ECAPA-TDNN model, combining features of Res2Net and SENet. In this architecture, a dense layer precedes and follows the dilated convolutional layer, processing contextual information from a single frame. The first dense layer reduces the dimensionality of the features, while the second restores the feature count to its original level. Subsequently, an SE block adjusts the weights for each channel. The entire structure enhances information flow through skip connections. The Res2Net module enhances the processing power of the central convolutional layer through internal hierarchical residual connections, enabling it to effectively handle multi-scale features. This design not only improves model performance but also significantly reduces the number of parameters required by the model.

[0093] Finally, after capturing audio features using the ECAPA-TDNN model, a fully connected layer is appended to the output of the ECAPA-TDNN model to map the feature space to the category space. Specifically, the output of the final fully connected layer of the bird sound event detection model has two nodes, whose output values, after being processed by a softmax function, represent the probabilities of bird sound samples and noise samples, respectively. The output of the final fully connected layer of the bird sound recognition model has 179 nodes, whose output values, after being processed by a softmax function, represent the probabilities of 179 different bird species.

[0094] That is, taking the construction of the bird sound recognition model as an example, Figure 7 The illustration shows an overall schematic diagram of the bird sound recognition model in the large-scale bird sound recognition method based on deep learning technology according to an embodiment of this application. Figure 7As shown, the specific model construction process is as follows: Frequency band segmentation preprocessing is performed on the spectrograms in the bird sound recognition dataset to generate sub-bands Z with different frequency band features; L2 normalization is performed on the segmented frequency band features, and fully connected layers are connected to increase the complex correlation between features; a sparse matrix-based acceleration strategy is adopted, using the COO (Coordinate Format) storage format, and matrix multiplication is performed on the non-zero elements in the sparse matrix to improve computational efficiency and resource utilization; the ECAPA-TDNN model is used to capture the speech features of the spectrograms, where the ECAPA-TDNN model combines dilated convolution, Res2Net structure, and SENet module; a fully connected layer is connected after the output of the ECAPA-TDNN model to map the feature space to the category space. The fully connected layer of the bird sound event detection model outputs two nodes, whose output values, after passing through the softmax function, represent the probabilities of bird sound samples and noise samples, respectively. The fully connected layer of the bird sound recognition model outputs 179 nodes, whose output values, after passing through the softmax function, represent the probabilities of 179 bird species, respectively.

[0095] In summary, the large-scale bird sound recognition method based on deep learning technology described in the embodiments of this application is clarified. This method collects bird sound audio using recording equipment in the project area, trains a bird sound event detection model using the Bird audio detection challenge 2018 dataset, and performs category prediction using field recording data to separate valid bird sounds from background noise. Simultaneously, it combines the bird species catalog from the China Birdwatching Recording Center and audio files from Xeno-Canto to construct a bird sound recognition dataset. The bird sound recognition model is trained using the background noise and bird sound recognition dataset and then validated. Finally, the valid bird sound data is input into the model for label prediction to identify bird sounds. This invention aims to conduct extensive bird sound recognition in project areas under construction using deep learning technology, thereby saving data processing costs and improving recognition accuracy in complex scenarios.

[0096] Example 2:

[0097] Figure 8 The illustration shows a schematic diagram of a large-scale bird sound recognition system based on deep learning technology according to an embodiment of this application. Figure 8As shown, the large-scale bird sound recognition system 10 based on deep learning technology according to an embodiment of this application includes: a bird sound audio acquisition module 110, used to acquire bird sound audio near the project area using a project area recording device to obtain field recording data; and a bird sound event detection model training module 120, used to train a model from the Birdaudio detection challenge 2018-DCASE Task. The three datasets obtained in step 3 serve as bird sound event detection datasets for training the bird sound event detection model. The bird sound category prediction module 130 inputs the field recording data into the trained bird sound event detection model for category prediction, obtaining valid bird sound data and background noise data. The bird sound recognition dataset acquisition module 140 obtains a catalog of bird species recorded in the project area from the China Bird Watching Recording Center, downloads corresponding bird species audio files from Xeno-Canto, and performs data preprocessing on the audio files to obtain the bird sound recognition dataset. The bird sound recognition model training module 150 uses the background noise data and the bird sound recognition dataset to train the bird sound recognition model. The model validation module 160 validates the trained bird sound recognition model. The label prediction module 170 inputs the valid bird sound data into the bird sound recognition model for label prediction, obtaining the recognition result.

[0098] Here, those skilled in the art will understand that the specific functions and operations of each step in the aforementioned large-scale bird sound recognition system 10 based on deep learning technology have been referenced above. Figure 2 The large-scale bird sound recognition method based on deep learning technology has been described in detail, and therefore its repeated description will be omitted.

[0099] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit of the technical solutions of the present invention.

Claims

1. A large-scale bird sound recognition method based on deep learning technology, characterized in that, The method comprises the following steps: S1, using a project area recording device to collect bird sound audio near the project area to obtain field recording data; S2, using three data sets obtained from the bird sound event detection data set of the Bird audio detection challenge 2018-DCASE Task3 as the bird sound event detection data set for training the bird sound event detection model; S3, inputting the field recording data into the trained bird sound event detection model for category prediction to obtain effective bird sound data and background noise data; S4, obtaining a bird species list recorded in the project area from the China Birdwatching Recording Center, downloading audio files of corresponding bird species in Xeno-Canto, and performing data preprocessing operations on the audio files to obtain a bird sound recognition data set; S5, training a bird sound recognition model using the background noise data and the bird sound recognition data set; S6, verifying the trained bird sound recognition model; S7, inputting the effective bird sound data into the bird sound recognition model for label prediction to obtain a recognition result; In the construction process of the bird sound event detection model and the bird sound recognition model, the spectrograms in the bird sound event detection data set and the spectrograms in the bird sound recognition data set are respectively taken as inputs, and an ECAPA-TDNN model is selected as a model basic architecture, and the specific model construction process is as follows: The spectrograms in the bird sound event detection data set and the spectrograms in the bird sound recognition data set are respectively subjected to frequency band segmentation preprocessing to generate subbands Z with different frequency band features; The segmented frequency band features are subjected to L2 normalization processing, and a fully connected layer is connected to increase the complex correlation between features; An acceleration strategy based on a sparse matrix is adopted, a COO storage format is used, and matrix multiplication operation is performed on non-zero elements in the sparse matrix to improve calculation efficiency and resource utilization; An ECAPA-TDNN model is used to capture speech features of the spectrograms, wherein the ECAPA-TDNN model combines an expanded convolution, a Res2Net structure and an SE-Net module; A fully connected layer is connected after the output of the ECAPA-TDNN model to map the feature space to the category space; The fully connected layer of the bird sound event detection model outputs 2 nodes, and the values output by the nodes respectively represent the probabilities of bird sound samples and noise samples after being subjected to a softmax function, and the fully connected layer of the bird sound recognition model outputs 179 nodes, and the values output by the nodes respectively represent the probabilities of 179 bird species after being subjected to a softmax function. 2.The method of claim 1, wherein, The S4 comprises: S41, manually checking spectrograms and waveform graphs of the audio files to filter out low-quality and incorrectly labeled audios; S42, cutting long audios every 30 seconds, and if the last piece of audio is less than 30 seconds, it is retained if it is greater than 15 seconds, and discarded if it is less than 15 seconds; S43, deleting original audio segments with a length less than 2 seconds. 3.The method of claim 2, wherein, The S5 comprises: S51, unify the sampling rate of the acoustic signals in the bird sound recognition data set to 32,000 kHz, and convert the audio format to a high-frequency lossless wav format to obtain preprocessed audio data; S52, randomly slice the preprocessed audio data, and unify the slice length to 10s; S53, normalize the amplitude of the sliced audio data; S54, mix the background noise data into the normalized audio data with a probability of 50% to obtain audio sample data; S55, perform spectral enhancement on the log mel spectrogram of the audio sample data, including randomly masking 0-5 frames in the time domain and 0-10 channels in the frequency domain; S56, use speech speed disturbance, volume enhancement, and Gaussian noise addition technology to train the model. 4.The method of claim 3, wherein, In the S56 process, a step-by-step fusion strategy is used to integrate the angle interval-based loss function into the cross-entropy loss function, and the specific process is as follows: In the initial stage of training, the cross-entropy loss function is relied on for optimization, at which time The formula is:​ As the model converges, The start is gradually increased, introducing the angle interval-based loss function in steps, whose formula is: In the later stages of training, grows to higher values close to 1, at which point the loss function based on angular separation dominates, and the influence of the cross-entropy loss diminishes, which is given by: in, It is the number of samples. It is the number of categories. It is the first Features of each input sample These are the weights of the classification layer. It is the correct category. It is an increase in angular interval. It is an adjustable scaling factor. This represents the angle between the i-th sample and its correct class weight. It's about training stride length. To follow the training step length Gradually changing weighting coefficients. 5.The method of claim 4, wherein, In the S56 process, an adaptive learning rate adjustment strategy is used to improve the training efficiency and generalization ability of the model, and the specific operation is as follows: In the early stage of model training, the learning rate is gradually increased through the warmup strategy, and the Adam optimizer is used to accelerate the convergence of the model; Set the batch size to 64; During the model training process, the system dynamically adjusts the number of training epochs according to the change of the loss function; In the later stage of training, the learning rate is gradually reduced in a cosine annealing manner for fine tuning. 6.The method of claim 5, wherein, In order to compare and evaluate the performance of each model in the method, the following evaluation indicators are used: wherein, TP represents true positive samples, TN represents true negative samples, FP represents false positive samples, FN represents false negative samples. 7.The method of claim 6, wherein, In the S7, two strategies are used to determine the correct recognition result, which are: The first strategy takes the bird sound recognition model output probability 0.7 as the threshold for judging the correct, that is, the audio sample with output probability greater than or equal to 0.7 is taken as the model correct recognition sample; The second strategy further adopts the "continuous enhancement recognition strategy" to post-process the model recognition result based on the first strategy. 8.The method of claim 7, wherein, The second strategy includes: When a bird species is recognized, if the same bird sound is recognized within 30s before and after, but the model output probability is less than 0.7 and greater than 0.5, it is also taken as a correct recognition sample.

9. A large-scale bird sound recognition system based on deep learning technology, characterized by, It includes: A bird sound audio acquisition module for acquiring bird sound audio near the project area using project area recording equipment to obtain field recording data; A bird sound event detection model training module for using three data sets obtained from the Bird audio detection challenge 2018 - DCASE Task 3 competition as bird sound event detection data sets to train a bird sound event detection model; A bird sound category prediction module for inputting the field recording data into the trained bird sound event detection model for category prediction to obtain effective bird sound data and background noise data; The bird sound recognition dataset obtaining module is configured to obtain a bird species list recorded in the project area from the China Birdwatching Recording Center, download audio files of corresponding bird species in Xeno-Canto, and perform data preprocessing operations on the audio files to obtain a bird sound recognition dataset; The bird sound recognition model training module is configured to train a bird sound recognition model using the background noise data and the bird sound recognition dataset; The model verification module is configured to verify the trained bird sound recognition model. The label prediction module is configured to input the effective bird sound data into the bird sound recognition model for label prediction to obtain a recognition result. The bird sound event detection model and the bird sound recognition model are constructed based on the method of any one of claims 1 to 8.

Images