Large-scale bird sound recognition method based on deep learning technology

Abstract

The invention relates to the crossing field of artificial intelligence and bird ecology, and particularly discloses a large-scale bird sound recognition method based on a deep learning technology, and the method comprises the steps: collecting a bird sound audio through a project region recording device, training a bird sound event detection model through a Bird audio detection challenge 2018 data set, carrying out the class prediction through field recording data, and carrying out the recognition of a bird sound event. Effective bird sound and background noise are separated; constructing a bird sound recognition data set in combination with a bird species catalog of the Chinese bird watching recording center and an audio file of Xen-Canto, training a bird sound recognition model by using background noise and the bird sound recognition data set, and performing verification; and finally, inputting the effective bird sound data into the model to carry out label prediction so as to identify the bird sound. The invention aims to carry out wide bird voice recognition on a project area under construction through a deep learning technology, so that the data processing cost is saved, and the recognition accuracy in a complex scene is improved.

Description

Technical Field

[0001] The present application relates to the intersection of artificial intelligence and bird ecology, and more specifically, to a large-scale bird sound recognition method based on deep learning technology. Background Art

[0002] In the field of bird ecology and environmental monitoring, experts have been exploring how to monitor and study birds more effectively. Active sound monitoring is a traditional method that captures bird sounds through manual recordings, but this method is costly and may affect the natural behavior of birds. With technological advances, passive sound monitoring technology has become popular. It automatically records sounds by installing sensors, reducing human bias, but requires a lot of manual analysis and identification later.

[0003] Many researchers have introduced artificial intelligence (AI) into passive monitoring, especially deep learning algorithms, which has brought revolutionary changes to this field. AI can process large amounts of data, improve monitoring efficiency, and reduce manpower requirements. For example, the AI ​​model based on the DenseNet121 network structure achieved a high accuracy rate of 96.9% in bird sound recognition.

[0004] However, most current algorithms can only identify limited bird sound audio in a given data set, but their comprehensive recognition capabilities in noisy natural environments involving multiple bird species are not ideal. This is not only due to the lack of robustness of the algorithm itself, but also because of the lack of sufficiently diverse and reliable bird sound recordings to train AI models.

[0005] In addition, most current AI research focuses on identifying bird species, while there is less research on the relationship between bird vocalization behavior and the ecological environment. In practical applications, AI's recognition ability in complex acoustic scenes is insufficient, limiting its application in long-term, large-scale biodiversity research.

[0006] Therefore, a large-scale bird sound recognition method based on deep learning technology is provided. Summary of the invention

[0007] In order to solve the above technical problems, this application is proposed. This application provides a large-scale bird sound recognition method based on deep learning technology, which aims to carry out extensive bird sound monitoring in the project area under construction through deep learning technology. It builds a complete innovative bird sound recognition framework based on deep learning technology based on spectrum segmentation technology and ECAPA-TDNN model.

[0008] Specifically, according to one aspect of the present application, a large-scale bird sound recognition method based on deep learning technology is provided, which includes:

[0009] S1. Use recording equipment in the project area to collect bird audio near the project area to obtain field recording data;

[0010] S2. Use three datasets obtained from the Bird audio detection challenge 2018-DCASETask 3 as bird sound event detection datasets to train the bird sound event detection model.

[0011] S3, inputting the field recording data into the trained bird sound event detection model for category prediction to obtain effective bird sound data and background noise data;

[0012] S4. Obtain a catalog of bird species recorded in the project area from the China Bird Watching Record Center, download audio files of corresponding bird species in Xeno-Canto, and perform data preprocessing on the audio files to obtain a bird sound recognition dataset;

[0013] S5. Using the background noise data and the bird sound recognition data set to train a bird sound recognition model;

[0014] S6. Verify the trained bird sound recognition model;

[0015] S7, inputting the valid bird sound data into the bird sound recognition model to perform label prediction to obtain a recognition result;

[0016] In the process of constructing the bird sound event detection model and the bird sound recognition model, the spectrogram in the bird sound event detection dataset and the spectrogram in the bird sound recognition dataset are respectively used as input, and the ECAPA-TDNN model is selected as the model infrastructure. The specific model construction process is:

[0017] The frequency spectrograms in the bird sound event detection dataset and the frequency spectrograms in the bird sound recognition dataset are preprocessed by frequency band segmentation to generate sub-bands Z with different frequency band characteristics;

[0018] The segmented frequency band features are L2 normalized and connected to a fully connected layer to increase the complex correlation between features;

[0019] Adopt an acceleration strategy based on sparse matrices, use the COO (Coordinate Format) storage format, and perform matrix multiplication operations on non-zero elements in sparse matrices to improve computing efficiency and resource utilization;

[0020] Use the ECAPA-TDNN model to capture speech features of the spectrogram, where the ECAPA-TDNN model combines dilated convolution, Res2Net structure, and SENet module;

[0021] A fully connected layer is added after the output of the ECAPA-TDNN model to map the feature space to the category space;

[0022] Among them, the fully connected layer of the bird sound event detection model outputs 2 nodes, and its output values ​​represent the probabilities of bird sound samples and noise samples respectively after passing through the softmax function. The fully connected layer of the bird sound recognition model outputs 179 nodes after passing through the softmax function, which represent the probabilities of 179 species of birds respectively.

[0023] Optionally, the S4 includes: S41, manually checking the spectrogram and waveform of the audio file to filter out low-quality and incorrectly labeled audio; S42, cutting the long audio every 30 seconds. If the last audio segment is less than 30 seconds, it will be retained if it is greater than 15 seconds, and discarded if it is less than 15 seconds; S43, deleting original audio segments with a length of less than 2 seconds.

[0024] Optionally, the S5 includes: S51, unifying the sampling rate of the acoustic signal in the bird sound recognition data set to 32,000kHz, and converting the audio format to a wav format without high-frequency loss to obtain preprocessed audio data; S52, randomly slicing the preprocessed audio data, and the slice length is unified to 10s; S53, normalizing the amplitude of the sliced ​​audio data; S54, mixing the background noise data into the normalized audio data with a probability of 50% to obtain audio sample data; S55, spectrally enhancing the logarithmic Mel spectrogram of the audio sample data, including randomly masking 0 to 5 frames in the time domain and randomly masking 0 to 10 channels in the frequency domain; S56, using techniques such as speech rate perturbation, volume enhancement and adding Gaussian noise to train the model.

[0025] In particular, in the S56 process, a step-by-step fusion strategy is adopted to integrate the loss function based on the angle interval into the cross entropy loss function. The specific process is: in the early stage of training, the cross entropy loss function L is relied on CE To optimize, at this time β(t)≈0, the formula is:

[0026]

[0027] As the model gradually converges, β(t) begins to increase gradually, and the loss function L based on the angle interval is gradually introduced. AAM The angle interval in is given by:

[0028] L fuse (t) = L CE +β(t)·L AAM ;

[0029] In the later stages of training, β(t) grows to a higher value close to 1, at which point the loss function L based on the angle interval AAM It dominates, and the influence of cross entropy loss gradually decreases, and its formula is:

[0030]

[0031] Where N is the number of samples, C is the number of categories, and x i is the feature of the i-th input sample, W j is the weight of the classification layer, y i is the correct category, m is the incremental angular interval, s is an adjustable scale factor, Represents the angle between the i-th sample and its correct category weight, t is the training step length, and β(t) is the weight coefficient that gradually changes with the training step length t.

[0032] In particular, in the S56 process, an adaptive learning rate adjustment strategy is used to improve the training efficiency and generalization ability of the model. The specific operations are: in the early stage of model training, the learning rate is gradually increased through the warmup strategy, and the Adam optimizer is used to accelerate the convergence of the model; the batch size is set to 64; during the model training process, the system dynamically adjusts the number of training epochs according to the changes in the loss function; in the later stage of training, the learning rate is gradually reduced by cosine annealing for fine tuning.

[0033] Optionally, in S6, in order to compare and evaluate the performance of each model in this method, the following evaluation indicators are used:

[0034]

[0035] Among them, TP represents true positive samples, TN represents true negative samples, FP represents false positive samples, and FN represents false negative samples.

[0036] Optionally, S7 uses two strategies to determine the correct recognition result, namely: the first strategy takes the bird sound recognition model output probability of 0.7 as the correct judgment threshold, that is, the audio sample with an output probability greater than or equal to 0.7 is used as the model correct recognition sample; the second strategy, based on the first strategy, further adopts the "continuous enhancement recognition strategy" to post-process the model recognition result. In particular, the second strategy includes: when a bird is recognized, if the same bird sound is recognized within 30 seconds before and after but its model output probability is less than 0.7 and greater than 0.5, it is also output as a correct recognition sample.

[0037] Specifically, according to another aspect of the present application, a large-scale bird sound recognition system based on deep learning technology is provided, which includes:

[0038] The bird sound audio collection module is used to collect bird sound audio near the project area using the project area recording equipment to obtain field recording data;

[0039] The bird sound event detection model training module is used to use the three data sets obtained from the bird sound event detection competition Bird audiodetection challenge 2018-DCASE Task 3 as bird sound event detection data sets for training the bird sound event detection model;

[0040] A bird sound category prediction module is used to input the field recording data into the trained bird sound event detection model for category prediction to obtain effective bird sound data and background noise data;

[0041] The bird sound recognition data set acquisition module is used to obtain the bird species catalog recorded in the project area from the China Bird Watching Record Center, download the audio files of the corresponding bird species in Xeno-Canto, and perform data preprocessing operations on the audio files to obtain the bird sound recognition data set;

[0042] A bird sound recognition model training module, used to train a bird sound recognition model using the background noise data and the bird sound recognition data set;

[0043] Model verification module, used to verify the trained bird sound recognition model;

[0044] The label prediction module is used to input the valid bird sound data into the bird sound recognition model to perform label prediction to obtain a recognition result.

[0045] Compared with the prior art, the present application provides a large-scale bird sound recognition method based on deep learning technology. In the high-dimensional feature matrix generated after the spectrum graph is divided into frequency bands, the method adopts a sparse matrix-based acceleration strategy to reduce the storage requirements of the feature matrix, and adopts an adaptive learning rate adjustment strategy during the model training process, which can improve the operation speed and significantly reduce memory usage, thereby saving data processing costs. In addition, the method selects the ECAPA-TDNN model as the basic architecture. The ECAPA-TDNN model uses dilated convolution and Res2Net to obtain multi-scale context information, and the SE-Res2Block model in the ECAPA-TDNN model combines Res2Net with SENet. This structure helps to process multi-scale features, reduces the number of model parameters, and can improve the accuracy of bird sound recognition in complex scenarios. And it is applied to bird activity and behavior analysis to evaluate the impact of engineering on the ecological environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0047] Figure 1 The figure illustrates a visual flowchart of a large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application.

[0048] Figure 2 The figure illustrates a flow chart of a large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application.

[0049] Figure 3 The figure illustrates a flowchart of step S4 in the large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application.

[0050] Figure 4 The figure illustrates a flowchart of step S5 in the large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application.

[0051] Figure 5 The figure illustrates a schematic diagram of a frequency band segmentation module in a large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application.

[0052] Figure 6 The diagram illustrates the network structure of SE-Res2Block and ECAPA-TDNN in the large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application.

[0053] Figure 7 The figure illustrates an overall schematic diagram of a bird sound recognition model in a large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application.

[0054] Figure 8 A schematic diagram of a large-scale bird sound recognition system based on deep learning technology according to an embodiment of the present application is illustrated. DETAILED DESCRIPTION

[0055] Below, the embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0056] Embodiment 1:

[0057] Figure 1The figure illustrates a visual flowchart of a large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application. Figure 1 The entire process of the large-scale bird sound recognition method based on deep learning technology in the embodiment of the present application is intuitively demonstrated, from data collection to the final recognition result. This graphical method can help users quickly understand the entire process of bird sound recognition, including data collection, processing, model training and recognition steps.

[0058] Figure 2 The flowchart of the method for large-scale bird sound recognition based on deep learning technology according to an embodiment of the present application is shown. Figure 1 As shown, according to the large-scale bird sound recognition method based on deep learning technology in an embodiment of the present application, the method includes: S1, using the project area recording equipment to collect bird sound audio near the project area to obtain field recording data; S2, using three data sets obtained from the bird sound event detection competition Bird audio detection challenge 2018-DCASE Task 3 as bird sound event detection data sets, for training the bird sound event detection model; S3, inputting the field recording data into the trained bird sound event detection model for category prediction to obtain valid bird sound data and background noise data; S4, obtaining a directory of bird species recorded in the project area from the China Bird Watching Record Center, downloading audio files of corresponding bird species in Xeno-Canto, and performing data preprocessing operations on the audio files to obtain a bird sound recognition data set; S5, using the background noise data and the bird sound recognition data set to train the bird sound recognition model; S6, verifying the trained bird sound recognition model; S7, inputting the valid bird sound data into the bird sound recognition model for label prediction to obtain a recognition result.

[0059] In an embodiment of the present application, step S1 uses a project area recording device to collect bird sound audio near the project area to obtain field recording data. It should be understood that the bird sound data in the natural environment is highly diverse, including different types of birds, different calls, and interference from different environmental noises. At the same time, considering that field recordings can capture the real sounds of birds in the natural environment, including sounds in various natural backgrounds, this is crucial for training and testing AI models to identify different types of bird sounds. Therefore, in order to provide more realistic and diverse data in order to train a more accurate and robust AI model, before identifying and predicting bird sounds, the project area recording device is first used to collect bird sound audio near the project area to obtain field recording data. In this way, it helps the model better adapt to the complexity of the real world and improve its robustness in practical applications.

[0060] In an embodiment of the present application, in step S2, three data sets obtained from the bird sound event detection competition Bird audiodetection challenge 2018-DCASE Task 3 are used as bird sound event detection data sets for training the bird sound event detection model. It should be understood that the bird sound event detection model can automatically identify and distinguish bird sound events in audio (such as effective bird sounds, invalid background noise, etc.), which is a key step in automated bird sound monitoring. Considering that the competition data set usually contains a variety of different bird sounds and background noise, such diversity helps to train a model with strong robustness to noise and interference. Based on this, three data sets obtained from the bird sound event detection competition Bird audiodetection challenge 2018-DCASE Task 3 are used as data sets for training the bird sound event detection model. In this way, by training with data collected in a variety of environments, the model can better handle various noises and interferences that may be encountered in practical applications.

[0061] In the embodiment of the present application, the step S3 inputs the field recording data into the trained bird sound event detection model for category prediction to obtain effective bird sound data and background noise data. It should be understood that the effective bird sound and background noise in the field recording data can be distinguished through category prediction, and these two different data play an important role in the embodiment of the present application. Among them, the effective bird sound data contains actual bird sound events, which can be used for further bird sound recognition model training to identify and classify different birds. These data are of great value for studying bird behavior, population dynamics and biodiversity; and although the background noise data does not contain bird sounds, these data are equally important for model training, and they can help the model learn to distinguish the audio features of bird sounds and non-bird sounds, and improve the model's recognition ability in complex environments. Therefore, the field recording data is further predicted by the trained bird sound event detection model, so as to accurately screen out effective bird sound data.

[0062] In the embodiment of the present application, the step S4 obtains the bird species catalog recorded in the project area from the China Bird Watching Record Center, downloads the audio files of the corresponding bird species in Xeno-Canto, and performs data preprocessing operations on the audio files to obtain a bird sound recognition data set. It should be understood that the bird species catalog provided by the China Bird Watching Record Center has high scientificity and accuracy, and Xeno-Canto is an international platform with bird sound samples from all over the world, which provides a wide variety and reliability for the data set. Therefore, these data are used as data sets for bird sound recognition model training. In addition, considering that the audio annotations are added by the uploader himself, there may be inaccuracies, and verification is required to minimize the impact of wrong labels on the results; and most of the audio files are within a length of 2s to 5min. For longer audio files, they may contain a large amount of noise and silent parts, which occupy most of the audio duration, while the actual bird calls account for a small proportion, and long audio clips need to be manually edited to remove those long useless segments. Therefore, it is necessary to further preprocess these data sets.

[0063] Specifically, Figure 3 FIG. 4 is a flowchart of step S4 in a large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application. Figure 3 As shown, the step S4 includes: S41, manually checking the spectrum and waveform of the audio file to filter out low-quality and incorrectly labeled audio; S42, cutting the long audio every 30 seconds, if the last audio segment is less than 30 seconds, it is retained if it is greater than 15 seconds, and discarded if it is less than 15 seconds; S43, deleting the original audio segment with a length of less than 2 seconds.

[0064] In particular, this dataset construction method is highly versatile and extensible. Bird sound datasets in various complex areas can be constructed according to this method, and it has broad application prospects.

[0065] In the embodiment of the present application, the step S5 uses the background noise data and the bird sound recognition data set to train the bird sound recognition model. It should be understood that the background noise data is interference data in the real environment obtained from the field recording data, which can help the model learn to distinguish between bird sounds and environmental noise. The bird sound recognition data set contains a catalog of bird species recorded in the project area, which can provide the features required for the model to learn to recognize different bird sounds. Using the combined data of the two to train the bird sound recognition model can improve the model's recognition accuracy of bird sounds and better adapt to complex acoustic scenes. Therefore, the background noise data and the bird sound recognition data set are used to train the bird sound recognition model. However, since the annotations of the audio file are added by the uploader himself, there may be errors. And the differences in the signal-to-noise ratio and the sound collection environment (equipment / sampling rate / temperature / weather, etc.) between the training and the field measured recordings are likely to cause the impact of spatial domain mismatch. Based on this, before model training, all audio samples must be formatted, randomly sliced, normalized, and data enhanced.

[0066] Specifically, Figure 4 FIG. 1 is a flowchart of step S5 in a large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application. Figure 4 As shown, the step S5 includes: S51, unifying the sampling rate of the acoustic signal in the bird sound recognition data set to 32,000kHz, and converting the audio format to a wav format without high-frequency loss to obtain preprocessed audio data; S52, randomly slicing the preprocessed audio data, and the slice length is unified to 10s; S53, normalizing the amplitude of the sliced ​​audio data; S54, mixing the background noise data into the normalized audio data with a probability of 50% to obtain audio sample data; S55, spectrally enhancing the logarithmic Mel spectrogram of the audio sample data, including randomly masking 0 to 5 frames in the time domain and randomly masking 0 to 10 channels in the frequency domain; S56, using speech rate perturbation, volume enhancement, and adding Gaussian noise to train the model.

[0067] In particular, it should be understood that the loss function is used to evaluate the degree of error between the model's prediction results and the actual results. In the multi-category speech classification task, the cross-entropy loss function is the most commonly used optimization target. It trains the classifier by minimizing the difference between the predicted probability of the true category and the target distribution. However, it also has some obvious limitations: (1) Lack of sufficient inter-class discrimination: The cross-entropy loss focuses on improving the accuracy of the prediction, but does not explicitly encourage features of different categories to maintain a sufficiently large distance in the feature space. Therefore, for high-similarity categories, the model may blur the boundaries between categories and cannot distinguish features of different categories well. (2) The aggregation of intra-class features is not tight enough: The cross-entropy loss does not impose sufficient constraints on the feature vectors of samples of the same category, resulting in samples of the same category being more dispersed in the feature space, affecting the accuracy of classification.

[0068] In order to deal with the above shortcomings of cross entropy loss, some loss functions based on angular margin have been proposed in recent years to improve the discriminative ability of the model. Among them, AAM-Softmax (Additive Angular Margin Softmax) loss explicitly enhances the inter-class distance by introducing the angular margin m, forcing different classes to have a larger angular separation, while promoting a tighter aggregation of samples within a class.

[0069] However, directly combining AAM-Softmax with the cross entropy loss function may introduce instability in the early training stage of the model. Therefore, in order to ensure the stability of the model in the early stage of training and gradually improve the ability to distinguish between classes, a gradual fusion strategy is adopted to gradually integrate AAM-Softmax into the cross entropy loss function. This gradual fusion method avoids the model being too affected by the angle interval in the early stage, thereby maintaining the stability of training.

[0070] Specifically, in the S56 process, a step-by-step fusion strategy is adopted to integrate the loss function based on the angle interval into the cross entropy loss function. The specific process is: in the early stage of training, the cross entropy loss function L is relied on. CE To optimize, at this time β(t)≈0, the formula is:

[0071]

[0072] As the model gradually converges, β(t) begins to increase gradually, and the loss function L based on the angle interval is gradually introduced. AAM The angle interval in is given by:

[0073] L fuse (t) = L CE +β(t)·L AAM ;

[0074] In the later stages of training, β(t) grows to a higher value close to 1, at which point the loss function L based on the angle interval AAM It dominates, and the influence of cross entropy loss gradually decreases, and its formula is:

[0075]

[0076] Where N is the number of samples, C is the number of categories, and x i is the feature of the i-th input sample, W j is the weight of the classification layer, y i is the correct category, m is the increased angular margin, the purpose of which is to force a larger distance between different categories; s is an adjustable scale factor, the purpose of which is to enhance the stability of the value during training; Represents the angle between the i-th sample and its correct category weight, t is the training step length, and β(t) is the weight coefficient that gradually changes with the training step length t.

[0077] Furthermore, in order to improve the training efficiency and generalization ability of the model, during the training process, the Adma optimizer is used to gradually fuse the loss function and adjust the learning rate by combining the cosine annealing learning rate reduction strategy with warmup.

[0078] Specifically, in the S56 process, an adaptive learning rate adjustment strategy is used to improve the training efficiency and generalization ability of the model. The specific operations are: in the early stage of model training, the learning rate is gradually increased through the warmup strategy, and the Adam optimizer is used to accelerate the convergence of the model; the batch size is set to 64; during the model training process, the system dynamically adjusts the number of training epochs according to the changes in the loss function; in the later stage of training, the learning rate is gradually reduced by cosine annealing for fine tuning.

[0079] In the embodiment of the present application, the step S6 verifies the trained bird sound recognition model. It should be understood that model verification is a key step to ensure that the AI ​​model is effective, reliable and fair in practical applications. Through this step, the generalization ability of the model can be improved, its performance in new environments can be ensured, and the trust and acceptance of the model by all stakeholders can be enhanced.

[0080] Specifically, in step S6, in order to compare and evaluate the performance of each model in this method, the following evaluation indicators are used:

[0081]

[0082]

[0083] Among them, TP represents true positive samples, TN represents true negative samples, FP represents false positive samples, and FN represents false negative samples.

[0084] In the embodiment of the present application, in step S7, the valid bird sound data is input into the bird sound recognition model for label prediction to obtain a recognition result. It should be understood that by inputting the valid bird sound data into the trained model, the label of the bird sound can be accurately predicted. This automated bird sound recognition model can greatly improve the efficiency and accuracy of bird monitoring, reduce the workload of manual listening identification, and make bird ecology and environmental monitoring more efficient.

[0085] Specifically, in step S7, two strategies are used to determine the correct recognition result, namely: the first strategy takes the bird sound recognition model output probability of 0.7 as the correct judgment threshold, that is, the audio sample with an output probability greater than or equal to 0.7 is used as the model correct recognition sample; the second strategy, based on the first strategy, further adopts the "continuous enhancement recognition strategy" to post-process the model recognition result. Among them, the second strategy includes: when a bird is recognized, if the same bird sound is recognized within 30 seconds before and after, but its model output probability is less than 0.7 and greater than 0.5, it is also output as a correct recognition sample.

[0086] In particular, in the embodiment of the present application, it is also crucial to construct a bird sound recognition framework (i.e., the bird sound event detection model and the bird sound recognition model). In the process of constructing the bird sound event detection model and the bird sound recognition model, the spectrogram in the bird sound event detection dataset and the spectrogram in the bird sound recognition dataset are respectively used as input, and the ECAPA-TDNN model is selected as the model infrastructure. The specific process is as follows:

[0087] First, considering that the bird sound audio data is continuous, directly using the ECAPA-TDNN model for multi-scale feature extraction may cause the model to fail to fully capture the subtle features of different frequency bands, thereby affecting the accuracy of recognition. Band segmentation can divide the spectrogram into a set of non-overlapping frequency bands, and generate a series of sub-bands with different frequency band characteristics after entering the fully connected layer. Therefore, before the input of the ECAPA-TDNN model, the spectrogram in the bird sound event detection dataset and the spectrogram in the bird sound recognition dataset are first pre-processed by band segmentation. In this way, by dividing the input spectrogram into multiple sub-bands and processing the features of each sub-band separately, this method can more accurately capture the detailed information in each frequency band, thereby improving the performance of the model in identifying bird sounds. This processing method helps to overcome the shortcomings that the original ECAPA-TDNN model may have in processing frequency information.

[0088] Specifically, Figure 5 FIG. 1 is a schematic diagram of a frequency band segmentation module in a large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application. Figure 5 As shown in the figure, the specific segmentation scheme is: the spectrum graph is divided into a set of non-overlapping frequency bands, and each of them is passed through a fully connected layer to generate a series of sub-bands Z with different frequency band characteristics, including 20 low-frequency sub-bands with a bandwidth of 200Hz, 8 sub-bands of 500Hz, and 4 high-frequency sub-bands of 2kHz. Then the merged sub-band features are L2 normalized, and finally connected to two fully connected layers for feature mapping to increase the complex correlation between features.

[0089] However, in the process of extracting features from frequency bands, the spectrum will generate high-dimensional feature matrices after frequency band segmentation. These matrices contain a large number of invalid or near-zero elements, especially in high-frequency bands, where the effective information of features is sparse. In order to solve this problem, an acceleration strategy based on sparse matrices is proposed to significantly improve the computational efficiency and resource utilization of the system.

[0090] Specifically, in the feature matrix after frequency band segmentation, sparse matrices can be used to represent those non-zero feature elements. When performing matrix multiplication, only these non-zero elements are calculated, which can avoid unnecessary operations and speed up the processing. In addition, COO (Coordinate Format) is used to store sparse matrices. This storage method can greatly reduce the storage space required for the feature matrix, especially when processing a large amount of bird sound data, which can significantly reduce the use of memory.

[0091] Next, the ECAPA-TDNN model is used to capture the speech features of the spectrogram. Specifically, Figure 6 The diagram shows the network structure of SE-Res2Block and ECAPA-TDNN in the large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application. Figure 6As shown, the embodiment of the present application adopts SE-Res2Block and ECAPA-TDNN network models, wherein the ECAPA-TDNN model is a neural network based on a time delay neural network (TDNN), and is currently one of the best single models for speaker recognition. Based on TDNN, it can capture contextual information of different scales by using dilated convolutions and introducing the Res2Net structure. In addition, it also integrates the SENet module, which enables the network to automatically learn and adjust the weights of each feature channel to extract more refined speech features. This makes ECAPA-TDNN more advantageous than other networks such as ResNet and MobileNet in recognizing the subtle differences in the sounds of multiple types of birds.

[0092] SE-Res2Block is a key component of the ECAPA-TDNN model, which combines the features of Res2Net and SENet. In this structure, there is a dense layer before and after the dilated convolution layer to process the information of 1 frame of context. The first dense layer is used to reduce the dimension of the features, while the second dense layer restores the number of features to the original level. Subsequently, an SE block adjusts the weights of each channel. The entire structure enhances the flow of information through skip connections. The Res2Net module enhances the processing power of the central convolution layer through internal hierarchical residual connections, enabling it to effectively process multi-scale features. This design not only improves the performance of the model, but also significantly reduces the number of parameters required by the model.

[0093] Finally, after the ECAPA-TDNN model is used to capture the audio features, a fully connected layer is added to the output of the ECAPA-TDNN model to map the feature space to the category space. The output of the last fully connected layer of the bird sound event detection model has 2 nodes, and the output values ​​represent the probabilities of the bird sound samples and the noise samples respectively after the softmax function. The output of the last fully connected layer of the bird sound recognition model has 179 nodes, and the output values ​​represent the probabilities of 179 species of birds respectively after the softmax function.

[0094] That is, taking the construction of the bird sound recognition model as an example, Figure 7 The figure shows the overall schematic diagram of the bird sound recognition model in the large-scale bird sound recognition method based on deep learning technology according to an embodiment of the present application. Figure 7As shown in the figure, the specific model construction process is as follows: the frequency spectrum in the bird sound recognition dataset is preprocessed by frequency band segmentation to generate sub-bands Z with different frequency band features; the frequency band features after segmentation are L2 normalized and connected to the fully connected layer to increase the complex correlation between features; an acceleration strategy based on sparse matrices is adopted, and the COO (Coordinate Format) storage format is used to perform matrix multiplication operations on non-zero elements in the sparse matrix to improve computational efficiency and resource utilization; the ECAPA-TDNN model is used to capture the speech features of the frequency spectrum, where the ECAPA-TDNN model combines dilated convolution, Res2Net structure and SENet module; a fully connected layer is connected to the output of the ECAPA-TDNN model to map the feature space to the category space, where the fully connected layer of the bird sound event detection model outputs 2 nodes, and the output values ​​thereof represent the probabilities of bird sound samples and noise samples respectively after the softmax function, and the fully connected layer of the bird sound recognition model outputs 179 nodes, and the output values ​​thereof represent the probabilities of 179 species of birds respectively after the softmax function.

[0095] In summary, the large-scale bird sound recognition method based on deep learning technology according to the embodiment of the present application is explained, which collects bird sound audio through recording equipment in the project area, trains a bird sound event detection model using the Bird audio detection challenge2018 data set, and uses field recording data for category prediction to separate effective bird sounds and background noise; at the same time, combining the bird species catalog of the China Bird Watching Recording Center and the audio files of Xeno-Canto, a bird sound recognition data set is constructed, and the bird sound recognition model is trained and verified using background noise and bird sound recognition data sets; finally, the effective bird sound data is input into the model for label prediction to identify the bird sound. The present invention aims to carry out extensive bird sound recognition in the project area under construction through deep learning technology, thereby saving data processing costs and improving recognition accuracy in complex scenarios.

[0096] Embodiment 2:

[0097] Figure 8 FIG. 1 is a schematic diagram of a large-scale bird sound recognition system based on deep learning technology according to an embodiment of the present application. Figure 8As shown, the large-scale bird sound recognition system 10 based on deep learning technology according to the embodiment of the present application includes: a bird sound audio collection module 110, which is used to use the project area recording equipment to collect bird sound audio near the project area to obtain field recording data; a bird sound event detection model training module 120, which is used to collect bird sound audio near the project area using the project area recording equipment ... The three data sets obtained in 3 are used as bird sound event detection data sets for training the bird sound event detection model; the bird sound category prediction module 130 is used to input the field recording data into the trained bird sound event detection model for category prediction to obtain effective bird sound data and background noise data; the bird sound recognition data set acquisition module 140 is used to obtain the bird species catalog recorded in the project area from the China Bird Watching Record Center, and download the audio files of the corresponding bird species in Xeno-Canto, and perform data preprocessing operations on the audio files to obtain the bird sound recognition data set; the bird sound recognition model training module 150 is used to use the background noise data and the bird sound recognition data set to train the bird sound recognition model; the model verification module 160 is used to verify the trained bird sound recognition model; the label prediction module 170 is used to input the effective bird sound data into the bird sound recognition model for label prediction to obtain the recognition result.

[0098] Here, those skilled in the art will appreciate that the specific functions and operations of each step in the large-scale bird sound recognition system 10 based on deep learning technology have been described in detail above. Figure 2 The description of the large-scale bird sound recognition method based on deep learning technology has been introduced in detail, and therefore, its repeated description will be omitted.

[0099] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit of the technical solution of the present invention.

Claims

1. A large-scale bird sound recognition method based on deep learning technology, characterized in that: include: S1. Use recording equipment in the project area to collect bird audio near the project area to obtain field recording data; S2. Three datasets obtained from the Bird audio detection challenge 2018-DCASE Task 3 are used as bird sound event detection datasets to train the bird sound event detection model. S3, inputting the field recording data into the trained bird sound event detection model for category prediction to obtain effective bird sound data and background noise data; S4. Obtain a catalog of bird species recorded in the project area from the China Bird Watching Record Center, download audio files of corresponding bird species in Xeno-Canto, and perform data preprocessing on the audio files to obtain a bird sound recognition dataset; S5. Using the background noise data and the bird sound recognition data set to train a bird sound recognition model; S6. Verify the trained bird sound recognition model; S7, inputting the valid bird sound data into the bird sound recognition model to perform label prediction to obtain a recognition result; In the process of constructing the bird sound event detection model and the bird sound recognition model, the spectrogram in the bird sound event detection dataset and the spectrogram in the bird sound recognition dataset are respectively used as input, and the ECAPA-TDNN model is selected as the model infrastructure. The specific model construction process is: The frequency spectrograms in the bird sound event detection dataset and the frequency spectrograms in the bird sound recognition dataset are preprocessed by frequency band segmentation to generate sub-bands Z with different frequency band characteristics; The segmented frequency band features are L2 normalized and connected to a fully connected layer to increase the complex correlation between features; Adopt an acceleration strategy based on sparse matrices, use the COO (Coordinate Format) storage format, and perform matrix multiplication operations on non-zero elements in sparse matrices to improve computing efficiency and resource utilization; Use the ECAPA-TDNN model to capture speech features of the spectrogram, where the ECAPA-TDNN model combines dilated convolution, Res2Net structure, and SENet module; A fully connected layer is added after the output of the ECAPA-TDNN model to map the feature space to the category space; Among them, the fully connected layer of the bird sound event detection model outputs 2 nodes, and the output values ​​represent the probabilities of bird sound samples and noise samples respectively after passing through the softmax function. The fully connected layer of the bird sound recognition model outputs 179 nodes, and the output values ​​represent the probabilities of 179 species of birds respectively after passing through the softmax function.

2. The large-scale bird sound recognition method based on deep learning technology according to claim 1 is characterized in that: The S4 comprises: S41, manually checking the spectrogram and waveform of the audio file to filter out low-quality and incorrectly labeled audio; S42: Cut the long audio every 30 seconds. If the last audio segment is less than 30 seconds, it will be retained if it is greater than 15 seconds, and discarded if it is less than 15 seconds. S43: Delete the original audio clips whose length is less than 2 seconds.

3. The large-scale bird sound recognition method based on deep learning technology according to claim 2 is characterized in that: The S5 comprises: S51, unifying the sampling rate of the acoustic signals in the bird sound recognition data set to 32,000 kHz, and converting the audio format to a wav format without high-frequency loss to obtain preprocessed audio data; S52, randomly slicing the preprocessed audio data, with the length of the slices being uniformly 10 seconds; S53, normalizing the amplitude of the sliced ​​audio data; S54, mixing the background noise data into the normalized audio data with a probability of 50% to obtain audio sample data; S55, performing spectrum enhancement on the logarithmic Mel spectrogram of the audio sample data, including randomly masking 0 to 5 frames in the time domain and randomly masking 0 to 10 channels in the frequency domain; S56, uses techniques such as speech rate perturbation, volume enhancement, and adding Gaussian noise to train the model.

4. The large-scale bird sound recognition method based on deep learning technology according to claim 3 is characterized in that: In the S56 process, a step-by-step fusion strategy is adopted to integrate the loss function based on the angle interval into the cross entropy loss function. The specific process is as follows: Rely on the cross entropy loss function L in the early stage of training CE To optimize, at this time β(t)≈0, the formula is: As the model gradually converges, β(t) begins to increase gradually, and the loss function L based on the angle interval is gradually introduced. AAM The angle interval in is given by: L fuse (t)=L CE +β(t)·L AAM ; In the later stages of training, β(t) grows to a higher value close to 1, at which point the loss function L based on the angle interval AAM It dominates, and the influence of cross entropy loss gradually decreases, and its formula is: Where N is the number of samples, C is the number of categories, and x i is the feature of the i-th input sample, W j is the weight of the classification layer, y i is the correct category, m is the incremental angular interval, s is an adjustable scale factor, Represents the angle between the i-th sample and its correct category weight, t is the training step length, and β(t) is the weight coefficient that gradually changes with the training step length t.

5. The large-scale bird sound recognition method based on deep learning technology according to claim 4 is characterized in that: In the S56 process, an adaptive learning rate adjustment strategy is used to improve the training efficiency and generalization ability of the model. The specific operations are as follows: In the early stage of model training, the learning rate is gradually increased through the warmup strategy, and the Adam optimizer is used to accelerate the convergence of the model; Set the batch size to 64; During model training, the system dynamically adjusts the number of training epochs based on changes in the loss function; In the later stages of training, the learning rate is gradually reduced using cosine annealing for fine tuning.

6. The large-scale bird sound recognition method based on deep learning technology according to claim 5 is characterized in that: In S6, in order to compare and evaluate the performance of each model in this method, the following evaluation indicators are used: Among them, TP represents true positive samples, TN represents true negative samples, FP represents false positive samples, and FN represents false negative samples.

7. The large-scale bird sound recognition method based on deep learning technology according to claim 6 is characterized in that: S7 uses two strategies to determine the correct recognition result, namely: The first strategy takes the output probability of the bird sound recognition model as 0.7 as the correct threshold, that is, the audio samples with an output probability greater than or equal to 0.7 are regarded as samples correctly recognized by the model; The second strategy, based on the first strategy, further adopts the "continuous enhancement recognition strategy" to post-process the model recognition results.

8. The large-scale bird sound recognition method based on deep learning technology according to claim 7 is characterized in that: The second strategy includes: When a bird species is identified, if the same bird sound is identified within 30 seconds before and after but its model output probability is less than 0.7 and greater than 0.5, it will also be output as a correctly identified sample.

9. A large-scale bird sound recognition system based on deep learning technology, characterized in that: include: The bird sound audio collection module is used to collect bird sound audio near the project area using the project area recording equipment to obtain field recording data; The bird sound event detection model training module is used to use the three data sets obtained from the bird sound event detection competition Bird audio detection challenge 2018-DCASE Task 3 as bird sound event detection data sets for training the bird sound event detection model; A bird sound category prediction module is used to input the field recording data into the trained bird sound event detection model for category prediction to obtain effective bird sound data and background noise data; The bird sound recognition data set acquisition module is used to obtain the bird species catalog recorded in the project area from the China Bird Watching Record Center, download the audio files of the corresponding bird species in Xeno-Canto, and perform data preprocessing operations on the audio files to obtain the bird sound recognition data set; A bird sound recognition model training module, used to train a bird sound recognition model using the background noise data and the bird sound recognition data set; Model verification module, used to verify the trained bird sound recognition model; The label prediction module is used to input the valid bird sound data into the bird sound recognition model to perform label prediction to obtain a recognition result.

Images