Bird sound event detection method and system oriented to national key protection bird monitoring

Abstract

The invention discloses a bird sound event detection method and system oriented to national key protection bird monitoring, and the method comprises the steps: collecting bird sound data and environment noise data of national key protection birds, and constructing a bird sound spectrogram data set and a noisy bird sound data set; constructing a bird sound noise reduction model and a target detection model; training and jointly optimizing a bird sound noise reduction model and a target detection model according to the bird sound spectrogram data set and the noisy bird sound data set; an image-audio mapping module is constructed and used for converting an image detection result of the target detection model into a sound event detection result of the audio; and obtaining to-be-detected bird sound data, performing retrieval and sound source localization on the target bird sound according to the bird sound noise reduction model, the target detection model and the image-audio mapping module, and determining species, time frequency and direction information of the target bird sound. The method can improve the detection effect of the field bird sound audio, further improves the classification precision, and can be widely applied to the technical field of acoustic information processing.

Description

Technical Field

[0001] The present invention relates to the technical field of acoustic information processing, and in particular to a bird sound event detection method and system for monitoring national key protected birds. Background Art

[0002] Nationally protected birds are often key species in ecosystems, and their presence and health directly affect the stability and diversity of ecosystems. In recent years, bird sound monitoring has played a vital role in bird research, especially in terms of non-invasive monitoring, the possibility of continuous monitoring, and large-scale data collection. The application of this technology has greatly promoted the development of bird ecology and conservation biology. As a non-invasive technology, bird sound monitoring has minimal interference with birds and their habitats compared to traditional research methods such as capture and tagging. Using sound monitoring equipment, researchers can collect data without direct contact with birds, thereby reducing the impact on the natural behavior of birds. Audio detection and identification of these nationally protected birds can help monitor and protect these species. However, there is usually a lot of background noise in the wild environment, such as wind, rain, human activities, etc., which may interfere with the detection and identification of bird sounds. In particular, many nationally protected birds are very rare, and some birds have very long intervals between calls, making it relatively difficult to obtain the corresponding bird calls.

[0003] Bird sound detection and classification is a key area of ​​bioacoustic research. Bird sounds usually occur in complex natural environments. These environmental features may cause overlap of sound events, making it more difficult to detect and classify bird sounds. Many methods based on traditional signal processing rely on extracting basic time-frequency features from audio signals, such as Mel-frequency cepstral coefficients (MFCC), pitch, energy, etc., and then detecting sound events through rule-based or template matching methods. Such methods are simple to implement and do not require complex algorithms and model training. However, such methods rely heavily on pre-defined rules or templates, can only capture some simple and significant features, and cannot handle complex time series patterns and spectral changes, and have poor flexibility. In recent years, deep learning has developed rapidly in the field of bird sound detection and classification. Bird sound detection methods based on CNN and CRNN can automatically extract complex time-frequency features from raw audio data, reduce dependence on manually designed features, and can capture richer sound patterns and details. They can usually achieve good detection accuracy in processing detection tasks involving multiple sound events. In the current deep learning bird sound detection methods, most CNN-based networks usually convert audio data into other features such as mel-spectrograms before inputting them into the network. CNN extracts local features through multiple convolutional layers and pooling layers, and finally performs classification or detection through fully connected layers. This method is good at capturing local patterns in audio and is suitable for detecting time-localized sound events, i.e., segment-level sound events. CRNN combines the advantages of CNN and RNN. First, the audio data is passed through a convolutional layer to extract time-frequency features, which is similar to a pure CNN model. Then, these features are passed through a recurrent neural network (RNN), usually a long short-term memory network (LSTM) or a gated recurrent unit (GRU), to capture time series dependencies in audio. CRNN is therefore able to simultaneously process the time-frequency features and temporal dynamics of audio, and is suitable for detecting sound events at the frame level.

[0004] At present, the traditional signal processing bird sound detection methods are not flexible and adaptable enough when facing complex and changeable real-life scenarios, and it is difficult to capture and process complex sound event characteristics; the bird sound detection methods based on CNN often only focus on the classification information of bird sounds in fixed-length audio clips and ignore the specific and variable time range information of bird sounds; the methods based on CRNN often only focus on the classification and time range information of bird sound events in audio clips, but ignore the specific frequency range information and the location information of the actual sound source signal; in addition, many bird recognition models often only focus on the bird sounds that can be seen everywhere in daily life, but ignore the national key protected birds. And the actual field recording signals often contain a large amount of various noises, which will greatly affect the reasoning effect of the detection model. The shortcomings of the current detection methods in frequency analysis, noise interference prevention, missed detection prevention and environmental adaptability limit their effectiveness in actual application scenarios. Especially for the national protected birds that require long-term and extensive monitoring and are scarce in number, the limitations of existing methods make it difficult to guarantee the accuracy and comprehensiveness of monitoring data. Summary of the invention

[0005] The main purpose of the embodiments of the present invention is to propose a bird sound event detection method and system for monitoring national key protected birds, which can improve the detection effect of wild bird sound audio and further improve the classification accuracy.

[0006] To achieve the above-mentioned purpose, an embodiment of the present invention provides a bird sound event detection method for monitoring national key protected birds, comprising the following steps:

[0007] Collect bird sound data of nationally protected birds and environmental noise data;

[0008] Constructing a bird sound spectrogram dataset and a bird sound dataset with noise according to the bird sound data and the environmental noise data;

[0009] Build bird sound noise reduction model and target detection model;

[0010] According to the bird sound spectrogram dataset and the noisy bird sound dataset, the bird sound denoising model and the target detection model are trained and jointly optimized; wherein the bird sound denoising model is used to perform denoising on the noisy bird sound data, and the target detection model is used to detect target images;

[0011] Constructing an image-audio mapping module for converting the image detection result of the target detection model into a sound event detection result of the audio;

[0012] The bird sound data to be detected is obtained, and the target bird sound is retrieved and the sound source is located according to the bird sound denoising model, the target detection model and the image-audio mapping module to determine the species, time-frequency and direction information of the target bird sound.

[0013] In some embodiments, the collecting of bird sound data of nationally protected birds and environmental noise data comprises the following steps:

[0014] According to the national first- and second-level protected bird lists, the sound data of these birds are collected, including data downloaded from the bird sound data website, or field recording data; the sampling rate is not less than 32kHz, and the quantization accuracy is not less than 16 bits;

[0015] The collected data is processed in a unified manner and subjected to noise reduction to obtain clean bird sound data audio files after screening;

[0016] According to the bird species, the screened audio files are connected to obtain the audio files of each bird species;

[0017] Environmental noise data is obtained based on the noise data sampling rate requirements, quantization accuracy requirements and noise types; among them, noise types include target noises in the actual national key bird monitoring areas, including: wind noise, car noise, rain noise, stream human voice, cricket chirping, katydid chirping, cicada chirping and frog croaking.

[0018] In some embodiments, constructing a bird sound spectrogram dataset and a bird sound dataset with noise according to the bird sound data and the environmental noise data comprises the following steps:

[0019] Convert the acquired bird sound data from the time domain to the frequency domain to generate a bird sound spectrogram;

[0020] Adjust the bird sound spectrogram image size and color depth;

[0021] Use image annotation tools to draw bounding boxes for each target bird sound on the bird sound map, label the category, and save the annotation file;

[0022] The bird sound spectrogram and the corresponding annotation files constitute the bird sound spectrogram dataset;

[0023] Noise data is added to the acquired clean bird sound data to obtain the noisy bird sound data set.

[0024] In some embodiments, converting the acquired bird sound data from the time domain to the frequency domain to generate a bird sound spectrogram comprises the following steps:

[0025] Note the number of data sampling points P of a bird sound signal, set the frame length flen to 2048, and the frame shift hlen to 512, and get the number of frames fn of the bird sound signal. Each frame signal after framing is x(p,λ), where p is the sampling point number and λ is the frame number. The calculation formula for the number of frames of the bird sound signal is:

[0026] The current frame x(p,λ) is overlapped with the previous frame x(p,λ-1) by a length of overlap=512 to obtain the target frame x o (p,λ), the total length of the frame after stacking is the frame length flen;

[0027] For the target frame x o (p,λ) is windowed, the window type is Hamming window w(p,ψ), where ψ is 0.46, the window length is equal to the frame length point flen=2048, and all the windowed frames x are obtained. w (p,λ);

[0028] For each frame x w (p, λ) performs a DFT operation of P = 2048 points, and takes the modulus of the DFT operation result to obtain the amplitude spectrum X(f, λ) of each frame signal, where f represents the frequency point and λ is the frame number. f Analyze the frequency points to generate the bird sound spectrogram;

[0029] The step of adjusting the image size and color depth of the bird sound spectrogram comprises the following steps:

[0030] The calculation parameters for generating the spectrogram are unified, where the STFT window size is 2048, the frame shift is 512, and the image resolution DPI is 300;

[0031] Set the vertical axis scale of the spectrogram to a linear scale and the color map to the 'inferno' mapping table; by setting the mapping table, frequency components of different intensities are mapped to specific colors in the bird sound spectrogram;

[0032] The step of adding noise data to the acquired clean bird sound data to obtain the noisy bird sound data set comprises the following steps:

[0033] Set the length of a bird sound signal to z seconds, the sampling rate to 32kHz, and the number of sampling points required to be z*32000. When the number of sampling points of the selected noise signal is greater than z*32000, randomly select z*32000 continuous sampling points from it;

[0034] After obtaining clean bird sound signals and noise signals of the same length, the noise is added to the clean bird sound signal; one type of noise is selected each time, so that the proportion of noisy bird sound signals of each noise type in the entire noisy bird sound signal is uniform, and the number of noisy bird sound signals of different noise types is ensured to be the same, and the signals are mixed according to the following formula:

[0035]

[0036] Among them, t represents the time of the signal, z(t) is the noisy bird sound signal, z1(t) and n(t) are clean bird sound signals and noise signals respectively, ξ is the gain coefficient of n(t) in the mixing process, and snr represents the signal-to-noise ratio of clean bird sound signals and noise signals.

[0037] In some embodiments, the bird sound denoising model includes a data input layer, an initial causal convolution layer, an expansion residual block, a skip connection summary and a final convolution output layer; the construction of the bird sound denoising model includes the following steps:

[0038] The input shape of the data input layer is (IN, 1), where IN represents the length of the input signal, which is used to receive the noisy bird sound signal;

[0039] The initial causal convolution layer uses a causal convolution kernel to process the input signal so that each output depends on the current and previous input values. The number of convolution kernels in the initial causal convolution layer is 128, and the convolution kernel size is 3. The process can be expressed as:

[0040] h causal =σ(W causal *x+b causal )

[0041] Among them, W causal and b causal are the causal convolution kernel weight and bias, respectively, h causal is the output of causal convolution, σ is the ReLU activation function;

[0042] There are 30 dilated residual blocks in total, each of which includes a dilated convolutional layer, an activation function, a gating mechanism, a 1x1 convolutional layer, a skip connection, and a residual connection; the number of convolution kernels in the dilated convolutional layer is 256, the convolution kernel size is 3, the dilation rate increases in power from 1 to 512 (1, 2, 4, 8, 16, 32, 64, 128, 256, 512), there are 10 dilation rates, and the number of stacks is 3; the activation function includes the hyperbolic tangent tanh function and the sigmoid function, where the hyperbolic tangent tanh function is applied to the first part of the output of the dilated convolutional layer to generate h tanh , the sigmoid function is applied to the second output of the dilated convolutional layer to generate hsigma ; The gating mechanism converts h tanh and h sigma Combine to generate h gate ; Then, h gate Generate h through a 1x1 convolutional layer 1x1 The main function of the 1x1 convolution layer is to adjust the dimension of the feature map through linear transformation and generate residual output and jump connection output. The number of convolution kernels is 256, of which the number of residual channels and jump connections are 128 respectively; the jump connection part is from h 1x1 The skip connection output is extracted for aggregation in subsequent layers. The residual connection part adds the residual output of the 1x1 convolutional layer to the input signal to generate the output of the dilated residual block.

[0043] The skip connection aggregation generates a comprehensive feature representation by summing the skip connection outputs of all dilated residual blocks; wherein the comprehensive feature representation contains the features of all layers, and provides a rich feature representation for the final output layer through feature information at different scales and different expansion rates. The formula for the summation operation is:

[0044]

[0045] Among them, s d represents the skip connection output of the d-th dilated residual block, D is the total number of dilated residual blocks; s represents the summation result;

[0046] The final convolution output layer consists of three convolution layers, wherein the first convolution layer has 2048 convolution kernels and a convolution kernel size of 3, which are used to extract deeper features from the comprehensive feature representation of the jump connection; the second convolution layer has 256 convolution kernels and a convolution kernel size of 3, which compresses the feature map to a smaller dimension; the third convolution layer is a 1x1 convolution layer, whose convolution kernel number and size are both 1, which is used to perform a linear transformation on the feature map to generate the final denoised bird sound signal.

[0047] In some embodiments, the building of the target detection model comprises the following steps:

[0048] Construct the Backbone network. The Backbone part includes the Stem Layer starting layer, the convolution modules from Stage 1 to Stage 4, the CSP layer, and an SPPF layer. The Stem Layer starting layer consists of a convolution module, which includes a two-dimensional convolution layer, a convolution operation followed by batch normalization and a SiLU activation function. The Stem Layer starting layer is used to preliminarily extract the low-level features of the image. Its operation process can be expressed by the following formula:

[0049] y=SiLU(BatchNorm(Conv2d(x,W,b)))

[0050] Where x is the input image, W and b are the convolution kernel weight parameter and bias respectively;

[0051] Four Stage Layer layers are used to gradually extract and integrate features. Each Stage contains a convolution module and a CSP layer. The CSP layer is used to integrate features and reduce the amount of calculation. It includes two convolution modules and a residual connection and two DarknetBottleneck modules. Each convolution module includes a convolution layer, followed by a batch normalization layer and a SiLU activation function.

[0052] The DarknetBottleneck module consists of two convolutional layers for efficient extraction of complex features. The first is a 1×1 convolutional layer to reduce the number of channels and improve computational efficiency, followed by a 3×3 convolutional layer to extract local features. Finally, the input and output are connected through a residual connection to enhance gradient transfer.

[0053] The SPPF layer is used for multi-scale feature integration to enhance the model's ability to detect objects of different scales. It includes three pooling operations with pooling kernel sizes of 5, 9, and 13, which connect the pooling features of different scales to form a richer feature representation.

[0054] The Neck part includes TopDown and BottomUp paths, which are used for top-down feature fusion and bottom-up feature extraction respectively. The TopDown path first performs an upsampling operation to upsample the feature map of the deeper layer to make its size match that of the feature map of the shallower layer, and then connects the downsampled feature map with the corresponding deeper feature map through a connection operation. The BottomUp path fuses the feature map of the shallower layer with the feature map of the deeper layer through a downsampling operation.

[0055] The head part includes the basic convolution layer and the convolution layer dedicated to bounding box regression and classification; the basic convolution layer is used to preliminarily process the input feature map and extract more compact and useful features;

[0056] Use the soft non-maximum suppression algorithm with Gaussian attenuation to attenuate the confidence of overlapping candidate boxes.

[0057] In some embodiments, the training and joint optimization of the bird sound denoising model and the target detection model based on the bird sound spectrogram dataset and the noisy bird sound dataset includes the following steps:

[0058] The bird sound denoising network and target detection network are pre-trained based on the noisy bird sound dataset and the bird sound spectrogram dataset respectively;

[0059] Construct a joint optimization framework for bird sound denoising and detection, and cascade the bird sound denoising network with the target detection network, including:

[0060] Define the denoising-detection comprehensive loss function, where the denoising model uses mean square error loss, mean square error loss Loss MSE The calculation process is as follows:

[0061]

[0062] Where M represents the total number of samples. is the value of the kth sample predicted by the model, y k is the true value of the kth sample;

[0063] The denoising-detection comprehensive loss function is defined as:

[0064] Loss total =μLoss yolo +γLoss MSE

[0065] Among them, Loss yolo =Loss EWA +CIoU+DFL, μ and γ are weight parameters; Loss EWA stands for energy-weighted attention loss; CIoU stands for complete intersection-over-union loss; DFL stands for distributional focus loss;

[0066] Freeze the initial causal convolutional layer and the first 9 dilated residual blocks of the bird sound denoising network, and freeze the Stem Layer starting layer and Stage 1 convolutional module of the target detection network;

[0067] Set the number of epochs and batchsize to 50 and 32 respectively;

[0068] The Adam optimizer is used as the stochastic batch gradient descent optimizer, and the initial learning rate lnitial_lr is set to 0.0001;

[0069] During the training process, the model training is completed when the validation set loss value does not decrease after 10 epochs.

[0070] In some embodiments, the image-audio mapping module is used to convert the image detection result of the target detection model into the sound event detection result of the audio, including the following steps:

[0071] Let the sampling rate of a bird sound audio signal be Sr, the total duration of the audio be Dur, and the width and height of the corresponding spectrogram image be set to width and height respectively. According to the duration and sampling rate of the audio signal, the total number of sampling points L can be calculated:

[0072] L=Dur*Sr

[0073] Where L represents the total number of sampling points of the audio signal, the sampling rate Sr of the audio signal refers to the number of samples collected per second, which is used for quantization of digital audio, and the audio duration Dur is the total duration of the audio signal;

[0074] The number of sampling points S corresponding to each pixel on the time axis pixel for:

[0075]

[0076] Among them, width represents the width of the image;

[0077] Temporal resolution refers to the length of time that each pixel corresponds to on the time axis, and is calculated as follows:

[0078]

[0079] Among them, t pixel Indicates the length of time corresponding to each pixel on the time axis;

[0080] Frequency resolution refers to the frequency bandwidth corresponding to each pixel on the frequency axis. According to the Nyquist sampling theorem, f pixel :

[0081]

[0082] Among them, f pixel Indicates the frequency bandwidth corresponding to each pixel on the frequency axis;

[0083] According to the input image pixel coordinate range (x min ,x max ), the corresponding audio signal time range can be calculated:

[0084] t start =x min *t pixel

[0085] t end =x max *t pixel

[0086] Among them, t start and t end Respectively represent the starting point and ending point of time;

[0087] According to the input image pixel coordinate range (y min ,y max ), calculate the corresponding audio signal frequency range:

[0088] f low =(height-y max )*f pixel

[0089] f high =(height-y min )*f pixel

[0090] Among them, f low and f high Represent the lower and upper limits of the frequency respectively; the image pixel coordinates (y min ,y max ) represents the frequency interval selected on the image, which is converted into the actual frequency range through the frequency resolution.

[0091] In some embodiments, the step of acquiring the bird sound data to be detected, retrieving and locating the sound source of the target bird sound according to the bird sound denoising model, the target detection model, and the image-audio mapping module, and determining the species, time-frequency, and direction information of the target bird sound comprises the following steps:

[0092] The multi-channel bird sound signal collected on site is first converted into a single-channel signal, and then sliced ​​at a fixed length and converted into a spectrogram image. When slicing, the relative time position information of each sliced ​​audio segment relative to the original long audio is recorded, and then the trained target detection model is used for detection to obtain the corresponding detection results, which include the bird category and the image pixel coordinate information of the rectangular detection frame; when the model detects that the image contains the shape of the target bird sound, the image pixel coordinate range of each rectangular detection frame is converted into the corresponding time and frequency range, and the start and end time period, the maximum and minimum frequency range and the main frequency of the target bird sound are determined and output as the result, where the calculation process of the main frequency is: first calculate the rectangular detection frame B i The maximum internal energy value E max,i , and then calculate the maximum energy value E max,i The coordinates (x e ,y e ), the coordinate of the maximum energy value (x e ,y e ) is converted into a corresponding time point and frequency point, and the frequency point is determined as the main frequency;

[0093] According to the relative time position information of each sliced ​​audio segment relative to the original long audio, the start and end time periods of the target bird sound relative to the sliced ​​audio segment are converted into the start and end time periods of the original long audio, thereby completing the target bird sound retrieval for the long recording file;

[0094] Each channel of the four-channel bird sound signal collected on site is converted into a corresponding spectrogram, and the bird sound signal of each channel is detected using the trained model to obtain the start and end time t of the target bird sound in each channel. start and t end And the maximum and minimum frequencies f low and f high , compare the start and end time of each channel with the maximum and minimum frequencies, and take the minimum value of the start time, the maximum value of the end time, the lowest value of the lowest frequency, and the highest value of the highest frequency of all channels as the new [t start , t end , f low , f high ], for each channel of the four-channel bird sound signal, according to the start and end time t start and t end The signal is intercepted and then filtered through a linear phase FIR bandpass filter. The frequency band of the bandpass filter is set to the minimum frequency and the maximum frequency f low and f high The frequency range between

[0095] The arrival time difference is calculated using the generalized cross-correlation algorithm for the intercepted and bandpass filtered target bird data of each channel;

[0096] After cross-correlating the signals of the target bird data of each channel after truncation and bandpass filtering, three delay estimates D are obtained: 12 , D 13 , D 14 , and then perform azimuth Calculation, the calculation formula is:

[0097] Another aspect of the embodiment of the present invention further provides a bird sound event detection system for monitoring national key protected birds, including:

[0098] The first module is used to collect bird sound data of nationally protected birds and environmental noise data;

[0099] The second module is used to construct a bird sound spectrogram dataset and a bird sound dataset with noise according to the bird sound data and the environmental noise data;

[0100] The third module is used to build a bird sound noise reduction model and a target detection model;

[0101] The fourth module is used to train and jointly optimize the bird sound denoising model and the target detection model according to the bird sound spectrogram dataset and the noisy bird sound dataset; wherein the bird sound denoising model is used to perform denoising on the noisy bird sound data, and the target detection model is used to detect the target image;

[0102] A fifth module is used to construct an image-audio mapping module, which is used to convert the image detection result of the target detection model into a sound event detection result of the audio;

[0103] The sixth module is used to obtain the bird sound data to be detected, retrieve and locate the target bird sound according to the bird sound denoising model, the target detection model and the image-audio mapping module, and determine the species, time-frequency and direction information of the target bird sound.

[0104] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0105] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0106] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0107] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a bird sound event detection method and system for monitoring national key protected birds, the scheme collects bird sound data and environmental noise data of national key protected birds; constructs a bird sound spectrogram data set and a noisy bird sound data set based on the bird sound data and the environmental noise data; constructs a bird sound denoising model and a target detection model; trains and jointly optimizes the bird sound denoising model and the target detection model based on the bird sound spectrogram data set and the noisy bird sound data set; wherein the bird sound denoising model is used to denoise the noisy bird sound data, and the target detection model is used to detect the target image; constructs an image-audio mapping module for converting the image detection result of the target detection model into a sound event detection result for the audio; obtains the bird sound data to be detected, retrieves and locates the target bird sound according to the bird sound denoising model, the target detection model and the image-audio mapping module, and determines the species, time-frequency and direction information of the target bird sound. The present invention can improve the detection effect of the bird sound audio in the wild, and thus improve the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0109] Figure 2 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0110] Figure 3 is a diagram of a bird sound noise reduction network structure provided by an embodiment of the present invention;

[0111] Figure 4 It is a structural diagram of Backbone and CSP layers provided by an embodiment of the present invention;

[0112] Figure 5 is a target bird sound retrieval flow chart provided by an embodiment of the present invention;

[0113] Figure 6 is a flow chart of applying the detection result provided by an embodiment of the present invention to sound source localization;

[0114] Figure 7 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0115] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0116] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0117] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0118] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0119] In order to overcome the shortcomings of the existing bird sound event detection technology for monitoring national key protected birds, the embodiment of the present invention provides a bird sound event detection method for monitoring national key protected birds based on target detection, which converts the image detection results of the target detection algorithm into the start and end time and the maximum and minimum frequency range and the main frequency of the sound event inside the audio signal. This method can also have a very good detection effect on the bird sound audio with low signal-to-noise ratio in actual field recordings. While having a high recall rate, the accuracy rate can also be well guaranteed to prevent the missed detection of rare birds. This method can simultaneously focus on the detection of the time position information and frequency range information of the bird sound event. The detected frequency range information can dynamically change according to the different call forms of different birds, thereby improving the classification accuracy of different bird sounds. These detection results can be well applied to the retrieval of target birds in actual field recordings obtained for a long time and in large quantities. In addition, these detection results also include the horizontal angle information of the sound source positioning.

[0120] The bird sound event detection method and system for national key protected bird monitoring provided by the embodiment of the present invention relate to the field of acoustic information processing technology. The bird sound event detection method for national key protected bird monitoring provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a bird sound event detection method for national key protected bird monitoring, etc., but is not limited to the above forms.

[0121] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0122] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0123] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0124] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0125] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0126] Based on the example Figure 1In the implementation environment shown, an embodiment of the present invention provides a bird sound event detection method for monitoring national key protected birds. The following is explained using the example of the bird sound event detection method for monitoring national key protected birds applied in the server 101. It can be understood that the method can also be applied to the terminal 102.

[0127] Reference Figure 2 , Figure 2 The flowchart of the bird sound event detection method for monitoring national key protected birds applied to a server provided in an embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method may include the following steps:

[0128] S1. Collect bird sound data of nationally protected birds and environmental noise data;

[0129] S2, construct bird sound spectrogram dataset and bird sound with noise dataset;

[0130] S3, construct a bird sound noise reduction model;

[0131] S4, build target detection model;

[0132] S5, train and jointly optimize the bird sound denoising and target detection models, and set the optimizer, learning rate, and learning rate strategy;

[0133] S6, constructing an image-audio mapping module;

[0134] S7. Target bird sound retrieval and sound source localization.

[0135] 1. The specific details of step S1 are:

[0136] (1) According to the national first- and second-level protected bird lists, collect the sound data of these birds. This data can be downloaded from bird sound data websites (such as xeno-canto) or actual field recording data.

[0137] (2) Bird sound data requirements:

[0138] The sampling rate shall not be less than 32kHz and the quantization accuracy shall not be less than 16 bits. There is no restriction on the geographical area of ​​the bird species involved in the data.

[0139] (3) Assume that the total number of bird sound categories is V, and the total duration of bird sound data of each category is T v , v = 1, 2, ... V, requiring the total duration T of each category of data in the bird sound dataset v No less than 600 seconds. Each category can contain multiple audio files, and the length of each file cannot be shorter than 10 seconds.

[0140] (4) The data collected above were standardized into the following formats: audio format: wav, sampling frequency: 32kHz, quantization accuracy: 16 bits, number of audio channels: mono. Audio processing software (such as Adobe Audition) was used to perform noise reduction processing to obtain clean bird sound data, and then manual inspection was performed to ensure that the signal-to-noise ratio of the clean bird sound data was not less than 30dB.

[0141] (5) Connect the audio files selected in the previous step according to the bird species. Each species corresponds to a long audio file. Slice the audio file, and the duration of each slice is between 5 and 15 seconds. The bird song structure of each slice should be complete, and the bird song fragments that were truncated during slicing should be discarded.

[0142] (6) Obtaining environmental noise data: The minimum sampling rate requirement for noise data is 32 kHz, and the minimum quantization accuracy requirement is 16 bits. The noise types should be common noises in the actual national key bird monitoring areas (such as wind, car sounds, rain, stream sounds, human voices, cricket sounds, katydid sounds, cicada sounds, and frog sounds). The total duration of each category of noise data should be no less than 1200 seconds. Each category of data can include several audio files, and the duration of each audio file in each category cannot be less than 120 seconds.

[0143] 2. The specific details of step S2 are:

[0144] (1) Construct a bird sound spectrogram dataset. First, convert the clean bird sound data obtained in S1 from the time domain to the frequency domain. The specific calculation process is as follows:

[0145] 1) Record the number of data sampling points P of a bird sound signal, set the frame length flen to 2048, the frame shift hlen to 512, and get the number of frames fn of the bird sound signal. Each frame signal after framing is x(p,λ), where p is the sampling point number and λ is the frame number.

[0146]

[0147] 2) Overlap the current frame x(p,λ) with the previous frame x(p,λ-1) by a length of overlap=512 to obtain x o (p,λ), the total length of the frame after stacking is still flen, p=0,1,…,flen-1.

[0148]

[0149] 3) For x o (p,λ) is windowed, and the window type is a Hamming window w(p,ψ), where ψ is 0.46 and the window length is equal to the frame length flen=2048, thus obtaining all the windowed frames xw (p,λ).

[0150]

[0151] x w (p,λ)=x0(p,λ)*w(p,ψ),0≤p≤flen-1#(4)

[0152] 4) For each frame x w (p, λ) performs a DFT operation of P = 2048 points, and takes the modulus of the DFT operation result to obtain the amplitude spectrum X(f, λ) of each frame signal, where f represents the frequency point and λ is the frame number. Due to the symmetry of Fourier transform, only the first P points of the spectrum are processed. f The frequency points are analyzed, among which

[0153]

[0154] (2) Convert the generated bird sound spectrogram into a standard image format and adjust the image size and color depth to suit subsequent processing. This includes the following steps:

[0155] 1) Unify the calculation parameters for generating the spectrogram, STFT window size: 2048, frame shift: 512, image resolution DPI: 300

[0156] 2) The image size should be no less than 640*640 to avoid excessive loss of image details, and it should not be too high to avoid slow training. The vertical axis scale of the spectrogram is set to linear scale, and the color mapping is set to 'inferno'. 'Inferno' is a color mapping table that usually gradients from black (low intensity) to yellow, red, and then to white (high intensity), making the details very clear. By setting this mapping table, frequency components of different intensities are mapped to specific colors in the bird sound spectrogram, making the image more intuitive and easy to analyze.

[0157] 3) Use the image annotation tool LabelImg to draw a bounding box for each target bird sound on the bird sound spectrogram and mark the category (marked as the bird name). The annotation file is saved in yolo_txt format, that is, each image corresponds to a txt file, and each line of the file is the information of a target, including class, x_center, y_center, width, and height information.

[0158] 4) The above bird sound spectrograms and the corresponding annotation files constitute a bird sound spectrogram dataset, and the stratified sampling strategy is used to divide the dataset into a training set and a validation set with a ratio of 7:3.

[0159] 5) Add the clean bird sound signal obtained in S1 to the noise data to obtain the noisy bird sound data set. The noise adding process is as follows:

[0160] 1) Assume that the length of a bird sound signal is z seconds. Considering that the sampling rate is 32kHz, the number of sampling points required is z*32000. Select a noise signal. When the number of sampling points of the selected noise signal is greater than z*32000 (that is, the noise duration exceeds z seconds), randomly select z*32000 continuous sampling points from it.

[0161] 2) After obtaining clean bird sound signals and noise signals of the same length, add the noise to the clean bird sound signal. Select one type of noise to be added each time, so that the proportion of noisy bird sound signals of each noise type in the entire noisy bird sound signal is uniform, and ensure that the number of noisy bird sound signals of different noise types is the same. Signal mixing is performed according to the following formula:

[0162]

[0163] Where t represents the time of the signal, z(t) is the noisy bird signal, z1(t) and n(t) are clean bird signals and noise signals, ξ is the gain coefficient of n(t) in the mixing process, snr represents the signal-to-noise ratio of clean bird signals and noise signals, measured in decibels (dB). When synthesizing the noisy bird signal of the training set, snr is selected in the range of {-15, -10, -7.5, -5, -2.5, 0, 2.5, 5, 7.5, 10, 12.5, 15}. When synthesizing the noisy bird signal of the validation set, snr is {-15, -10, -5, 0, 5, 10, 15}. The number of noisy bird signals of each signal-to-noise ratio is equal.

[0164] (6) The above-mentioned noisy bird sound data and the corresponding clean bird sound data are combined to form a noisy bird sound dataset, in which the clean bird sound data is used as a label, and a stratified sampling strategy is used to divide the dataset into a training set and a validation set, with a ratio of 7:3.

[0165] 3. The specific details of step S3 are:

[0166] (1) Figure 3 As shown in the figure, the bird sound denoising network includes a data input layer, an initial causal convolution layer, an expanded residual block, a skip connection summary, and a final output layer. First, the input shape of the data input layer is (IN, 1), where IN represents the length of the input signal, which is used to receive the noisy bird sound signal. The initial causal convolution layer uses a causal convolution kernel to process the input signal so that each output depends only on the current and previous input values. The number of convolution kernels is 128, and the convolution kernel size is 3. The process can be expressed as:

[0167] h causal =σ(Wcausal *x+b causal )#(7)

[0168] Among them, W causal and b causal are the causal convolution kernel weight and bias, respectively, h causal is the output of the causal convolution, and σ is the ReLU activation function.

[0169] The dilated residual block is the core part of the denoising network. There are 27 dilated residual blocks in total. Each dilated residual block includes a dilated convolution layer, an activation function, a gating mechanism, a 1x1 convolution layer, a skip connection, and a residual connection. The number of convolution kernels in the dilated convolution layer is 256, the convolution kernel size is 3, and the dilation rate increases in power from 1 to 512 (1, 2, 4, 8, 16, 32, 64, 128, 256, 512). There are 10 dilation rates and the number of stacks is 3. The activation function includes the hyperbolic tangent tanh function and the sigmoid function. The hyperbolic tangent tanh function is applied to the first part of the output of the dilated convolution layer to generate h tanh , the sigmoid function is applied to the second output of the dilated convolutional layer to generate h sigma The gating mechanism converts h into tanh and h sigma Combine to generate h gate Then, h gate Generate h through a 1x1 convolutional layer 1x1 The main function of the 1x1 convolution layer is to adjust the dimension of the feature map through linear transformation and generate residual output and skip connection output. The number of convolution kernels is 256, and the number of residual channels and skip connections is 128 respectively. The skip connection part is from h 1x1 The skip connection output is extracted for aggregation in subsequent layers, and the residual connection part adds the residual output of the 1x1 convolutional layer to the input signal to generate the output of the dilated residual block.

[0170] (2) Skip connection aggregation: By summing up the skip connection outputs of all dilated residual blocks, a comprehensive feature representation is generated. This comprehensive feature representation contains the features of all layers and provides a rich feature representation for the final output layer through feature information at different scales and different dilation rates. The summation operation formula is:

[0171]

[0172] Among them, s d represents the skip connection output of the d-th dilated residual block, and D is the total number of dilated residual blocks.

[0173] The final convolution output layer consists of three convolution layers, of which the first convolution layer has 2048 convolution kernels and a convolution kernel size of 3, which is used to extract deeper features from the comprehensive feature representation of the jump connection. The second convolution layer has 256 convolution kernels and a convolution kernel size of 3, which further compresses the feature map to a smaller dimension. The third convolution layer is a 1x1 convolution layer, whose convolution kernel number and size are both 1, which is used to perform a linear transformation on the feature map to generate the final denoised bird sound signal.

[0174] 4. The specific details of step S4 are:

[0175] (1), reference Figure 4 , build a Backbone network, such as Figure 4 As shown in (a), the Backbone part includes the StemLayer starting layer, the convolution modules from Stage1 to Stage4, the CSP layer, and an SPPF layer. The Stem Layer starting layer consists of a convolution module (ConvModule). This module includes a two-dimensional convolution layer (Conv2d) with a convolution kernel size of 3x3, a step size of 2, and a padding of 1. The convolution operation is followed by batch normalization (BatchNorm) and SiLU activation function. This layer is mainly used to preliminarily extract low-level features of the image. Its operation process can be expressed by the following formula:

[0176] y=SiLU(BatchNorm(Conv2d(x,W,b)))#(9)

[0177] Among them, x is the input image, W and b are the convolution kernel weight parameter and bias respectively.

[0178] The four StageLayer layers are the key parts of Backbone for gradually extracting and integrating features. Each Stage contains a convolution module (ConvModule) and a CSP layer.

[0179] like Figure 4 As shown in (b), the CSP layer is used for feature integration and computational reduction, including two convolution modules (ConvModule) and a residual connection and two DarknetBottleneck modules. Each convolution module includes a convolution layer with a convolution kernel size of 3×3 and a stride of 1, followed by a batch normalization layer (BatchNorm) and a SiLU activation function.

[0180] The DarknetBottleneck module consists of two convolutional layers to extract complex features more efficiently. The first is a 1×1 convolutional layer to reduce the number of channels and improve computational efficiency, followed by a 3×3 convolutional layer to extract local features. Finally, the input and output are connected through a residual connection to enhance gradient transfer.

[0181] The SPPF layer is used for multi-scale feature integration to enhance the model’s ability to detect targets of different scales. It includes three pooling operations with pooling kernel sizes of 5, 9, and 13, respectively. It connects pooling features of different scales to form a richer feature representation.

[0182] (2) The Neck part includes the TopDown and BottomUp paths, which are used for top-down feature fusion and bottom-up feature extraction, respectively. The TopDown path first performs an upsampling operation (Upsample) to upsample the feature map of the deeper layer to match its size with the feature map of the shallower layer, and then connects the downsampled feature map with the corresponding deeper feature map through the concatenation operation (Concat). The BottomUp path fuses the feature map of the shallower layer with the feature map of the deeper layer through the downsampling (DownSample) operation. The concatenation operation concatenates the feature map along the channel dimension to form a new feature map containing multi-scale information. Then, the CSP layer (CSPLayer) is used to process the concatenated feature map. The CSP layer (CSPLayer) contains two convolution modules, each of which includes a convolution layer (the convolution kernel size is 3×3 and the stride is 1), a batch normalization layer (BatchNorm2d) and a SiLU activation function. The TopDown path and the BottomUp path can be expressed as:

[0183] TopDown(x1,x2)=CSPLayer(Concat(Upsample(x1),x2))#(10)

[0184] BottomUp(x2,x1)=CSPLayer(Concat(DownSample(x2),x1))#(11)

[0185] Among them, x1 is a deeper feature map, that is, a feature map from a later layer in Backbone, which usually has a smaller spatial resolution and higher semantic information, and x2 is a shallower feature map, that is, a feature map from an earlier layer in Backbone, which usually has a larger spatial resolution and lower semantic information.

[0186] (3) The Head part includes a basic convolutional layer (ConvModule) and a convolutional layer (Conv2d) dedicated to bounding box regression and classification. The basic convolutional layer (ConvModule) is used to preliminarily process the input feature map and extract more compact and useful features. This layer includes a convolution operation, batch normalization (BatchNorm), and an activation function (SiLU). The convolution kernel size of the convolution operation is set to 3×3, and the stride is set to 1. The convolution layer (Conv2d) dedicated to bounding box regression and classification is responsible for the final detection output. This layer includes two different branches, one for bounding box regression and the other for target classification. The convolution kernel size of the convolution operation of each branch is set to 1×1, and the stride is set to 1. Through the processing of this layer, the bounding box regression branch outputs the predicted bounding box position and size, while the classification branch outputs the predicted target category.

[0187] During training, the bounding box regression loss and classification loss need to be calculated separately. To define the energy-weighted attention loss function (EWA Loss), we first need to extract the relevant parameters of each detection box i, including the center coordinates, width and height of the bounding box, and the confidence C i And the category prediction score P i In addition, it is also necessary to calculate the energy mean E in the detection box i , the calculation process of the energy mean is as follows:

[0188]

[0189] Among them, |B i | indicates the detection box B i The number of pixels in the image, S(x,y) is the value of the spectrogram at position (x,y).

[0190] The confidence level C i , energy mean E i And the category prediction score P i The input feature vector is concatenated as the input of a weighted network. The weighted network first performs a nonlinear transformation on the input feature vector through a fully connected layer to obtain an intermediate feature vector. This nonlinear transformation is applied to the output of the fully connected layer through the rectified linear unit (ReLU) activation function. In the second fully connected layer, the network calculates the attention weights for each input feature, which are normalized by the softmax function to ensure that their sum is 1. Next, the attention weights are applied to the original input features, and the features are weighted by element-wise multiplication to obtain a weighted feature vector. Subsequently, the weighted confidence C is calculated through the third and fourth fully connected layers respectively. iw And the category prediction score P iwThe output of the third fully connected layer is mapped to the confidence range through the sigmoid activation function, and the output of the fourth fully connected layer is mapped to the category prediction score range through the same activation function. The calculation process of the energy-weighted attention loss function (EWA Loss) can be expressed as:

[0191]

[0192] Loss EWA =Loss C +Loss p #(15)

[0193] Among them, N represents the total number of detection boxes, C represents the total number of categories, is the true label of the jth category of the i-th detection box.

[0194] The bounding box regression loss is defined as the complete intersection-over-union loss (CioU Loss) and the distribution focal loss (DFLLoss). The calculation process of CioU Loss is as follows:

[0195]

[0196] Among them, IoU is the intersection over union ratio, A is the intersection area of ​​the predicted bounding box and the true bounding box, and B is their union area. d is the Euclidean distance between the center points of the predicted bounding box and the true bounding box, c is the diagonal length of the minimum closure area containing the predicted bounding box and the true bounding box, α is a coefficient to measure the consistency of the aspect ratio of the bounding box, v is a measure of the difference in the aspect ratio of the predicted bounding box and the true bounding box, and W G and H G are the width and height of the ground-truth bounding box, W P and H P are the width and height of the predicted bounding box, respectively.

[0197] The calculation process of DFL Loss is as follows:

[0198] DFL(s a ,s a+1 )=-((y a+1 -y)log(s a )+(yy a )log(s a+1 ))#(20)

[0199] Among them, s a and a+1 is the predicted probability distribution value, y is the target value, and y a is the left integer part of the target value, y a+1 is the right integer part of the target value.

[0200] (5) The soft non-maximum suppression algorithm (Soft-NMS) with Gaussian attenuation is used to attenuate the confidence of overlapping candidate boxes, thereby retaining more detection information, which is beneficial to improving the recall rate of the model while maintaining a high precision, thereby improving the overall detection performance.

[0201]

[0202] Among them, S j ′ is the candidate box B j The updated confidence score, S j It is candidate box B j The original confidence score, IoU(B i ,B j ) represents the candidate box B i and B j The intersection-and-union ratio between them, g is the parameter that controls the decay rate, and its value is set to 0.5.

[0203] 5. The specific details of step S5 are:

[0204] (1) Use the noisy bird sound dataset and bird sound spectrogram dataset constructed in S2 to pre-train the bird sound denoising network and target detection network respectively. Use the early stopping strategy in the pre-training process. When the validation set loss value does not decrease after 10 epochs, the model training is completed.

[0205] (2) Construct a joint optimization framework for bird sound denoising and detection, and cascade the bird sound denoising network with the target detection network. Specifically, the output of the bird sound denoising network is used as the input of the target detection network, where the denoising network outputs a processed clean bird sound signal. The signal will be converted into a corresponding spectrogram before being input into the target detection network. The conversion process is consistent with the process of constructing the bird sound spectrogram dataset in S2. The parameter settings from time domain to frequency domain and the size of the generated spectrogram are also consistent with those in S2, so the labels of the bird sound spectrogram dataset in S2 can be shared. The generated spectrogram is then input into the target detection network for training. The specific training process is:

[0206] 1) Define the denoising-detection comprehensive loss function, where the denoising model uses the mean square error (MSE) loss. The calculation process of the MSE loss is as follows:

[0207]

[0208] Where M represents the total number of samples. is the value of the kth sample predicted by the model, y k is the true value of the kth sample.

[0209] The denoising-detection comprehensive loss function is defined as:

[0210] Loss total =μLoss yolo +γLoss MSE #(twenty three)

[0211] Among them, Loss yolo =Loss EWA +CIoU+DFL, μ and γ are weight parameters. The initial values ​​of μ and γ are set to 1, and are set to L after 10 training iterations. yolo and L MSE The reciprocal of the relative value.

[0212] 2) Freeze the initial causal convolutional layer and the first 9 dilated residual blocks of the bird sound denoising network, and freeze the Stem Layer starting layer and Stage 1 convolutional module of the target detection network. These frozen layers have learned stable low-level features in the pre-training stage. Keeping them unchanged will help focus training resources on optimizing higher-level features.

[0213] 3) Set the number of epochs and batch sizes to 50 and 32 respectively.

[0214] 4) Optimizer selection: Use Adam optimizer as the stochastic batch gradient descent optimizer, and set the initial learning rate lnitial_lr to 0.0001.

[0215] 5) During the training process, the model training is completed when the validation set loss value is observed to have not decreased after 10 epochs.

[0216] 6. The specific details of step S6 are:

[0217] (1) The task of sound event detection is to identify which sound events occur in a piece of audio and when. It not only needs to obtain the start and end time of the event, but also needs to classify the detected events. By constructing an image-audio mapping module, the target detection results of the image can be converted into the sound event detection results of the audio. First, it is necessary to obtain the parameters of the audio signal and its corresponding spectrogram image. The specific process is as follows: let the sampling rate of a bird sound audio signal be Sr, the total duration of the audio be Dur, and the width and height of the corresponding spectrogram image be set to width and height respectively. According to the duration and sampling rate of the audio signal, the total number of sampling points can be calculated:

[0218] L=Dur*Sr#(24)

[0219] Where L represents the total number of sampling points of the audio signal. The sampling rate Sr of the audio signal refers to the number of samples collected per second, which is used for quantization of digital audio. In digital audio processing, the higher the sampling rate, the more sound details can be captured, but it also increases the amount of data. The audio duration Dur is the total duration of the audio signal in seconds. The width and height of the image determine the resolution of the spectrogram used to represent the audio signal.

[0220] The number of sampling points corresponding to each pixel on the time axis is:

[0221]

[0222] Here, S pixel Indicates the number of sampling points corresponding to each pixel on the time axis, which determines the resolution on the time axis. The larger the width of the image, the shorter the time corresponding to each pixel, and the higher the time resolution.

[0223] (2) Calculation of time and frequency resolution

[0224] Temporal resolution refers to the length of time that each pixel corresponds to on the time axis, and is calculated as follows:

[0225]

[0226] Among them, t pixel Indicates the length of time that each pixel corresponds to on the time axis, in seconds. It determines the fineness of the image on the time axis. By combining the sampling rate and image width, the temporal details of the audio signal can be accurately located.

[0227] Frequency resolution represents the frequency bandwidth corresponding to each pixel on the frequency axis, which can be obtained from the Nyquist sampling theorem:

[0228]

[0229] Among them, f pixel Indicates the frequency bandwidth corresponding to each pixel on the frequency axis, in Hertz. The Nyquist sampling theorem shows that the sampling rate should be at least twice the highest frequency of the signal to avoid aliasing. The resolution of the image height can accurately represent the frequency components of the audio signal.

[0230] (3) Image pixel coordinates are converted to time range:

[0231] According to the input image pixel coordinate range (x min ,x max ), the corresponding audio signal time range can be calculated:

[0232] t start=x min *t pixel #(28)

[0233] t end =x max *t pixel #(29)

[0234] Among them, t start and t end Respectively represent the time start point and end point. These values ​​can accurately locate the specific range of the audio signal on the time axis by combining the image pixel coordinates with the time resolution. Image pixel coordinates (x min ,x max ) represents the time interval selected on the image, which is combined with the time length corresponding to each pixel and converted into the actual audio signal time range.

[0235] (4) Image pixel coordinates converted to frequency range

[0236] According to the input image pixel coordinate range (y min ,y max ), the corresponding audio signal frequency range can be calculated:

[0237] f low =(height-y max )*f pixel #(30)

[0238] f high =(height-y min )*f pixel #(31)

[0239] Among them, f low and f high Represent the lower and upper limits of the frequency respectively. Image pixel coordinates (y min ,y max ) represents the frequency interval selected on the image, which is converted into the actual frequency range through the frequency resolution.

[0240] 7. The specific details of step S7 are:

[0241] (1)Reference Figure 5 , the four-channel bird sound signal collected on site is first converted into a single-channel signal, and then sliced ​​into fixed lengths and converted into a spectrogram image. When slicing, the relative time position information of each sliced ​​audio segment relative to the original long audio is recorded. Then, the target detection model trained in steps S3, S4, and S5 is used for detection to obtain the corresponding detection results, which include the bird category, the image pixel coordinates of the rectangular detection box (x min ,y min ,xmax ,y max ) and other information. When the model detects that the image contains the shape of the target bird sound, it will use a rectangular frame to frame the shape of the target bird sound. The size of the rectangular frame will change dynamically according to the change of the size of the target bird sound shape. According to the above steps, the image pixel coordinate range of each rectangular detection frame is converted into the corresponding time and frequency range, that is, the start and end time period, maximum and minimum frequency range and main frequency of the target bird sound are calculated and output as the result. The calculation process of the main frequency is:

[0242] 1) Calculate the rectangular detection box B i The maximum internal energy value E max,i

[0243]

[0244] Where S(x,y) is the value of the spectrogram at position (x,y).

[0245] 2) Then calculate the maximum energy value E max,i The coordinates (x e ,y e ), according to the above steps, the coordinate of the maximum energy value (x e ,y e ) is converted into a corresponding time point and frequency point, and the frequency point is the main frequency. Then, according to the relative time position information of each sliced ​​audio segment relative to the original long audio, the start and end time period of the target bird sound relative to the sliced ​​audio segment is converted into the start and end time period of the original long audio, thereby achieving the purpose of retrieving the target bird sound from the long recording file.

[0246] (2)Reference Figure 6 , each channel of the four-channel bird sound signal collected on site (the channel range is channel 1, channel 2, channel 3, channel 4, a total of four channels) is converted into a corresponding spectrogram, and then the bird sound signal of each channel is detected using the above-mentioned trained model to obtain the start and end time t of the target bird sound in each channel start and t end And the maximum and minimum frequencies f low and f high Compare the start and end time of each channel with the maximum and minimum frequencies, and take the minimum value of the start time, the maximum value of the end time, the lowest value of the lowest frequency, and the highest value of the highest frequency of all channels as the new [t start , t end , f low , f high According to the frequency characteristics of the target bird sounds, a linear phase FIR bandpass filter is used to enhance the target signal and suppress the non-target signal. That is, each channel of the four-channel bird sound signal is filtered according to the start and end time tstart and t end The signal is intercepted and then filtered through a linear phase FIR bandpass filter. The frequency band of the bandpass filter is set to the minimum frequency and the maximum frequency f low and f high The frequency range between .

[0247] (3) The generalized cross-correlation algorithm (GCC) is used to calculate the time difference of arrival (TDOA) of the target bird data in each channel after truncation and bandpass filtering.

[0248] (4) In summary, after cross-correlating the signals of the target bird data of each channel after truncation and bandpass filtering, three delay estimates can be obtained, which are D 12 , D 13 , D 14 , and then perform azimuth Calculation, the calculation formula is as follows:

[0249]

[0250] The present invention successfully detects the start and end time, maximum and minimum frequency range and main frequency of various bird calls in a multi-category mixed national key protected bird sound data test set, can effectively retrieve the target bird sounds, and can also achieve efficient detection in a low signal-to-noise ratio environment. The average error of the target bird azimuth calculated from the four-channel recording signal collected on site is greatly reduced. The details are shown in Table 1 below:

[0251] Table 1 Results of azimuth positioning of sound source signals when R = 50m

[0252]

[0253] As shown in Table 1, when the sound source distance is 50 m, the average error of azimuth estimation for the Fork-tailed Sunbird song is 0.78°.

[0254] In summary, compared with the prior art, the present invention has the following advantages:

[0255] 1. Accurately monitor the issues of national key protected birds;

[0256] 2. Solve the problems of traditional bird sound detection methods lacking accurate time range information, frequency range information, main frequency information, poor detection effect in low signal-to-noise ratio environment, missed detection and no direction information;

[0257] Another aspect of the embodiment of the present invention further provides a bird sound event detection system for monitoring national key protected birds, including:

[0258] The first module is used to collect bird sound data of nationally protected birds and environmental noise data;

[0259] The second module is used to construct a bird sound spectrogram dataset and a bird sound dataset with noise according to the bird sound data and the environmental noise data;

[0260] The third module is used to build a bird sound noise reduction model and a target detection model;

[0261] The fourth module is used to train and jointly optimize the bird sound denoising model and the target detection model according to the bird sound spectrogram dataset and the noisy bird sound dataset; wherein the bird sound denoising model is used to perform denoising on the noisy bird sound data, and the target detection model is used to detect the target image;

[0262] A fifth module is used to construct an image-audio mapping module, which is used to convert the image detection result of the target detection model into a sound event detection result of the audio;

[0263] The sixth module is used to obtain the bird sound data to be detected, retrieve and locate the target bird sound according to the bird sound denoising model, the target detection model and the image-audio mapping module, and determine the species, time-frequency and direction information of the target bird sound

[0264] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0265] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned bird sound event detection method for monitoring national key protected birds when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0266] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0267] See also Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0268] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0269] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the bird sound event detection method for national key protected bird monitoring according to the embodiment of the present invention;

[0270] Input / output interface 703, used to implement information input and output;

[0271] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0272] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0273] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0274] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned bird sound event detection method for monitoring national key protected birds.

[0275] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0276] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0277] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0278] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0279] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0280] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0281] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0282] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0283] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0284] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0285] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0286] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0287] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0288] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A bird sound event detection method for monitoring national key protected birds, characterized in that: The following steps are involved: Collect bird sound data of nationally protected birds and environmental noise data; Constructing a bird sound spectrogram dataset and a bird sound dataset with noise according to the bird sound data and the environmental noise data; Build bird sound noise reduction model and target detection model; According to the bird sound spectrogram dataset and the noisy bird sound dataset, the bird sound denoising model and the target detection model are trained and jointly optimized; wherein the bird sound denoising model is used to perform denoising on the noisy bird sound data, and the target detection model is used to detect target images; Constructing an image-audio mapping module for converting the image detection result of the target detection model into a sound event detection result of the audio; The bird sound data to be detected is obtained, and the target bird sound is retrieved and the sound source is located according to the bird sound denoising model, the target detection model and the image-audio mapping module to determine the species, time-frequency and direction information of the target bird sound.

2. A bird sound event detection method for monitoring national key protected birds according to claim 1, characterized in that: The method of collecting bird sound data of nationally protected birds and environmental noise data comprises the following steps: According to the national first- and second-level protected bird lists, the sound data of these birds are collected, including data downloaded from the bird sound data website, or field recording data; the sampling rate is not less than 32kHz, and the quantization accuracy is not less than 16 bits; The collected data is processed in a unified manner and subjected to noise reduction to obtain clean bird sound data audio files after screening; According to the bird species, the screened audio files are connected to obtain the audio files of each bird species; Environmental noise data is obtained based on the noise data sampling rate requirements, quantization accuracy requirements and noise types; among them, noise types include target noises in the actual national key bird monitoring areas, including: wind noise, car noise, rain noise, stream human voice, cricket chirping, katydid chirping, cicada chirping and frog croaking.

3. The bird sound event detection method for monitoring national key protected birds according to claim 1, characterized in that: The step of constructing a bird sound spectrogram dataset and a bird sound dataset with noise according to the bird sound data and the environmental noise data comprises the following steps: Convert the acquired bird sound data from the time domain to the frequency domain to generate a bird sound spectrogram; Adjust the bird sound spectrogram image size and color depth; Use image annotation tools to draw bounding boxes for each target bird sound on the bird sound map, label the category, and save the annotation file; The bird sound spectrogram and the corresponding annotation files constitute the bird sound spectrogram dataset; Noise data is added to the acquired clean bird sound data to obtain the noisy bird sound data set.

4. The bird sound event detection method for monitoring national key protected birds according to claim 3, characterized in that: The step of converting the acquired bird sound data from the time domain to the frequency domain to generate a bird sound spectrogram comprises the following steps: Note the number of data sampling points P of a bird sound signal, set the frame length flen to 2048, and the frame shift hlen to 512, and get the number of frames fn of the bird sound signal. Each frame signal after framing is x(p,λ), where p is the sampling point number and λ is the frame number. The calculation formula for the number of frames of the bird sound signal is: The current frame x(p,λ) is overlapped with the previous frame x(p,λ-1) by a length of overlap=512 to obtain the target frame x o (p,λ), the total length of the frame after stacking is the frame length flen; For the target frame x o (p,λ) is windowed, the window type is Hamming window w(p,ψ), where ψ is 0.46, the window length is equal to the frame length point flen=2048, and all the windowed frames x are obtained. w (p,λ); For each frame x w (p,λ) performs a DFT operation of P = 2048 points, and takes the modulus of the DFT operation result to obtain the amplitude spectrum X(f,λ) of each frame signal, where f represents the frequency point and λ is the frame number. The first P of the spectrum is completed. f Analyze the frequency points to generate the bird sound spectrogram; The step of adjusting the image size and color depth of the bird sound spectrogram comprises the following steps: The calculation parameters for generating the spectrogram are unified, where the STFT window size is 2048, the frame shift is 512, and the image resolution DPI is 300; Set the vertical axis scale of the spectrogram to a linear scale and the color map to the 'inferno' mapping table; by setting the mapping table, frequency components of different intensities are mapped to specific colors in the bird sound spectrogram; The step of adding noise data to the acquired clean bird sound data to obtain the noisy bird sound data set comprises the following steps: Set the length of a bird sound signal to z seconds, the sampling rate to 32kHz, and the number of sampling points required to be z*32000. When the number of sampling points of the selected noise signal is greater than z*32000, randomly select z*32000 continuous sampling points from it; After obtaining clean bird sound signals and noise signals of the same length, the noise is added to the clean bird sound signal; one type of noise is selected each time, so that the proportion of noisy bird sound signals of each noise type in the entire noisy bird sound signal is uniform, and the number of noisy bird sound signals of different noise types is ensured to be the same, and the signals are mixed according to the following formula: Among them, t represents the time of the signal, z(t) is the noisy bird sound signal, z1(t) and n(t) are clean bird sound signals and noise signals respectively, ξ is the gain coefficient of n(t) in the mixing process, and snr represents the signal-to-noise ratio of clean bird sound signals and noise signals.

5. The bird sound event detection method for monitoring national key protected birds according to claim 1, characterized in that: The bird sound denoising model comprises a data input layer, an initial causal convolution layer, an expansion residual block, a skip connection summary and a final convolution output layer; the construction of the bird sound denoising model comprises the following steps: The input shape of the data input layer is (IN, 1), where IN represents the length of the input signal, which is used to receive the noisy bird sound signal; The initial causal convolution layer uses a causal convolution kernel to process the input signal so that each output depends on the current and previous input values. The number of convolution kernels in the initial causal convolution layer is 128, and the convolution kernel size is 3. The process can be expressed as: h causal =σ(W causal *x+b causal ) Among them, W causal and b causal are the causal convolution kernel weight and bias, respectively, h causal is the output of causal convolution, σ is the ReLU activation function; There are 30 dilated residual blocks in total, each of which includes a dilated convolutional layer, an activation function, a gating mechanism, a 1x1 convolutional layer, a skip connection, and a residual connection; the number of convolution kernels in the dilated convolutional layer is 256, the convolution kernel size is 3, the dilation rate increases in power from 1 to 512 (1, 2, 4, 8, 16, 32, 64, 128, 256, 512), there are 10 dilation rates, and the number of stacks is 3; the activation function includes the hyperbolic tangent tanh function and the sigmoid function, where the hyperbolic tangent tanh function is applied to the first part of the output of the dilated convolutional layer to generate h tanh , the sigmoid function is applied to the second output of the dilated convolutional layer to generate h sigma ; The gating mechanism converts h tanh and h sigma Combine to generate h gate ; Then, h gate Generate h through a 1x1 convolutional layer 1x1 The main function of the 1x1 convolution layer is to adjust the dimension of the feature map through linear transformation and generate residual output and jump connection output. The number of convolution kernels is 256, of which the number of residual channels and jump connections are 128 respectively; the jump connection part is from h 1x1 The skip connection output is extracted for aggregation in subsequent layers. The residual connection part adds the residual output of the 1x1 convolutional layer to the input signal to generate the output of the dilated residual block. The skip connection aggregation generates a comprehensive feature representation by summing the skip connection outputs of all dilated residual blocks; wherein the comprehensive feature representation contains the features of all layers, and provides a rich feature representation for the final output layer through feature information at different scales and different expansion rates. The formula for the summation operation is: Among them, s d represents the skip connection output of the d-th dilated residual block, D is the total number of dilated residual blocks; s represents the summation result; The final convolution output layer consists of three convolution layers, wherein the first convolution layer has 2048 convolution kernels and a convolution kernel size of 3, which are used to extract deeper features from the comprehensive feature representation of the jump connection; the second convolution layer has 256 convolution kernels and a convolution kernel size of 3, which compresses the feature map to a smaller dimension; the third convolution layer is a 1x1 convolution layer, whose convolution kernel number and size are both 1, which is used to perform a linear transformation on the feature map to generate the final denoised bird sound signal.

6. The bird sound event detection method for monitoring national key protected birds according to claim 1, characterized in that: The target detection model is constructed, comprising the following steps: Construct the Backbone network. The Backbone part includes the Stem Layer starting layer, the convolution modules from Stage 1 to Stage 4, the CSP layer, and an SPPF layer. The Stem Layer starting layer consists of a convolution module, which includes a two-dimensional convolution layer, a convolution operation followed by batch normalization and a SiLU activation function. The Stem Layer starting layer is used to preliminarily extract the low-level features of the image. Its operation process can be expressed by the following formula: y=SiLU(BatchNorm(Conv2d(x,W,b))) Where x is the input image, W and b are the convolution kernel weight parameter and bias respectively; Four Stage Layer layers are used to gradually extract and integrate features. Each Stage contains a convolution module and a CSP layer. The CSP layer is used to integrate features and reduce the amount of calculation. It includes two convolution modules and a residual connection and two DarknetBottleneck modules. Each convolution module includes a convolution layer, followed by a batch normalization layer and a SiLU activation function. The DarknetBottleneck module consists of two convolutional layers for efficient extraction of complex features. The first is a 1×1 convolutional layer to reduce the number of channels and improve computational efficiency, followed by a 3×3 convolutional layer to extract local features. Finally, the input and output are connected through a residual connection to enhance gradient transfer. The SPPF layer is used for multi-scale feature integration to enhance the model's ability to detect objects of different scales. It includes three pooling operations with pooling kernel sizes of 5, 9, and 13, which connect the pooling features of different scales to form a richer feature representation. The Neck part includes TopDown and BottomUp paths, which are used for top-down feature fusion and bottom-up feature extraction respectively. The TopDown path first performs an upsampling operation to upsample the feature map of the deeper layer to make its size match that of the feature map of the shallower layer, and then connects the downsampled feature map with the corresponding deeper feature map through a connection operation. The BottomUp path fuses the feature map of the shallower layer with the feature map of the deeper layer through a downsampling operation. The head part includes the basic convolution layer and the convolution layer dedicated to bounding box regression and classification; the basic convolution layer is used to preliminarily process the input feature map and extract more compact and useful features; Use the soft non-maximum suppression algorithm with Gaussian attenuation to attenuate the confidence of overlapping candidate boxes.

7. The bird sound event detection method for monitoring national key protected birds according to claim 1, characterized in that: The method of training and jointly optimizing the bird sound denoising model and the target detection model according to the bird sound spectrogram dataset and the noisy bird sound dataset comprises the following steps: The bird sound denoising network and target detection network are pre-trained based on the noisy bird sound dataset and the bird sound spectrogram dataset respectively; Construct a joint optimization framework for bird sound denoising and detection, and cascade the bird sound denoising network with the target detection network, including: Define the denoising-detection comprehensive loss function, where the denoising model uses mean square error loss, mean square error loss Loss MSE The calculation process is as follows: Where M represents the total number of samples. is the value of the kth sample predicted by the model, y k is the true value of the kth sample; The denoising-detection comprehensive loss function is defined as: Loss total =μLoss yolo +γLoss MSE Among them, Loss yolo =Loss EWA +CIoU+DFL, μ and γ are weight parameters; Loss EWA stands for energy-weighted attention loss; CIoU stands for complete intersection-over-union loss; DFL stands for distributional focus loss; Freeze the initial causal convolutional layer and the first 9 dilated residual blocks of the bird sound denoising network, and freeze the StemLayer starting layer and Stage 1 convolutional module of the object detection network; Set the number of epochs and batchsize to 50 and 32 respectively; The Adam optimizer is used as the stochastic batch gradient descent optimizer, and the initial learning rate lnitial_lr is set to 0.0001; During the training process, the model training is completed when the validation set loss value does not decrease after 10 epochs.

8. The bird sound event detection method for monitoring national key protected birds according to claim 1, characterized in that: The image-audio mapping module is constructed to convert the image detection result of the target detection model into a sound event detection result for the audio, comprising the following steps: Let the sampling rate of a bird sound audio signal be Sr, the total duration of the audio be Dur, and the width and height of the corresponding spectrogram image be set to width and height respectively. According to the duration and sampling rate of the audio signal, the total number of sampling points L can be calculated: L=Dur*Sr Where L represents the total number of sampling points of the audio signal, the sampling rate Sr of the audio signal refers to the number of samples collected per second, which is used for quantization of digital audio, and the audio duration Dur is the total duration of the audio signal; The number of sampling points S corresponding to each pixel on the time axis pixel for: Among them, width represents the width of the image; Temporal resolution refers to the length of time that each pixel corresponds to on the time axis, and is calculated as follows: Among them, t pixel Indicates the length of time corresponding to each pixel on the time axis; Frequency resolution refers to the frequency bandwidth corresponding to each pixel on the frequency axis. According to the Nyquist sampling theorem, f pixel : Among them, f pixel Indicates the frequency bandwidth corresponding to each pixel on the frequency axis; According to the input image pixel coordinate range (x min ,x max ), the corresponding audio signal time range can be calculated: t start =x min *t pixel t end =x max *t pixel Among them, t start and t end Respectively represent the starting point and ending point of time; According to the input image pixel coordinate range (y min ,y max ), calculate the corresponding audio signal frequency range: f low =(height-y max )*f pixel f high =(height-y min )*f pixel Among them, f low and f high Represent the lower and upper limits of the frequency respectively; the image pixel coordinates (y min ,y max ) represents the frequency interval selected on the image, which is converted into the actual frequency range through the frequency resolution.

9. The bird sound event detection method for monitoring national key protected birds according to claim 1, characterized in that: The step of obtaining the bird sound data to be detected, retrieving and locating the target bird sound according to the bird sound denoising model, the target detection model and the image-audio mapping module, and determining the species, time-frequency and direction information of the target bird sound comprises the following steps: The four-channel bird sound signal collected on site is first converted into a single-channel signal, and then sliced ​​at a fixed length and converted into a spectrogram image. When slicing, the relative time position information of each sliced ​​audio segment relative to the original long audio is recorded, and then the trained target detection model is used for detection to obtain the corresponding detection results. The detection results include the bird category and the image pixel coordinate information of the rectangular detection frame; when the model detects that the image contains the shape of the target bird sound, it will use a rectangular frame to frame the shape of the target bird sound. The size of the rectangular frame will change dynamically according to the change of the size of the target bird sound shape, and the image pixel coordinate range of each rectangular detection frame is converted into the corresponding time and frequency range, and the start and end time period, maximum and minimum frequency range and main frequency of the target bird sound are determined and output as the result, where the calculation process of the main frequency is: first calculate the rectangular detection frame B i The maximum internal energy value E max,i , and then calculate the maximum energy value E max,i The coordinates (x e ,y e ), the coordinate of the maximum energy value (x e ,y e ) is converted into a corresponding time point and frequency point, and the frequency point is determined as the main frequency; According to the relative time position information of each sliced ​​audio segment relative to the original long audio, the start and end time periods of the target bird sound relative to the sliced ​​audio segment are converted into the start and end time periods of the original long audio, thereby completing the target bird sound retrieval for the long recording file; Each channel of the four-channel bird sound signal collected on site is converted into a corresponding spectrogram, and the bird sound signal of each channel is detected using the trained model to obtain the start and end time t of the target bird sound in each channel. start and t end And the maximum and minimum frequencies f low and f high , compare the start and end time of each channel with the maximum and minimum frequencies, and take the minimum value of the start time, the maximum value of the end time, the lowest value of the lowest frequency, and the highest value of the highest frequency of all channels as the new [t start , t end , f low , f high ], for each channel of the four-channel bird sound signal, according to the start and end time t start and t end The signal is intercepted and then filtered through a linear phase FIR bandpass filter. The frequency band of the bandpass filter is set to the minimum frequency and the maximum frequency f low and f high The frequency range between The arrival time difference is calculated using the generalized cross-correlation algorithm for the intercepted and bandpass filtered target bird data of each channel; After cross-correlating the signals of the target bird data of each channel after truncation and bandpass filtering, three delay estimates D are obtained: 12 , D 13 , D 14 , and then perform azimuth Calculation, the calculation formula is:

10. A bird sound event detection system for monitoring national key protected birds, characterized in that: include: The first module is used to collect bird sound data of nationally protected birds and environmental noise data; The second module is used to construct a bird sound spectrogram dataset and a bird sound dataset with noise according to the bird sound data and the environmental noise data; The third module is used to build a bird sound noise reduction model and a target detection model; The fourth module is used to train and jointly optimize the bird sound denoising model and the target detection model according to the bird sound spectrogram dataset and the noisy bird sound dataset; wherein the bird sound denoising model is used to perform denoising on the noisy bird sound data, and the target detection model is used to detect the target image; A fifth module is used to construct an image-audio mapping module, which is used to convert the image detection result of the target detection model into a sound event detection result of the audio; The sixth module is used to obtain the bird sound data to be detected, retrieve and locate the target bird sound according to the bird sound denoising model, the target detection model and the image-audio mapping module, and determine the species, time-frequency and direction information of the target bird sound.

Images