Birdsong recognition model training method, recognition method and storage medium

Abstract

The invention discloses a bird song recognition model training method, a recognition method and a storage medium. The invention performs first processing on a pre-training audio data set, a pre-training bird sound data set and a target bird sound data set respectively to obtain corresponding first training data. The first processing comprises: performing first frame processing on the pre-training audio data set, the pre-training bird sound data set and the target bird sound data set respectively and determining a first amplitude spectrum and first slice data; performing second frame processing according to the first slice data and determining a second amplitude spectrum, determining a first Mel spectrum according to the second amplitude spectrum and a Mel filter group and performing frequency-time masking processing; performing second processing on a negative sample data set to obtain second training data, training a bird song recognition model according to the first training data and the second training data to obtain a target bird song recognition model, and improving the recognition accuracy of the target bird song recognition model. The invention can be widely applied to the field of sound recognition.

Description

Technical Field

[0001] The present invention relates to the field of sound recognition, and in particular to a bird song recognition model training method, a recognition method and a storage medium. Background Art

[0002] There are nearly 10,000 bird species in the world. Relevant data show that the population of some bird species is showing a downward trend. Human protection of birds has reached an urgent point. The richness of bird species and the activity of birds can often infer the environmental quality of the area. Therefore, how to monitor birds is an important step in protecting birds.

[0003] Current bird monitoring methods include traditional manual sampling, line sampling, and manually deployed infrared cameras. These traditional methods suffer from low efficiency and limited coverage, and the placement of infrared cameras is also affected by the shooting angle and power supply. In reality, compared to observing morphological features such as beaks or feathers through photos or videos, the biological information contained in bird songs varies significantly, and this difference can help distinguish between different species. Therefore, bird song-based bird monitoring is an emerging method that can extract and analyze the information contained in bird songs to identify bird species and analyze the distribution of birds within ecological zones, thereby helping to understand biodiversity and monitor ecosystem balance. However, current bird song monitoring methods rely on template matching classification algorithms or deep learning. Template matching-based classification algorithms lack robustness and are limited in application scenarios. Deep learning algorithms also perform poorly when data is scarce or of low quality, resulting in poor recognition performance of trained models. Therefore, a solution is needed. Summary of the Invention

[0004] In view of this, in order to solve the above technical problems, the purpose of the present invention is to provide a bird song recognition model training method, recognition method and storage medium.

[0005] The technical solution adopted in the present invention is:

[0006] A bird song recognition model training method, comprising:

[0007] Acquire a data set; the data set includes a pre-trained audio data set, a pre-trained bird sound data set, a target bird sound data set, and a negative sample data set;

[0008] performing a first processing on the pre-training audio dataset, the pre-training bird sound dataset, and the target bird sound dataset, respectively, to obtain first training data corresponding to the pre-training audio dataset, the pre-training bird sound dataset, and the target bird sound dataset; the first processing comprising: performing a first framing processing on the pre-training audio dataset, the pre-training bird sound dataset, and the target bird sound dataset, respectively, and determining a first amplitude spectrum based on the first framing processing result; determining first slice data consisting of valid frames based on the first amplitude spectrum; performing a second framing processing on the first slice data and determining a second amplitude spectrum based on the second framing processing result; determining a first mel spectrum based on the second amplitude spectrum and a mel filter bank; and performing frequency-time masking processing on the first mel spectrum;

[0009] Performing a second processing on the negative sample data set to obtain second training data;

[0010] The bird song recognition model is trained according to the first training data and the second training data to obtain a target bird song recognition model.

[0011] Furthermore, the performing first frame segmentation processing on the pre-trained audio dataset, the pre-trained bird sound dataset, and the target bird sound dataset respectively includes:

[0012] Frame the pre-trained audio dataset, the pre-trained bird sound dataset, and the target bird sound dataset according to a preset frame length and a preset frame shift, respectively, to obtain corresponding frame data;

[0013] Determine the first frame and the second frame from the frame data respectively; the second frame is the previous frame of the first frame;

[0014] After stacking the first frame and the second frame, the second frame is used as the new first frame, and the process returns to the step of determining the second frame until the first frame is the last frame of the frame data, thereby obtaining stacked frame data corresponding to the frame data; the stacked frame data is the first frame processing result.

[0015] Furthermore, determining the first amplitude spectrum according to the first frame processing result includes:

[0016] Performing a first windowing process on the frame stack data to obtain a first windowing process result;

[0017] Performing Fourier transform processing on the first windowing processing result;

[0018] The modulus of the Fourier transform operation result is determined to obtain a first amplitude spectrum of each frame of data in the frame stack data.

[0019] Further, determining first slice data consisting of valid frames according to the first amplitude spectrum includes:

[0020] Calculating the decibel value of each frequency point of the first amplitude spectrum to obtain the amplitude decibel value of each frequency point;

[0021] Calculating an average value based on the amplitude decibel value to determine the average decibel value of each frame of data; each frame of data includes a number of frequency points;

[0022] The data corresponding to the average decibel value being greater than or equal to the decibel threshold is regarded as a valid frame;

[0023] Splicing the valid frames in chronological order to obtain spliced ​​data;

[0024] The spliced ​​data is subjected to a first slicing process according to a preset slicing length to obtain first slicing data.

[0025] Furthermore, the mel filter bank includes at least one mel filter, and determining the first mel spectrum according to the second amplitude spectrum and the mel filter bank includes:

[0026] calculating the square of the second amplitude spectrum to obtain a frequency spectrum;

[0027] Determining an actual frequency upper limit and an actual frequency lower limit according to the bird sound vocalization frequencies of various bird species in the pre-trained bird sound dataset and the target bird sound dataset;

[0028] According to the conversion equation between Mel frequency and actual frequency, the upper limit of the bandpass frequency of the Mel filter group corresponding to the upper limit of the actual frequency and the lower limit of the bandpass frequency of the Mel filter group corresponding to the lower limit of the actual frequency are calculated;

[0029] The frequency spectrum is processed according to the Mel filter to obtain a first Mel spectrum; the center frequency of the Mel filter is determined according to the bandpass frequency upper limit and the bandpass frequency lower limit.

[0030] Furthermore, performing frequency-time masking processing on the first mel spectrum includes:

[0031] Determine the upper and lower limits of bird song frequencies for various bird species;

[0032] Calculating a first difference between the upper limit of the bird singing frequency and a preset shielding width and determining a first threshold; the first threshold is greater than or equal to the lower limit of the bird singing frequency and less than or equal to the first difference;

[0033] Calculating the sum of the first threshold and the preset shielding width to obtain a second threshold;

[0034] Determining a shielded frequency channel range according to the first threshold and the second threshold, and performing frequency channel shielding on the first mel spectrum according to the shielded frequency channel range;

[0035] Determining the total number of frames in the second framing process, calculating a second difference between the total number of frames and a preset time frame mask width and determining a third threshold; the third threshold is greater than or equal to 1 and less than or equal to the second difference;

[0036] Calculating the sum of the third threshold and the preset time frame masking width to obtain a fourth threshold;

[0037] The masking time channel range is determined based on the third threshold and the fourth threshold, and the frequency channel masking result is subjected to time channel masking based on the masking time channel range and the second preset channel number to obtain first training data corresponding to the pre-trained audio dataset, the pre-trained bird sound dataset, and the target bird sound dataset.

[0038] Furthermore, performing a second processing on the negative sample data set to obtain second training data includes:

[0039] Performing a second slicing process on the negative sample data set to obtain second slicing data;

[0040] Perform a third frame processing on the second slice data and determine a third amplitude spectrum according to the third frame processing result; determine a second Mel spectrum according to the third amplitude spectrum and the Mel filter bank and use it as the second training data.

[0041] Furthermore, the bird song recognition model includes a feature extraction backbone module and a bird song recognition detection block, and the bird song recognition detection block includes a first branch and a second branch; the bird song recognition model is trained based on the first training data and the second training data to obtain a target bird song recognition model, including:

[0042] Constructing third training data based on the second training data and the first training data corresponding to the target bird sound dataset;

[0043] Inputting the first training data corresponding to the pre-trained audio dataset and the pre-trained bird sound dataset, and the third training data, into a feature extraction backbone module for feature extraction processing, aggregating the embedded features obtained by the feature extraction processing in the time dimension to obtain a sub-segment level prediction result; the feature extraction backbone module is trained based on hierarchical transfer learning;

[0044] Inputting the sub-segment-level prediction results into the first branch and performing a first convolution process to obtain segment-level prediction results of the bird sounds of each sub-segment;

[0045] The sub-segment-level prediction result is input into the second branch, subjected to a second convolution process and input into the attention block for classification, the output of the attention block is multiplied by the segment-level prediction result and aggregated based on the time of the sub-segment to obtain a frame-level prediction result;

[0046] A loss function value is calculated according to the segment-level prediction result or the frame-level prediction result, and a target bird song recognition model is determined by fine-tuning according to the loss function value.

[0047] The present invention also provides a method for identifying bird calls, comprising:

[0048] The bird song data to be detected is input into the target bird song recognition model to obtain a recognition result; the recognition result includes a segment-level recognition result or a frame-level recognition result, the segment-level recognition result includes the bird song category in the bird song data to be detected, and the frame-level recognition result includes the bird song category in the bird song data to be detected and the start time and end time of the bird song.

[0049] The present invention also provides a bird song recognition device, comprising a processor and a memory;

[0050] The memory stores a program;

[0051] The processor executes the program to implement the method.

[0052] The present invention also provides a computer-readable storage medium, wherein the storage medium stores a program, and the program implements the method when executed by a processor.

[0053] The present invention has the following beneficial effects: by performing a first processing on a pre-training audio dataset, a pre-training bird sound dataset, and a target bird sound dataset, respectively, first training data corresponding to the pre-training audio dataset, the pre-training bird sound dataset, and the target bird sound dataset are obtained. The first processing includes: performing a first frame processing on the pre-training audio dataset, the pre-training bird sound dataset, and the target bird sound dataset, respectively, and determining a first amplitude spectrum based on the first frame processing result; determining first slice data consisting of valid frames based on the first amplitude spectrum, which can enhance the data set and is beneficial to improving the quality of the first training data; performing a second frame processing on the first slice data and determining a second amplitude spectrum based on the second frame processing result; determining a first mel spectrum based on the second amplitude spectrum and a mel filter bank; and performing frequency and time masking processing on the first mel spectrum, making the first training data more robust and expanding the applicable scenarios; performing a second processing on the negative sample dataset to obtain second training data, and training a bird song recognition model based on the first training data and the second training data to obtain a target bird song recognition model, which is beneficial to improving the recognition accuracy of the target bird song recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the steps of the bird song recognition model training method of the present invention;

[0055] Figure 2 Schematic diagram of a bird song recognition model according to a specific embodiment of the present invention;

[0056] Figure 3 Schematic diagram of a hierarchical transfer learning pre-model according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0058] The terms "first," "second," "third," and "fourth," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, rather than to describe a specific order. In addition, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0059] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] like Figure 1 As shown, the embodiment of the present invention provides a bird song recognition model training method, including steps S100-S400:

[0061] S100: Acquire a data set.

[0062] In an embodiment of the present invention, the data set includes a pre-trained audio data set, a pre-trained bird sound data set, a target bird sound data set, and a negative sample data set. Optionally, the data of the pre-trained audio data set may include audio data (signals) such as human speech, various animal calls, and various musical instrument sounds; the negative sample data set may include natural noise (wind, rain, running water, etc.), urban noise (traffic noise, construction noise, social life noise), etc., without specific limitation, and specific noise categories that may be encountered in actual projects may be added; the pre-trained bird sound data set contains a large amount of audio data of different bird species; the target bird sound data set contains audio data of a variety of bird species, and the number of bird species is smaller than the pre-trained bird sound data set, and the model trained with the pre-trained bird sound data set is then fitted to the target bird sound data set, so that the final model trained in this way has higher robustness than the model directly fitting the target bird sound data set, avoiding overfitting of the model due to insufficient number of bird species in the target bird sound data set. It should be noted that each data set contains segment-level annotated data (including real labels), which can be derived from various bird-loving recording websites. Optionally, for the quantitative relationship of the number of categories of the above data sets, it is generally set to N sound >N bird-source >N bird-target >N negative (including but not limited to 100>50>20>10). For the total duration of each category of the above dataset, it can be set to K sound >K bird-source >K bird-target >K negative At the same time, the total duration of each category of data in each dataset is required to be no less than 10 minutes, and each category of data can include several audio files / data. Table 1 shows the relevant information of the dataset.

[0063] Table 1

[0064]

[0065] S200: Perform first processing on the pre-training audio dataset, the pre-training bird sound dataset, and the target bird sound dataset to obtain first training data corresponding to the pre-training audio dataset, the pre-training bird sound dataset, and the target bird sound dataset.

[0066] It should be noted that the pre-training audio dataset is first processed to obtain the first training data corresponding to the pre-training audio dataset, the pre-training bird sound dataset is first processed to obtain the first training data corresponding to the pre-training bird sound dataset, and the target bird sound dataset (local target bird sound dataset) is first processed to obtain the first training data corresponding to the target bird sound dataset (local target bird sound dataset).

[0067] Optionally, the first processing includes: performing a first frame processing on the pre-trained audio dataset, the pre-trained bird sound dataset, and the target bird sound dataset, respectively, and determining a first amplitude spectrum based on the first frame processing results, determining first slice data consisting of valid frames based on the first amplitude spectrum, performing a second frame processing on the first slice data and determining a second amplitude spectrum based on the second frame processing results, determining a first Mel spectrum based on the second amplitude spectrum and a Mel filter group, and performing frequency-time masking processing on the first Mel spectrum.

[0068] Specifically, the pre-training audio data set, the pre-training bird sound data set, and the target bird sound data set are respectively subjected to the first frame segmentation process, including steps S211-S213:

[0069] S211 , dividing the pre-trained audio dataset, the pre-trained bird sound dataset, and the target bird sound dataset into frames according to a preset frame length and a preset frame shift, and obtaining corresponding frame data.

[0070] Optionally, the pre-trained audio dataset, the pre-trained bird sound dataset, and the target bird sound dataset are each framed according to a preset frame length and a preset frame shift. In this embodiment of the present invention, the number of data sampling points N of an audio file is recorded, the frame length is set to wlen (preset frame length, including but not limited to 1024), and the frame shift inc is set to (preset frame shift, including but not limited to 256). The number of frames nf of the audio file is obtained, where λ is the frame number, λ∈[1,nf].

[0071]

[0072] Each frame after the scoring frame is x in (n,λ),x in (n,λ) constitutes the frame data, where n is the sampling point number, n=0,1,...,N F -1, where N F The total length of the frame after stacking (can be set to an integer power of 2, such as 1024).

[0073] S212 , determining a first frame and a second frame from the frame data respectively; the second frame is a previous frame of the first frame.

[0074] S213, after stacking the first frame and the second frame, use the second frame as the new first frame, and return to the step of determining the second frame until the first frame is the last frame of the frame data, and obtain the frame stacking data corresponding to the frame data; the frame stacking data is the first frame processing result.

[0075] Specifically, the current frame is determined, and the current frame is overlapped with the adjacent frame, for example, the current frame is overlapped with the previous frame or the current frame is overlapped with the next frame, which is not specifically limited. In the embodiment of the present invention, the second frame is x in(n,λ-1), the first frame is x in (n,λ) as an example, the adjacent frames are stacked until the first frame is the last frame of the frame data (that is, there is no previous frame), thereby determining the frame stack data corresponding to the frame data. Specifically, the current frame x in (n,λ) and the previous frame x in (n,λ-1) performs frame stacking to obtain the frame stacking result x on (n,λ), the frame stacking formula is as follows:

[0076]

[0077] Among them, N L N is the sampling point number at the beginning of the frame stack. F is the total length of the frame after stacking, each x on (n,λ) constitutes the frame stacking data, that is, x on (n,λ) is a portion of the frame-stacked data. Specifically, the frame-stacked data of the pre-training audio dataset, the frame-stacked data of the pre-training bird sound dataset, and the frame-stacked data of the target bird sound dataset are obtained by frame-stacked.

[0078] Specifically, determining the first amplitude spectrum according to the first frame processing result includes steps S221-S223:

[0079] S221 . Perform a first windowing process on the frame stacking data to obtain a first windowing process result.

[0080] Specifically, for x on (n,λ) is windowed. In the embodiment of the present invention, the Hamming window is used as an example, and is not specifically limited in other embodiments. The Hamming window w(n,α), where w() is a vector, α is 0.46, and the window length (i.e., the length of w(n,α), the value range of n) is equal to the frame length point N F (total length), thus obtaining all windowed frames x w (λ,n) (result of the first windowing process):

[0081]

[0082] x w (n,λ)=x on (n,λ)*w(n,α)0≤n≤N F -1 (4)

[0083] S222: Perform Fourier transform processing on the first windowing processing result.

[0084] S223 : Determine the modulus value of the Fourier transform operation result to obtain a first amplitude spectrum of each frame of the frame-stacked data.

[0085] Optionally, the Fourier transform operation processing may be performed using DFT (Discrete Fourier Transform) or FFT (Fast Fourier Transform), without specific limitation. In the embodiment of the present invention, DFT is used as an example for description. Specifically:

[0086] For each frame x w (n,λ) performs Fourier transform operation on N points (generally a power of 2, such as 1024), and takes the modulus of the Fourier transform operation result to obtain the amplitude spectrum of each frame signal (data). The first amplitude spectrum X(k,λ), where k represents the frequency point, λ is the frame number, and j is the imaginary unit. Due to the symmetry of Fourier transform, only the first N of the spectrum are processed. f Frequency points are analyzed, where N f =N / 2+1.

[0087]

[0088] Specifically, determining the first slice data consisting of valid frames according to the first amplitude spectrum includes steps S231-S235:

[0089] S231 , calculating the decibel value of each frequency point of the first amplitude spectrum to obtain the amplitude decibel value of each frequency point.

[0090] S232: Calculate the average value based on the amplitude decibel value to determine the average decibel value of each frame of data.

[0091] It should be noted that each frame of data includes several frequency points.

[0092] Specifically, the decibel value is calculated at each frequency point of X(k,λ), and the amplitude decibel value dB(k,λ) at each frequency point of each frame is obtained. Finally, the average decibel value dB is calculated. a (λ) represents the decibel value of this frame signal:

[0093] dB(k,λ)=20lg|X(k,λ)|0≤k≤N f -1 (6)

[0094]

[0095] S233: The data corresponding to the average decibel value being greater than or equal to the decibel threshold is regarded as a valid frame.

[0096] S234: Splice the valid frames in chronological order to obtain spliced ​​data.

[0097] S235 : Perform first slicing processing on the spliced ​​data according to a preset slicing length to obtain first slicing data.

[0098] In an embodiment of the present invention, a decibel threshold ζ is set, and the size of the decibel threshold ζ can be set according to actual needs. Optionally, in an embodiment of the present invention, taking the value range of ζ as 20dB-30dB as an example, when the average decibel value is lower than the decibel threshold ζ, the frame (data) is an invalid frame, and when the average decibel value is greater than or equal to the decibel threshold ζ, the frame (data) is a valid frame, thereby counting all valid frames of an audio data, and splicing the valid frames in chronological order in the time dimension to obtain spliced ​​data Y_ping. Then, the spliced ​​data Y_ping is subjected to a first slicing process according to a preset slicing length to obtain a first slicing data Y_cut. Optionally, the first slicing process can adopt the Hamming window method, and with reference to formula (3), the slicing window length (preset slicing length) is set to N cut (including but not limited to 64000), if there is a length less than N cut The splicing data needs to be padded with zeros to N cut , thus converting the completed splicing data Y_ping into a certain number of length N cut The first slice data Y_cut.

[0099] Optionally, a second framing process is performed on the first slice data and a second amplitude spectrum is determined based on the second framing process result. The second framing process is similar to the first framing process, and the first slice data is framed and stacked to obtain the second framing process result. Determining the second amplitude spectrum based on the second framing process result is similar to determining the first amplitude spectrum based on the first framing process result. The framing process result is windowed, Fourier transformed, and the modulus of the Fourier transform process result is determined to obtain the second amplitude spectrum of each frame data in the frame stacked data (second framing process result). Details will not be repeated here.

[0100] In an embodiment of the present invention, the mel filter bank includes at least one mel filter; specifically, determining the first mel spectrum according to the second amplitude spectrum and the mel filter bank includes steps S241-S244:

[0101] S241. Calculate the square of the second amplitude spectrum to obtain a frequency spectrum.

[0102] Alternatively, assuming that the second amplitude spectrum is X p (k1,λ1), then the frequency spectrum of the first slice data is X p 2 (k1,λ1), where k1 represents the frequency point and λ1 represents the frame number.

[0103] S242. Determine an actual frequency upper limit and an actual frequency lower limit based on the bird sound frequencies of various bird species in the pre-trained bird sound dataset and the target bird sound dataset.

[0104] Specifically, the actual upper frequency limit is f h and the actual frequency lower limit is f l , for example, f l =200Hz, f h =14000Hz, the highest frequency among all bird calls is taken as f h , the lowest frequency of all bird calls is taken as f l .

[0105] S243. Calculate the upper passband frequency limit of the Mel filter group corresponding to the upper limit of the actual frequency and the lower passband frequency limit of the Mel filter group corresponding to the lower limit of the actual frequency according to the conversion equation between the Mel frequency and the actual frequency.

[0106] Specifically, the transformation equation is:

[0107]

[0108] Among them, F mel (f) is the conversion result, f is the frequency, and the upper and lower limits of the bandpass frequency of the Mel filter group are determined by calculation to obtain the upper limit of the bandpass frequency F mel (f h ) and the lower limit of the passband frequency F mel (f l ).

[0109] S244 : Process the frequency spectrum according to the Mel filter to obtain a first Mel spectrum.

[0110] It should be noted that the center frequency of the Mel filter is determined according to the upper limit and lower limit of the passband frequency.

[0111] In the embodiment of the present invention, a Mel filter bank is constructed to filter the frequency range of bird sounds (birdsong) (e.g., f∈[f l ,f h ]) Set several (Mel filter) bandpass filters H(k1,m), 1≤m≤M, where k1 represents the frequency point, λ1 represents the frame number, m is the filter index, and M is the total number of Mel filters. It should be noted that the value of M can be adjusted as needed and is not specifically limited. The larger the value of M, the better the frequency resolution of the obtained Mel spectrum. In the embodiment of the present invention, M is 256. The expression of H(k1,m) is shown in formula (9). Each Mel filter has a triangular filtering characteristic, and its center frequency is f(m) (the center frequency of the mth filter), as shown in formula (10):

[0112]

[0113]

[0114] N1 is the length of DFT (or FFT), f s is the sampling frequency, F mel The inverse function of (f), b represents the Mel frequency, Converts the Mel frequency value b (in mel) to frequency (in hz).

[0115] In the embodiment of the present invention, the frequency spectrum X of the first slice data Y_cut is p 2 (k1,λ1) obtains the first Mel spectrum of the first slice data Y_cut through the Mel filter H(k1,m) See formula (11), so we can Consider it as a spectrogram with a time dimension of λ1∈[1,247] and a frequency dimension of m∈[1,256]:

[0116]

[0117] Wherein, nf1 is the number of divided frames for the second processing.

[0118] Specifically, the first mel spectrum is subjected to frequency-time masking processing, which includes frequency channel masking and time channel masking, that is, a certain range of passbands are selected and set to 0 in the horizontal axis direction (frequency) and the vertical axis direction (time) of the spectrum.

[0119] Specifically, frequency channel shielding includes steps S251-S253;

[0120] S251. Determine the upper limit and lower limit of bird song frequencies corresponding to various bird species.

[0121] Optionally, the upper limit of the bird sound frequency f can be determined based on the bird sound frequencies of various bird species in the pre-trained bird sound dataset or the target bird sound dataset. 鸟max And the lower limit of bird song frequency f 鸟min , or by searching the data to determine the upper limit of bird song frequency f corresponding to various bird species 鸟max And the lower limit of bird song frequency f 鸟min , no specific limitation is given.

[0122] S252. Calculate a first difference between the upper limit of the bird singing frequency and the preset shielding width and determine a first threshold; the first threshold is greater than or equal to the lower limit of the bird singing frequency and less than or equal to the first difference.

[0123] S253: Calculate the sum of the first threshold and the preset shielding width to obtain a second threshold.

[0124] S254: Determine a masked frequency channel range according to the first threshold and the second threshold, and perform frequency channel masking on the first mel spectrum according to the masked frequency channel range.

[0125] Specifically, the shielding frequency channel range is [m1, m1+f * ), where m1∈[f 鸟min , f 鸟max -f * ], where f * The preset shielding width is adjustable. For example, N% (N∈[0,20)) of the corresponding bird species’ sound frequency bandwidth can be set. The first threshold is m1, and the second threshold is m1+f * , and then set the number of frequency shielding channels N according to the shielding frequency channel range F (For example, 2) frequency channel masking is performed on the first mel spectrum to obtain a frequency channel masking result.

[0126] Specifically, the time channel shielding includes steps S261-S263:

[0127] S261, determining the total number of frames in the second framing process, calculating a second difference between the total number of frames and a preset time frame mask width and determining a third threshold; the third threshold is greater than or equal to 1 and less than or equal to the second difference.

[0128] Optionally, the total number of frames divided in the second frame division process is the total number of frames after the first slice data is divided and stacked.

[0129] S262: Calculate the sum of the third threshold and the preset time frame masking width to obtain a fourth threshold.

[0130] S263. Determine a masked time channel range based on the third threshold and the fourth threshold, and perform time channel masking on the frequency channel masking result based on the masked time channel range to obtain first training data corresponding to the pre-trained audio dataset, the pre-trained bird sound dataset, and the target bird sound dataset.

[0131] Specifically, the masked time range channel is [λ2,λ2+T * ), where λ2∈[1, total number of frames - T * ], T * It is an adjustable preset time frame mask width. For example, it can be set to N% of the bird species’ sound frequency bandwidth (N∈[0,20)). λ2 is the third threshold, λ2+T * is the fourth threshold, and then the number of shielding time channels N is set according to the shielding time range channel T, (for example 2), perform time channel masking on the frequency channel masking result, and set N to zero in the time channel λ2∈[1,nf] range T It should be noted that, in the embodiment of the present invention, the number of shielding time channels N T Number of channels N with frequency shielding F The same; the first training data corresponding to the final pre-training audio dataset, pre-training bird sound dataset and target bird sound dataset are Mel spectrograms processed by frequency channel masking and time channel masking.

[0132] S300: Perform a second process on the negative sample data set to obtain second training data.

[0133] Specifically, step S300 includes steps S310-S320:

[0134] S310 , performing a second slicing process on the negative sample data set to obtain second slicing data.

[0135] S320 , performing a third frame processing on the second slice data and determining a third amplitude spectrum according to the third frame processing result, and determining a second mel spectrum according to the third amplitude spectrum and the mel filter bank and using the second training data.

[0136] Similarly, the second slice processing principle is the same as the first slice processing principle, and the third framing process is similar to the first framing process. The second slice data is framed and stacked to obtain the third framing process result. Determining the third amplitude spectrum based on the third framing process result is similar to determining the first amplitude spectrum based on the first framing process result. The framing process result is windowed, Fourier transformed, and the modulus of the Fourier transform result is determined to obtain the third amplitude spectrum of each frame of the frame-stacked data (the third framing process result). Similarly, the method for determining the second mel spectrum based on the third amplitude spectrum and the mel filter bank is the same as the method for determining the first mel spectrum based on the second amplitude spectrum and the mel filter bank, and will not be repeated here.

[0137] S400: Training a bird song recognition model according to the first training data and the second training data to obtain a target bird song recognition model.

[0138] like Figure 2 As shown, in the embodiment of the present invention, the bird sound recognition model includes a feature extraction backbone module (CNN**_Blocks) and a bird sound recognition detection block (ASC_SED_Block), and the bird sound recognition detection block includes a first branch and a second branch.

[0139] Optionally, CNN**_Blocks consists of 6 feature extraction blocks (CNN*_Block), and CNN*_Block consists of a convolutional layer (CNN layer), a batch normalization layer (BN layer), and a pooling layer (pooling layer). The convolution kernel size of each CNN*_Block is set to 3×3, and the number of convolution kernels in the convolution layers of the first to sixth CNN*_Blocks is set to 64, 128, 256, 512, 1024, and 2048 respectively. The Pooling layer of each CNN*_Block is set to the maximum pooling mode with a pooling size of 2×2.

[0140] Specifically, step S400 includes steps S410-S450:

[0141] S410 : Construct third training data based on the second training data and the first training data corresponding to the target bird sound dataset.

[0142] In this embodiment of the present invention, the second training data is included as a negative sample class in the first training data corresponding to the target bird sound dataset (local target bird sound dataset), forming the third training data. Optionally, the third training data can be allocated as needed, including but not limited to dividing the training set into a validation set in a ratio of 8:2. The third training data is used for fine-tuning the bird sound recognition model ASC_SED_MODEL.

[0143] S420. Input the first training data and the third training data corresponding to the pre-trained audio dataset and the pre-trained bird sound dataset into the feature extraction backbone module for feature extraction processing, aggregate the embedded features obtained by the feature extraction processing in the time dimension, and obtain the prediction results at the sub-segment level; the feature extraction backbone module is obtained based on hierarchical transfer learning training.

[0144] In an embodiment of the present invention, the pre-trained audio dataset and the first training data corresponding to the pre-trained bird sound dataset are divided into a training set and a validation set in a ratio of 8:2, which are used to train a pre-model of hierarchical transfer learning using a hierarchical transfer learning strategy to obtain a feature extraction backbone module.

[0145] Specifically, a hierarchical transfer learning pre-model CNN_MODEL is constructed. The structure of CNN_MODEL is as follows: Figure 3 As shown in the figure, it consists of feature CNN_Blocks, CNN*_Blocks and fully connected layers. By adopting a hierarchical transfer learning strategy on CNN_MODEL, the feature extraction backbone module CNN**_Blocks after transfer learning is obtained. The specific transfer learning strategy training process is as follows:

[0146] 1) For CNN_Blocks, set the classification fully connected layer FC (first fully connected layer) according to the category of the first training data corresponding to the pre-training audio dataset, input the first training data corresponding to the pre-training audio dataset to obtain the first embedding feature (Embedding1), and then set the number of small batch samples for training to 32, and use the initial learning rate of 1x10 -3 The Adam optimizer is used to optimize the model. After training the model for 50 epochs (periods), the model is saved and the feature extraction backbone module CNN**_Blocks completed in the first training can be obtained.

[0147] 2) For CNN*_Blocks, copy the weights of CNN_Blocks to CNN*_Blocks, add a new classification fully connected layer FC (second fully connected layer) according to the category of the first training data corresponding to the pre-trained bird sound dataset, input the first training data corresponding to the pre-trained bird sound dataset to obtain the second embedding feature (Embedding2), set the number of small batch samples for training to 32, and use an initial learning rate of 1x10 -4 The Adam optimizer is used to optimize the model. The model is fine-tuned on the third training data for 50 epochs and then saved. A new feature extraction backbone module CNN**_Blocks is obtained, which serves as the feature extraction backbone module in the bird song recognition model.

[0148] Specifically, the first training data and the third training data corresponding to the pre-trained audio dataset and the pre-trained bird sound dataset are input into the feature extraction backbone module in the bird sound recognition model for feature extraction processing. The embedded features (Embedding) obtained by the feature extraction processing are aggregated in the time dimension to obtain the sub-segment level prediction results (batch_size, 2048, sub-segment), and the sub-segment level prediction results are used as the input of the bird sound recognition detection block ASC_SED_Block.

[0149] S430: Input the sub-segment-level prediction results into the first branch, perform a first convolution process, and obtain segment-level prediction results of the bird sounds of each sub-segment.

[0150] Specifically, the first convolution processing of the sub-segment-level prediction results includes: convolving the sub-segment-level prediction results with several (the number of categories in the target bird sound dataset) 1x1 convolution kernels, and then processing them with a sigmoid activation function (Formula 12) to obtain the segment-level prediction results segmentwise_output of the bird sounds of each sub-segment.

[0151]

[0152] Among them, x1 is the tensor input to the sigmoid function σ(), that is, the segment-level prediction result of S430.

[0153] S440. Input the prediction result at the sub-segment level into the second branch, perform a second convolution process and input the attention block for classification, multiply the output of the attention block with the segment-level prediction result and aggregate based on the time of the sub-segment to obtain the frame-level prediction result.

[0154] Specifically, the second convolution process includes a number of 1x1 convolution kernels (the number of categories in the target bird sound dataset), followed by a tanh activation function and a softmax function (Equations 13-14). The attention block formula for classification is shown in (15):

[0155]

[0156]

[0157] att_norm=softmax(tanh(x1)) (15)

[0158] Among them, x1 is the prediction result of the sub-segment level, P(i) represents the probability of the i-th category, and V i is the tanh(x1) corresponding to category index i, and K is the total number of bird categories. It should be noted that the Softmax function can be used to convert the output values ​​of multiple classifications into relative probabilities. The final output of the attention block (att_norm) is multiplied by the clip-level prediction result to obtain the classification prediction results of each sub-clip. The sub-clip is then aggregated on the time axis to obtain the classification prediction result of the total clip, clipwise_output, which is the frame-level prediction result.

[0159] S450: Calculate a loss function value based on the segment-level prediction result or the frame-level prediction result, and determine a target bird song recognition model through fine-tuning based on the loss function value.

[0160] Optionally, the loss function value can be calculated based on the segment-level prediction result or the frame-level prediction result and the true result (true label) by the binary cross entropy (BCE) loss function, and the initial learning rate is 1x10 -5 Fine-tune the Adam optimizer and train for a preset number of epochs (e.g., 10) or terminate the training when the loss function value is less than or equal to the loss threshold to determine the target bird song recognition model.

[0161] Specifically, taking the segment-level prediction results as an example, the segment-level prediction results and the segment-level true labels are used to calculate the loss function value through the binary cross entropy (BCE) loss function:

[0162]

[0163] Among them, Loss is the loss function value, is the predicted value (segment-level prediction result), y i is the true label at the segment level, and N2 is the number of bird categories.

[0164] The present invention also provides a method for identifying bird sounds, including:

[0165] The bird song data to be detected is input into the target bird song recognition model to obtain the recognition result.

[0166] Recognition results include either segment-level or frame-level recognition results. Segment-level recognition results are weakly labeled outputs, including the bird song category in the bird song data to be detected. Frame-level recognition results are strongly labeled outputs, including the bird song category in the bird song data to be detected, as well as the start and end times of the bird song. As shown in Table 2, Table 2 shows the recognition results of the target bird song recognition model, and Table 3 shows the detection results for the long recording sedtest1.wav. Table 2 shows, from left to right, the precision, recall, F1 score, and number of test audios. Table 3 shows, from left to right, the audio file name, bird category name, start time (s), end time (s), maximum confidence, and average confidence.

[0167] Table 2

[0168]

[0169] Table 3

[0170]

[0171]

[0172] The target bird song recognition model is obtained by the bird song recognition model training method in the embodiment of the present invention:

[0173] 1. For recordings of bird calls collected in the wild, the data enhancement method of syllable splicing and reconstruction in this invention (the process of obtaining sliced ​​data) effectively solves the serious mismatch between the training data required for bird song recognition models (short recordings of individual birds) and the recording data used in monitoring applications (usually long recordings of multiple species singing simultaneously), achieving the effect of purifying the training data required for the model.

[0174] 2. Dynamic masking of the data spectrogram (frequency channel masking, time channel masking) can effectively improve the robustness of the bird song recognition model in detecting and identifying bird sounds, so that the target bird song recognition model can better adapt to the prediction and identification of bird sounds in certain complex sound environments.

[0175] 3. The feature extraction backbone module obtained through multi-level transfer learning has a strong feature extraction capability. The embedding features (Embedding) of various bird spectrograms mined by the feature extraction backbone module have strong discriminability. The test recognition effect is shown in Table 2.

[0176] 4. The bird sound recognition detection block of the present invention can provide segment-level and frame-level predictions, and can label bird tags on the time axis for long audio recording data. The test detection results are shown in Table 3.

[0177] An embodiment of the present invention further provides a bird song recognition device, the device comprising a processor and a memory;

[0178] The memory is used to store programs;

[0179] The processor is configured to execute a program to implement the bird song recognition model training method and bird song recognition method according to embodiments of the present invention. The device according to embodiments of the present invention can implement bird song recognition model training and bird song recognition functions. The device can be any intelligent terminal, including a mobile phone, tablet computer, computer, or vehicle-mounted computer.

[0180] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0181] An embodiment of the present invention further provides a computer-readable storage medium storing a program, which is executed by a processor to complete the bird song recognition model training method and the bird song recognition method as described in the aforementioned embodiment of the invention.

[0182] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the bird song recognition model training method and the bird song recognition method of the aforementioned embodiment of the invention.

[0183] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0184] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0185] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other may be through interfaces, or indirect coupling or communication connection between devices or units, which may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the objectives of the present embodiments according to actual needs. Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in either hardware or software functional units.

[0186] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0187] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A bird song recognition model training method, characterized in that: include: Get the data set; The data set includes a pre-trained audio data set, a pre-trained bird sound data set, a target bird sound data set, and a negative sample data set; Performing first processing on the pre-training audio data set, the pre-training bird sound data set, and the target bird sound data set, respectively, to obtain first training data corresponding to the pre-training audio data set, the pre-training bird sound data set, and the target bird sound data set; The first processing includes: performing first frame processing on the pre-trained audio data set, the pre-trained bird sound data set, and the target bird sound data set, respectively, and determining a first amplitude spectrum according to the first frame processing result, determining first slice data consisting of valid frames according to the first amplitude spectrum, performing second frame processing on the first slice data and determining a second amplitude spectrum according to the second frame processing result, determining a first Mel spectrum according to the second amplitude spectrum and a Mel filter bank, and performing frequency-time masking processing on the first Mel spectrum; Performing a second process on the negative sample data set to obtain second training data; Training a bird song recognition model according to the first training data and the second training data to obtain a target bird song recognition model; The bird sound recognition model includes a feature extraction backbone module and a bird sound recognition detection block, and the bird sound recognition detection block includes a first branch and a second branch; the bird sound recognition model is trained according to the first training data and the second training data to obtain a target bird sound recognition model, including: Constructing third training data according to the second training data and the first training data corresponding to the target bird sound data set; Inputting the pre-trained audio data set and the first training data corresponding to the pre-trained bird sound data set and the third training data into a feature extraction backbone module for feature extraction processing, aggregating the embedded features obtained by the feature extraction processing in the time dimension, and obtaining a prediction result at the sub-segment level; the feature extraction backbone module is obtained based on hierarchical transfer learning training; Inputting the sub-segment level prediction results into the first branch, performing a first convolution process, and obtaining segment-level prediction results of the bird sounds of each sub-segment; Input the prediction result of the sub-segment level into the second branch, perform a second convolution process and input it into the attention block for classification, multiply the output of the attention block with the segment level prediction result and aggregate based on the time of the sub-segment to obtain a frame level prediction result; A loss function value is calculated according to the segment-level prediction result or the frame-level prediction result, and a target bird song recognition model is determined by fine-tuning according to the loss function value.

2. The bird song recognition model training method according to claim 1, characterized in that: The performing first frame segmentation processing on the pre-trained audio data set, the pre-trained bird sound data set, and the target bird sound data set respectively includes: The pre-trained audio data set, the pre-trained bird sound data set, and the target bird sound data set are respectively divided into frames according to a preset frame length and a preset frame shift to obtain corresponding divided frame data respectively; Determine the first frame and the second frame from the frame data respectively; the second frame is the previous frame of the first frame; After stacking the first frame and the second frame, the second frame is used as the new first frame, and the step of determining the second frame is returned until the first frame is the last frame of the frame data, and the frame stacking data corresponding to the frame data is obtained; the frame stacking data is the first frame processing result.

3. The bird song recognition model training method according to claim 2, characterized in that: The determining of the first amplitude spectrum according to the first frame processing result includes: Performing a first windowing process on the frame stacking data to obtain a first windowing process result; Performing Fourier transform operation processing on the first windowing processing result; The modulus of the Fourier transform operation processing result is determined to obtain a first amplitude spectrum of each frame of data in the frame stack data.

4. The bird song recognition model training method according to claim 1, characterized in that: The determining, according to the first amplitude spectrum, first slice data consisting of valid frames comprises: Calculate the decibel value of each frequency point of the first amplitude spectrum to obtain the decibel value of the amplitude at each frequency point; Calculating the average value according to the amplitude decibel value to determine the average decibel value of each frame of data; each frame of data includes a number of frequency points; The data corresponding to the average decibel value being greater than or equal to the decibel threshold is taken as a valid frame; Splicing the valid frames in chronological order to obtain spliced ​​data; The spliced ​​data is subjected to a first slicing process according to a preset slicing length to obtain first slicing data.

5. The bird song recognition model training method according to claim 1, characterized in that: The Mel filter bank includes at least one Mel filter, and determining the first Mel spectrum according to the second amplitude spectrum and the Mel filter bank includes: Calculating the square of the second amplitude spectrum to obtain a frequency spectrum; Determine an actual frequency upper limit and an actual frequency lower limit according to the bird sound sound frequencies of various bird species in the pre-trained bird sound dataset and the target bird sound dataset; According to the conversion equation between the Mel frequency and the actual frequency, the upper limit of the bandpass frequency of the Mel filter group corresponding to the upper limit of the actual frequency is calculated, and the lower limit of the bandpass frequency of the Mel filter group corresponding to the lower limit of the actual frequency is calculated; The frequency spectrum is processed according to the Mel filter to obtain a first Mel spectrum; the center frequency of the Mel filter is determined according to the upper limit of the passband frequency and the lower limit of the passband frequency.

6. The bird song recognition model training method according to claim 1, characterized in that: The performing frequency-time masking processing on the first mel spectrum includes: Determine the upper and lower limits of bird song frequencies for various bird species; Calculate a first difference between the upper limit of the bird singing frequency and a preset shielding width and determine a first threshold; the first threshold is greater than or equal to the lower limit of the bird singing frequency and less than or equal to the first difference; Calculating the sum of the first threshold and the preset shielding width to obtain a second threshold; Determine a shielded frequency channel range according to the first threshold and the second threshold, and perform frequency channel shielding on the first Mel spectrum according to the shielded frequency channel range; Determine the total number of frames in the second frame segmentation process, calculate a second difference between the total number of frames and a preset time frame mask width, and determine a third threshold; the third threshold is greater than or equal to 1 and less than or equal to the second difference; Calculating the sum of the third threshold and the preset time frame masking width to obtain a fourth threshold; The masking time channel range is determined according to the third threshold and the fourth threshold, and the frequency channel masking result is subjected to time channel masking according to the masking time channel range and the second preset channel number to obtain the first training data corresponding to the pre-trained audio data set, the pre-trained bird sound data set and the target bird sound data set.

7. The bird song recognition model training method according to claim 1, characterized in that: The performing a second processing on the negative sample data set to obtain second training data includes: Performing a second slicing process on the negative sample data set to obtain second slicing data; Perform a third frame processing on the second slice data and determine a third amplitude spectrum according to the third frame processing result; determine a second Mel spectrum according to the third amplitude spectrum and the Mel filter bank and use it as the second training data.

8. A method for identifying bird calls, characterized in that: include: The bird sound data to be detected is input into the target bird sound recognition model obtained by the bird sound recognition model training method as described in any one of claims 1 to 7 to obtain a recognition result; the recognition result includes a segment-level recognition result or a frame-level recognition result, the segment-level recognition result includes the bird sound category in the bird sound data to be detected, and the frame-level recognition result includes the bird sound category in the bird sound data to be detected and the start time and end time of the bird sound.

9. A computer-readable storage medium, characterized in that: The storage medium stores a program, and when the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images