Bird chirp separation model, separation method, system, equipment and medium

Abstract

The invention provides a bird chirp separation model, a separation method, a system, equipment and a medium. The model is based on a Deeplabv3plus encoder-decoder structure, and comprises an Xception network, the Middle flow of which comprises three convolution layers and is repeated for 16 times; replacing the maximum pooling layer with a separable convolution; the ASPP module comprises a convolution layer, three expansion convolution layers and a global average pooling layer; the PSA module is used for providing a dual attention mechanism in two dimensions of a channel and a space; and the output layer generates a plurality of sound spectrum amplitude masks corresponding to specific directions. The separation method comprises the following steps: collecting a multi-source aliasing sound signal based on an R-channel microphone array; extracting spectrum features and phase difference features of the multi-channel twitter signal as input of the model, and obtaining sound spectrum amplitude masks in 360 / R directions; multiplying the frequency spectrum of the reference channel by the mask coefficient to obtain a frequency spectrum characteristic of each sound source direction; and obtaining a time domain signal of a separated sound source based on ISTFT and phase characteristics. According to the invention, the bird chirp separation effect is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of bioacoustic analysis, and in particular to a bird song separation model and separation method, system, equipment and medium. Background Art

[0002] In recent years, with the increasing attention paid to biodiversity, automatic and semi-automatic bioacoustic analysis methods have developed rapidly, among which bird song analysis has become an important means of evaluating biodiversity. As an important part of the ecosystem, the number and diversity of bird species can directly reflect the level of biodiversity in the habitat. However, in real wild environments, multiple bird individuals may make sounds at the same time, forming complex overlapping sound signals. This phenomenon is called the bird acoustic "cocktail party problem" (BACPP). BACPP is one of the biggest obstacles to the automatic processing of wild bird song records, which seriously affects the accuracy of bird song recognition.

[0003] In related technologies, the automatic sound source separation technology is used to solve BACPP. This technology effectively solves the overlap problem by separating the aliased signal into multiple components corresponding to different sound sources, and extracts useful sound information from the noisy environment. When the sound source signal of a single individual bird cannot be obtained, the chirping signal of an individual bird can be separated from the aliased bird chirping signal by using the chirping separation technology, thereby improving the accuracy of bird chirping recognition. At the same time, distinguishing the chirping of individual birds is also a prerequisite for many bird behavior studies, and is of great significance to the study of bird population and individual statistics, bird individual identification, and bird interactions.

[0004] At present, bird song separation methods are mainly divided into two categories: traditional methods and deep learning methods.

[0005] Traditional methods usually use a "cascade" system, such as the semi-automatic bird song analysis system proposed by Kojima et al. (2016), which integrates the detection, positioning, separation and recognition functions of bird songs. Although it improves the recognition accuracy to a certain extent, there is a problem of error accumulation, resulting in a suboptimal final output. The bird song analysis system proposed by Gabriel et al. (2019) that combines the MUSIC algorithm, the GHDSS algorithm and the CNN can effectively distinguish different types of bird songs, but it also faces the problem of error accumulation.

[0006] Deep learning methods, such as the BiocppNet method proposed by Bermant et al. (2021), have achieved certain results, but the separation effect may be limited because the single-channel audio does not contain the spatial distribution information of the target sound source and the interference source. In addition, since the structure adopts a codec structure separation model, the deep downsampling operation of the encoder will reduce the spatial and channel resolution of the input features, resulting in the loss of shallow key features and affecting the separation effect. Summary of the invention

[0007] The present invention provides a bird song separation model and separation method, system, equipment and medium, which solve the problem of how to improve the separation effect of related bird song separation technology.

[0008] In order to achieve the above objectives, this application adopts the following technical solutions:

[0009] In the first aspect, a bird song separation model is provided, which is based on the Deeplabv3plus model encoder-decoder structure, including:

[0010] Xception network, the middle flow of which includes 3 convolutional layers repeated 16 times; all maximum pooling layers are replaced by depthwise separable convolutions with a stride of 2; after each convolutional layer, a batch normalization layer and a Relu activation function are added;

[0011] The ASPP module includes a convolution layer with a convolution kernel size of 1×1, three dilated convolution layers with a convolution kernel size of 3×3, and a global average pooling layer; the dilation rates of the three dilated convolutions are 6, 12, and 18 respectively;

[0012] In the decoder, the features obtained by the ASPP module are fused with the shallow features in the Entry flow of the Xception network, and then the bilinear interpolation method is used to obtain features consistent with the input size;

[0013] PSA module, which is used to combine the dual attention mechanism of channel and space dimensions to enhance feature representation capabilities;

[0014] The output layer is configured as the last convolutional layer to generate multiple spectrum amplitude masks corresponding to specific directions.

[0015] In a first possible implementation manner of the first aspect, the model acquires a multi-source aliased sound signal, where the multi-source aliased sound signal includes a sound signal spectrum feature and a phase difference feature;

[0016] The output mask feature dimension is consistent with the input aliased sound signal spectrogram resolution.

[0017] In a second possible implementation manner of the first aspect, the PSA module includes:

[0018] Channel Attention Branch A ch (X)∈R C×1×1 :

[0019] Spatial Attention Branch A sp (X)∈R 1×H×W : A sp (X) = F SG [σ3(F SM (σ1(F GP (W q (X))))×σ2(W v (X)))]

[0020] Among them, F SG Represents the Sigmoid operation layer, W v , W q and W z There are three 1×1 convolutional layers, σ1 and σ2 represent the reshape layer, and × represents the matrix dot product operation; F SM represents the Softmax operation, F GP Represents a global pooling operation.

[0021] In a second aspect, a method for separating bird calls is provided, which uses the bird call separation model as described in the first aspect to separate bird calls, comprising:

[0022] Collect multi-source aliased beeping signals based on an R-channel microphone array; a channel where one of the microphones in the R-channel microphone array is located is configured as a reference channel;

[0023] Extract the spectrum characteristics and phase difference characteristics of multi-channel bird song signals;

[0024] The extracted frequency spectrum features and phase difference features are used as inputs of the bird song separation model. The model predicts amplitude spectrum masks in multiple directions according to a set angular resolution R to obtain 360 / R directions of sound spectrum amplitude masks;

[0025] Multiplying the spectrum of the reference channel and the mask coefficient of the sound spectrum amplitude mask to obtain the spectrum characteristics of each sound source direction;

[0026] The time domain signal of the separated sound source is obtained by inverse Fourier transform (ISTFT) and combining the phase characteristics of the reference microphone.

[0027] In a first possible implementation manner of the second aspect, the R-channel microphone array is specifically configured as: an 8-channel microphone array;

[0028] The 8-channel microphone array is configured as 8 microphones arranged in a circular array with a set distance as the radius, and the 8 microphones are on the same horizontal plane; adjacent microphones are spaced 45 degrees apart, and the microphone at the 0° direction is recorded as the reference microphone;

[0029] The intervals between the sound sources collected by the 8-channel microphone array are not less than 45 degrees.

[0030] Based on the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, the step of extracting the frequency spectrum features and phase difference features of the multi-channel bird song signal includes:

[0031] The multi-source mixed sound signal is processed by windowing and time-frequency analysis to obtain the corresponding STFT features;

[0032] The IPD between each non-reference microphone and the reference microphone is calculated.

[0033] In a third possible implementation manner of the second aspect, before performing the step of obtaining a time domain signal of a separated sound source by performing an inverse Fourier transform (ISTFT) in combination with a phase characteristic of a reference microphone,

[0034] Set the sound energy threshold and use it to filter the direction of the sound source.

[0035] In a third aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the bird song separation method as described in the first aspect.

[0036] In a fourth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the bird song separation method as described in the first aspect are implemented.

[0037] The bird song separation model and method proposed in this application is based on the codec structure and polarized self-attention mechanism. It is an end-to-end solution that avoids the error accumulation of multiple processing functions. The bird song separation model based on the codec structure and polarized self-attention mechanism uses the spectral characteristics and phase difference characteristics of the input aliased sound signal as input, effectively utilizing the spatial position characteristics of the sound source signal; secondly, in order to reduce the loss of shallow features of the input information, the application adds polarized self-attention at the bottleneck of the codec structure, ensuring the maximum resolution of the input features in both spatial and channel dimensions, so that it can effectively process multi-source aliased sound signals and improve the separation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1A schematic flow chart of a method for separating bird calls provided in an embodiment of the present application;

[0039] Figure 2 A schematic structural diagram of an Xception network provided in an embodiment of the present application;

[0040] Figure 3 A schematic structural diagram of an ASPP module provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of bilinear interpolation provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of the overall architecture of a separation model provided in an embodiment of the present application;

[0043] Figure 6 A schematic flow chart of a PSA module provided in an embodiment of the present application

[0044] Figure 7 A schematic flow chart of another method for separating bird calls provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of the structure of an 8-channel microphone array provided in an embodiment of the present application;

[0046] Fig. 9 A schematic flowchart of a sound source signal reconstruction provided in an embodiment of the present application;

[0047] Fig.10 A structural block diagram of a bird song separation system provided in an embodiment of the present application;

[0048] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the technical solutions in the embodiments of the present application are clearly described. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present application.

[0050] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0051] The description of the method flow in the specification of the present application and the steps of the flowchart in the drawings of the present specification do not have to be strictly executed according to the step numbers, and the method steps can be executed in a different order. Moreover, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0052] The bird song separation model and separation method, system, equipment and medium provided in the embodiments of the present application are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0053] First, the application scenarios of the bird song separation model and separation method, system, equipment and medium in the embodiments of the present application are described in detail.

[0054] Kojima et al. (2016) proposed a semi-automatic bird song analysis system using spatial cues, which simultaneously integrated the detection, localization, separation and recognition functions of birdsong. Experimental data from bird song recordings in the wild showed that the recognition accuracy obtained by the system was better than that of traditional robot audio-visual methods (Nakajima et al., 2010). Gabriel et al. (2019) proposed a bird song analysis system that combines the MUSIC (multiple signal classification) algorithm, the GHDSS (geometric high-order decorrelation-based source separation) algorithm and the CNN (convolutional neural networks) to perform sound source localization, separation and recognition tasks in turn. Experimental results show that the system can effectively distinguish different types of bird songs. Tolkova and Klinck (2022) proposed the use of acoustic vector sensors (AVSs) to perform the direction of arrival estimation and overlapping sound separation of sound source signals: first, the direction of arrival of each sound source is calculated, then the direction of arrival of the sound source is resolved into the angular distribution of each sound source, and finally the sound source signal is reconstructed.

[0055] Experimental results show that the system is robust to environmental noise and has a higher fidelity in reconstructing the sound source signal. However, the biggest disadvantage of the "cascade" method is that each independent function will produce errors, and the accumulation of multiple errors will result in a suboptimal final output.

[0056] Bermant et al. (2021) proposed a lightweight and modular deep learning method, BiocppNet, for the problem of sound source separation of different biological groups. This method also uses a deep neural network model U-net with a codec structure as the backbone network. The initial layer of the encoder is a high-pass filter. After the initial layer, there is a feature extraction encoder, which contains several options of one-dimensional convolution blocks, short-time Fourier transform (STFT) filter banks, and logarithmic STFT filter banks. The feature extraction encoder is followed by a two-dimensional U-net core. The architecture consists of n (4 for macaques, 3 for dolphins, and 4 for bats) downsampling convolution blocks, an intermediate convolution block, and n upsampling convolution blocks. The downsampling block consists of two 2D convolutional layers. The number of convolution kernels in each convolution layer increases with the depth of the model. The convolution layer is activated and pooled by the ReLU function. The upsampling block uses bilinear interpolation with a proportional coefficient corresponding to the downsampling pooling size. The model concatenates the pre-convolution features in the codec with the upsampled features using skip connections. All convolutional layers in the downsampling, intermediate, and upsampling modules include batch normalization after the activation function to stabilize and speed up training and promote regularization. The U-net returns N masks, which are then multiplied by the original encoded representation of the mixed waveform, and finally the inverse transform decoder is used to obtain the original waveform corresponding to the model's prediction of the separated utterance. The method uses the scale-invariant signal-distortion ratio (SI-SDR) and downstream classification accuracy to evaluate the reconstruction quality of the separated sound source signal.

[0057] Bermant et al. (2021) proposed a lightweight and modular deep learning method, BiocppNet, for the problem of sound source separation of different biological groups, and used the scale-invariant signal-distortion ratio (SI-SDR) and downstream classification accuracy to evaluate the reconstruction quality of the separated waveform. However, since single-channel audio does not contain the spatial distribution information of the target sound source and the interference source, the separation effect may be limited. In addition, since the structure adopts a codec structure separation model, the deep downsampling operation of the encoder will reduce the spatial and channel resolution of the input features, resulting in the loss of shallow key features.

[0058] Based on the fact that the single-channel sound source separation method in the above-mentioned related technology does not have the ability to collect the spatial features of multiple sound source signals, when multiple sound sources overlap in the time dimension, the separation performance of the single-channel sound source separation method will be limited. This application is based on a multi-channel microphone array. On the basis of using the spectral features of the reference channel of the aliased chirping signal as the input of the separation model, the phase difference features between the channels of the aliased chirping signal are further used as auxiliary inputs to jointly train the separation model, so that the separation model can not only learn the spectral features of the aliased chirping signal, but also learn the correlation features between different channels. In this way, the spatial position information of the chirping source signal is effectively utilized, and the separation results can better understand the acoustic interactions of bird species in time and space. In view of the problem that the downsampling operation of the encoder structure of the existing separation model reduces the resolution of the input features and causes the loss of key information of the input features, this application proposes a bird song separation method based on a codec structure and a polarization self-attention mechanism. The polarization self-attention mechanism maintains the original resolution of the input features in both space and channel. The representative features of multi-source aliased chirping signals can be effectively extracted through this structure.

[0059] In general, based on the spectral characteristics of the reference channel of the aliased chirping signal as the input of the separation model, the present application further utilizes the phase difference characteristics between the channels of the aliased chirping signal as the auxiliary input, and jointly trains the separation model, so that the separation model can not only learn the spectral characteristics of the aliased chirping signal, but also learn the correlation characteristics between different channels. In this way, the spatial position information of the chirping signal is effectively utilized, and the separation results can better understand the acoustic interactions of bird species in time and space.

[0060] See also Figure 1 ,like Figure 1 As shown, the overall architecture of the present application, for the input 8-channel mixed sound signal, the spectrum and phase difference characteristics are extracted through STFT (short-time Fourier transform, STFT) as the input of the separation model, and the separation model predicts the amplitude spectrum mask in 360 / R directions according to the set output angle resolution R, and multiplies the predicted mask coefficient with the spectrum of the reference channel to obtain the spectrum characteristics of each sound source direction, and finally obtains the time domain signal of the separated sound source through inverse Fourier transform (inverse short-time Fourier transform, ISTFT).

[0061] The embodiment of the present application provides a bird song separation model. The separation model adopts the encoder-decoder structure Deeplabv3plus. The present application adopts Xception as the backbone network of the encoder part of Deeplabv3plus. In order to take into account both accuracy and lightness, the original Xception is further improved in Deeplabv3plus, mainly including: (1) deepening the original network structure, increasing the middle layer (Middle flow) from the original 8 convolution layers to 16 layers to obtain a deeper convolution structure; (2) replacing all the maximum pooling layers (maxpooling) of the original network with a depthwise separable convolution with a stride of 2, and retaining the Entry flow structure to ensure the speed of operation and memory efficiency; (3) adding a batch normalization layer BN layer and a Relu activation function after each convolution kernel of size 3×3.

[0062] Specifically: See Figure 2 , the Xception network in Deeplabv3plus is as follows Figure 2 As shown in the figure, the convolutional neural network Xception consists of Entry flow, Middle flow and Exit flow, where the Entry flow contains 11 convolutional layers, the Middle flow contains 3 convolutional layers and is repeated 16 times, and the Exit flow contains 6 convolutional layers. Jump connections are used in the Entry flow, Middleflow and Exit flow.

[0063] In the specific implementation process, the input feature of Xception first passes through the Entry flow. The Entry flow contains two regular convolutional layers and three separable convolutional blocks. A residual block connection is added to each module to add the input and output and send them to the next module.

[0064] Then it enters the Middle flow, which repeats the separable convolution 16 times. Finally, it passes through the Exit flow, which contains two separable convolution modules. As the core module of Deeplabv3plus, the Atrous Spatial Pyramid Pooling (ASPP) can further extract multi-scale information from the features output by Xception.

[0065] The schematic diagram of the ASPP module is as follows Figure 3As shown in the figure, it includes a convolution layer with a convolution kernel size of 1×1, three dilated convolution layers with a convolution kernel size of 3×3, and a global average pooling layer. The dilation rates of the three dilated convolutions are 6, 12, and 18 respectively. As the dilated convolution coefficient continues to increase, the effect of the convolution kernel will become smaller and smaller. When the size of the dilated convolution kernel is consistent with the input feature, the convolution can be equivalent to a 1×1 convolution kernel. Therefore, a global average pooling layer is added to the ASPP structure in Deeplabv3plus to perform global average pooling on the input features along the channel direction.

[0066] Finally, the features extracted from the above multi-scales are concatenated (Concat), and the number of fused feature channels is converted to the expected value using 1×1 convolution. The ASPP structure is used to learn the characteristics of bird sound sources with different shape characteristics. For example, the sound of the Oriental Great Reed Warbler is urgent, and the spectrogram distribution is relatively dense, while the sound of the cuckoo is gentle, so the spectrogram distribution is relatively sparse; in the decoder module, the features obtained by ASPP are fused with the shallow features in the Entry flow in Xception, and then the bilinear interpolation method is used to obtain features consistent with the input size.

[0067] The principle of bilinear interpolation can be found in Figure 4 ,like Figure 4 The following is a schematic diagram of bilinear interpolation. Here, it is assumed that the element value at point P needs to be calculated and Q is known. 11 , Q 21 , Q 12 The coordinates and element values ​​of point P. First, use the single linear difference about X to calculate the element values ​​of R1 and R2 respectively, and then use the single linear difference about y to calculate the element value of point P.

[0068] Since the downsampling operation of the Deeplabv3plus encoder reduces the resolution of the input features, the features connecting the encoder and decoder are smaller in spatial dimension than the input and output features. Although the model attempts to learn more low-level features through skip connections between the encoder and decoder, it still does not completely solve the problem of loss of key information of the input features caused by reducing the resolution of the input features in the downsampling operation. In order to improve the above problems, this application introduces a "parallel" structured PSA module at the bottleneck of Deeplabv3plus, see Figure 5 , the overall architecture of the separation model is as follows Figure 5 shown.

[0069] The polarized attention mechanism (PSA) module is a dual attention mechanism that combines channel and spatial dimensions, see Figure 6 , Figure 6Schematic diagram of a PSA module shown in a "parallel" configuration.

[0070] Among them, the channel attention branch A ch (X)∈R C×1×1 It can be expressed by formula (1):

[0071]

[0072] Among them, F SG Represents the Sigmoid operation layer, W v , W q and W z There are three 1×1 convolutional layers, σ1 and σ2 represent the reshape layer, and × represents the matrix dot product operation. SM Represents the Softmax operation, and the calculation process is shown in formula (2):

[0073]

[0074] For the input feature X, we first use a 1×1 convolution operation to convert it into features Q and V, where the channel of Q is completely compressed, while the channel dimension of V remains at a relatively high level (C / 2), maintaining a high resolution on the channel. Then, we perform matrix multiplication on Q and K, and then add a 1×1 convolution layer and a LayerNorm layer to increase the channel dimension to the original input size C. Finally, we use the Sigmoid function to keep all parameters between 0 and 1. The output of the channel attention branch is shown in formula (3):

[0075]

[0076] in, is the channel multiplication operator.

[0077] Spatial Attention Branch A sp (X)∈R 1×H×W As shown in formula (4):

[0078] A sp (X) = F SG [σ3(F SM (σ1(F GP (W q (X))))×σ2(W v (X)))] (4)

[0079] Among them, F GP Represents the global pooling operation, and the calculation process is shown in formula (5):

[0080]

[0081] Similar to the channel attention branch, the spatial attention branch first uses a 1×1 convolution to convert the input into features Q and V. Then, global pooling is used to compress the spatial dimension of Q and convert it into a size of 1×1, while the spatial dimension of V remains at a relatively large level (H×W). Since the spatial dimension of Q is compressed, Softmax is used to enhance the information of Q. Finally, Q and K are matrix multiplied, and then followed by a reshape and Sigmoid layer so that all parameters remain between 0 and 1. The output of the spatial attention branch is shown in formula (6):

[0082]

[0083] The overall layout of the "parallel" structure PSA is shown in formula (7):

[0084] PSA p (X) = Z ch +Z sp (7)

[0085] This application modifies the last convolution kernel of size 3×3 in the Deeplabv3plus model decoder to 1×1 to make the output mask feature dimension consistent with the input aliasing spectrogram resolution, activates it through the Sigmoid function, and controls the number of output masks through the number of convolution kernels in the last layer. Since the directional intervals of the bird sound sources in this application are all greater than 45°, the number of convolution kernels in the last layer is set to 8, and the model outputs the amplitude spectrum mask coefficients (SMM) of the 8 directional sound sources. The spectral magnitude mask (SMM), also known as FFT-MASK, is defined based on the STFT amplitude of the pure sound source and the mixed sound signal, as shown in formula (8):

[0086]

[0087] Among them, |S(t,f)| represents the sound spectrum value of the pure sound source, and |Y(t,f)| represents the sound spectrum value of the mixed sound signal. Unlike IRM, the upper limit value of SMM is not 1. In order to obtain the separated sound source signal, SMM or its estimated value is applied to the spectral amplitude value of the mixed sound source, and the phase of the mixed sound source (or the estimation of the phase of the pure sound source) is used to resynthesize the separated sound source.

[0088] The goal of model training is to continuously optimize the predicted spectrum from the input aliased signal. As shown in formula (9), the root mean square error (MSE) is used as the loss function for training the separation model, where: represents the element-wise product, X represents the spectrogram predicted by the separation model, and Y represents the spectrogram in the direction of the true source signal. Where f(X) is the SMM predicted by the model, X mag It is the spectrum value of the aliased beeping signal of the microphone reference channel.

[0089]

[0090] See also Figure 7 , the present application embodiment provides a method for separating bird calls, such as Figure 7 As shown, the separation method of the embodiment of the present application includes:

[0091] Step S1, collecting multi-source aliased sound signals based on an R-channel microphone array; a channel where one of the microphones in the R-channel microphone array is located is configured as a reference channel.

[0092] Exemplarily, multi-source mixed sound signals are collected based on an 8-channel microphone array; wherein the 8-channel microphone array is configured as 8 microphones arranged in a circular array with a set distance as a radius, the 8 microphones are on the same horizontal plane, adjacent microphones are spaced 45 degrees apart, and the microphone in the 0° direction is recorded as the reference microphone; the intervals between the sound sources collected by the 8-channel microphone array are not less than 45 degrees.

[0093] The separation method proposed in this application is applicable to multi-source aliased beeping signals collected by an 8-channel microphone array. Figure 8 The shown figure shows an 8-channel circular microphone array with a radius of 0.1m. In this microphone array, the microphones are aligned at 45° intervals and are located in the same horizontal plane. The microphone in the 0° direction is recorded as the reference microphone, and the interval between the sound sources is not less than 45.

[0094] Step S2, extracting the frequency spectrum features and phase difference features of the multi-channel bird sound signal.

[0095] Step S201, windowing processing is performed on the multi-source aliased sound signal and time-frequency analysis is performed to obtain corresponding STFT features.

[0096] Birdsong signals are non-stationary time series signals. Since the information expressed by the time domain signals of birdsong is not very intuitive, the singing signals are usually converted to the frequency domain for calculation and analysis. Through frequency domain feature analysis, we can intuitively understand the energy distribution of a signal in different frequency bands, and obtain the unique characteristics of the specific singing signal. In addition, frequency domain transformation also has the effect of dimensionality reduction. Short-time Fourier transform (STFT), also known as windowed or time-division Fourier transform, is a simple and effective time-frequency analysis method. Before the sound signal is Fourier transformed, it must first be windowed to ensure that the non-stationary signal is stable within the short time in the window. By moving the window function on the time axis, the STFT characteristics of the entire audio segment are obtained.

[0097] For example, the present application adds a Hamming window, the window length is 512, and the window overlap rate is 0.5. The aliased sound signal of each channel is subjected to STFT transformation, and the formula is as follows:

[0098]

[0099] In the formula, h(τ-t) is the window function, x i (τ) is the time domain waveform of a channel of the aliased sound signal.

[0100] Step S202 , calculating the IPD between each non-reference microphone and the reference microphone.

[0101] Spectral features are used as the main input features of deep neural networks because they have obvious spectral structures in the STFT domain that can reflect the time-frequency nature of speech. In addition, thanks to the advantage of microphone arrays in extracting spatial features of sound source signals, the spatial information of the target bird song source should also be used as the input features of the separation model. As the most effective feature reflecting sound source information, the phase difference feature (IPD) is defined as the phase difference between each microphone channel and the reference microphone channel signal. This feature can help the separation model to well characterize the spatial information of the bird song source, thereby helping the model to estimate a more accurate mask.

[0102] For example, the present application records the microphone in the 0° direction as the reference microphone channel m, and the microphone channels in other directions as n. Then the IPD between the nth microphone and the reference microphone m at the frequency f at time t can be expressed as:

[0103] θ t,f,m,n =STFT m (t,f)-STFT n (t,f) (11)

[0104] Among them, θ t,f,m,n is the phase difference characteristic, STFT m (t,f) and STFTn (t, f) are the STFT features of the reference microphone channel m and the non-reference microphone channel n respectively. Then the IPD features are normalized, and the normalized IPD can be expressed as:

[0105] sinIPD(t,f,m,n)=sin(θ t,f,m,n ) (12)

[0106] cosIPD(t,f,m,n)=cos(θ t,f,m,n ) (13)

[0107] Step S3, using the extracted frequency spectrum features and phase difference features as inputs of the bird song separation model, and the model predicts amplitude spectrum masks in multiple directions according to a set angular resolution R to obtain sound spectrum amplitude masks in 360 / R directions.

[0108] For the description of the bird song separation model in this step, please refer to the above-mentioned bird song separation model, which will not be described here to avoid repetition.

[0109] Step S4: multiply the spectrum of the reference channel by the mask coefficient of the sound spectrum amplitude mask to obtain the spectrum characteristics of each sound source direction.

[0110] Step S5, obtaining a time domain signal of the separated sound source by performing an inverse Fourier transform (ISTFT) in combination with the phase characteristics of the reference microphone.

[0111] Steps S4-S5 refer to Fig. 9 , Fig. 9 For the sound source signal reconstruction flow chart, the output spatial resolution R of the separation model proposed in this application is set to 45°. The model predicts the SMM in 8 directions of the microphone array, and the spectrograms in 8 directions are obtained using the spectrogram of the input mixed sound source reference channel and the SMM predicted by the separation model. Considering the actual application scenario, if two or more bird sound sources are particularly close or in the same direction, the angular resolution can be reduced to improve the separation quality. For example, when the angular resolution is set to 15°, the model predicts sound sources in 24 directions.

[0112] In some possible implementations, before performing the step of obtaining the time domain signal of the separated sound source by inverse Fourier transform (ISTFT) combined with the phase characteristics of the reference microphone, a sound energy threshold is set to filter the direction of the sound source by the sound energy threshold.

[0113] In the direction where there is no sound source, the spectrogram with zero energy value can be discarded by setting the sound energy threshold. For the direction where there is a sound source, the time domain signal is reconstructed from the predicted spectrogram and the phase characteristics of the reference microphone m through the inverse short Fourier transform (ISTFT).

[0114] The bird song separation model and separation method based on the codec structure and polarized self-attention mechanism proposed in this application is an end-to-end sound source separation method that avoids the error accumulation of multiple processing functions. The bird song separation model based on the codec structure and polarized self-attention mechanism uses the spectral characteristics and phase difference characteristics of the input aliased sound signal as input, effectively utilizing the spatial position characteristics of the sound source signal; secondly, in order to reduce the loss of shallow features of the input information, the present invention adds polarized self-attention at the bottleneck of the codec structure, ensuring the maximum resolution of the input features in both spatial and channel dimensions.

[0115] This application has demonstrated significant advantages in improving separation accuracy, enhancing model robustness, achieving efficient computing and lightweight, having strong adaptability and improving user experience. It is not only suitable for the separation of bird calls, but this method can also be extended to other bioacoustic fields, such as the sound analysis of marine mammals, and other application scenarios that require multi-source sound separation.

[0116] The following experimental data is provided to further illustrate the effects and advantages of this application.

[0117] In order to select the optimal system parameters suitable for this separation model, 800 sets of multi-source aliased sound data containing true values ​​were selected as the training set, and the Adam optimizer with a learning rate of 1×10-3 was used to optimize the root mean square error (MSE). The training batch size was set to 4, the model iteration number used 200 epochs, and the early stopping mechanism was used to save the optimal model as the final model. The relevant hyperparameter settings of the model are shown in Table 1:

[0118] Table 1 Model training parameter settings

[0119]

[0120] After determining the system parameters of the separation model, 800 sets of training set samples were imported into the separator for training, and finally a multi-source aliasing sound separation model with determined parameters was obtained.

[0121] For evaluation indicators: This application uses root mean square error (RMSE) and four evaluation indicators commonly used in speech separation: SDR (Source-to-Distortion Ratio), SIR (Source-to-Interference Ratio), SAR (Source-to-Artifact Ratio) and STOI (Short-Time Objective Intelligibility) as evaluation indicators of the separation model.

[0122] Test data separation results: This application tests 100 sets of sound data composed of 3 different types of sound sources, and compares the separation results of the Deeplabv3plus model. The test results are shown in Table 2.

[0123] Table 2 Model training parameter settings

[0124]

[0125] As can be seen from Table 2, when a single spectrogram feature spectrogram is used as input, the RMSE of this application is reduced by 0.68 compared with Deeplabv3plus, indicating that the error between the separated sound source signal and the real source signal is smaller. In terms of SDR, SIR, SAR and STOI, this application is improved by 0.30, 0.17, 0.17 and 0.01 respectively compared with Deeplabv3plus. When the fusion space sinIPDs and cosIPDs are used as input, the RMSE of this application is reduced by 0.08 compared with Deeplabv3plus, and SDR, SIR, SAR and STOI are improved by 0.08, 0.07, 0.03 and 0.02 respectively compared with Deeplabv3plus. The increase in SDR value indicates that the predicted sound source signal is less distorted, the increase in SIR value indicates that the predicted target sound source is less affected by interference, the increase in SAR indicates that the degree of suppression of the interference error of the application itself and other signals is smaller, and the increase in STOI value indicates that the intelligibility scores of the predicted sound source signal and the real source signal are higher.

[0126] This result has well verified the feasibility of separating the aliased sound signals in this application.

[0127] See also Fig.10 Corresponding to the above-mentioned bird song separation method embodiment, the present application embodiment provides a bird song separation system, the separation system comprising:

[0128] The signal acquisition module 1001 is used to collect multi-source aliased sound signals based on an R-channel microphone array; a channel where one of the microphones in the R-channel microphone array is located is configured as a reference channel;

[0129] A feature extraction module 1002 is used to extract the frequency spectrum features and phase difference features of the multi-channel bird sound signal;

[0130] The separation model module 1003 is based on the bird song separation model and is used to use the extracted frequency spectrum features and phase difference features as inputs of the bird song separation model. The model predicts amplitude spectrum masks in multiple directions according to a set angular resolution R to obtain 360 / R directions of sound spectrum amplitude masks;

[0131] A unidirectional spectrum feature acquisition module 1004 is used to multiply the spectrum of the reference channel and the mask coefficient of the sound spectrum amplitude mask to obtain the spectrum feature of each sound source direction;

[0132] The signal reconstruction module 1005 is used to obtain the time domain signal of the separated sound source by inverse Fourier transform (ISTFT) combined with the phase characteristics of the reference microphone.

[0133] The above-mentioned bird song separation system implements the steps and various processes of the above-mentioned bird song separation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0134] See also Fig.11 Corresponding to the above-mentioned bird song separation method embodiment, the embodiment of the present application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the above-mentioned bird song separation method embodiment and the various processes of the embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0135] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0136] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0137] Corresponding to the above-mentioned bird song separation method embodiment, the embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, the steps of the above-mentioned bird song separation method embodiment and the various processes of the embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0138] The processor is the processor in the electronic device described in the above embodiment of the present application. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0139] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0141] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A bird song separation model, characterized in that: Based on the Deeplabv3plus model encoder-decoder structure, including: Xception network, the middle flow of which includes 3 convolutional layers repeated 16 times; all maximum pooling layers are replaced by depthwise separable convolutions with a stride of 2; after each convolutional layer, a batch normalization layer and a Relu activation function are added; The ASPP module includes a convolution layer with a convolution kernel size of 1×1, three dilated convolution layers with a convolution kernel size of 3×3, and a global average pooling layer; the dilation rates of the three dilated convolutions are 6, 12, and 18 respectively; In the decoder, the features obtained by the ASPP module are fused with the shallow features in the Entry flow of the Xception network, and then the bilinear interpolation method is used to obtain features consistent with the input size; PSA module, which is used to combine the dual attention mechanism of channel and space dimensions to enhance feature representation capabilities; The output layer is configured as the last convolutional layer to generate multiple spectrum amplitude masks corresponding to specific directions.

2. The bird song separation model according to claim 1, characterized in that: The model acquires a multi-source aliased sound signal, wherein the multi-source aliased sound signal includes a sound signal spectrum feature and a phase difference feature; The output mask feature dimension is consistent with the input aliased sound signal spectrogram resolution.

3. The bird song separation model according to claim 1, characterized in that: The PSA module comprises: Channel Attention Branch A ch (X)∈R C×1×1 : Spatial Attention Branch A sp (X)∈R 1×H×W : Among them, F SG Represents the Sigmoid operation layer, W v , W q and W z There are three 1×1 convolutional layers, σ1 and σ2 represent the reshape layer, and × represents the matrix dot product operation; F SM represents the Softmax operation, F GP Represents a global pooling operation.

4. A method for separating bird calls, characterized in that: Using the bird song separation model according to any one of claims 1 to 3 to separate bird songs comprises: Collect multi-source aliased beeping signals based on an R-channel microphone array; a channel where one of the microphones in the R-channel microphone array is located is configured as a reference channel; Extract the spectrum characteristics and phase difference characteristics of multi-channel bird song signals; The extracted frequency spectrum features and phase difference features are used as inputs of the bird song separation model. The model predicts amplitude spectrum masks in multiple directions according to a set angular resolution R to obtain 360 / R directions of sound spectrum amplitude masks; Multiplying the spectrum of the reference channel and the mask coefficient of the sound spectrum amplitude mask to obtain the spectrum characteristics of each sound source direction; The time domain signal of the separated sound source is obtained by inverse Fourier transform (ISTFT) and combining the phase characteristics of the reference microphone.

5. The method for separating bird calls according to claim 4, characterized in that: The R-channel microphone array is specifically configured as: an 8-channel microphone array; The 8-channel microphone array is configured as 8 microphones arranged in a circular array with a set distance as the radius, and the 8 microphones are on the same horizontal plane; adjacent microphones are spaced 45 degrees apart, and the microphone at the 0° direction is recorded as the reference microphone; The intervals between the sound sources collected by the 8-channel microphone array are not less than 45 degrees.

6. The method for separating bird calls according to claim 5, characterized in that: The method of extracting the frequency spectrum characteristics and phase difference characteristics of the multi-channel bird sound signal includes: The multi-source aliased sound signal is subjected to windowing processing and time-frequency analysis to obtain the corresponding STFT features; The IPD between each non-reference microphone and the reference microphone is calculated.

7. The method for separating bird calls according to claim 4, characterized in that: The method also includes: Before performing the step of obtaining the time domain signal of the separated sound source by inverse Fourier transform (ISTFT) and combining the phase characteristics of the reference microphone, Set the sound energy threshold and use it to filter the direction of the sound source.

8. A bird song separation system, characterized in that: include: A signal acquisition module, used for collecting multi-source aliased sound signals based on an R-channel microphone array; a channel where one of the microphones in the R-channel microphone array is located is configured as a reference channel; A feature extraction module is used to extract the spectrum features and phase difference features of multi-channel bird sound signals; A separation model module, based on the bird song separation model according to any one of claims 1 to 3, used to use the extracted frequency spectrum features and phase difference features as inputs of the bird song separation model, and the model predicts amplitude spectrum masks in multiple directions according to a set angular resolution R to obtain sound spectrum amplitude masks in 360 / R directions; A unidirectional spectrum feature acquisition module, used for multiplying the spectrum of the reference channel and the mask coefficient of the sound spectrum amplitude mask to obtain the spectrum feature of each sound source direction; The signal reconstruction module is used to obtain the time domain signal of the separated sound source by inverse Fourier transform (ISTFT) combined with the phase characteristics of the reference microphone.

9. An electronic device, characterized in that: The electronic device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the bird song separation method as claimed in any one of claims 4 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the bird song separation method according to any one of claims 4 to 7 are implemented.

Images