Multi-sound-source detection and positioning method and system of convolutional recurrent neural network

Abstract

The invention provides a multi-sound-source detection and positioning method and system of a convolutional recurrent neural network. Amplitude and phase features of multi-channel audio are extracted, an embedded layer is used for generating a position code for each frame of audio, the features and the position codes are jointly input into a neural network, and the convolutional neural network is used in a network structure to learn distinguishing of sound source categories and sound source positioning according to features between channels. Features are extracted from different scales by using multiple groups of void spatial pyramid pooling among convolutional layers, translation invariance of a convolutional neural network is kept by using a spatial converter network, context information is learned by using a gating loop unit based on an attention mechanism, and the sound source position of the current frame is predicted in an auxiliary manner according to the information of the previous frame; global features are obtained through global average pooling processing, the global features and the position codes are combined and input into a parallel full connection layer, and a sound event prediction result and an arrival direction prediction result are output. According to the invention, respective positioning of simultaneous sounding of a plurality of sound sources is realized, and robustness is provided for reverberation and low signal-to-noise ratio environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital audio signal processing, and in particular relates to a convolutional recurrent neural network multi-sound source detection and positioning method and system. Background Art

[0002] Multi-sound source localization is a speech and audio signal processing technology that determines the spatial orientation (also known as DOAestimation, direction of arrival estimation) or distance of multiple active sound sources relative to a specified position based on the signals received by the microphone array. It is an extension of the sound source localization technology in terms of the number of sound sources. Sound source localization technology plays an important role in speech recognition, speech separation, wireless communication and other related speech and audio signal processing technology fields. In a listening scene containing multiple sound sources or speakers, this method can accurately locate each sound source, so that different sounds can be accurately rendered at the specified orientation during playback using various audio space rendering technologies to enhance the realism of the scene. Traditional sound source localization algorithms include three categories: methods based on relative delay estimation such as the time difference of arrival location algorithm (TDOA) and the generalized cross correlation algorithm based on phase weighting (GCC-PHAT); methods based on beamforming technology such as the steering response power method based on phase transformation weighting (SRP-PHAT); and methods based on signal subspace such as the multiple signal classification algorithm (MUSIC) and the signal parameter estimation method based on rotational invariance technology (ESPRIT). These traditional methods are based on the physical principles of acoustics, and most of them have problems such as insufficient spatial resolution and sensitivity to reverberation and noisy environments. With the widespread application of neural networks in various fields, sound source localization technology has also made new breakthroughs. This technology has significantly improved the accuracy and precision of positioning in environments with noise and reverberation. Using unaural audio data for testing, the sound source localization algorithm based on deep neural networks has a prediction accuracy of more than 99% and an average prediction error of less than 5% in the case of a single sound source, and a prediction accuracy of more than 70% and an average prediction error of less than 20% in the case of three sound sources, which is similar to the average prediction error of the best performing MUSIC algorithm among traditional methods; and using multiple sets of data with noise and reverberation for testing, the prediction error of the MUSIC algorithm rapidly increased to more than 50%, while the prediction error of the method based on deep neural networks was only about 10% lower than the unaural performance. This method has more advantages than other positioning networks, including but not limited to: the number of predictable sound sources exceeds 3, and the update of the network structure enables the algorithm to have the potential to predict more sound sources; the predicted angle range not only includes 360° all-round angles, but also supports the prediction of altitude angles in the range of -60° to 60°; it supports a variety of microphone array structures including linear and spherical, and has good generalization performance. Summary of the invention

[0003] To solve the above technical problems, the present invention proposes a multi-sound source detection and positioning method and system of a convolutional recurrent neural network, which classifies multiple sound sources that make sounds at the same time and predicts their spatial orientation, effectively distinguishes different sound sources at the same time and locates them independently, and is suitable for scenarios where multiple sound sources need to be positioned.

[0004] The technical solution of the method of the present invention is a multi-sound source detection and positioning method of a convolutional recurrent neural network, which specifically includes the following steps:

[0005] Step 1: Input multiple groups of multi-channel audio, perform discrete Fourier transform on each group of multi-channel audio, obtain the spectrum of each group of multi-channel audio, mark the true value of the event type and the true arrival direction of each frame of audio sound of each group of multi-channel audio, extract audio feature information of each group of multi-channel audio, obtain the amplitude feature sequence and phase feature sequence of each group of multi-channel audio, and concatenate the feature amplitude of each group of multi-channel audio and the phase feature of each group of multi-channel audio by head-to-tail splicing to obtain the feature sequence of each group of multi-channel audio;

[0006] Step 2: Calculate the relative position of each frame of audio in each group of multi-channel audio by sine-cosine alternating position coding to construct a position coding sequence for each group of multi-channel audio;

[0007] Step 3: construct a multi-sound source detection and localization network, input each frame of audio sound of each group of multi-channel audio, predict the event type and arrival direction through the multi-sound source detection and localization network, obtain the predicted value of the event type of each group of multi-channel audio and multi-frame audio, construct the sound event type prediction result sequence of each group of multi-channel audio, and the predicted value of the arrival direction of each group of multi-channel audio and multi-frame audio, construct the arrival direction prediction result sequence of each group of multi-channel audio, combine the sound event type true result sequence of each group of multi-channel audio and the sound event type prediction result sequence to construct a binary cross loss, combine the arrival direction true result sequence of each group of multi-channel audio and the arrival direction prediction result sequence to construct a mean square error loss, weightedly construct a weighted loss function model, and optimize the training through the adaptive moment estimation algorithm to obtain the optimized multi-sound source detection and localization network;

[0008] Step 4: Collect real-time multi-channel audio and input it into the optimized multi-sound source detection and localization network to predict the event type and arrival direction, and obtain the predicted event type and predicted arrival direction of the real-time multi-channel audio;

[0009] Preferably, each group of multi-channel audio in step 1 is composed of T frames of audio sounds;

[0010] Each group of multi-channel audio in step 1 is used for training;

[0011] The actual value of the event type of each frame of audio sound of each group of multi-channel audio is marked in step 1, as follows:

[0012] Construct a true result sequence of the sound event type of each group of multi-channel audio by using the true value of the event type of each group of multi-channel audio multi-frame audio sound;

[0013] The true value of the arrival direction of each frame of audio for each group of multi-channel audio marked in step 1 is as follows:

[0014] Constructing a true result sequence of the direction of arrival of each group of multi-channel audio by using the true value of the direction of arrival of each group of multi-channel audio multi-frame audio sound;

[0015] Preferably, the multi-sound source detection and localization network in step 3 is composed of an improved multi-layer 2D convolutional layer, a gated recurrent unit optimized by an attention mechanism, a global average pooling layer, and a fully connected layer cascaded in sequence;

[0016] The improved multi-layer 2D convolutional layer is used to input the feature sequence of each group of multi-channel audio.

[0017] The improved multi-layer 2D convolution layer is composed of a first improved convolution module, a second improved convolution module, ..., an Mth improved convolution module connected in cascade sequence;

[0018] The first improved convolution module, the second improved convolution module, ..., and the M-1 improved convolution module have the same structure;

[0019] The i-th improved convolution module is composed of a convolution layer, a first atrous spatial pooling layer, a second atrous spatial pooling layer, a third atrous spatial pooling layer, a fusion module, and a spatial transformer network cascade;

[0020] The convolutional layer is connected to the first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer in sequence respectively;

[0021] The first void space pooling layer is connected to the fusion module;

[0022] The second atrous spatial pooling layer is connected to the fusion module;

[0023] The third atrous spatial pooling layer is connected to the fusion module; the fusion module is connected to the spatial transformer network, i∈[1,M-1];

[0024] The i-th improved convolution module is input with the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtains the convolution feature sequence of each group of multi-channel audio of the i-th improved convolution module through improved convolution processing, and outputs it to the i+1-th improved convolution module;

[0025] If i=1, the input of the first improved convolution module is the feature sequence of each group of multi-channel audio;

[0026] The convolution layer of the i-th improved convolution module is used to input the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtain the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module through convolution processing, and output them to the first void space pooling layer, the second void space pooling layer, and the third void space pooling layer of the i-th improved convolution module respectively;

[0027] The first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer of the i-th improved convolution module respectively use convolution kernels with the same convolution kernel size and with the atrous rates from low to high;

[0028] The first atrous spatial pooling layer, the second atrous spatial pooling layer, and the third atrous spatial pooling layer of the i-th improved convolution module respectively sample the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module in sequence to obtain the first scale feature map, the second scale feature map, and the third scale feature map of each group of multi-channel audio of the i-th improved convolution module, and output them to the fusion module of the i-th improved convolution module;

[0029] The fusion module of the i-th improved convolution module splices the first scale feature map, the second scale feature map, and the third scale feature of each group of multi-channel audio of the i-th improved convolution module end to end to obtain a fused scale feature map of each group of multi-channel audio of the i-th improved convolution module, and outputs it to the spatial transformer network of the i-th improved convolution module;

[0030] The spatial transformer network of the i-th improved convolution module performs translation and rotation transformation on the fusion scale feature map of each group of multi-channel audio of the i-th improved convolution module, amplifies and translates the key features to the center of the feature map, and outputs the key feature focused feature map of each group of multi-channel audio of the i-th improved convolution module;

[0031] The Mth improved convolution module is composed of a cascade of a convolution layer and a maximum pooling layer;

[0032] The M-th improved convolution module is inputted with the convolution feature sequence of each group of multi-channel audio of the M-1-th improved convolution module, and sequentially passes through the convolution layer of the M-th improved convolution module and the maximum pooling layer to perform maximum redundancy processing, so as to obtain a multi-layer convolution feature sequence of each group of multi-channel audio, and outputs it to the gated recurrent unit optimized by the attention mechanism;

[0033] The attention mechanism optimized gated recurrent unit inputs the multi-layer convolution feature sequence of each group of multi-channel audio, obtains the attention mechanism feature sequence of each group of multi-channel audio, and outputs it to the global average pooling layer;

[0034] The global average pooling layer is used to perform global average pooling on the attention mechanism feature sequence of each group of multi-channel audio, concatenate the sequence with the position encoding sequence of each group of multi-channel audio, obtain the global feature sequence of each group of multi-channel audio, and output it to the fully connected layer;

[0035] The fully connected layer is composed of a sound event type fully connected layer and an arrival direction fully connected layer in parallel;

[0036] The sound event type fully connected layer is connected to the global average pooling layer, and the arrival direction fully connected layer is connected to the global average pooling layer;

[0037] The sound event type fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the event type of each frame of audio sound of each group of multi-channel audio; and construct the sound event type prediction result sequence of each group of multi-channel audio through the predicted value of the event type of each group of multi-channel audio and multi-frame audio;

[0038] The arrival direction fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the arrival direction of each frame of audio sound of each group of multi-channel audio; and construct the arrival direction prediction result sequence of each group of multi-channel audio through the predicted values ​​of the arrival direction of each group of multi-channel audio frames;

[0039] The weighted loss function model described in step 3 is specifically defined as follows:

[0040] Loss = αBCELoss + βMSE

[0041] Among them, BCELoss represents the binary cross entropy loss function, MSE represents the mean square error loss function, α represents the weight of the binary cross entropy loss function, and β represents the weight of the mean square error loss function;

[0042] The binary cross entropy loss function is defined as follows:

[0043]

[0044] Where N represents the total number of groups of multi-channel audio, n j Indicates the total number of frames of the audio sound of the j-th group of multi-channel audio, y i_j represents the true value of the event type of each frame of each group of multi-channel audio marked in step 1, specifically the binary true value 0 or 1 of the sound event type of the i-th frame of the j-th group of multi-channel audio, p(yi_j =1) represents the prediction result sequence of the sound event type of each group of multi-channel audio in step 3, specifically refers to the predicted value of the sound event type of the i-th frame of the j-th group of multi-channel audio, that is, the probability that the model predicts that the sound event type of the i-th frame is 1;

[0045] The mean square error loss function is as follows:

[0046]

[0047] Where N represents the total number of multi-channel audio groups, n j Indicates the total number of frames of the j-th group of multi-channel audio, x i_j represents the true value of the arrival direction of each frame of each group of multi-channel audio marked in step 1, specifically the true value of the arrival direction of the i-th frame of the j-th group of multi-channel audio, x′ i_j represents the predicted value of the arrival direction of each frame of each group of multi-channel audio in step 3, specifically the predicted value of the arrival direction of the i-th frame of the j-th group of multi-channel audio;

[0048] The technical solution of the system of the present invention is a multi-sound source detection and positioning system of a convolutional recurrent neural network, comprising:

[0049] Sample label construction module, position coding sequence generation module, multi-sound source detection and positioning network optimization module, real-time multi-channel audio prediction module;

[0050] The sample label construction module is used to input multiple groups of multi-channel audio, perform discrete Fourier transform on each group of multi-channel audio, obtain the spectrum of each group of multi-channel audio, mark the true value of the event type and the true arrival direction of each frame of audio sound of each group of multi-channel audio, extract audio feature information from each group of multi-channel audio, obtain the amplitude feature sequence and phase feature sequence of each group of multi-channel audio, and obtain the feature sequence of each group of multi-channel audio by concatenating the feature amplitude of each group of multi-channel audio and the phase feature of each group of multi-channel audio through head-to-tail splicing;

[0051] The position coding sequence generation module is used to calculate the relative position of each frame of audio of each group of multi-channel audio in each group of multi-channel audio by sine-cosine alternating position coding, so as to construct a position coding sequence of each group of multi-channel audio;

[0052] The multi-sound source detection and positioning network optimization module is used to construct a multi-sound source detection and positioning network, input each group of multi-channel audio each frame of audio sound, predict event type and arrival direction through the multi-sound source detection and positioning network, obtain the predicted value of the event type of each group of multi-channel audio multi-frame audio to construct a sound event type prediction result sequence of each group of multi-channel audio, and the predicted value of the arrival direction of each group of multi-channel audio multi-frame audio to construct a prediction result sequence of the arrival direction of each group of multi-channel audio, combine the sound event type real result sequence of each group of multi-channel audio and the sound event type prediction result sequence to construct a binary cross loss, combine the arrival direction real result sequence of each group of multi-channel audio and the arrival direction prediction result sequence to construct a mean square error loss, weightedly construct a weighted loss function model, and optimize and train the optimized multi-sound source detection and positioning network through adaptive moment estimation algorithm optimization;

[0053] The real-time multi-channel audio prediction module is used to collect real-time multi-channel audio and input it into the optimized multi-sound source detection and positioning network to predict event type and arrival direction, so as to obtain the predicted event type and predicted arrival direction of the real-time multi-channel audio.

[0054] Compared with the existing multi-sound source localization algorithm, the advantages of the present invention are: the present method uses two sets of parallel network branches to simultaneously perform the sound event detection task and the arrival angle estimation task, which is a good combination of the existing sound source localization algorithm and the active sound source detection method, and the detection and classification of the active sound source also plays an auxiliary role in the localization task; the present method uses a layer of global average pooling instead of a fully connected layer to reduce the number of network parameters and prevent overfitting; the present method introduces position coding and makes good use of the context information between audio frames; when performing the arrival angle estimation task, the present method can estimate the three-dimensional spatial orientation, while most algorithms that perform two tasks simultaneously can only be carried out in the horizontal orientation; compared with previous methods, the number and category of overlapping sound sources detected by the present method have been increased from one to two sound sources to three sound sources, and the category has been increased from multiple different categories to a mixture of the same type and different types, and the method actually has the potential to detect more sound sources and has good versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 : A flowchart of the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In specific implementation, the method proposed in the technical solution of the present invention can be implemented by technical personnel in this field using computer software technology to realize automatic operation process. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of the present invention and computer equipment running the corresponding computer program, should also be within the scope of protection of the present invention.

[0058] The following is combined with Figure 1 The specific implementation of the present invention is a method and system for detecting and locating multiple sound sources using a convolutional recurrent neural network, as follows:

[0059] Reference Figure 1 The present invention proposes a multi-sound source detection and positioning method of a convolutional recurrent neural network, comprising the following steps:

[0060] The input is a multi-source audio signal. The data used for training and testing is a public dataset provided by Tampere University of Technology, the Ambisonics format anechoic synthetic impulse response dataset (TUT Sound Events2018-Ambisonics, Anechoic and Synthetic Impulse Response dataset, ANSIM). In this dataset, each audio segment is 30 seconds long, with a sampling rate of 44.1kHz and contains four channels.

[0061] It should be noted that the audio parameters and object types specified herein are merely examples for illustrating the implementation process of the present invention and are not intended to limit the present invention.

[0062] Step 1: Input multiple groups of multi-channel audio, perform discrete Fourier transform on each group of multi-channel audio, obtain the spectrum of each group of multi-channel audio, mark the true value of the event type and the true arrival direction of each frame of audio sound of each group of multi-channel audio, extract audio feature information of each group of multi-channel audio, obtain the amplitude feature sequence and phase feature sequence of each group of multi-channel audio, and concatenate the feature amplitude of each group of multi-channel audio and the phase feature of each group of multi-channel audio by head-to-tail splicing to obtain the feature sequence of each group of multi-channel audio;

[0063] Each group of multi-channel audio in step 1 is composed of T frames of audio sound;

[0064] In the embodiment of the present invention, the total number of frames T constructed by one record is 5166 frames;

[0065] In the embodiment of the present invention, the frame length is specified to be 22ms, the number of Fourier transform points M is 512, and the window function uses a Hamming window with a 50% frame offset, as shown in the following formula:

[0066]

[0067] That is, a 512-point spectrum is obtained for each channel and each audio frame.

[0068] It should be noted that the frame length, window function type and transformation method specified here are only used to illustrate the specific implementation steps of the present invention and are not used to limit the present invention.

[0069] In the embodiment of the present invention, the sequence size of each set of feature sequences of multi-channel audio is C×T×2×M / 2.

[0070] In the embodiment of the present invention, C=4, T=5166, and M=512.

[0071] Each group of multi-channel audio in step 1 is used for training;

[0072] The actual value of the event type of each frame of audio sound of each group of multi-channel audio is marked in step 1, as follows:

[0073] Construct a true result sequence of the sound event type of each group of multi-channel audio by using the true value of the event type of each group of multi-channel audio multi-frame audio sound;

[0074] The true value of the arrival direction of each frame of audio for each group of multi-channel audio marked in step 1 is as follows:

[0075] Constructing a true result sequence of the direction of arrival of each group of multi-channel audio by using the true value of the direction of arrival of each group of multi-channel audio multi-frame audio sound;

[0076] Step 2: Calculate the relative position of each frame of audio in each group of multi-channel audio by sine-cosine alternating position coding to construct a position coding sequence for each group of multi-channel audio;

[0077] In this embodiment, the encoder is in the following form:

[0078] PE(pos,2i)=sin(pos / 10000 2i / d ),PE(pos,2i+1)=cos(pos / 10000 2i+1 / d )

[0079] Among them, PE represents the position code of the pos position, pos represents the position of the audio frame in the entire audio sequence, 2i represents an even frame, 2i+1 represents an odd frame, and d represents the frame length of the audio sequence. Generally speaking, d=T, so in this embodiment, d=5166.

[0080] Step 3: construct a multi-sound source detection and localization network, input each frame of audio sound of each group of multi-channel audio, predict the event type and arrival direction through the multi-sound source detection and localization network, obtain the predicted value of the event type of each group of multi-channel audio and multi-frame audio, construct the sound event type prediction result sequence of each group of multi-channel audio, and the predicted value of the arrival direction of each group of multi-channel audio and multi-frame audio, construct the arrival direction prediction result sequence of each group of multi-channel audio, combine the sound event type true result sequence of each group of multi-channel audio and the sound event type prediction result sequence to construct a binary cross loss, combine the arrival direction true result sequence of each group of multi-channel audio and the arrival direction prediction result sequence to construct a mean square error loss, weightedly construct a weighted loss function model, and optimize the training through the adaptive moment estimation algorithm to obtain the optimized multi-sound source detection and localization network;

[0081] The multi-sound source detection and localization network described in step 3 is composed of an improved multi-layer 2D convolutional layer, a gated recurrent unit optimized by an attention mechanism, a global average pooling layer, and a fully connected layer cascaded in sequence;

[0082] The improved multi-layer 2D convolutional layer is used to input the feature sequence of each group of multi-channel audio.

[0083] The improved multi-layer 2D convolution layer is composed of a first improved convolution module, a second improved convolution module, ..., an Mth improved convolution module which are sequentially cascaded;

[0084] The first improved convolution module, the second improved convolution module, ..., and the M-1 improved convolution module have the same structure;

[0085] The i-th improved convolution module is composed of a convolution layer, a first atrous spatial pooling layer, a second atrous spatial pooling layer, a third atrous spatial pooling layer, a fusion module, and a spatial transformer network cascade;

[0086] The convolutional layer is connected to the first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer in sequence respectively;

[0087] The first void space pooling layer is connected to the fusion module;

[0088] The second atrous spatial pooling layer is connected to the fusion module;

[0089] The third atrous spatial pooling layer is connected to the fusion module; the fusion module is connected to the spatial transformer network, i∈[1,M-1];

[0090] The i-th improved convolution module is input with the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtains the convolution feature sequence of each group of multi-channel audio of the i-th improved convolution module through improved convolution processing, and outputs it to the i+1-th improved convolution module;

[0091] If i=1, the input of the first improved convolution module is the feature sequence of each group of multi-channel audio;

[0092] The convolution layer of the i-th improved convolution module is used to input the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtain the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module through convolution processing, and output them to the first void space pooling layer, the second void space pooling layer, and the third void space pooling layer of the i-th improved convolution module respectively;

[0093] In this embodiment, the dilated spatial pooling uses a convolution kernel of 3×3 size and a dilation rate of 1 to 3.

[0094] The first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer of the i-th improved convolution module respectively use convolution kernels with the same convolution kernel size and with the atrous rates from low to high;

[0095] The first atrous spatial pooling layer, the second atrous spatial pooling layer, and the third atrous spatial pooling layer of the i-th improved convolution module respectively sample the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module in sequence to obtain the first scale feature map, the second scale feature map, and the third scale feature map of each group of multi-channel audio of the i-th improved convolution module, and output them to the fusion module of the i-th improved convolution module;

[0096] The fusion module of the i-th improved convolution module splices the first scale feature map, the second scale feature map, and the third scale feature of each group of multi-channel audio of the i-th improved convolution module end to end to obtain a fused scale feature map of each group of multi-channel audio of the i-th improved convolution module, and outputs it to the spatial transformer network of the i-th improved convolution module;

[0097] The spatial transformer network of the i-th improved convolution module performs translation and rotation transformation on the fusion scale feature map of each group of multi-channel audio of the i-th improved convolution module, amplifies and translates the key features to the center of the feature map, and outputs the key feature focused feature map of each group of multi-channel audio of the i-th improved convolution module;

[0098] The Mth improved convolution module is composed of a cascade of a convolution layer and a maximum pooling layer;

[0099] The M-th improved convolution module is inputted with the convolution feature sequence of each group of multi-channel audio of the M-1-th improved convolution module, and sequentially passes through the convolution layer of the M-th improved convolution module and the maximum pooling layer to perform maximum redundancy processing, so as to obtain a multi-layer convolution feature sequence of each group of multi-channel audio, and outputs it to the gated recurrent unit optimized by the attention mechanism;

[0100] The attention mechanism optimized gated recurrent unit inputs the multi-layer convolution feature sequence of each group of multi-channel audio, obtains the attention mechanism feature sequence of each group of multi-channel audio, and outputs it to the global average pooling layer;

[0101] The global average pooling layer is used to perform global average pooling on the attention mechanism feature sequence of each group of multi-channel audio, concatenate the sequence with the position encoding sequence of each group of multi-channel audio, obtain the global feature sequence of each group of multi-channel audio, and output it to the fully connected layer;

[0102] The fully connected layer is composed of a sound event type fully connected layer and an arrival direction fully connected layer in parallel;

[0103] The sound event type fully connected layer is connected to the global average pooling layer, and the arrival direction fully connected layer is connected to the global average pooling layer;

[0104] The sound event type fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the event type of each frame of audio sound of each group of multi-channel audio; and construct the sound event type prediction result sequence of each group of multi-channel audio through the predicted value of the event type of each group of multi-channel audio and multi-frame audio;

[0105] The arrival direction fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the arrival direction of each frame of audio sound of each group of multi-channel audio; and construct the arrival direction prediction result sequence of each group of multi-channel audio through the predicted values ​​of the arrival direction of each group of multi-channel audio frames;

[0106] In the embodiment of the present invention, the predicted value of the event type of each group, each channel, and each frame of audio is output as the activity value of N sound source types. Among the N categories, the predicted active sound source sound event type outputs 1, and the inactive sound source sound event type outputs 0;

[0107] In the embodiment of the present invention, N=11;

[0108] In the embodiment of the present invention, the predicted value of the arrival direction of each frame of audio per channel per group is output as the predicted value of the arrival direction of N sound source types, specifically the three-dimensional Cartesian coordinates x, y, z predicted by N sound event types.

[0109] In the embodiment of the present invention, the range of x is [-1, 1], the range of y is [-1, 1], and the range of z is [-1, 1].

[0110] The weighted loss function model described in step 3 is specifically defined as follows:

[0111] Loss = αBCELoss + βMSE

[0112] Among them, BCELoss represents the binary cross entropy loss function, MSE represents the mean square error loss function, α represents the weight of the binary cross entropy loss function, and β represents the weight of the mean square error loss function;

[0113] In the embodiment of the present invention, α=0.5, β=0.5;

[0114] The binary cross entropy loss function is defined as follows:

[0115]

[0116] Where N represents the total number of groups of multi-channel audio, n j Indicates the total number of frames of the audio sound of the j-th group of multi-channel audio, y i_j represents the true value of the event type of each frame of each group of multi-channel audio marked in step 1, specifically the binary true value 0 or 1 of the sound event type of the i-th frame of the j-th group of multi-channel audio, p(y i_j =1) represents the prediction result sequence of the sound event type of each group of multi-channel audio in step 3, specifically refers to the predicted value of the sound event type of the i-th frame of the j-th group of multi-channel audio, that is, the probability that the model predicts that the sound event type of the i-th frame is 1;

[0117] N should be determined based on the appropriate audio dataset size required for your training;

[0118] In the embodiment of the present invention, N=6000;

[0119] The mean square error loss function is as follows:

[0120]

[0121] Where N represents the total number of multi-channel audio groups, n j Indicates the total number of frames of the j-th group of multi-channel audio, x i_j represents the true value of the arrival direction of each frame of each group of multi-channel audio marked in step 1, specifically the true value of the arrival direction of the i-th frame of the j-th group of multi-channel audio, x′ i_j represents the predicted value of the arrival direction of each frame of each group of multi-channel audio in step 3, specifically the predicted value of the arrival direction of the i-th frame of the j-th group of multi-channel audio;

[0122] Step 4: Collect real-time multi-channel audio and input it into the optimized multi-sound source detection and localization network to predict the event type and arrival direction, and obtain the predicted event type and predicted arrival direction of the real-time multi-channel audio;

[0123] The present invention uses a network model to simultaneously learn the type and spatial orientation of the sound source, so that the algorithm has the ability to distinguish multiple simultaneously active sound sources and estimate their spatial orientations respectively; while performing the sound event detection task, the present invention performs the task of estimating the spatial orientation in a three-dimensional Cartesian coordinate system, which is the most tasks performed among algorithms of the same type; the present invention predicts at most three simultaneously occurring sound sources, which is the most predicted number of sound sources among algorithms of the same type; the present invention has good robustness to reverberation and low signal-to-noise ratio environmental conditions; the present invention has the potential to expand the prediction of more sound sources.

[0124] Experimental results analysis

[0125]

[0126] This method conducts comparative experiments with existing algorithms for the sound event type prediction task and the arrival direction estimation task. The experimental results are shown in the table above. The dataset ANSYN represents an audio dataset collected in a reverberation-free environment, and RESYN Room1, 2, and 3 represent audio datasets collected in three different reverberation environments. F represents the proportion of correctly predicted frames in each group of multi-channel audio in the total number of frames in each group of multi-channel audio in the sound event type prediction task, where the correctly predicted frames include: the sound type does not appear in the frame audio and is predicted not to appear, and the sound type appears in the frame audio and is predicted to appear; the arrival direction estimation error represents the average value of the angular difference between the predicted sound source direction and the actual sound source direction in each frame of each group of multi-channel audio in the arrival direction estimation task; the frame recall rate represents the ratio of the number of frames with a predicted arrival direction estimation error less than N (degrees) in each frame of each group of multi-channel audio in the arrival direction estimation task to the total number of frames in each group of multi-channel audio;

[0127] In this embodiment, N=5;

[0128] Obviously, the higher the F value, the better the sound event type prediction performance of the method; the smaller the arrival direction estimation error and the higher the frame recall rate, the better the arrival direction estimation performance of the method. This method performs well in terms of F value and frame recall rate indicators, especially in high reverberation and multi-sound source environments, proving the effectiveness and superiority of the algorithm.

[0129] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0130] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A method for detecting and locating multiple sound sources using a convolutional recurrent neural network, characterized in that: The following steps are involved: Step 1: Input multiple groups of multi-channel audio, perform discrete Fourier transform on each group of multi-channel audio, obtain the spectrum of each group of multi-channel audio, mark the true value of the event type and the true arrival direction of each frame of audio sound of each group of multi-channel audio, extract audio feature information of each group of multi-channel audio, obtain the amplitude feature sequence and phase feature sequence of each group of multi-channel audio, and concatenate the feature amplitude of each group of multi-channel audio and the phase feature of each group of multi-channel audio by head-to-tail splicing to obtain the feature sequence of each group of multi-channel audio; Step 2: Calculate the relative position of each frame of audio in each group of multi-channel audio by sine-cosine alternating position coding to construct a position coding sequence for each group of multi-channel audio; Step 3: construct a multi-sound source detection and localization network, input each frame of audio sound of each group of multi-channel audio, predict the event type and arrival direction through the multi-sound source detection and localization network, obtain the predicted value of the event type of each group of multi-channel audio and multi-frame audio, construct the sound event type prediction result sequence of each group of multi-channel audio, and the predicted value of the arrival direction of each group of multi-channel audio and multi-frame audio, construct the arrival direction prediction result sequence of each group of multi-channel audio, combine the sound event type true result sequence of each group of multi-channel audio and the sound event type prediction result sequence to construct a binary cross loss, combine the arrival direction true result sequence of each group of multi-channel audio and the arrival direction prediction result sequence to construct a mean square error loss, weightedly construct a weighted loss function model, and optimize the training through the adaptive moment estimation algorithm to obtain the optimized multi-sound source detection and localization network; Step 4: Collect real-time multi-channel audio and input it into the optimized multi-sound source detection and localization network to predict the event type and arrival direction, and obtain the predicted event type and predicted arrival direction of the real-time multi-channel audio.

2. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 1, characterized in that: Each group of multi-channel audio in step 1 is composed of T frames of audio sound; Each group of multi-channel audio in step 1 is used for training; The actual value of the event type of each frame of audio sound of each group of multi-channel audio is marked in step 1, as follows: Construct a true result sequence of the sound event type of each group of multi-channel audio by using the true value of the event type of each group of multi-channel audio multi-frame audio sound; The true value of the arrival direction of each frame of audio for each group of multi-channel audio marked in step 1 is as follows: The true result sequence of the arrival direction of each group of multi-channel audio is constructed by using the true value of the arrival direction of each group of multi-channel audio multi-frame audio sound.

3. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 2, characterized in that: The multi-sound source detection and localization network described in step 3 is composed of an improved multi-layer 2D convolutional layer, a gated recurrent unit optimized by an attention mechanism, a global average pooling layer, and a fully connected layer cascaded in sequence; The improved multi-layer 2D convolutional layer is used to input the feature sequence of each group of multi-channel audio; The attention mechanism optimized gated recurrent unit inputs the multi-layer convolution feature sequence of each group of multi-channel audio, obtains the attention mechanism feature sequence of each group of multi-channel audio, and outputs it to the global average pooling layer; The global average pooling layer is used to perform global average pooling on the attention mechanism feature sequence of each group of multi-channel audio, concatenate the sequence with the position encoding sequence of each group of multi-channel audio, obtain the global feature sequence of each group of multi-channel audio, and output it to the fully connected layer; The fully connected layer is composed of a sound event type fully connected layer and an arrival direction fully connected layer in parallel.

4. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 3, characterized in that: The improved multi-layer 2D convolution layer is composed of a first improved convolution module, a second improved convolution module, ..., an Mth improved convolution module which are sequentially cascaded; The first improved convolution module, the second improved convolution module, ..., and the M-1 improved convolution module have the same structure; The i-th improved convolution module is composed of a convolution layer, a first atrous spatial pooling layer, a second atrous spatial pooling layer, a third atrous spatial pooling layer, a fusion module, and a spatial transformer network cascade; The convolutional layer is connected to the first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer in sequence respectively; The first void space pooling layer is connected to the fusion module; The second atrous spatial pooling layer is connected to the fusion module; The third atrous spatial pooling layer is connected to the fusion module; the fusion module is connected to the spatial transformer network, i∈[1,M-1]; The i-th improved convolution module is input with the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtains the convolution feature sequence of each group of multi-channel audio of the i-th improved convolution module through improved convolution processing, and outputs it to the i+1-th improved convolution module; If i=1, the input of the first improved convolution module is the feature sequence of each group of multi-channel audio; The Mth improved convolution module is composed of a cascade of a convolution layer and a maximum pooling layer; The Mth improved convolution module inputs the convolution feature sequence of each group of multi-channel audio of the M-1th improved convolution module, and sequentially performs convolution processing through the convolution layer of the Mth improved convolution module and the maximum pooling layer to obtain a multi-layer convolution feature sequence of each group of multi-channel audio, and outputs it to the gated recurrent unit optimized by the attention mechanism.

5. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 4, characterized in that: The convolution layer of the i-th improved convolution module is used to input the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtain the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module through convolution processing, and output them to the first void space pooling layer, the second void space pooling layer, and the third void space pooling layer of the i-th improved convolution module respectively; The first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer of the i-th improved convolution module respectively use convolution kernels with the same convolution kernel size and with the atrous rates from low to high; The first atrous spatial pooling layer, the second atrous spatial pooling layer, and the third atrous spatial pooling layer of the i-th improved convolution module respectively sample the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module in sequence to obtain the first scale feature map, the second scale feature map, and the third scale feature map of each group of multi-channel audio of the i-th improved convolution module, and output them to the fusion module of the i-th improved convolution module.

6. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 5, characterized in that: The fusion module of the i-th improved convolution module splices the first scale feature map, the second scale feature map, and the third scale feature of each group of multi-channel audio of the i-th improved convolution module end to end to obtain a fused scale feature map of each group of multi-channel audio of the i-th improved convolution module, and outputs it to the spatial transformer network of the i-th improved convolution module; The spatial transformer network of the i-th improved convolution module performs translation and rotation transformation on the fusion scale feature map of each group of multi-channel audio of the i-th improved convolution module, enlarges and translates the key features to the center of the feature map, and outputs the key feature focused feature map of each group of multi-channel audio of the i-th improved convolution module.

7. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 6, characterized in that: The sound event type fully connected layer is connected to the global average pooling layer, and the arrival direction fully connected layer is connected to the global average pooling layer; The sound event type fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the event type of each frame of audio sound in each group of multi-channel audio; and construct the sound event type prediction result sequence of each group of multi-channel audio through the predicted values ​​of the event types of multiple frames of audio in each group of multi-channel audio.

8. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 7, characterized in that: The arrival direction fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the arrival direction of each frame of audio sound of each group of multi-channel audio; and construct the arrival direction prediction result sequence of each group of multi-channel audio through the predicted values ​​of the arrival direction of each group of multi-channel audio frames.

9. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 8, characterized in that: The weighted loss function model described in step 3 is specifically defined as follows: Loss = αBCELoss + βMSE Among them, BCELoss represents the binary cross entropy loss function, MSE represents the mean square error loss function, α represents the weight of the binary cross entropy loss function, and β represents the weight of the mean square error loss function; The binary cross entropy loss function is defined as follows: Where N represents the total number of groups of multi-channel audio, n j Indicates the total number of frames of the audio sound of the j-th group of multi-channel audio, y i_j represents the true value of the event type of each frame of each group of multi-channel audio marked in step 1, specifically the binary true value 0 or 1 of the sound event type of the i-th frame of the j-th group of multi-channel audio, p(y i_j =1) represents the prediction result sequence of the sound event type of each group of multi-channel audio in step 3, specifically refers to the predicted value of the sound event type of the i-th frame of the j-th group of multi-channel audio, that is, the probability that the model predicts that the sound event type of the i-th frame is 1; The mean square error loss function is as follows: Where N represents the total number of multi-channel audio groups, n j Indicates the total number of frames of the j-th group of multi-channel audio, x i_j represents the true value of the arrival direction of each frame of each group of multi-channel audio marked in step 1, specifically the true value of the arrival direction of the i-th frame of the j-th group of multi-channel audio, x′ i_j It represents the predicted value of the arrival direction of each frame of audio sound of each group of multi-channel audio in step 3, specifically refers to the predicted value of the arrival direction of the i-th frame of the j-th group of multi-channel audio.

10. A convolutional recurrent neural network multi-sound source detection and positioning system, characterized in that: include: Sample label construction module, position coding sequence generation module, multi-sound source detection and positioning network optimization module, real-time multi-channel audio prediction module; The sample label construction module is used to input multiple groups of multi-channel audio, perform discrete Fourier transform on each group of multi-channel audio, obtain the spectrum of each group of multi-channel audio, mark the true value of the event type and the true arrival direction of each frame of audio sound of each group of multi-channel audio, extract audio feature information from each group of multi-channel audio, obtain the amplitude feature sequence and phase feature sequence of each group of multi-channel audio, and obtain the feature sequence of each group of multi-channel audio by concatenating the feature amplitude of each group of multi-channel audio and the phase feature of each group of multi-channel audio through head-to-tail splicing; The position coding sequence generation module is used to calculate the relative position of each frame of audio of each group of multi-channel audio in each group of multi-channel audio by sine-cosine alternating position coding, so as to construct a position coding sequence of each group of multi-channel audio; The multi-sound source detection and positioning network optimization module is used to construct a multi-sound source detection and positioning network, input each group of multi-channel audio each frame of audio sound, predict event type and arrival direction through the multi-sound source detection and positioning network, obtain the predicted value of the event type of each group of multi-channel audio multi-frame audio to construct a sound event type prediction result sequence of each group of multi-channel audio, and the predicted value of the arrival direction of each group of multi-channel audio multi-frame audio to construct a prediction result sequence of the arrival direction of each group of multi-channel audio, combine the sound event type real result sequence of each group of multi-channel audio and the sound event type prediction result sequence to construct a binary cross loss, combine the arrival direction real result sequence of each group of multi-channel audio and the arrival direction prediction result sequence to construct a mean square error loss, weightedly construct a weighted loss function model, and optimize and train the optimized multi-sound source detection and positioning network through adaptive moment estimation algorithm optimization; The real-time multi-channel audio prediction module is used to collect real-time multi-channel audio and input it into the optimized multi-sound source detection and positioning network to predict event type and arrival direction, so as to obtain the predicted event type and predicted arrival direction of the real-time multi-channel audio.

Images