A convolutional recurrent neural network multi-sound source detection and localization method and system

Abstract

The present invention proposes a multi-sound source detection and localization method and system using a convolutional recurrent neural network. The present invention extracts the amplitude and phase features of multi-channel audio, uses an embedding layer to generate a position code for each frame of audio, and inputs the features and position code into a neural network. In the network structure, a convolutional neural network is used to learn to distinguish and localize sound source categories based on inter-channel features. Multiple groups of dilated spatial pyramid pooling are used between convolutional layers to extract features from different scales. A spatial transformer network is used to maintain the translation invariance of the convolutional neural network. An attention-based gated recurrent unit is used to learn contextual information. The sound source location of the current frame is assisted by information from previous frames. Global features are obtained using global average pooling. The global features and position code are combined and input into parallel fully connected layers, which output sound event prediction results and arrival direction prediction results. The present invention achieves the separate localization of multiple sound sources simultaneously emitting sound, and is robust to reverberant and low signal-to-noise ratio environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital audio signal processing, and in particular relates to a multi-sound source detection and positioning method and system based on a convolutional recurrent neural network. Background Art

[0002] Multi-source localization is an audio signal processing technique that determines the spatial orientation (also known as DOA estimation) or distance of multiple active sound sources relative to a specified location based on signals received by a microphone array. It extends sound source localization technology by increasing the number of sound sources. Sound source localization plays an important role in related audio signal processing technologies, such as speech recognition, speech separation, and wireless communications. In listening scenarios involving multiple sound sources or speakers, this method accurately localizes each sound source. This allows the use of various audio spatial rendering techniques to accurately render different sounds at specific locations during playback, enhancing the realism of the scene. Traditional sound source localization algorithms fall into three main categories: those based on relative delay estimation, such as the time difference of arrival (TDOA) algorithm and the phase-weighted generalized cross-correlation algorithm (GCC-PHAT); those based on beamforming, such as the phase-shift-weighted steering response power method (SRP-PHAT); and those based on signal subspaces, such as the multiple signal classification algorithm (MUSIC) and the signal parameter estimation method using rotational invariance techniques (ESPRIT). These traditional methods are based on the physical principles of acoustics, and most suffer from issues such as insufficient spatial resolution and sensitivity to reverberant and noisy environments. With the widespread application of neural networks in various fields, sound source localization technology has also achieved new breakthroughs. This technology has significantly improved the accuracy and precision of localization in environments containing noise and reverberation. Using undisturbed audio data, a sound source localization algorithm based on a deep neural network achieved a prediction accuracy exceeding 99% and an average prediction error below 5% for a single sound source, and an accuracy exceeding 70% and an average prediction error below 20% for three sound sources. This performance is similar to the best-performing traditional method, the MUSIC algorithm, in terms of average prediction error. However, when testing multiple sets of data containing noise and reverberation, the prediction error of the MUSIC algorithm rapidly increased to over 50%, while the prediction error of the deep neural network-based method was only approximately 10% lower than the undisturbed performance. This method has more advantages than other positioning networks, including but not limited to: the number of predictable sound sources exceeds 3, and the update of the network structure enables the algorithm to have the potential to predict more sound sources; the predicted angle range not only includes 360° all-round angles, but also supports the prediction of elevation angles in the range of -60° to 60°; it supports various microphone array structures including linear and spherical, and has good generalization performance. Summary of the Invention

[0003] To solve the above technical problems, the present invention proposes a multi-sound source detection and positioning method and system based on a convolutional recurrent neural network, which classifies multiple sound sources that sound simultaneously and predicts their spatial orientation, effectively distinguishes different sound sources at the same time and locates them independently, and is suitable for scenarios where multiple sound sources need to be positioned.

[0004] The technical solution of the method of the present invention is a multi-sound source detection and localization method using a convolutional recurrent neural network, which specifically includes the following steps:

[0005] Step 1: Input multiple groups of multi-channel audio, perform discrete Fourier transform on each group of multi-channel audio, obtain the spectrum of each group of multi-channel audio, mark the true value of the event type and the true value of the arrival direction of each frame of audio in each group of multi-channel audio, extract audio feature information from each group of multi-channel audio, obtain the amplitude feature sequence and phase feature sequence of each group of multi-channel audio, and concatenate the feature amplitude and phase features of each group of multi-channel audio by end-to-end splicing to obtain the feature sequence of each group of multi-channel audio;

[0006] Step 2: Calculate the relative position of each frame of audio in each group of multi-channel audio by alternating sine and cosine position coding to construct a position coding sequence for each group of multi-channel audio;

[0007] Step 3: Construct a multi-sound source detection and localization network, input each frame of audio for each group of multi-channel audio, predict the event type and arrival direction through the multi-sound source detection and localization network, obtain the predicted value of the event type of each group of multi-channel audio and multi-frame audio, construct a sound event type prediction result sequence for each group of multi-channel audio, and the predicted value of the arrival direction of each group of multi-channel audio and multi-frame audio, construct a sequence of arrival direction prediction results for each group of multi-channel audio, combine the true result sequence of the sound event type of each group of multi-channel audio and the sound event type prediction result sequence to construct a binary cross loss, combine the true result sequence of the arrival direction of each group of multi-channel audio and the arrival direction prediction result sequence to construct a mean square error loss, weightedly construct a weighted loss function model, and optimize and train the optimized multi-sound source detection and localization network through the adaptive moment estimation algorithm;

[0008] Step 4: Collect real-time multi-channel audio and input it into the optimized multi-sound source detection and localization network to predict the event type and arrival direction, and obtain the predicted event type and predicted arrival direction of the real-time multi-channel audio;

[0009] Preferably, each group of multi-channel audio in step 1 is composed of T frames of audio sound;

[0010] Each set of multi-channel audio in step 1 is used for training;

[0011] The actual value of the event type of each frame of audio sound in each group of multi-channel audio is marked in step 1, as follows:

[0012] Construct a true result sequence of the sound event type of each group of multi-channel audio by using the true value of the event type of each group of multi-channel audio and multi-frame audio sound;

[0013] The true value of the arrival direction of each frame of each multi-channel audio group marked in step 1 is as follows:

[0014] Constructing a true result sequence of the direction of arrival of each group of multi-channel audio by using the true value of the direction of arrival of each group of multi-channel audio multi-frame audio sound;

[0015] Preferably, the multi-sound source detection and localization network in step 3 is composed of an improved multi-layer 2D convolutional layer, an attention mechanism-optimized gated recurrent unit, a global average pooling layer, and a fully connected layer cascaded in sequence;

[0016] The improved multi-layer 2D convolutional layer is used to input the feature sequence of each group of multi-channel audio,

[0017] The improved multi-layer 2D convolution layer is composed of a first improved convolution module, a second improved convolution module, ..., an Mth improved convolution module cascaded in sequence;

[0018] The first improved convolution module, the second improved convolution module, ..., and the M-1 improved convolution module have the same structure;

[0019] The i-th improved convolution module is composed of a convolution layer, a first atrous spatial pooling layer, a second atrous spatial pooling layer, a third atrous spatial pooling layer, a fusion module, and a spatial transformer network cascade;

[0020] The convolutional layer is connected to the first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer in sequence respectively;

[0021] The first atrous spatial pooling layer is connected to the fusion module;

[0022] The second atrous spatial pooling layer is connected to the fusion module;

[0023] The third atrous spatial pooling layer is connected to the fusion module; the fusion module is connected to the spatial transformer network, i∈[1,M-1];

[0024] The i-th improved convolution module receives as input the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtains the convolution feature sequence of each group of multi-channel audio of the i-th improved convolution module through improved convolution processing, and outputs it to the i+1-th improved convolution module;

[0025] If i=1, the input of the first improved convolution module is the feature sequence of each group of multi-channel audio;

[0026] The convolution layer of the i-th improved convolution module is used to input the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtain the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module through convolution processing, and output it to the first void space pooling layer, the second void space pooling layer, and the third void space pooling layer of the i-th improved convolution module respectively;

[0027] The first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer of the i-th improved convolution module respectively use convolution kernels with the same size and atrous rates from low to high;

[0028] The first atrous spatial pooling layer, the second atrous spatial pooling layer, and the third atrous spatial pooling layer of the i-th improved convolution module respectively sample the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module in sequence to obtain the first scale feature map, the second scale feature map, and the third scale feature map of each group of multi-channel audio of the i-th improved convolution module, and output them to the fusion module of the i-th improved convolution module;

[0029] The fusion module of the i-th improved convolution module concatenates the first scale feature map, the second scale feature map, and the third scale feature of each group of multi-channel audio of the i-th improved convolution module end to end to obtain a fused scale feature map of each group of multi-channel audio of the i-th improved convolution module, and outputs the fused scale feature map to the spatial transformer network of the i-th improved convolution module;

[0030] The spatial transformer network of the i-th improved convolution module performs a translation and rotation transformation on the fused scale feature map of each group of multi-channel audio of the i-th improved convolution module, amplifies and translates the key features to the center of the feature map, and outputs a key feature focused feature map of each group of multi-channel audio of the i-th improved convolution module;

[0031] The M-th improved convolution module is composed of a cascade of convolutional layers and maximum pooling layers;

[0032] The Mth improved convolution module inputs the convolution feature sequence of each group of multi-channel audio of the M-1th improved convolution module, sequentially performs convolution processing through the convolution layer of the Mth improved convolution module and performs maximum redundancy processing through the maximum pooling layer to obtain a multi-layer convolution feature sequence of each group of multi-channel audio, and outputs it to the gated recurrent unit optimized by the attention mechanism;

[0033] The attention mechanism optimized gated recurrent unit inputs the multi-layer convolution feature sequence of each group of multi-channel audio, obtains the attention mechanism feature sequence of each group of multi-channel audio, and outputs it to the global average pooling layer;

[0034] The global average pooling layer is used to perform global average pooling on the attention mechanism feature sequence of each group of multi-channel audio, concatenate it with the position encoding sequence of each group of multi-channel audio, obtain a global feature sequence of each group of multi-channel audio, and output it to the fully connected layer;

[0035] The fully connected layer is composed of a sound event type fully connected layer and an arrival direction fully connected layer in parallel;

[0036] The sound event type fully connected layer is connected to the global average pooling layer, and the arrival direction fully connected layer is connected to the global average pooling layer;

[0037] The sound event type fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the event type of each frame of audio sound in each group of multi-channel audio; and construct the sound event type prediction result sequence of each group of multi-channel audio through the predicted value of the event type of each group of multi-channel audio and multi-frame audio;

[0038] The arrival direction fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the arrival direction of each frame of audio sound in each group of multi-channel audio; and construct the arrival direction prediction result sequence of each group of multi-channel audio through the predicted value of the arrival direction of each group of multi-channel audio and multiple frames of audio;

[0039] The weighted loss function model described in step 3 is specifically defined as follows:

[0040] Loss = αBCELoss + βMSE

[0041] Among them, BCELoss represents the binary cross entropy loss function, MSE represents the mean square error loss function, α represents the weight of the binary cross entropy loss function, and β represents the weight of the mean square error loss function;

[0042] The binary cross entropy loss function is defined as follows:

[0043]

[0044] Where N represents the total number of multi-channel audio groups, n j Indicates the total number of frames of the audio sound of the j-th group of multi-channel audio, y i_j represents the true value of the event type of each frame of each group of multi-channel audio marked in step 1, specifically the binary true value 0 or 1 of the sound event type of the i-th frame of the j-th group of multi-channel audio, p(yi_j =1) represents the sequence of sound event type prediction results for each group of multi-channel audio in step 3, specifically the predicted value of the sound event type of the i-th frame of the j-th group of multi-channel audio, that is, the probability that the model predicts that the sound event type of the i-th frame is 1;

[0045] The mean square error loss function is as follows:

[0046]

[0047] Among them, N represents the total number of multi-channel audio groups, n j Indicates the total number of frames of the j-th group of multi-channel audio, x i_j represents the true value of the arrival direction of each frame of each multi-channel audio group marked in step 1, specifically the true value of the arrival direction of the i-th frame of the j-th multi-channel audio group, x′ i_j The predicted value of the arrival direction of each frame of each multi-channel audio group in step 3, specifically the predicted value of the arrival direction of the i-th frame of the j-th multi-channel audio group;

[0048] The technical solution of the system of the present invention is a multi-sound source detection and positioning system based on a convolutional recurrent neural network, comprising:

[0049] Sample label construction module, position coding sequence generation module, multi-sound source detection and positioning network optimization module, real-time multi-channel audio prediction module;

[0050] The sample label construction module is used to input multiple groups of multi-channel audio, perform discrete Fourier transform on each group of multi-channel audio, obtain the spectrum of each group of multi-channel audio, mark the true value of the event type and the true value of the arrival direction of each frame of audio in each group of multi-channel audio, extract audio feature information from each group of multi-channel audio, obtain the amplitude feature sequence and phase feature sequence of each group of multi-channel audio, and concatenate the feature amplitude and phase features of each group of multi-channel audio by end-to-end splicing to obtain the feature sequence of each group of multi-channel audio;

[0051] The position coding sequence generation module is used to calculate the relative position of each frame of each group of multi-channel audio in each group of multi-channel audio through sinusoidal and cosine alternating position coding to construct a position coding sequence for each group of multi-channel audio;

[0052] The multi-sound source detection and positioning network optimization module is used to construct a multi-sound source detection and positioning network, input each group of multi-channel audio and each frame of audio sound, predict the event type and arrival direction through the multi-sound source detection and positioning network, obtain the predicted value of the event type of each group of multi-channel audio and multi-frame audio to construct a sound event type prediction result sequence for each group of multi-channel audio, and the predicted value of the arrival direction of each group of multi-channel audio and multi-frame audio to construct an arrival direction prediction result sequence for each group of multi-channel audio, combine the sound event type true result sequence of each group of multi-channel audio and the sound event type prediction result sequence to construct a binary cross loss, combine the arrival direction true result sequence of each group of multi-channel audio and the arrival direction prediction result sequence to construct a mean square error loss, weightedly construct a weighted loss function model, and optimize and train the optimized multi-sound source detection and positioning network through the adaptive moment estimation algorithm;

[0053] The real-time multi-channel audio prediction module is used to collect real-time multi-channel audio and input it into the optimized multi-sound source detection and localization network to predict event type and arrival direction, thereby obtaining the predicted event type and predicted arrival direction of the real-time multi-channel audio.

[0054] Compared with the existing multi-sound source localization algorithm, the advantages of the present invention are: this method uses two sets of parallel network branches to simultaneously perform sound event detection tasks and arrival angle estimation tasks, which is a good combination of the existing sound source localization algorithm and the active sound source detection method. The detection and classification of active sound sources also plays an auxiliary role in the positioning task; this method uses a layer of global average pooling instead of a fully connected layer to reduce the number of network parameters and prevent overfitting; this method introduces position encoding and makes good use of the context information between audio frames; when performing the arrival angle estimation task, this method can estimate the three-dimensional spatial orientation, while most algorithms that perform two tasks simultaneously can only perform in the horizontal orientation; compared with previous methods, the number and categories of overlapping sound sources detected by this method have been increased from one to two sound sources to three sound sources, and the categories have been increased from multiple different categories to a mixture of the same type and different types. In addition, this method actually has the potential to detect more sound sources and has good versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 : A flowchart of the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0058] The following is combined with Figure 1 The specific embodiment of the present invention is a multi-sound source detection and localization method and system based on a convolutional recurrent neural network, as follows:

[0059] Reference Figure 1 The present invention proposes a multi-sound source detection and localization method based on a convolutional recurrent neural network, comprising the following steps:

[0060] The input is multi-source audio signals. The training and testing data are from the Tampere University of Technology's public dataset, the TUT Sound Events 2018-Ambisonics, Anechoic and Synthetic Impulse Response dataset (ANSIM). Each audio segment in this dataset is 30 seconds long, sampled at 44.1 kHz, and contains four channels.

[0061] It should be noted that the audio parameters and object types specified herein are merely examples for illustrating the implementation process of the present invention and are not intended to limit the present invention.

[0062] Step 1: Input multiple groups of multi-channel audio, perform discrete Fourier transform on each group of multi-channel audio, obtain the spectrum of each group of multi-channel audio, mark the true value of the event type and the true value of the arrival direction of each frame of audio in each group of multi-channel audio, extract audio feature information from each group of multi-channel audio, obtain the amplitude feature sequence and phase feature sequence of each group of multi-channel audio, and concatenate the feature amplitude and phase features of each group of multi-channel audio by end-to-end splicing to obtain the feature sequence of each group of multi-channel audio;

[0063] Each group of multi-channel audio in step 1 is composed of T frames of audio sound;

[0064] In the embodiment of the present invention, the total number of frames T constructed by one record is 5166 frames;

[0065] In the embodiment of the present invention, the frame length is specified to be 22ms, the number of Fourier transform points M is 512, and the window function uses a Hamming window with a 50% frame offset, as shown in the following formula:

[0066]

[0067] That is, a 512-point spectrum is obtained for each channel and each audio frame.

[0068] It should be noted that the frame length, window function type, and transformation method specified herein are merely examples for illustrating the specific implementation steps of the present invention and are not intended to limit the present invention.

[0069] In the embodiment of the present invention, the sequence size of each set of multi-channel audio feature sequences is C×T×2×M / 2.

[0070] In the embodiment of the present invention, C=4, T=5166, and M=512.

[0071] Each set of multi-channel audio in step 1 is used for training;

[0072] The actual value of the event type of each frame of audio sound in each group of multi-channel audio is marked in step 1, as follows:

[0073] Construct a true result sequence of the sound event type of each group of multi-channel audio by using the true value of the event type of each group of multi-channel audio and multi-frame audio sound;

[0074] The true value of the arrival direction of each frame of each multi-channel audio group marked in step 1 is as follows:

[0075] Constructing a true result sequence of the direction of arrival of each group of multi-channel audio by using the true value of the direction of arrival of each group of multi-channel audio multi-frame audio sound;

[0076] Step 2: Calculate the relative position of each frame of audio in each group of multi-channel audio by alternating sine and cosine position coding to construct a position coding sequence for each group of multi-channel audio;

[0077] In this embodiment, the encoder is in the following form:

[0078] PE(pos,2i)=sin(pos / 10000 2i / d ),PE(pos,2i+1)=cos(pos / 10000 2i+1 / d )

[0079] Wherein, PE represents the position code of the pos position, pos represents the position of the audio frame in the entire audio sequence, 2i represents an even frame, 2i+1 represents an odd frame, and d represents the frame length of the audio sequence. Generally speaking, d=T, so in this embodiment, d=5166.

[0080] Step 3: Construct a multi-sound source detection and localization network, input each frame of audio for each group of multi-channel audio, predict the event type and arrival direction through the multi-sound source detection and localization network, obtain the predicted value of the event type of each group of multi-channel audio and multi-frame audio, construct a sound event type prediction result sequence for each group of multi-channel audio, and the predicted value of the arrival direction of each group of multi-channel audio and multi-frame audio, construct a sequence of arrival direction prediction results for each group of multi-channel audio, combine the true result sequence of the sound event type of each group of multi-channel audio and the sound event type prediction result sequence to construct a binary cross loss, combine the true result sequence of the arrival direction of each group of multi-channel audio and the arrival direction prediction result sequence to construct a mean square error loss, weightedly construct a weighted loss function model, and optimize and train the optimized multi-sound source detection and localization network through the adaptive moment estimation algorithm;

[0081] The multi-sound source detection and localization network described in step 3 is composed of an improved multi-layer 2D convolutional layer, a gated recurrent unit with optimized attention mechanism, a global average pooling layer, and a fully connected layer cascaded in sequence;

[0082] The improved multi-layer 2D convolutional layer is used to input the feature sequence of each group of multi-channel audio,

[0083] The improved multi-layer 2D convolution layer is composed of a first improved convolution module, a second improved convolution module, ..., an Mth improved convolution module cascaded in sequence;

[0084] The first improved convolution module, the second improved convolution module, ..., and the M-1 improved convolution module have the same structure;

[0085] The i-th improved convolution module is composed of a convolution layer, a first atrous spatial pooling layer, a second atrous spatial pooling layer, a third atrous spatial pooling layer, a fusion module, and a spatial transformer network cascade;

[0086] The convolutional layer is connected to the first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer in sequence respectively;

[0087] The first atrous spatial pooling layer is connected to the fusion module;

[0088] The second atrous spatial pooling layer is connected to the fusion module;

[0089] The third atrous spatial pooling layer is connected to the fusion module; the fusion module is connected to the spatial transformer network, i∈[1,M-1];

[0090] The i-th improved convolution module receives as input the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtains the convolution feature sequence of each group of multi-channel audio of the i-th improved convolution module through improved convolution processing, and outputs it to the i+1-th improved convolution module;

[0091] If i=1, the input of the first improved convolution module is the feature sequence of each group of multi-channel audio;

[0092] The convolution layer of the i-th improved convolution module is used to input the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtain the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module through convolution processing, and output it to the first void space pooling layer, the second void space pooling layer, and the third void space pooling layer of the i-th improved convolution module respectively;

[0093] In this embodiment, the dilated spatial pooling uses a convolution kernel of 3×3 and a dilation ratio of 1 to 3.

[0094] The first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer of the i-th improved convolution module respectively use convolution kernels with the same size and atrous rates from low to high;

[0095] The first atrous spatial pooling layer, the second atrous spatial pooling layer, and the third atrous spatial pooling layer of the i-th improved convolution module respectively sample the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module in sequence to obtain the first scale feature map, the second scale feature map, and the third scale feature map of each group of multi-channel audio of the i-th improved convolution module, and output them to the fusion module of the i-th improved convolution module;

[0096] The fusion module of the i-th improved convolution module concatenates the first scale feature map, the second scale feature map, and the third scale feature of each group of multi-channel audio of the i-th improved convolution module end to end to obtain a fused scale feature map of each group of multi-channel audio of the i-th improved convolution module, and outputs the fused scale feature map to the spatial transformer network of the i-th improved convolution module;

[0097] The spatial transformer network of the i-th improved convolution module performs a translation and rotation transformation on the fused scale feature map of each group of multi-channel audio of the i-th improved convolution module, amplifies and translates the key features to the center of the feature map, and outputs a key feature focused feature map of each group of multi-channel audio of the i-th improved convolution module;

[0098] The M-th improved convolution module is composed of a cascade of convolutional layers and maximum pooling layers;

[0099] The Mth improved convolution module inputs the convolution feature sequence of each group of multi-channel audio of the M-1th improved convolution module, sequentially performs convolution processing through the convolution layer of the Mth improved convolution module and performs maximum redundancy processing through the maximum pooling layer to obtain a multi-layer convolution feature sequence of each group of multi-channel audio, and outputs it to the gated recurrent unit optimized by the attention mechanism;

[0100] The attention mechanism optimized gated recurrent unit inputs the multi-layer convolution feature sequence of each group of multi-channel audio, obtains the attention mechanism feature sequence of each group of multi-channel audio, and outputs it to the global average pooling layer;

[0101] The global average pooling layer is used to perform global average pooling on the attention mechanism feature sequence of each group of multi-channel audio, concatenate it with the position encoding sequence of each group of multi-channel audio, obtain a global feature sequence of each group of multi-channel audio, and output it to the fully connected layer;

[0102] The fully connected layer is composed of a sound event type fully connected layer and an arrival direction fully connected layer in parallel;

[0103] The sound event type fully connected layer is connected to the global average pooling layer, and the arrival direction fully connected layer is connected to the global average pooling layer;

[0104] The sound event type fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the event type of each frame of audio sound in each group of multi-channel audio; and construct the sound event type prediction result sequence of each group of multi-channel audio through the predicted value of the event type of each group of multi-channel audio and multi-frame audio;

[0105] The arrival direction fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the arrival direction of each frame of audio sound in each group of multi-channel audio; and construct the arrival direction prediction result sequence of each group of multi-channel audio through the predicted value of the arrival direction of each group of multi-channel audio and multiple frames of audio;

[0106] In the embodiment of the present invention, the predicted value of the event type of each audio frame per channel per group is output as the activity value of N sound source types. Among the N categories, the predicted active sound source sound event type is output as 1, and the inactive sound source sound event type is output as 0.

[0107] In the embodiment of the present invention, N=11;

[0108] In an embodiment of the present invention, the predicted value of the arrival direction of each group, each channel, and each frame of audio is output as the predicted value of the arrival direction of N sound source types, specifically the three-dimensional Cartesian coordinates x, y, and z predicted for N sound event types.

[0109] In the embodiment of the present invention, the range of x is [-1, 1], the range of y is [-1, 1], and the range of z is [-1, 1].

[0110] The weighted loss function model described in step 3 is specifically defined as follows:

[0111] Loss = αBCELoss + βMSE

[0112] Among them, BCELoss represents the binary cross entropy loss function, MSE represents the mean square error loss function, α represents the weight of the binary cross entropy loss function, and β represents the weight of the mean square error loss function;

[0113] In the embodiment of the present invention, α=0.5, β=0.5;

[0114] The binary cross entropy loss function is defined as follows:

[0115]

[0116] Where N represents the total number of multi-channel audio groups, n j Indicates the total number of frames of the audio sound of the j-th group of multi-channel audio, y i_j represents the true value of the event type of each frame of each group of multi-channel audio marked in step 1, specifically the binary true value 0 or 1 of the sound event type of the i-th frame of the j-th group of multi-channel audio, p(y i_j =1) represents the sequence of sound event type prediction results for each group of multi-channel audio in step 3, specifically the predicted value of the sound event type of the i-th frame of the j-th group of multi-channel audio, that is, the probability that the model predicts that the sound event type of the i-th frame is 1;

[0117] N should be determined based on the appropriate audio dataset size required for training.

[0118] In the embodiment of the present invention, N=6000;

[0119] The mean square error loss function is as follows:

[0120]

[0121] Among them, N represents the total number of multi-channel audio groups, n j Indicates the total number of frames of the j-th group of multi-channel audio, x i_j represents the true value of the arrival direction of each frame of each multi-channel audio group marked in step 1, specifically the true value of the arrival direction of the i-th frame of the j-th multi-channel audio group, x′ i_j The predicted value of the arrival direction of each frame of each multi-channel audio group in step 3, specifically the predicted value of the arrival direction of the i-th frame of the j-th multi-channel audio group;

[0122] Step 4: Collect real-time multi-channel audio and input it into the optimized multi-sound source detection and localization network to predict the event type and arrival direction, and obtain the predicted event type and predicted arrival direction of the real-time multi-channel audio;

[0123] The present invention uses a network model to simultaneously learn the type and spatial orientation of sound sources, so that the algorithm has the ability to distinguish multiple simultaneously active sound sources and estimate their spatial orientations separately; while performing the sound event detection task, the present invention also performs the task of estimating the spatial orientation in a three-dimensional Cartesian coordinate system, which is the most tasks performed among algorithms of the same type; the present invention can predict up to three simultaneously occurring sound sources, which is the largest number of predicted sound sources among algorithms of the same type; the present invention has good robustness to reverberation and low signal-to-noise ratio environmental conditions; and the present invention has the potential to expand the prediction of more sound sources.

[0124] Experimental results analysis

[0125]

[0126] This method conducts comparative experiments with existing algorithms for the sound event type prediction task and the arrival direction estimation task. The experimental results are shown in the table above. The dataset ANSYN represents an audio dataset collected in a reverberation-free environment, and RESYN Room1, 2, and 3 represent audio datasets collected in three different reverberation environments. F represents the proportion of correctly predicted frames in each group of multi-channel audio in the total number of frames in each group of multi-channel audio in the sound event type prediction task, where the correctly predicted frames include: the sound type in the frame audio does not appear and is predicted not to appear, and the sound type in the frame audio appears and is predicted to appear; the arrival direction estimation error represents the average value of the angular difference between the predicted sound source direction and the actual sound source direction in each frame of each group of multi-channel audio in the arrival direction estimation task; the frame recall rate represents the ratio of the number of frames with a predicted arrival direction estimation error less than N (degrees) in each frame of each group of multi-channel audio in the arrival direction estimation task to the total number of frames in each group of multi-channel audio;

[0127] In this embodiment, N=5;

[0128] Clearly, a higher F-value indicates a better sound event type prediction performance; smaller DOA estimation errors and higher frame recall rates indicate better DOA estimation performance. This method performs well in both F-value and frame recall, particularly in highly reverberant and multi-source environments, demonstrating its effectiveness and superiority.

[0129] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0130] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A method for detecting and localizing multiple sound sources using a convolutional recurrent neural network, characterized in that: The following steps are involved: Step 1: Input multiple groups of multi-channel audio, perform discrete Fourier transform on each group of multi-channel audio, obtain the spectrum of each group of multi-channel audio, mark the true value of the event type and the true value of the arrival direction of each frame of audio in each group of multi-channel audio, extract audio feature information from each group of multi-channel audio, obtain the amplitude feature sequence and phase feature sequence of each group of multi-channel audio, and concatenate the feature amplitude and phase features of each group of multi-channel audio by end-to-end splicing to obtain the feature sequence of each group of multi-channel audio; Step 2: Calculate the relative position of each frame of audio in each group of multi-channel audio by alternating sine and cosine position coding to construct a position coding sequence for each group of multi-channel audio; Step 3: Construct a multi-sound source detection and localization network, input each frame of audio for each group of multi-channel audio, predict the event type and arrival direction through the multi-sound source detection and localization network, obtain the predicted value of the event type of each group of multi-channel audio and multi-frame audio, construct a sound event type prediction result sequence for each group of multi-channel audio, and the predicted value of the arrival direction of each group of multi-channel audio and multi-frame audio, construct a sequence of arrival direction prediction results for each group of multi-channel audio, combine the true result sequence of the sound event type of each group of multi-channel audio and the sound event type prediction result sequence to construct a binary cross loss, combine the true result sequence of the arrival direction of each group of multi-channel audio and the arrival direction prediction result sequence to construct a mean square error loss, weightedly construct a weighted loss function model, and optimize and train the optimized multi-sound source detection and localization network through the adaptive moment estimation algorithm; Step 4: Collect real-time multi-channel audio and input it into the optimized multi-source detection and localization network to predict the event type and arrival direction, and obtain the predicted event type and predicted arrival direction of the real-time multi-channel audio.

2. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 1, wherein: Each group of multi-channel audio in step 1 is composed of T frames of audio sound; Each set of multi-channel audio in step 1 is used for training; The actual value of the event type of each frame of audio sound in each group of multi-channel audio is marked in step 1, as follows: Construct a true result sequence of the sound event type of each group of multi-channel audio by using the true value of the event type of each group of multi-channel audio and multi-frame audio sound; The true value of the arrival direction of each frame of each multi-channel audio group marked in step 1 is as follows: The true result sequence of the arrival direction of each group of multi-channel audio is constructed by using the true value of the arrival direction of each group of multi-channel audio multi-frame audio sound.

3. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 2, wherein: The multi-sound source detection and localization network described in step 3 is composed of an improved multi-layer 2D convolutional layer, a gated recurrent unit with optimized attention mechanism, a global average pooling layer, and a fully connected layer cascaded in sequence; The improved multi-layer 2D convolutional layer is used to input the feature sequence of each group of multi-channel audio; The attention mechanism optimized gated recurrent unit inputs the multi-layer convolution feature sequence of each group of multi-channel audio, obtains the attention mechanism feature sequence of each group of multi-channel audio, and outputs it to the global average pooling layer; The global average pooling layer is used to perform global average pooling on the attention mechanism feature sequence of each group of multi-channel audio, concatenate it with the position encoding sequence of each group of multi-channel audio, obtain a global feature sequence of each group of multi-channel audio, and output it to the fully connected layer; The fully connected layer is composed of a sound event type fully connected layer and an arrival direction fully connected layer in parallel.

4. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 3, wherein: The improved multi-layer 2D convolution layer is composed of a first improved convolution module, a second improved convolution module, ..., an Mth improved convolution module cascaded in sequence; The first improved convolution module, the second improved convolution module, ..., and the M-1 improved convolution module have the same structure; The i-th improved convolution module is composed of a convolution layer, a first atrous spatial pooling layer, a second atrous spatial pooling layer, a third atrous spatial pooling layer, a fusion module, and a spatial transformer network cascade; The convolutional layer is connected to the first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer in sequence respectively; The first atrous spatial pooling layer is connected to the fusion module; The second atrous spatial pooling layer is connected to the fusion module; The third atrous spatial pooling layer is connected to the fusion module; the fusion module is connected to the spatial transformer network, i∈[1,M-1]; The i-th improved convolution module receives as input the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtains the convolution feature sequence of each group of multi-channel audio of the i-th improved convolution module through improved convolution processing, and outputs it to the i+1-th improved convolution module; If i=1, the input of the first improved convolution module is the feature sequence of each group of multi-channel audio; The M-th improved convolution module is composed of a cascade of convolutional layers and maximum pooling layers; The M-th improved convolution module inputs the convolution feature sequence of each group of multi-channel audio of the M-1-th improved convolution module, and sequentially performs convolution processing through the convolution layer of the M-th improved convolution module and maximum pooling processing through the maximum pooling layer to obtain a multi-layer convolution feature sequence of each group of multi-channel audio, and outputs it to the gated recurrent unit optimized by the attention mechanism.

5. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 4, wherein: The convolution layer of the i-th improved convolution module is used to input the convolution feature sequence of each group of multi-channel audio of the i-1-th improved convolution module, obtain the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module through convolution processing, and output it to the first void space pooling layer, the second void space pooling layer, and the third void space pooling layer of the i-th improved convolution module respectively; The first atrous space pooling layer, the second atrous space pooling layer, and the third atrous space pooling layer of the i-th improved convolution module respectively use convolution kernels with the same size and atrous rates from low to high; The first atrous spatial pooling layer, the second atrous spatial pooling layer, and the third atrous spatial pooling layer of the i-th improved convolution module respectively sample the convolution processing feature map of each group of multi-channel audio of the i-th improved convolution module in sequence to obtain the first scale feature map, the second scale feature map, and the third scale feature map of each group of multi-channel audio of the i-th improved convolution module, and output them to the fusion module of the i-th improved convolution module.

6. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 5, wherein: The fusion module of the i-th improved convolution module concatenates the first scale feature map, the second scale feature map, and the third scale feature of each group of multi-channel audio of the i-th improved convolution module end to end to obtain a fused scale feature map of each group of multi-channel audio of the i-th improved convolution module, and outputs the fused scale feature map to the spatial transformer network of the i-th improved convolution module; The spatial transformer network of the i-th improved convolution module performs a translation and rotation transformation on the fusion scale feature map of each group of multi-channel audio of the i-th improved convolution module, amplifies and translates the key features to the center of the feature map, and outputs a key feature focused feature map of each group of multi-channel audio of the i-th improved convolution module.

7. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 6, wherein: The sound event type fully connected layer is connected to the global average pooling layer, and the arrival direction fully connected layer is connected to the global average pooling layer; The sound event type fully connected layer is used to input the global feature sequence of each group of multi-channel audio to obtain the predicted value of the event type of each frame of audio sound in each group of multi-channel audio; and construct the sound event type prediction result sequence of each group of multi-channel audio through the predicted value of the event type of each group of multi-channel audio and multiple frames of audio.

8. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 7, wherein: The arrival direction fully connected layer is used to input the global feature sequence of each group of multi-channel audio, obtain the predicted value of the arrival direction of each frame of audio sound in each group of multi-channel audio; and construct the arrival direction prediction result sequence of each group of multi-channel audio through the predicted value of the arrival direction of each group of multi-channel audio and multiple frames of audio.

9. The method for multiple sound source detection and localization using a convolutional recurrent neural network according to claim 8, wherein: The weighted loss function model described in step 3 is specifically defined as follows: Loss = αBCELoss + βMSE Among them, BCELoss represents the binary cross entropy loss function, MSE represents the mean square error loss function, α represents the weight of the binary cross entropy loss function, and β represents the weight of the mean square error loss function; The binary cross entropy loss function is defined as follows: Where N represents the total number of multi-channel audio groups, n j Indicates the total number of frames of the audio sound of the j-th group of multi-channel audio, y i_j represents the true value of the event type of each frame of each group of multi-channel audio marked in step 1, specifically the binary true value 0 or 1 of the sound event type of the i-th frame of the j-th group of multi-channel audio, p(y i_j =1) represents the sequence of sound event type prediction results for each group of multi-channel audio in step 3, specifically the predicted value of the sound event type of the i-th frame of the j-th group of multi-channel audio, that is, the probability that the model predicts that the sound event type of the i-th frame is 1; The mean square error loss function is as follows: Among them, N represents the total number of multi-channel audio groups, n j Indicates the total number of frames of the j-th group of multi-channel audio, x i_j represents the true value of the arrival direction of each frame of each multi-channel audio group marked in step 1, specifically the true value of the arrival direction of the i-th frame of the j-th multi-channel audio group, x′ i_j It represents the predicted value of the arrival direction of each frame of each group of multi-channel audio in step 3, specifically the predicted value of the arrival direction of the i-th frame of the j-th group of multi-channel audio.

10. A convolutional recurrent neural network multi-sound source detection and positioning system, characterized in that: include: Sample label construction module, position coding sequence generation module, multi-sound source detection and positioning network optimization module, real-time multi-channel audio prediction module; The sample label construction module is used to input multiple groups of multi-channel audio, perform discrete Fourier transform on each group of multi-channel audio, obtain the spectrum of each group of multi-channel audio, mark the true value of the event type and the true value of the arrival direction of each frame of audio in each group of multi-channel audio, extract audio feature information from each group of multi-channel audio, obtain the amplitude feature sequence and phase feature sequence of each group of multi-channel audio, and concatenate the feature amplitude and phase features of each group of multi-channel audio by end-to-end splicing to obtain the feature sequence of each group of multi-channel audio; The position coding sequence generation module is used to calculate the relative position of each frame of each group of multi-channel audio in each group of multi-channel audio through sinusoidal and cosine alternating position coding to construct a position coding sequence for each group of multi-channel audio; The multi-sound source detection and positioning network optimization module is used to construct a multi-sound source detection and positioning network, input each group of multi-channel audio and each frame of audio sound, predict the event type and arrival direction through the multi-sound source detection and positioning network, obtain the predicted value of the event type of each group of multi-channel audio and multi-frame audio to construct a sound event type prediction result sequence for each group of multi-channel audio, and the predicted value of the arrival direction of each group of multi-channel audio and multi-frame audio to construct an arrival direction prediction result sequence for each group of multi-channel audio, combine the sound event type true result sequence of each group of multi-channel audio and the sound event type prediction result sequence to construct a binary cross loss, combine the arrival direction true result sequence of each group of multi-channel audio and the arrival direction prediction result sequence to construct a mean square error loss, weightedly construct a weighted loss function model, and optimize and train the optimized multi-sound source detection and positioning network through the adaptive moment estimation algorithm; The real-time multi-channel audio prediction module is used to collect real-time multi-channel audio and input it into the optimized multi-sound source detection and localization network to predict event type and arrival direction, thereby obtaining the predicted event type and predicted arrival direction of the real-time multi-channel audio.

Images