Sound source localization method based on frequency domain dynamic convolution and conformer

Abstract

The application provides a sound source positioning method based on a frequency domain dynamic convolution and a Conformer. First, the multi-channel audio captured by a microphone array is subjected to logarithmic amplitude spectrum and phase spectrum calculation and serial fusion. The fused features are input into a CNN-Conformer network, the network outputs a direct path phase difference, then the spatial response power spectrum is calculated through the direct path phase difference output by the network, and finally the spatial response power spectrum is subjected to spectral peak searching to obtain the final sound source position. The model combines a frequency domain dynamic CNN and a Conformer module, can release the translational invariance of convolution in the frequency dimension, alleviate the positioning error caused by different phase differences corresponding to different frequencies in the time-frequency spectrum, and effectively capture the time-frequency mode and long-term dependence in the time-frequency spectrum. Compared with the existing CRNN model, the method has stronger spatial feature capturing capability and can effectively improve the sound source positioning precision.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and sound source positioning, and particularly relates to a sound source positioning method based on frequency domain dynamic convolution and Conformer. BACKGROUND

[0002] Sound source positioning is a technology for estimating the position or direction of arrival of a sound source from signals captured by audio sensors. Sound source positioning mainly focuses on the spatial feature information of sound, and obtains the position of the target sound source according to the inter-channel and intra-channel clues of the microphone array. At present, sound source positioning has been widely applied to indoor meetings, intelligent robots, and hearing aids. In the actual environment, there will be background noise, such as the sound similar to Gaussian white noise emitted by fans, air conditioners, etc., and the sound similar to impulse noise emitted by closing and knocking doors, etc., and reverberation will occur during the propagation of sound. These noises and reverberations will cause changes in the phase and amplitude of the speech signals received by the microphone array. Compared with traditional methods, the nonlinear mapping relationship model trained by the deep learning method is more robust and adaptive in the noise and reverberation environment. And the method with time-frequency spectrum as the feature has higher positioning accuracy, which often uses two-dimensional convolution to convolve in the time and frequency axes. Since at any time point, any source at the same position can obtain the same DOA (Direction of Arrival) result, the translational invariance of convolution in the time axis is conducive to DOA estimation. However, since the phase difference of the same source position varies with frequency, the DOA result will be affected by the frequency distribution, so the translational invariance of convolution in the frequency axis is not conducive to DOA estimation.

[0003] In order to improve the accuracy of sound source positioning, the network structure needs to be changed in combination with the characteristics of the time-frequency spectrum. Compared with traditional sound source positioning features, the time-frequency spectrum belongs to signal-level features and contains more original information, so the network structure needs to extract positioning-related features and eliminate useless features. Moreover, since the time-frequency spectrum contains the sparsity characteristics of the time-frequency unit, it is not easy to contain the information of multiple sound sources in a time-frequency unit, and the multi-source problem can be simplified as a single-source positioning problem in a single time-frequency unit. Therefore, how to better extract positioning-related features and fully utilize the characteristics of the time-frequency pattern has become the key to effectively improving the performance of sound source positioning. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a sound source positioning method based on frequency domain dynamic convolution and Conformer, which can release the translational invariance of convolution in the frequency dimension, improve the physical consistency of the model and the time-frequency pattern of the sound source position, and train the network to focus on the short-term and long-term spatio-temporal context information of the time-frequency spectrum, thereby obtaining better sound source positioning effect.

[0005] First, the log amplitude spectrum and phase spectrum of the multi-channel audio captured by the microphone array are calculated and fused in series. The fused features are input into the CNN-Conformer network, and the network outputs the direct path phase difference. Then, the spatial response power spectrum is calculated based on the direct path phase difference output by the network, and finally the spatial response power spectrum is searched for spectral peaks to obtain the final sound source position. The model of the application combines the frequency domain dynamic CNN and the Conformer module, which can release the translational invariance of convolution in the frequency dimension, alleviate the positioning error caused by different phase differences corresponding to different frequencies in the time-frequency spectrum, and effectively capture the time-frequency patterns and long-term dependencies in the time-frequency spectrum. Compared with the existing CRNN model, the method has stronger spatial feature capturing ability and can effectively improve the sound source positioning accuracy.

[0006] The technical solutions adopted by the application to solve the technical problems are:

[0007] A sound source positioning method based on frequency domain dynamic convolution and Conformer, which first calculates the log amplitude spectrum and phase spectrum of the multi-channel audio captured by the microphone array and fuses them in series. The fused features are input into the CNN-Conformer network, and the network outputs the direct path phase difference. Then, the spatial response power spectrum is calculated, and finally the spatial response power spectrum is searched for spectral peaks to obtain the final sound source position.

[0008] Further, the CNN-Conformer model combines a CNN module and a Conformer module; wherein the CNN module is used to extract short-term spatio-temporal features; and the Conformer module is used to extract long-term spatio-temporal features.

[0009] Further, the CNN module is composed of 3 causal convolution blocks and 2 frequency domain dynamic convolution blocks, and each convolution block is composed of 2 convolution layers. The output of each convolution module is compressed using max pooling in the frequency and time dimensions, which preserves the spatio-temporal context information while downsampling. Then it is input into the Conformer module. Finally, a fully connected layer is input and activated using a tanh activation function to obtain the direct path phase difference.

[0010] Further, the frequency domain dynamic convolution uses a frequency adaptive kernel to enhance the frequency dependence of two-dimensional convolution. First, average pooling is performed on the input in the time dimension, and then two one-dimensional convolution layers are applied along the channel axis to extract frequency adaptive attention weights from the input.

[0011] Further, the direct path phase difference output by the CNN-Conformer model is used to calculate the power response spectrum, and the specific calculation formula is as follows:

[0012]

[0013] wherein, θ is the candidate angle, n' is the current time frame index, M is the number of microphones, F is the number of frequency points, m, m' are the indices of a pair of microphones, is the direct path phase difference, r mm' (θ) is the vector representation of all frequencies.

[0014] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the sound source positioning method based on frequency domain dynamic convolution and Conformer as described above when executing the program.

[0015] A non-transitory computer readable storage medium having stored thereon a computer program, the computer program implementing the steps of the sound source positioning method based on frequency domain dynamic convolution and Conformer as described above when executed by a processor.

[0016] Compared with the prior art, the present application and its preferred solutions can release the translation invariance of convolution in the frequency domain dimension, improve the physical consistency of the model and the time-frequency pattern of the sound source position, enable the network to pay more attention to short-term and long-term spatiotemporal features, and ultimately obtain better sound source positioning effect. BRIEF DESCRIPTION OF DRAWINGS

[0017] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments:

[0018] Figure 1 is a schematic diagram of the implementation process of the method of the embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the features and advantages of the patent more obvious and easy to understand, the following examples are specifically described as follows:

[0020] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used in the specification have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0021] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or combination thereof.

[0022] As Figure 1As shown, the embodiment provides a sound source positioning method based on frequency domain dynamic convolution and Conformer, comprising the following steps:

[0023] Step S1: First, calculate the log amplitude spectrum and phase spectrum of the multi-channel audio captured by the microphone array, and perform serial fusion.

[0024] Step S2: Input the fused features into the CNN-Conformer network, and the network outputs the direct path phase difference.

[0025] Step S3: Calculate the spatial response power spectrum through the direct path phase difference output by the network in step 2.

[0026] Step S4: Perform spectral peak search on the spatial response power spectrum to obtain the final sound source position.

[0027] In this embodiment, the CNN-Conformer model efficiently fuses the CNN module and the Conformer module; the CNN module is used to extract short-term spatiotemporal features, and the Conformer module is used to extract long-term spatiotemporal features.

[0028] In this embodiment, the CNN module is composed of 3 causal convolution blocks and 2 frequency domain dynamic convolution blocks, and each convolution block is composed of 2 convolution layers; the maximum pooling is used to compress the output of each convolution module in the frequency and time dimensions, which retains the spatiotemporal context information while rapidly downsampling; then input into the Conformer module; finally input into the full connection layer, and use the tanh activation function to activate, to obtain the direct path phase difference.

[0029] In this embodiment, the frequency domain dynamic convolution uses a frequency adaptive kernel to enhance the frequency dependence of two-dimensional convolution, first performs average pooling on the input in the time dimension, then applies two one-dimensional convolution layers along the channel axis to extract frequency adaptive attention weights from the input.

[0030] In this embodiment, the direct path phase difference output by the CNN-Conformer model is used to calculate the power response spectrum, and the specific calculation formula is as follows:

[0031]

[0032] Where, θ is the candidate angle, n' is the current time frame number, M is the number of microphones, F is the number of frequency points, m and m' are the serial numbers of a pair of microphone pairs, is the direct path phase difference, r mm' (θ) is the vector representation of all frequencies.

[0033] In summary, in view of the fact that the existing method has limited attention to the space-time features in the time-frequency spectrogram and cannot release the translational equivariance of convolution in the time-frequency spectrogram, resulting in limited accuracy of sound source positioning, a sound source positioning method based on frequency domain dynamic convolution and Conformer is proposed. The network includes a CNN module and a Conformer module. The frequency domain dynamic convolution block in the CNN module generates a frequency dynamic convolution kernel by designing two one-dimensional convolutions to extract short-term space-time features and eliminate the translational equivariance of the time-frequency spectrogram in the frequency domain axis, thereby improving the physical consistency of the model with the time-frequency pattern of the sound source position. The Confomrer module is used to extract long-term space-time features. Compared with existing methods, the sound source positioning performance is better.

[0034] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application in other forms. Any person skilled in the art can modify or change the above-mentioned technical content to obtain equivalent embodiments. However, any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments shall still fall within the protection scope of the present application.

[0035] The present patent is not limited to the above-mentioned best embodiment, and anyone can derive other various forms of sound source positioning method based on frequency domain dynamic convolution and Conformer under the inspiration of the present patent. Any equivalent change and modification made according to the scope of the present patent application shall fall within the scope of the present patent.

Claims

1. A sound source localization method based on frequency domain dynamic convolution and Conformer, characterized by: First, the logarithmic amplitude spectrum and phase spectrum of the multi-channel audio captured by the microphone array are calculated and fused in series. The fused features are input into the CNN-Conformer network, which outputs the direct path phase difference. The spatial response power spectrum is then calculated, and finally a spectral peak search is performed on the spatial response power spectrum to obtain the final sound source location. The CNN-Conformer network integrates the CNN module and the Conformer module; wherein the CNN module is used to extract short-term spatiotemporal features; and the Conformer module is used to extract long-term spatiotemporal features. The CNN module consists of three causal convolution blocks and two frequency domain dynamic convolution blocks, each of which consists of two convolution layers. The output of each convolution module is compressed using maximum pooling in the frequency and time dimensions, preserving spatiotemporal context information while downsampling. The output is then fed into the Conformer module. Finally, the output is fed into a fully connected layer and activated using the tanh activation function to obtain the direct path phase difference. The direct path phase difference output by the CNN-Conformer network is used to calculate the power response spectrum. The specific calculation formula is as follows: in, is the candidate angle, is the current time frame number, is the number of microphones, is the number of frequency points, m, is the serial number of a pair of microphones, is the direct path phase difference, is the vector representation of all frequencies.

2. The sound source localization method based on frequency domain dynamic convolution and Conformer according to claim 1, characterized in that: The frequency-domain dynamic convolution adopts a frequency-adaptive kernel to enhance the frequency dependence of two-dimensional convolution. It first average-pools the input in the time dimension, and then applies two one-dimensional convolution layers along the channel axis to extract frequency-adaptive attention weights from the input.

3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the sound source localization method based on frequency domain dynamic convolution and Conformer are implemented as claimed in claim 1 or 2.

4. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the sound source localization method based on frequency domain dynamic convolution and Conformer are implemented as claimed in claim 1 or 2.

Images