Sound source localization method based on self-attention mechanism

Abstract

The invention discloses a sound source localization method based on a self-attention mechanism, and the method comprises the steps: dividing an input multi-channel audio signal into non-overlapping time blocks, connecting an amplitude spectrogram with a phase spectrogram for each channel to form a feature tensor, and obtaining a feature map; inputting the feature map into a convolutional neural network (CNN) for feature extraction, performing position coding on a first feature, and mapping a second sequence feature to a query key and value matrix through linear change; calculating the dot product between the query and the key, inputting the standard deep feature into an FFN feedforward neural network, and carrying out two times of linear changes and one time of ReLU activation to obtain a target deep feature; and the deep features of the target enter and exit a full connection layer, and a DOA trajectory prediction value of the sound source position in a three-dimensional space is obtained. The global context information in the audio signal can be more effectively captured and modeled, so that the accuracy of sound source positioning is improved.

Description

Technical Field

[0001] The present invention relates to the field of sound signal acquisition and processing, and in particular to a sound source localization method based on a self-attention mechanism. Background Art

[0002] Sound source localization aims to estimate the position of one or more sound sources relative to an arbitrary reference point based on multi-channel microphone signals. It has been widely used in various scenarios, such as human-computer interaction, noise control, and speech enhancement. Accurate sound source localization can significantly improve system performance and user experience. For example, in human-computer interaction, accurate sound source localization can enhance the effectiveness of speech recognition systems, allowing robots or smart assistants to respond to users' voice commands more accurately. In terms of noise control and speech enhancement, sound source localization technology helps to identify and suppress noise sources, thereby improving the quality of signal processing.

[0003] Sound source localization is a long-standing and widely studied problem. Traditional sound source localization methods mainly rely on array signal processing techniques, such as time difference method and beamforming method. These methods perform well in relatively simple and ideal environments. However, in complex sound field environments, traditional methods often face many limitations. For example, sound signals in actual environments are affected by multipath effects such as reflection and scattering, which complicates the signal path and makes it difficult for traditional methods to accurately estimate the location of the sound source. At the same time, traditional methods have poor adaptability to dynamically changing scenes and are difficult to effectively track moving sound sources.

[0004] In recent years, the rapid development of neural networks and deep learning technologies has provided new solutions to the problem of sound source localization. Such data-driven models demonstrate strong adaptability and generalization capabilities by learning patterns and features from massive amounts of data. Deep learning models excel in handling complex and nonlinear problems, and can maintain high positioning accuracy in environments with severe noise interference and multipath effects.

[0005] The development of the self-attention mechanism provides an important means for deep learning models to further improve their performance. It can effectively capture long-distance dependencies in sequence data and has achieved remarkable results in natural language processing, computer vision and other fields. The core idea is to dynamically adjust the feature representation by calculating the correlation weights between elements in the sequence, thereby improving the model's ability to capture key information. Summary of the invention

[0006] The purpose of the present invention is to solve the above problems and design a sound source localization method based on the self-attention mechanism.

[0007] To achieve the above object, the technical solution of the present invention is that, further, in the above-mentioned sound source localization method based on the self-attention mechanism, the sound source localization method comprises the following steps:

[0008] The input multi-channel audio signal is divided into non-overlapping time blocks, and each channel of each time block is subjected to STFT short-time Fourier transform to obtain a complex value containing amplitude and phase information;

[0009] Extracting the magnitude spectrum and phase spectrum of the complex value, and for each channel, connecting the magnitude spectrum and the phase spectrum to form a feature tensor to obtain a feature map;

[0010] Inputting the feature map into a convolutional neural network (CNN) to extract features and obtain first sequence features;

[0011] Position encoding the first feature to obtain a second sequence feature, and mapping the second sequence feature to a query key and value matrix through a linear transformation;

[0012] Calculate the dot product between the query and the key, then perform a scaling operation to obtain the attention weights, and normalize the attention weights through the softmax function; calculate the attention score of each head, and concatenate the outputs of each attention head to get the result of multi-head attention;

[0013] The result of multi-head attention is mapped back to the original feature space through linear transformation. The output of multi-head self-attention is added to the input second sequence feature through residual connection, and layer normalization is performed to obtain standard deep features.

[0014] The standard deep features are input into a FFN feedforward neural network, and two linear changes and one ReLU activation are performed to obtain the target deep features;

[0015] The target deep features are input and output into a fully connected layer to obtain the DOA trajectory prediction value of the sound source position in the three-dimensional space.

[0016] Furthermore, in the above-mentioned sound source localization method based on the self-attention mechanism, the input multi-channel audio signal is divided into non-overlapping time blocks, and each channel of each time block is subjected to STFT short-time Fourier transform to obtain a complex value containing amplitude and phase information, including:

[0017] The formula for STFT short-time Fourier transform is as follows:

[0018]

[0019] Where X c (n, k) represents the STFT result of channel c at the nth frame and the kth frequency point, x c(n) represents the input signal of channel c, w(m) represents the Hamming window function, M is the window length of STFT, k=0, 1, .., M-1 represents the frequency index, and j is the imaginary unit.

[0020] Further, in the above-mentioned sound source localization method based on the self-attention mechanism, the extracting of the amplitude spectrum and the phase spectrum of the complex value, for each channel, connecting the amplitude spectrum and the phase spectrum to form a feature tensor to obtain a feature map, includes:

[0021] Extract the amplitude spectrum of the nth frame and the kth frequency point |X c The method of (n, k)| is:

[0022]

[0023] The method to extract the phase spectrum is:

[0024] θ c (n, k) = atan2(Im(X c (n, k)), Re(X c (n, k)));

[0025] The formula for the feature map is as follows:

[0026] F=[A1Θ1A2Θ2…A C Θ c ].

[0027] Furthermore, in the above-mentioned sound source localization method based on the self-attention mechanism, the convolutional neural network CNN has three layers, each layer uses a convolution kernel of size 3 and contains 64 filters, and the output after each convolution is batch normalized.

[0028] Furthermore, in the above-mentioned sound source localization method based on the self-attention mechanism, the position encoding of the first feature to obtain the second sequence feature includes:

[0029] The expression of the second sequence characteristic is as follows:

[0030] Z input =Z+PE

[0031] Among them, PE (pos) =W pos , Z is the first feature.

[0032] Furthermore, in the above-mentioned sound source localization method based on the self-attention mechanism, mapping the second sequence feature to a query key and value matrix through linear change includes:

[0033] The input feature Z is transformed into inputMapped to query, key, and value matrices. For the i-th head, these matrices are represented as: Q i =Z input W i Q , K i =Z input W i K and V i =Z input W i V .Notice is the weight matrix of query, key and value, d k is the dimension of each attention head.

[0034] Its beneficial effect is that the present invention, by combining the self-attention mechanism, can more effectively capture and model the global context information in the audio signal, thereby improving the accuracy of sound source localization. The present invention uses CNN for feature extraction, which can extract more robust high-level features in a variety of sound field environments and reduce sensitivity to background noise and environmental changes. In addition, the multi-head self-attention mechanism can enhance the model's ability to pay attention to the positions of different sound sources, further improving the robustness of positioning. Traditional methods have poor positioning accuracy in multi-sound source environments, and the model of the present invention can simultaneously process and locate multiple sound sources by learning the characteristics and time-frequency relationships of multiple sound sources, thereby improving performance in multi-sound source scenarios. The method of the present invention is highly flexible and can adapt to microphone arrays of different numbers and arrangements. By adjusting the parameters and structure of the model, it can be easily expanded to more complex sound field environments and application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0036] Figure 1 Schematic diagram of a first embodiment of a sound source localization method based on a self-attention mechanism in an embodiment of the present invention. Figure 2 Schematic diagram of a second embodiment of a sound source localization method based on a self-attention mechanism in an embodiment of the present invention; Figure 3 Schematic diagram of a third embodiment of a sound source localization method based on a self-attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0039] The present invention will be described in detail below in conjunction with the accompanying drawings. Figure 1 As shown, a sound source localization method based on a self-attention mechanism comprises the following steps:

[0040] Step 101: divide the input multi-channel audio signal into non-overlapping time blocks, perform STFT short-time Fourier transform on each channel of each time block, and obtain a complex value containing amplitude and phase information;

[0041] Specifically, in this embodiment, the input multi-channel audio signal is divided into non-overlapping time blocks. Assume that an audio signal x(t) with C channels is given and is divided into non-overlapping time blocks with a duration of 500 milliseconds.

[0042] Perform a short-time Fourier transform (STFT) on each channel of each time block,

[0043] The formula for STFT short-time Fourier transform is as follows:

[0044]

[0045] Where X c (n, k) represents the STFT result of channel c at the nth frame and the kth frequency point, x c (n) represents the input signal of channel c, w(m) represents the Hamming window function, M is the window length of STFT, k=0, 1, .., M-1 represents the frequency index, and j is the imaginary unit.

[0046] The result of STFT calculation is a complex value containing amplitude and phase information. Extract the amplitude spectrum and phase spectrum respectively. Extract the amplitude spectrum of the nth frame and the kth frequency point |X c The method of (n, k)| is: The method to extract the phase spectrum is: c (n, k) = atan2(Im(Xc (n, k)), Re(X c (n, k))).

[0047] Step 102: extracting a complex-valued amplitude spectrum and a phase spectrum, and for each channel, connecting the amplitude spectrum and the phase spectrum to form a feature tensor to obtain a feature map;

[0048] Specifically, in this embodiment, for each channel c, the amplitude spectrum and the phase spectrum are connected to form a feature tensor, that is, F = [A1θ1A2θ2…A C Θ C ].

[0049] Step 103: input the feature map into a convolutional neural network (CNN) to extract features and obtain first sequence features;

[0050] Specifically, in this embodiment, the extracted feature map is first input into a convolutional neural network (CNN) for feature extraction. The CNN has three layers, each layer uses a convolution kernel of size 3 and contains 64 filters, and the output after each convolution is batch normalized.

[0051] After CNN, a two-layer fully connected network is used to flatten the output for subsequent processing. The features at this time are recorded as Z.

[0052] Step 104: Position-encode the first feature to obtain a second sequence feature, and map the second sequence feature to a query key and value matrix through linear transformation;

[0053] Specifically, in this embodiment, Z is position-encoded to retain the sequence information of the time frame, that is, Z input =Z+PE, where PE (pos) =W pos .

[0054] The input feature Z is transformed into input Mapped to query, key, and value matrices. For the i-th head, these matrices are represented as: Q i =Z input W i Q , K i =Z input W i K and V i =Z input W i V .Notice is the weight matrix of query, key and value, d k is the dimension of each attention head.

[0055] Step 105: Calculate the dot product between the query and the key, then perform a scaling operation to obtain the attention weight, and normalize the attention weight through the softmax function; calculate the attention score of each head, and concatenate the output of each attention head to obtain the result of multi-head attention;

[0056] Specifically, in this embodiment,

[0057] The dot product between the query and the key is calculated, and then scaled to obtain the attention weights. The attention weights are normalized by the softmax function, i.e.

[0058] Calculate the attention score of each head, i.e.

[0059] Step 106: The result of the multi-head self-attention is mapped back to the original feature space through a linear transformation, and the output of the multi-head self-attention is added to the input second sequence feature through a residual connection, and layer normalization is performed to obtain a standard deep feature;

[0060] Specifically, in this embodiment, the output of each attention head is connected to obtain the result of multi-head attention. Finally, the result is mapped back to the original feature space through linear transformation: MultiHead(Q, K, V) = Concat(head1, ..., head h )W O ,here is the output linear transformation matrix.

[0061] Step 107: Input the standard deep features into a FFN feedforward neural network, perform two linear changes and one ReLU activation, and obtain the target deep features;

[0062] Specifically, in this embodiment, the output of the multi-head self-attention is added to the input Z through a residual connection input Then perform layer normalization:

[0063] Z attention =LayerNorm(Z input +MultiHead(Q, K, V)),

[0064] in μ and σ are the mean and standard deviation of the input x, respectively, and ∈ is a small constant to prevent division by zero.

[0065] Step 108: The target deep features are input and output into a fully connected layer to obtain the DOA trajectory prediction value of the sound source position in the three-dimensional space.

[0066] Specifically, in this embodiment, the output is input into a feed-forward neural network (FFN), which undergoes two linear transformations and one ReLU activation: Z ffn =max(0, Z attention W1+b1)W2+b2, where W1 and W2 are weight matrices, b1 and b2 are bias vectors. Finally, Z attention Added to Z via residual connection ffn And normalize it to get the final deep feature: Z = LayerNorm (Z attention +Z ffn ).

[0067] The deep features are input into a fully connected layer to compress the high-dimensional encoding features. A fully connected layer with 3N output nodes is constructed, where N represents the number of the largest sound events in the audio. Each sound event is represented by three nodes, corresponding to the X, Y, and Z coordinates of the sound source position. After the above steps, the DOA trajectory prediction value of the sound source position in three-dimensional space is finally obtained.

[0068] Prepare a dataset containing multi-channel audio signals and their corresponding sound source location labels. The dataset should include audio samples in various environments, and use mean square error to iteratively train the above-constructed model.

[0069] The beneficial effect is that by dividing the input multi-channel audio signal into non-overlapping time blocks, performing STFT short-time Fourier transform on each channel of each time block to obtain a complex value containing amplitude and phase information; extracting the amplitude spectrum and phase spectrum of the complex value, for each channel, connecting the amplitude spectrum and the phase spectrum to form a feature tensor to obtain a feature map; inputting the feature map into a convolutional neural network CNN for feature extraction to obtain a first sequence feature; position encoding the first feature to obtain a second sequence feature, and mapping the second sequence feature to a query key and value matrix through linear transformation; calculating the dot product between the query and the key, and then performing a scaling operation to obtain attention The attention weight is normalized by the softmax function; the attention score of each head is calculated, and the output of each attention head is connected to obtain the result of multi-head attention; the result of multi-head attention is mapped back to the original feature space through linear transformation, and the output of multi-head self-attention is added to the input second sequence feature through residual connection, and layer normalization is performed to obtain standard deep features; the standard deep features are input into an FFN feedforward neural network, and two linear changes and one ReLU activation are performed to obtain the target deep features; the target deep features are input and output into a fully connected layer to obtain the DOA trajectory prediction value of the sound source position in three-dimensional space. It can more effectively capture and model the global context information in the audio signal, thereby improving the accuracy of sound source localization. It can extract more robust high-level features in a variety of sound field environments and reduce sensitivity to background noise and environmental changes. In addition, the multi-head self-attention mechanism can enhance the model's ability to pay attention to different sound source positions, further improving the robustness of positioning. Traditional methods have poor positioning accuracy in multi-sound source environments, but the model of the present invention can simultaneously process and locate multiple sound sources by learning the characteristics and time-frequency relationship of multiple sound sources, thereby improving the performance in multi-sound source scenarios. The method of the present invention is highly flexible and can adapt to microphone arrays of different numbers and arrangements. By adjusting the parameters and structure of the model, it can be easily expanded to more complex sound field environments and application scenarios.

[0070] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A sound source localization method based on self-attention mechanism, characterized in that: The sound source localization method comprises the following steps: The input multi-channel audio signal is divided into non-overlapping time blocks, and each channel of each time block is subjected to STFT short-time Fourier transform to obtain a complex value containing amplitude and phase information; Extracting the magnitude spectrum and phase spectrum of the complex value, and for each channel, connecting the magnitude spectrum and the phase spectrum to form a feature tensor to obtain a feature map; Inputting the feature map into a convolutional neural network (CNN) to extract features and obtain first sequence features; Position encoding the first feature to obtain a second sequence feature, and mapping the second sequence feature to a query key and value matrix through a linear transformation; Calculate the dot product between the query and the key, then perform a scaling operation to obtain the attention weights, and normalize the attention weights through the softmax function; calculate the attention score of each head, and concatenate the outputs of each attention head to get the result of multi-head attention; The result of multi-head attention is mapped back to the original feature space through linear transformation. The output of multi-head self-attention is added to the input second sequence feature through residual connection, and layer normalization is performed to obtain standard deep features. The standard deep features are input into a FFN feedforward neural network, and two linear changes and one ReLU activation are performed to obtain the target deep features; The target deep features are input and output into a fully connected layer to obtain the DOA trajectory prediction value of the sound source position in the three-dimensional space.

2. A sound source localization method based on self-attention mechanism, characterized in that: The input multi-channel audio signal is divided into non-overlapping time blocks, and each channel of each time block is subjected to STFT short-time Fourier transform to obtain a complex value containing amplitude and phase information, including: The formula for STFT short-time Fourier transform is as follows: Where X c (n, k) represents the STFT result of channel c at the nth frame and the kth frequency point, x c (n) represents the input signal of channel c, w(m) represents the Hamming window function, M is the window length of STFT, k=0, 1, .., M-1 represents the frequency index, and j is the imaginary unit.

3. A sound source localization method based on self-attention mechanism, characterized in that: The extracting the amplitude spectrum and phase spectrum of the complex value, for each channel, connecting the amplitude spectrum and the phase spectrum to form a feature tensor, and obtaining a feature map, includes: Extract the amplitude spectrum of the nth frame and the kth frequency point |X c The method of (n, k)| is: The method to extract the phase spectrum is: θ c (n,k)=attan2(Im(X c (n,k)),Re(X c (n,k))); The formula for the feature map is as follows: F=[A1 Θ1A2Θ2…A C I C ]。 4. A sound source localization method based on self-attention mechanism, characterized in that: The convolutional neural network CNN has three layers in total. Each layer uses a convolution kernel of size 3 and contains 64 filters. The output after each convolution is batch normalized.

5. A sound source localization method based on self-attention mechanism, characterized in that: The performing position encoding on the first feature to obtain a second sequence feature comprises: The expression of the second sequence characteristic is as follows: WITH input =Z+PE Among them, PE (pos) =W pos , Z is the first feature.

6. A sound source localization method based on self-attention mechanism, characterized in that: Mapping the second sequence feature to a query key and value matrix by linear transformation includes: The input feature Z is transformed into input Mapped to query, key, and value matrices. For the i-th head, these matrices are represented as: Q i =Z input W i Q , K i =Z input W i K and V i =Z input W i V .Notice is the weight matrix of query, key and value, d k is the dimension of each attention head.

Images