Broadband sound source localization methods, devices, equipment and storage media

Abstract

This application discloses a broadband sound source localization method, apparatus, device, and storage medium, relating to the field of neural network technology. The method includes: acquiring a sound field signal in free space; calculating a covariance matrix based on the sound field signal; fusing and concatenating the covariance matrix to generate a covariance tensor; inputting the covariance tensor into an adaptive convolutional attention neural network model to obtain a subspace feature vector of the covariance tensor; calculating a spatial spectrum value based on the subspace feature vector and a preset scan vector; and determining the sound source location in the free space field using the spatial spectrum value. Using an adaptive convolutional attention neural network model for sound source localization prediction improves the flexibility and accuracy of sound source localization.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to broadband sound source localization methods, devices, equipment, and storage media. Background Technology

[0002] Acoustic imaging is an effective means of visualizing the power distribution of sound sources in a free sound field, and it has important application value in noise detection, equipment fault diagnosis, and other fields. In existing technologies, to achieve accurate sound source localization, traditional methods often use multi-channel, large-size microphone arrays for synchronous measurement. However, this approach suffers from high equipment complexity and cost. To address this, researchers have proposed asynchronous measurement methods, which involve sequentially moving a small prototype array to different locations to collect sound field signals, thereby expanding the array aperture and reducing hardware costs. Further improvements have been developed based on algorithms such as cyclic projection, block Hermitian matrix completion, and tensor singular value decomposition. Some solutions incorporate deep learning technology to adapt to the localization requirements of broadband non-stationary sound sources. However, existing deep learning-based asynchronous sound source localization methods have significant limitations. Their network structure design is fixed, only adapting to a preset number of measurement data, and cannot adapt to scenarios where the number of measurements changes during asynchronous measurement, resulting in insufficient flexibility in practical applications and difficulty in meeting diverse measurement needs under different sound field environments.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a broadband sound source localization method, apparatus, device, and storage medium, aiming to solve the technical problem of insufficient flexibility in sound source localization.

[0005] To achieve the above objectives, this application proposes a broadband sound source localization method, the method comprising:

[0006] Acquire the sound field signal of the free space field;

[0007] The covariance matrix is ​​calculated based on the sound field signal, and the covariance matrix is ​​fused and spliced ​​to generate a covariance tensor.

[0008] The covariance tensor is input into an adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor.

[0009] The spatial spectrum value is calculated based on the subspace feature vector and the preset scanning vector, and the sound source location of the free space field is determined by the spatial spectrum value.

[0010] In one embodiment, the sound field signal includes acoustic signals and coordinate information, and the step of acquiring the sound field signal of the free space field includes:

[0011] Set a microphone array with a preset number of channels, and move the microphone array to a preset measurement position in the free space field;

[0012] Asynchronous measurement is performed using the microphone array to collect acoustic signals at the preset measurement location, thereby obtaining the coordinate information of the preset measurement location.

[0013] In one embodiment, the step of calculating the covariance matrix based on the sound field signal and fusing and splicing the covariance matrix to generate a covariance tensor includes:

[0014] The sound field signal is transformed to obtain the frequency domain signal of the sound field signal;

[0015] Based on the frequency domain signal, calculate the covariance matrix of the frequency domain signal;

[0016] The number of matrices and the dimension of the covariance matrix are determined, and the covariance matrix is ​​concatenated based on the number of matrices and the dimension of the matrix to generate the covariance tensor of the frequency domain signal.

[0017] In one embodiment, the step of calculating the covariance matrix of the frequency domain signal based on the frequency domain signal includes:

[0018] Perform conjugate transpose calculation on the frequency domain signal to generate the conjugate transpose of the frequency domain signal;

[0019] Perform matrix multiplication on the frequency domain signal and its conjugate transpose to obtain the matrix product of the frequency domain signals;

[0020] The expected value of the matrix product is calculated to obtain the covariance matrix of the frequency domain signal.

[0021] In one embodiment, the adaptive convolutional attention neural network model includes a convolutional neural network module, a self-attention module, and a feedforward neural network module. The step of inputting the covariance tensor into the adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor includes:

[0022] The covariance tensor is split into amplitude data and phase data;

[0023] The amplitude data and the phase data are respectively input into the convolutional neural network module for feature encoding, and the first and second features of the covariance tensor are extracted.

[0024] The first feature and the second feature are respectively input into the self-attention module to capture the global dependencies in the covariance matrix, thereby obtaining the third feature and the fourth feature of the covariance matrix;

[0025] The third feature and the fourth feature are respectively input into the feedforward neural network module for nonlinear transformation and complexification to obtain the subspace feature vector of the covariance matrix.

[0026] In one embodiment, the step of calculating the spatial spectrum value based on the subspace feature vector and a preset scan vector, and determining the sound source location of the free space field using the spatial spectrum value, includes:

[0027] Based on the region to be scanned in the free space field, determine a number of scanning points and the corresponding scanning vectors for each scanning point;

[0028] Projection operations are performed based on the subspace feature vector and the scanning vector to obtain the spatial spectrum value of the scanning point;

[0029] Based on the spatial spectrum value, a spatial spectrum distribution map is obtained, and the location of the sound source in the free space field is determined according to the peak position in the spatial spectrum distribution map.

[0030] In one embodiment, the method further includes:

[0031] Obtain the synchronous measurement virtual array corresponding to the training signal obtained from asynchronous measurement;

[0032] Based on the training signal of the synchronous measurement virtual array, a synchronous covariance matrix is ​​constructed. The synchronous covariance matrix is ​​then decomposed into features, and the feature vector corresponding to the largest feature value is extracted as a supervision label.

[0033] The covariance tensor of asynchronous measurements is input into the adaptive convolutional attention neural network model. The adaptive convolutional attention neural network model is trained by minimizing the difference between the model's predicted output and the supervision label, resulting in the trained adaptive convolutional attention neural network model.

[0034] Furthermore, to achieve the above objectives, this application also proposes a broadband sound source localization device, which includes:

[0035] The sound field signal acquisition module is used to acquire the sound field signal of the free space field;

[0036] The variance tensor generation module is used to calculate the covariance matrix based on the sound field signal, and to fuse and splice the covariance matrix to generate a covariance tensor.

[0037] The feature vector generation module is used to input the covariance tensor into an adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor.

[0038] The sound source location determination module is used to calculate the spatial spectrum value based on the subspace feature vector and the preset scanning vector, and determine the sound source location of the free space field through the spatial spectrum value.

[0039] In addition, to achieve the above objectives, this application also proposes a broadband sound source localization device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the broadband sound source localization method as described above.

[0040] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the broadband sound source localization method described above.

[0041] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the broadband sound source localization method described above.

[0042] One or more technical solutions proposed in this application have at least the following technical effects:

[0043] This application proposes a broadband sound source localization method, apparatus, device, and storage medium. The method involves acquiring a sound field signal in free space; calculating a covariance matrix based on the sound field signal; fusing and concatenating the covariance matrix to generate a covariance tensor; inputting the covariance tensor into an adaptive convolutional attention neural network model to obtain a subspace feature vector; calculating a spatial spectrum value based on the subspace feature vector and a preset scan vector; and determining the sound source location in the free space field using the spatial spectrum value. The use of an adaptive convolutional attention neural network model for sound source localization prediction improves the flexibility and accuracy of sound source localization. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating an embodiment of the broadband sound source localization method of this application.

[0047] Figure 2 A schematic diagram of the structure of the adaptive convolutional attention neural network model provided in the embodiment of the broadband sound source localization method of this application;

[0048] Figure 3 An example diagram of the spatial spectral distribution map provided for the broadband sound source localization method of this application;

[0049] Figure 4 This is a schematic diagram of the module structure of the broadband sound source localization device according to an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the broadband sound source localization method in the embodiments of this application.

[0051] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0053] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of this application embodiment is as follows: acquiring the sound field signal of the free space field; calculating the covariance matrix based on the sound field signal, and fusing and splicing the covariance matrix to generate a covariance tensor; inputting the covariance tensor into an adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor; calculating the spatial spectrum value based on the subspace feature vector and a preset scanning vector, and determining the sound source location of the free space field through the spatial spectrum value.

[0055] In this embodiment, for ease of description, the following description uses a broadband sound source localization system as the execution subject.

[0056] Acoustic imaging is an effective means of visualizing the power distribution of sound sources in a free sound field, and it has important application value in noise detection, equipment fault diagnosis, and other fields. In existing technologies, to achieve accurate sound source localization, traditional methods often use multi-channel, large-size microphone arrays for synchronous measurement. However, this approach suffers from high equipment complexity and cost. To address this, researchers have proposed asynchronous measurement methods, which involve sequentially moving a small prototype array to different locations to collect sound field signals, thereby expanding the array aperture and reducing hardware costs. Further improvements have been developed based on algorithms such as cyclic projection, block Hermitian matrix completion, and tensor singular value decomposition. Some solutions incorporate deep learning technology to adapt to the localization requirements of broadband non-stationary sound sources. However, existing deep learning-based asynchronous sound source localization methods have significant limitations. Their network structure design is fixed, only adapting to a preset number of measurement data, and cannot adapt to scenarios where the number of measurements changes during asynchronous measurement, resulting in insufficient flexibility in practical applications and difficulty in meeting diverse measurement needs under different sound field environments.

[0057] This application provides a solution that improves the flexibility and accuracy of sound source localization by using an adaptive convolutional attention neural network model for sound source localization prediction.

[0058] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or broadband sound source localization system capable of performing the above functions. The following description uses a broadband sound source localization system as an example to illustrate this embodiment and the subsequent embodiments.

[0059] Based on this, embodiments of this application provide a broadband sound source localization method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the broadband sound source localization method of this application.

[0060] In this embodiment, the broadband sound source localization method includes steps S11 to S14:

[0061] Step S11: Obtain the sound field signal of the free space field.

[0062] It should be noted that free space refers to a spatial environment where sound waves can propagate freely without reflection or obstruction. The sound field signal is a signal generated by a sound source in a free space field, containing information such as changes in sound pressure, and serves as the foundational data for subsequent location analysis.

[0063] Specifically, a microphone array with a preset number of channels (e.g., 16 channels) is first selected. Based on the range of the free space field and the positioning accuracy requirements, K different measurement positions (e.g., 12) are planned and their coordinates are marked. Then, the microphone array is moved to each preset position in sequence by a mechanical moving device, and stays at each position for a preset time (e.g., 0.5 seconds). The array is then started to collect acoustic signals, and the coordinate information of the current position is recorded by the positioning module. Finally, K sets of sound field signals containing acoustic signals and corresponding coordinate information are collected.

[0064] For example, in an unobstructed, open laboratory (free space field), a 16-channel microphone array is used to plan 12 evenly distributed measurement positions with a spacing of 0.5 meters. After moving the array to the first position, acoustic signals are collected for 0.5 seconds, and the coordinates of that position (1.0m, 0.0m, 0.0m) are recorded simultaneously. This operation is repeated until the signals from all 12 positions are collected, resulting in 12 sets of sound field signals.

[0065] Step S12: Calculate the covariance matrix based on the sound field signal, and fuse and splice the covariance matrix to generate a covariance tensor.

[0066] It should be noted that the covariance matrix is ​​a matrix that describes the degree of linear correlation between multiple random variables, and here it is used to characterize the correlation between different channels of the sound field signal. The covariance tensor is a high-order tensor formed by concatenating multiple covariance matrices of the same dimension along a specific dimension, and is used to integrate the correlation information of multiple sets of measurements.

[0067] Specifically, a discrete Fourier transform is first performed on each group of sound field signals to convert the time-domain signals into frequency-domain signals; for each frequency point, the covariance matrix of each group of frequency-domain signals at that frequency is calculated; the number and dimensions of the covariance matrix are determined, and the covariance matrix is ​​spliced ​​along the first dimension (the dimension of the number of measurements) to generate a covariance tensor.

[0068] For example, discrete Fourier transforms are performed on 12 sets of sound field signals to obtain frequency domain signals at each frequency point; a frequency of ω=1000Hz is selected, and a 16×16 covariance matrix is ​​calculated for each set of frequency domain signals, resulting in a total of 12 16×16 covariance matrices; these 12 matrices are concatenated along the first dimension to generate a 12×16×16 covariance tensor, which contains the signal correlation information of 12 measurements at a frequency of 1000Hz.

[0069] Step S13: Input the covariance tensor into the adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor.

[0070] It should be noted that the Adaptive Convolutional Attention Neural Network (ACANN-NSM) is a deep learning model that integrates convolutional neural network modules, self-attention modules, and feedforward neural modules. It possesses the ability to handle inputs with variable shapes and extract key features. The subspace feature vector is the core vector representing the signal subspace and reflects the main characteristics of the sound source signal.

[0071] Specifically, the covariance tensor is first preprocessed and split into amplitude data and phase data. The two types of data are then input into a convolutional neural network module for feature encoding (1×3 kernel size, 1×2 stride, no padding) to extract local features. The encoded features are then input into a self-attention module (8 attention heads, 8 layers) to capture the global dependencies between different measurements. Finally, the attention-processed features are input into a feedforward neural network module (4 layers) for nonlinear transformation. The amplitude and phase information are fused through complexification, and a subspace feature vector with shape K×M×1 (12×16×1=192×1) is output.

[0072] For example, the 12×16×16 covariance tensor is decomposed into amplitude tensor and phase tensor, which are then input into a convolutional neural network module. After multiple convolutions, the output is 12×16×1 amplitude features and phase features. The two types of features are input into a self-attention module to learn the spatial correlation between 12 measurements and obtain optimized features. Then, a feedforward neural network module is used for nonlinear mapping, and finally, a 192×1 subspace feature vector is output. This vector accurately represents the signal subspace of the synchronous measurement virtual array.

[0073] Step S14: Calculate the spatial spectrum value based on the subspace feature vector and the preset scanning vector, and determine the sound source location of the free space field through the spatial spectrum value.

[0074] It should be noted that the scan vector is the guide vector from a certain scan point in the sound field to each channel of the microphone array, reflecting the spatial positional relationship between the scan point and the array.

[0075] Additionally, it should be noted that the spatial spectrum value is the result of the projection operation between the scan vector and the subspace feature vector, used to characterize the probability of a sound source existing at the scan point. The spatial spectrum distribution map is a visual representation of the spatial spectrum values ​​of all scan points.

[0076] Specifically, the area to be scanned in the free space field is determined, and the scanning points are divided according to the preset scanning interval. The scanning vector corresponding to each scanning point is calculated. Based on the MUSIC (Multiple Signal Classification) algorithm variant, the spatial spectrum value of each scanning point is calculated. The spatial spectrum values ​​of all scanning points are plotted into a spatial spectrum distribution map, and the peak position in the distribution map is the location of the sound source.

[0077] For example, the area to be scanned is set as x∈[-2m,2m], y∈[-2m,2m], and 1681 scanning points are obtained at a spacing of 0.1m. The 192×1 scanning vector corresponding to each scanning point is calculated. The scanning vector is projected onto the 192×1 subspace feature vector to obtain the spatial spectrum value of each scanning point. The spatial spectrum distribution map is plotted, and a significant peak is found at the coordinates (0.5m, 1.0m, 0.0m), which is determined to be the sound source location.

[0078] This embodiment uses the above-described scheme to acquire sound field signals through asynchronous measurements at multiple locations, integrates multiple sets of correlation information using the covariance tensor, and directly extracts signal subspace features through an adaptive neural network, thus avoiding the complex matrix recovery and eigenvalue decomposition process.

[0079] Based on the above implementation scheme, in one feasible implementation, the sound field signal includes acoustic signals and coordinate information, and the step of acquiring the sound field signal of the free space field includes S21~S22:

[0080] Step S21: Set a microphone array with a preset number of channels, and move the microphone array to a preset measurement position in the free space field.

[0081] It should be noted that the preset number of channels is the number of microphone units in the microphone array that is preset according to the sound source localization accuracy requirements. The more channels there are, the richer the signal acquisition dimensions are, and the higher the localization accuracy is usually. Common values ​​are 4 channels, 8 channels, 16 channels, 32 channels, etc.

[0082] Additionally, it should be noted that a microphone array is a signal acquisition device composed of multiple microphones arranged in a specific manner, and the number of channels determines the dimension of signal acquisition. The preset measurement positions are the pre-planned placement of the microphone array based on the free space field range, the possible distribution area of ​​the sound source, and the positioning resolution requirements. It usually needs to cover the spatial range where the sound source may exist. The number of positions (i.e., the number of measurements K) can be set to 3, 6, 12, 24, etc., as required.

[0083] Step S22: Perform asynchronous measurement using the microphone array to collect acoustic signals at the preset measurement location and obtain the coordinate information of the preset measurement location.

[0084] It should be noted that asynchronous measurement refers to the signal acquisition process of a microphone array at different preset measurement positions not overlapping or synchronizing in time. That is, after completing the acquisition at one position, it moves to the next position for acquisition. This is different from "synchronous measurement" (multiple arrays acquiring signals at different positions at the same time). It can reduce hardware costs, but it requires algorithms to compensate for the phase loss caused by time asynchrony.

[0085] Specifically, in one embodiment of this application, a channel microphone array is selected based on the positioning accuracy requirements, and b measurement positions are planned in a free space field. The array is manually moved to each position, and c seconds of acoustic signal are collected and the coordinates are recorded each time. In another embodiment of this application, a d channel microphone array is selected, e measurement positions are planned, and an automatic guide rail device is used to move the array to achieve unmanned data collection. F seconds of acoustic signal are collected at each position, and the coordinates are automatically recorded via a GPS module. In yet another embodiment of this application, for small spaces, a g channel microphone array is selected, h measurement positions are planned with a spacing of i meters, and acoustic signals and coordinate information are synchronously transmitted via a Bluetooth module.

[0086] For example, in a small conference room (free space field), a 4-channel microphone array is selected, and 3 measurement positions are planned with coordinates of (0.0m, 0.0m, 0.0m), (0.3m, 0.0m, 0.0m), and (0.15m, 0.26m, 0.0m). The array is manually moved to each position, and 0.5 seconds of acoustic signal is collected at each position. The acoustic signal and the corresponding coordinates are simultaneously transmitted to the processing terminal via Bluetooth, and finally 3 sets of sound field signals containing acoustic signal and coordinate information are obtained.

[0087] This embodiment, through the above-described scheme, adapts to the signal acquisition needs of different spatial scenarios by flexibly selecting the number of microphone array channels and the measurement position planning method. The automatic or manual movement mode can be switched according to actual conditions. The acquired sound field signal contains complete acoustic and positional information, providing accurate data support for subsequent covariance calculation and positioning. The flexibility and applicability of signal acquisition are significantly improved.

[0088] Based on the above implementation scheme, in one feasible implementation, the step of calculating the covariance matrix based on the sound field signal and fusing and splicing the covariance matrix to generate a covariance tensor includes S31~S33:

[0089] Step S31: Perform signal transformation on the sound field signal to obtain the frequency domain signal of the sound field signal.

[0090] It should be noted that signal transformation refers to the mathematical operation of converting a sound field signal in the time domain (a digital signal of sound pressure that changes with time) into a signal in the frequency domain (a signal amplitude / phase distribution that changes with frequency). The purpose is to extract the frequency characteristics of the signal to meet the localization requirements of broadband sound sources (multi-frequency components). In one embodiment of this application, discrete Fourier transform is used for signal transformation.

[0091] Additionally, it's important to clarify that frequency domain signals refer to the transformed form of the signal, with "frequency-amplitude / phase" as the core dimension. They reflect the energy distribution of the signal at different frequency points, such as the proportion of frequency components in the 20Hz~20kHz range of a broadband sound source, providing fundamental frequency domain data for subsequent covariance matrix calculations. Time domain signals, on the other hand, refer to the original form of the sound field signal, with "time-sound pressure value" as the core dimension. They reflect the signal's variation over time, such as a 44.1kHz sampling rate signal collected by a microphone within one second, which is represented as a digital sequence of length 44100.

[0092] Specifically, the asynchronously measured sound field signal is read, and the time-domain signal of each channel is processed to "remove DC component" (subtract the mean of the signal for that channel to eliminate static noise interference). Then, a Hanning window is added, with the window length equal to the number of sampling points in the time-domain signal, to avoid spectral leakage caused by signal truncation. Then, the discrete Fourier transform parameters are configured, and a discrete Fourier transform operation is performed on the windowed time-domain signal of each channel (e.g., calling the fft function in MATLAB or the numpy.fft.fft function in Python) to obtain a complex frequency domain signal matrix. The frequency domain signal of each measurement is in the form of "number of channels × number of frequency points", such as 4 channels corresponding to a 4×16385 complex matrix, where the matrix elements are the complex values ​​(amplitude + phase) of each frequency point.

[0093] Step S32: Calculate the covariance matrix of the frequency domain signal based on the frequency domain signal.

[0094] It should be noted that the covariance matrix is ​​a complex square matrix that describes the degree of linear correlation between different channels of a frequency domain signal. The matrix dimension is "number of channels × number of channels" (M × M), and the elements R... k(i,j) It reflects the correlation between the frequency domain signals of the i-th channel and the j-th channel. The larger the value, the stronger the correlation, which provides spatial correlation information between channels for subsequent sound source localization.

[0095] Specifically, the frequency domain signal is subjected to conjugate transpose processing; the frequency domain signal and its conjugate transpose are multiplied by matrix to obtain the matrix product of the frequency domain signal; the expected value of the matrix product is calculated to obtain the covariance matrix of the frequency domain signal.

[0096] Step S33: Determine the number of matrices and the matrix dimension of the covariance matrix, and concatenate the covariance matrix based on the number of matrices and the matrix dimension to generate the covariance tensor of the frequency domain signal.

[0097] It should be noted that the number of matrices refers to the total number of asynchronous measurements, that is, the number of effective covariance matrices obtained by calculation, denoted as K. For example, if there are 12 asynchronous measurements and no missing data, the number of matrices K=12. This number determines the size of the first dimension of the covariance tensor.

[0098] Additionally, it should be noted that the matrix dimension refers to the square matrix dimension of a single covariance matrix, which is determined by the number of channels M of the microphone array, i.e., M×M. For example, a 4-channel array corresponds to a 4×4 covariance matrix. The matrix dimension determines the size of the second and third dimensions of the covariance tensor.

[0099] Additionally, it should be noted that the covariance tensor is a third-order tensor formed by concatenating K M×M covariance matrices along the first dimension (the dimension of measurement frequency). Its form is K×M×M. Dimensional alignment during concatenation ensures that the channel dimensions and frequency characteristics of all covariance matrices are consistent, avoiding misalignment. The tensor can integrate spatial correlation information from multiple sets of asynchronous measurements, transforming scattered matrix data into structured high-order data, adapting to the input format of subsequent adaptive neural network models.

[0100] Specifically, the number of valid asynchronous measurements and the number of channels for each measurement are obtained, the number of matrices and the dimensions of each covariance matrix are determined, consistency checks and dimension adaptations are performed on each covariance matrix to ensure that the dimensions of each matrix are consistent, and then the covariance matrices are concatenated according to the measurement order to generate a covariance tensor.

[0101] Furthermore, the expression for generating the covariance tensor by concatenating the covariance matrix is ​​as follows:

[0102]

[0103] in, refers to frequency The covariance tensor at the location; This refers to the number of measurements taken asynchronously. This refers to the covariance matrix of the first measurement; It is a splicing operation along the first dimension.

[0104] For example, a Discrete Fourier Transform (Sampling frequency 44100Hz, FFT number 1024) was performed on 18 sets of sound field signals to obtain the frequency domain signals at each frequency point; a frequency ω=2000Hz was selected, and the 8×8 covariance matrix of each set of frequency domain signals was calculated, resulting in a total of 18 8×8 covariance matrices; after confirming that the number of matrices is 18 and the dimension is 8×8, the 18 matrices were sequentially concatenated along the first dimension (the dimension of the number of measurements) to generate an 18×8×8 covariance tensor, which completely preserves the correlation between the signal channels of the 18 measurements at a frequency of 2000Hz.

[0105] This embodiment supports multiple signal transformation methods through the above scheme. The appropriate transformation method can be selected according to the signal characteristics and computational efficiency. By specifying the number and dimensions of matrices for precise splicing, the generated covariance tensor can effectively integrate the frequency domain correlation information of multiple sets of measurements, providing the neural network model with structurally sound and informationally complete input data.

[0106] Based on the above implementation scheme, in one feasible implementation, the step of calculating the covariance matrix of the frequency domain signal based on the frequency domain signal includes S41~S43:

[0107] Step S41: Perform conjugate transpose calculation on the frequency domain signal to generate the conjugate transpose of the frequency domain signal.

[0108] It should be noted that the conjugate transpose of a frequency domain signal is an operation that transposes and conjugates a complex-valued frequency domain signal, used to construct the dual matrix for covariance calculation. The goal is to ensure that the frequency domain matrix and its conjugate transpose satisfy the multiplication dimension matching, while preserving the phase correlation of the signal.

[0109] Specifically, the conjugate transpose of the frequency domain signal is calculated to generate the conjugate transpose of the frequency domain signal.

[0110] Step S42: Perform matrix multiplication on the frequency domain signal and the conjugate transpose of the frequency domain signal to obtain the matrix product of the frequency domain signal.

[0111] It should be noted that matrix multiplication refers to multiplying the frequency domain signal matrix with its conjugate transpose to obtain an intermediate matrix that represents the correlation of the signal.

[0112] Specifically, matrix multiplication is performed on the frequency domain signal and its conjugate transpose to obtain the matrix product of the frequency domain signals.

[0113] Step S43: Calculate the expected value of the matrix product to obtain the covariance matrix of the frequency domain signal.

[0114] It should be noted that the mathematical expectation calculation is to perform a statistical average of the matrix products to eliminate the influence of random noise and obtain a stable covariance matrix.

[0115] Specifically, the expected value of the matrix product is calculated to obtain the covariance matrix of the frequency domain signal.

[0116] Furthermore, for The asynchronous measurement of this measurement, through The prototype array of the channel is moved to The different positions, the first Second-rate Covariance matrix obtained from asynchronous measurements for:

[0117]

[0118] in, Indicates the first In this measurement In frequency Discrete Fourier transform at the point, Represents the mathematical expectation. This indicates the conjugate transpose operation.

[0119] For example, a set of frequency domain signals pk(ω) is a complex matrix of 16×1024. Taking its conjugate transpose yields a matrix pk(ω) of 1024×16. H Compare pk(ω) of size 16 × 1024 with pk(ω) of size 1024 × 16. H Matrix multiplication is performed to obtain a 16×16 matrix product. The sum of the 1024 time points of this matrix product is then divided by 1024 to obtain the expected value, and finally a 16×16 covariance matrix is ​​obtained. This matrix accurately reflects the linear correlation between the 16 channels of the frequency domain signal.

[0120] This embodiment, through the above scheme, constructs the correlation matrix by using conjugate transpose and matrix multiplication, and combines it with a reasonable expectation estimation method, effectively reducing the impact of noise on the covariance matrix and improving the stability and reliability of calculating the covariance matrix.

[0121] Based on the above implementation scheme, in one feasible implementation, the adaptive convolutional attention neural network model includes a convolutional neural network module, a self-attention module, and a feedforward neural network module. The step of inputting the covariance tensor into the adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor includes S51~S54:

[0122] Step S51: The covariance tensor is split into amplitude data and phase data.

[0123] It should be noted that amplitude data is the magnitude part of the covariance tensor, reflecting the signal intensity characteristics; phase data is the argument part of the covariance tensor, reflecting the signal phase relationship. Amplitude reflects the signal intensity correlation, while phase reflects the signal time difference correlation; processing them separately can avoid mutual interference.

[0124] Specifically, the K×M×M covariance tensor is decomposed into K×M×M amplitude data and K×M×M phase data according to its complex attribute. The complex covariance tensor (with elements a+bj) is then decomposed into amplitude data (with element magnitudes) according to its complex attribute. The operations on the magnitude and phase data (element argument arctan2(b,a)) are intended to process the magnitude and phase information of complex numbers respectively.

[0125] Step S52: Input the amplitude data and the phase data into the convolutional neural network module for feature encoding, and extract the first feature and the second feature of the covariance tensor.

[0126] It should be noted that the Convolutional Neural Network (CNN) module is designed to encode features of nonholonomic covariance tensors, and its stride is [missing value]. And all layers have no padding, except for the last layer. Apart from the convolutional kernel, all other convolutional layers use The convolutional kernel. The shape of the hidden features output by the convolutional neural network module is... Combining all convolutional layers is to extract the last dimension of the data from... Shrink to Therefore, the number of layers in this module depends on the number of channels in the microphone array. .

[0127] Additionally, it should be noted that the first feature refers to the local features of the covariance tensor extracted after the amplitude data obtained from the covariance tensor decomposition is encoded by the convolutional neural network module. It focuses on the local correlation information of the covariance tensor under the amplitude dimension, and is a compression of the redundant dimensions of the amplitude data and preservation of key local features.

[0128] Additionally, it should be noted that the second feature refers to the local features of the covariance tensor extracted after the phase data obtained by covariance tensor decomposition is encoded by the convolutional neural network module. It focuses on the local phase correlation information of the covariance tensor under the phase dimension, and is a compression of the redundant dimensions of the phase data and preservation of key local features.

[0129] Specifically, the amplitude data and the phase data are respectively input into the convolutional neural network module. The convolutional neural network module contains multiple convolutional layers (except for the last convolutional kernel which is 1×2, the rest are 1×3, stride is 1×2, and there is no padding). The amplitude data and the phase data are respectively feature encoded, and the first feature (amplitude encoding feature) and the second feature (phase encoding feature) of K×M×1 are output.

[0130] Step S53: Input the first feature and the second feature into the self-attention module to capture the global dependencies in the covariance matrix, and obtain the third feature and the fourth feature of the covariance matrix.

[0131] It should be noted that the Self-Attention (SA) module is used to extract features and recover the spatial correlation between different measurements in the same spatial domain. To reduce network complexity while maintaining performance, after extensive training experiments, the number of attention heads and layers were both set to 8, and the model dimension was set to match the number of channels in the prototype array. same.

[0132] Additionally, it should be noted that the third feature refers to the optimized feature obtained after the first feature is input into the attention module and the module captures the global dependencies between multiple sets of measurements in the covariance matrix. Based on the local amplitude features of the first feature, it integrates the global correlation information between different measurements and different channels, thereby enhancing the global consistency of the feature.

[0133] Additionally, it should be noted that the fourth feature refers to the optimized feature obtained after the second feature is input into the attention module and the module captures the global dependencies between multiple sets of measurements in the covariance matrix. Based on the local phase features of the second feature, it integrates the global phase correlation information between different measurements and different channels, thereby enhancing the global consistency of the feature.

[0134] Specifically, the first and second features are input into the attention module (8 attention heads, 8-layer network, model dimension consistent with M) to capture the global dependencies between different measurements, thus obtaining the third and fourth features.

[0135] Step S54: Input the third feature and the fourth feature into the feedforward neural network module for nonlinear transformation and complexification to obtain the subspace feature vector of the covariance matrix.

[0136] It should be noted that the FeedForward Neural Network (FFNN) module includes a four-layer network to further enhance nonlinear prediction capabilities.

[0137] Specifically, the third and fourth features are input into the feedforward neural network module (4-layer fully connected network with ReLU activation function) for nonlinear transformation, and finally fused through complexification operation (magnitude feature as modulus, phase feature as argument) to output a K×M×1 subspace feature vector.

[0138] For better understanding, please refer to Figure 2 , Figure 2 This is a schematic diagram of the adaptive convolutional attention neural network model. First, the K×M×M covariance tensor is split into phase and amplitude data, which are then input into the adaptive convolutional attention neural network model. The model employs a two-branch parallel structure, processing the input amplitude and phase data separately. Each branch contains a convolutional neural network module consisting of three convolutional layers with kernel sizes of 1×3, 1×3, and 1×2, respectively, and a uniform stride of 1×2 without padding. This aims to effectively encode features and compress dimensions for the amplitude and phase data, extracting local features focused on the correlation between signal strength and phase. Then, the K×M×1 features processed by the convolutional module are fed into the self-attention module to capture the global dependencies between different asynchronous measurements, thereby integrating global information based on local features and enhancing the spatial consistency of the features. The processed features are then standardized and fed into the feedforward neural network module. This module uses fully connected layers and activation functions (such as ReLU) for nonlinear transformation to further improve the model's fitting ability. Finally, the features output from the two branches are fused through a "complexification" operation, whereby the output of the amplitude branch is used as the real part (or modulus) and the output of the phase branch is used as the imaginary part (or argument), recombine them into a complex-valued subspace feature vector with a dimension of KM×1. This vector is the network's prediction of the synchronous measurement virtual array signal subspace. Through this divide-and-conquer and fusion strategy, the entire network structure collaboratively extracts local and global features of the complex-valued covariance tensor, avoiding mutual interference between amplitude and phase information. This allows for adaptation to different measurement counts K without altering the network structure and significantly improves the accuracy of subspace feature vector prediction.

[0139] For example, the 18×16×16 covariance tensor is split into 18×16×16 amplitude data and 18×16×16 phase data; these are respectively input into a convolutional neural network module, and after 4 layers of convolution (the first 3 layers have 1×3 kernels, and the last layer has 1×2 kernels), the first and second features of 18×16×1 are output; the two types of features are input into a self-attention module (8 attention heads, model dimension 16) to learn the spatial correlation between 18 measurements, and the third and fourth features are obtained; these are then input into a feedforward neural network module for nonlinear transformation, and through complex fusion, the subspace feature vector of 18×16×1=288×1 is output, which accurately matches the signal subspace of the synchronous measurement virtual array.

[0140] This embodiment, through the above-described scheme, fully preserves the complete information of the complex numerical covariance tensor by splitting the amplitude and phase data for separate processing. The combination of the convolutional neural network module and the self-attention module enables the collaborative extraction of local features and global dependencies. The feedforward neural network module enhances the nonlinear fitting ability and improves the accuracy of the final output subspace feature vector. Moreover, the model can adapt to inputs with different number of measurements without adjusting its structure, and its adaptive ability is significantly better than that of traditional deep learning models.

[0141] Based on the above implementation scheme, in one feasible implementation, the step of calculating the spatial spectrum value based on the subspace feature vector and the preset scanning vector, and determining the sound source location of the free space field through the spatial spectrum value, includes S61~S63:

[0142] Step S61: Based on the area to be scanned in the free space field, determine a number of scanning points and the scanning vectors corresponding to the scanning points.

[0143] It should be noted that the area to be scanned refers to the preset free space field range where sound sources may exist. It is usually defined in a rectangular coordinate system or a polar coordinate system (e.g., x∈[-5m,5m], y∈[-5m,5m], z=0m). It needs to cover the potential distribution area of ​​sound sources. The size of the area is determined according to the positioning scenario (e.g., 5m×5m for small spaces, 50m×50m for large spaces).

[0144] Additionally, it should be noted that a scanning point refers to a discrete point within the area to be scanned, divided at a specific interval (e.g., 0.1m / 0.5m). Each scanning point represents a potential sound source location. The number of scanning points = (area length / interval + 1) × (area width / interval + 1). The smaller the interval, the higher the positioning accuracy, but the greater the computational load.

[0145] Additionally, it should be noted that the scan vector is a complex-valued steering vector used to describe the spatial relationship between the scan point and each channel of the microphone array.

[0146] Specifically, the scanning area and scanning interval are determined, the number of scanning points and the coordinates of the scanning points are determined based on the scanning area and scanning interval, and the scanning vector is calculated based on the coordinates of the scanning points and the channel array.

[0147] In one embodiment of this application, for a rectangular region (e.g., x∈[-3m,3m], y∈[-3m,3m], z=0m), the scanning interval is set to 0.1m, and the number of scanning points is calculated as (6 / 0.1+1)×(6 / 0.1+1)=61×61=3721. Based on the coordinates of each scanning point (xm, ym, 0) and the channel coordinates dk,m of the 12-times 4-channel array (k=1~12, m=1~4), the 48×1 scanning vector w(rm) is calculated according to the formula.

[0148] In another embodiment of this application, for a circular area (e.g., radius 5m, z=0m), a radial spacing of 0.2m (26 points from 0 to 5m) and an angular spacing of 10° (36 angles) are set, and the number of scanning points is 26×36=936. The polar coordinates (ρ,θ,0) are converted to rectangular coordinates (ρcosθ,ρsinθ,0), and then the scanning vector is calculated to adapt to the circular sound source distribution scene.

[0149] Step S62: Perform projection operation based on the subspace feature vector and the scanning vector to obtain the spatial spectrum value of the scanning point.

[0150] It should be noted that the spatial spectrum value refers to the value obtained by projecting the subspace feature vector output by the adaptive convolutional attention neural network model onto the preset scanning vector corresponding to each scanning point in the free space field to be scanned. Its core attribute is to quantitatively represent the degree of correlation between the corresponding scanning point and the sound source signal subspace. It is a key indicator reflecting the subspace matching degree between the scanning point and the sound source signal, and provides core data support for the subsequent generation of spatial spectrum distribution map and location of sound source peak position.

[0151] Additionally, it should be noted that projection operation refers to the mathematical operation of calculating the correlation between the scan vector and the feature vector of the subspace. Based on a variant of the MUSIC (Multiple signal classification) algorithm, the spatial spectrum value is calculated using the formula for calculating the spatial spectrum value. The larger the value, the higher the probability that a sound source exists at that scan point.

[0152] Specifically, the formula for calculating the spatial spectral value is as follows:

[0153]

[0154] in, This is the distance vector from the scan point to the array; for Spatial spectral values; for The scan vector; This is the conjugate transpose of the scan vector; For subspace eigenvectors; is the conjugate transpose of the eigenvectors of the subspace.

[0155] Step S63: Obtain a spatial spectrum distribution map based on the spatial spectrum value, and determine the location of the sound source in the free space field according to the peak position in the spatial spectrum distribution map.

[0156] It should be noted that a spatial spectrum distribution map is an image that visualizes the spatial spectrum values ​​of all scanned points according to their coordinate positions, such as a heat map (the darker the color, the higher the spectral value) or a three-dimensional surface map, which can intuitively show the distribution pattern of spatial spectrum values; the peak position refers to the point with the largest spectral value (single sound source) or the top N largest points (multiple sound sources) in the spatial spectrum distribution map, which corresponds to the actual position of the sound source and is determined by peak detection algorithms (such as thresholding and maximum suppression).

[0157] Specifically, for the case of a single sound source, single peak detection is performed, and the spatial spectrum values ​​are plotted as a heat map according to the coordinates of the scanning points (x-axis is the horizontal axis, y-axis is the vertical axis, and color maps the spectrum values). The coordinates of the scanning point with the largest spectrum value are determined by the "find maximum value" algorithm, which is the location of the sound source. If the difference between the maximum value and the second largest value exceeds 2 times (threshold), the single sound source is confirmed.

[0158] For example, the scanning region is set as x∈[-3m,3m], y∈[-3m,3m], z=0m, and 3601 scanning points are obtained at 0.1m intervals; the 192×1 scanning vector (KM=12×16=192) corresponding to each scanning point is calculated; the scanning vector is projected onto the 192×1 subspace feature vector to obtain the spatial spectrum value of each scanning point; the spatial spectrum distribution map is plotted, and the threshold is set to 70% of the highest spectrum value to select two peak positions (1.2m, 0.8m, 0.0m) and (-0.5m, -1.0m, 0.0m). It is verified that these two positions are the actual positions of the dual sound sources.

[0159] This embodiment, through the above-described scheme, can balance positioning accuracy and computational efficiency by flexibly setting the area to be scanned and the scanning interval. The fast projection algorithm and threshold filtering method improve the positioning speed.

[0160] Based on the above implementation scheme, in one feasible implementation, the method further includes steps S71 to S73:

[0161] Step S71: Obtain the synchronous measurement virtual array corresponding to the training signal obtained from asynchronous measurement.

[0162] It should be noted that the synchronous measurement virtual array refers to a "virtual synchronous array" built based on the measurement positions and number of channels of asynchronous measurements. Its size is K×M×1 (K is the number of asynchronous measurements, and M is the number of channels). It simulates the array structure of synchronous measurements, that is, it assumes that all microphone units of asynchronous measurements exist in space at the same time, forming a large array, which is used to generate ideal synchronous training data and provide a reference benchmark for supervised labels for neural networks.

[0163] Additionally, it should be noted that the training signal refers to the asynchronous measurement sound field signal used for model training. It includes multiple sets of signals with different number of measurements (e.g., K=6 / 12 / 18 / 24) and different number of channels (e.g., M=4 / 8 / 16) to ensure the model's generalization ability. Supervised training is achieved by comparing the signal with the synchronous measurement virtual array.

[0164] Specifically, training signals from asynchronous measurements are acquired (e.g., multiple sets of signals containing K=6, 12, 18, 24 measurements). Based on the measurement positions and number of channels of these signals, a corresponding synchronous measurement virtual array is constructed, and the synchronous training signal of the virtual array is generated by simulation (without time asynchronous error).

[0165] Furthermore, for synchronous measurements corresponding to asynchronous measurements, assuming that the array is the same as the virtual array of the asynchronous measurements, the training signal corresponding to the synchronous measurements... for:

[0166]

[0167] in, The training signal was collected at time t; It is the index of the time sampling point, with a value range from 1 to T (T is the total number of samples); This refers to the sound source signal component of the training signal; This refers to the noise signal component of the training signal.

[0168] Therefore, the synchronization covariance matrix corresponding to the training signal for:

[0169]

[0170] in, The synchronization covariance matrix corresponding to the training signal; This refers to synchronous measurement. In frequency The frequency domain matrix obtained by the discrete Fourier transform at the given location; Represents the mathematical expectation. This indicates the conjugate transpose operation.

[0171] Step S72: Construct a synchronization covariance matrix based on the training signal of the synchronization measurement virtual array, perform eigenvalue decomposition on the synchronization covariance matrix, and extract the feature vector corresponding to the largest eigenvalue as a supervision label.

[0172] It should be noted that the synchronous covariance matrix refers to the covariance matrix calculated based on the training signal of the synchronous virtual array. This matrix has no time asynchronous error and reflects the ideal channel correlation.

[0173] Additionally, it should be noted that the supervision label refers to the eigenvector corresponding to the largest eigenvalue in the synchronization covariance matrix. It represents the ideal signal subspace and is the target output for model training, used to guide the model in learning the mapping relationship from the covariance tensor to the signal subspace. Eigenvalue decomposition refers to the operation of decomposing the synchronization covariance matrix into the product of an eigenvector matrix and an eigenvalue diagonal matrix. The formula for eigenvalue decomposition is:

[0174]

[0175] in, It is the eigenvector matrix representing the signal subspace, and its column vectors are mutually orthogonal; This refers to the diagonal matrix of eigenvalues ​​in the signal subspace; This refers to the synchronization covariance matrix; It is the conjugate transpose of the eigenvector matrix representing the signal subspace.

[0176] Specifically, a discrete Fourier transform is performed on the training signal of the synchronous virtual array to obtain the corresponding frequency domain signal. Then, the covariance matrix corresponding to the training signal, i.e., the synchronization covariance matrix, is calculated. Eigenvalue decomposition is performed on the synchronization covariance matrix to obtain an eigenvector matrix and an eigenvalue diagonal matrix. The largest eigenvalue and its corresponding index are found in the eigenvalue diagonal matrix. Based on the index of the largest eigenvalue, the corresponding column vector is extracted from the eigenvector matrix to obtain a complex-valued eigenvector, which serves as the supervision label for model training. This label preserves the amplitude and phase information of the ideal signal subspace and is the target benchmark for model prediction.

[0177] Step S73: Input the covariance tensor of asynchronous measurement into the adaptive convolutional attention neural network model, and train the adaptive convolutional attention neural network model by minimizing the difference between the model prediction output and the supervision label to obtain the trained adaptive convolutional attention neural network model.

[0178] Specifically, the synchronous covariance matrix is ​​split into several sub-matrices according to the number of asynchronous measurements, and concatenated along the first dimension to generate a covariance tensor as the model input. Multiple sets of inputs and corresponding supervision labels are collected to form a training set. An adaptive convolutional attention neural network model is initialized, and model parameters are randomly initialized. The training set is input into the model in batches, and the model outputs predicted subspace feature vectors. The difference between the predicted subspace feature vectors and the supervision labels is calculated as the loss value. The gradient of the model parameters is calculated using the backpropagation algorithm, and the model parameters are updated using the Adam optimizer. After each training round, the loss is evaluated using a validation set. If the validation loss does not decrease after a preset number of rounds, an early stopping mechanism is triggered to avoid overfitting and complete the model training. After training, the model parameters are saved, and a validation dataset can be obtained to validate the trained model and ensure the accuracy of model predictions.

[0179] Furthermore, the model input is of size [missing information]. The complex numerical covariance tensor, where and These represent the number of measurements in asynchronous measurements and the number of channels in the prototype array, respectively. The model's label is the eigenvector corresponding to the largest eigenvalue of the synchronous measurement. By merging amplitude and phase predictions, the signal subspace is represented using complex-valued outputs. Through this deployment, the proposed model can learn a nonlinear approximation. Its optimization problem is formulated as follows:

[0180]

[0181]

[0182] in, It is a frequency unit; It is the eigenvector corresponding to the largest eigenvalue generated by the corresponding synchronous measurement; It is the covariance tensor generated by asynchronous measurement; It is a nonlinear mapping Used to generate feature vectors of the signal subspace ; This refers to the model's predicted output of the covariance tensor, i.e., the generated signal subspace feature vector; It refers to the norm coefficient (Flobenius norm), which measures the difference between the predicted vector and the label vector. The smaller the norm, the higher the model's prediction accuracy.

[0183] For example, training signals (M=16 channels) from K=6, 12, 18, and 24 measurements are obtained, and corresponding synchronous measurement virtual arrays are constructed (scales of 96×1, 192×1, 288×1, and 384×1, respectively). Discrete Fourier transform is performed on the synchronous training signals, the synchronous covariance matrix is ​​calculated, and eigenvalue decomposition is performed. The eigenvector corresponding to the largest eigenvalue is extracted as the supervision label. The covariance tensors from K=6, 12, 18, and 24 measurements (shapes of 6×16×16, 12×16×16, etc.) are input into the model, the F-norm is used as the loss function, and the Adam optimizer is used to train 1000 times. The model is tested using the covariance tensors from K=30 and 36 measurements (30×16×16 and 36×16×16).

[0184] This embodiment, through the above scheme, uses the ideal feature vector of the synchronous measurement virtual array as the supervision label, and through reasonable loss function and optimizer selection, enables the model to fully learn the mapping relationship between the asynchronous measurement covariance tensor and the synchronous signal subspace. The trained model can maintain high accuracy under different measurement numbers.

[0185] For example, please refer to Figure 3 , Figure 3 An example diagram of the spatial spectral distribution provided for broadband sound source localization methods, in Figure 3 (a) in the figure is the spatial spectral distribution of 30 asynchronous measurements; Figure 3 (b) in the figure is the spatial spectrum distribution of 36 asynchronous measurements. Figure 3 In this context, X represents the spatial coordinates of the sound source in the horizontal direction. Figure 3 In this context, N represents the spatial coordinates of the sound source in the vertical direction. Figure 3 The colors in the image represent the spatial spectrum values ​​corresponding to the sound source. According to... Figure 3 The location marked by the circle in the middle is the sound source location. It can be seen that the broadband sound source localization method of this application can clearly determine the location of the sound source. Furthermore, the spatial spectrum distribution map obtained by using 36 asynchronous measurements as a test set is still stable, which can demonstrate the accuracy and robustness of the broadband sound source localization method of this application.

[0186] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the broadband sound source localization method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0187] This application also provides a broadband sound source localization device, please refer to... Figure 4 The broadband sound source localization device includes:

[0188] The sound field signal acquisition module 401 is used to acquire the sound field signal of the free space field;

[0189] The variance tensor generation module 402 is used to calculate the covariance matrix based on the sound field signal and to fuse and splice the covariance matrix to generate a covariance tensor.

[0190] The feature vector generation module 403 is used to input the covariance tensor into the adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor.

[0191] The sound source location determination module 404 is used to calculate the spatial spectrum value based on the subspace feature vector and the preset scanning vector, and determine the sound source location of the free space field through the spatial spectrum value.

[0192] The broadband sound source localization device provided in this application, employing the broadband sound source localization method in the above embodiments, can solve the technical problem of insufficient flexibility in sound source localization. Compared with the prior art, the beneficial effects of the broadband sound source localization device provided in this application are the same as those of the broadband sound source localization method provided in the above embodiments, and other technical features in the broadband sound source localization device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0193] This application provides a broadband sound source localization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the broadband sound source localization method in the above embodiment 1.

[0194] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing a broadband sound source localization device according to embodiments of this application. The broadband sound source localization device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The broadband sound source localization device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0195] like Figure 5As shown, the broadband sound source localization device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the broadband sound source localization device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the broadband sound source localization device to communicate wirelessly or wiredly with other devices to exchange data. Although broadband sound source localization devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0196] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0197] The broadband sound source localization device provided in this application, employing the broadband sound source localization method in the above embodiments, can solve the technical problem of insufficient flexibility in sound source localization. Compared with the prior art, the beneficial effects of the broadband sound source localization device provided in this application are the same as those of the broadband sound source localization method provided in the above embodiments, and other technical features of this broadband sound source localization device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0198] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0199] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0200] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the broadband sound source localization method in the above embodiments.

[0201] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0202] The aforementioned computer-readable storage medium may be included in the broadband sound source localization device; or it may exist independently and not assembled into the broadband sound source localization device.

[0203] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the broadband sound source localization device, the broadband sound source localization device: acquires the sound field signal of the free space field; calculates the covariance matrix based on the sound field signal, and fuses and splices the covariance matrix to generate a covariance tensor; inputs the covariance tensor into an adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor; calculates the spatial spectrum value based on the subspace feature vector and a preset scan vector, and determines the sound source location of the free space field through the spatial spectrum value.

[0204] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0206] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0207] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described broadband sound source localization method, thereby solving the technical problem of insufficient flexibility in sound source localization. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the broadband sound source localization method provided in the above embodiments, and will not be repeated here.

[0208] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the broadband sound source localization method described above.

[0209] The computer program product provided in this application can solve the technical problem of insufficient flexibility in sound source localization. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the broadband sound source localization method provided in the above embodiments, and will not be repeated here.

[0210] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A broadband sound source localization method, characterized in that, The broadband sound source localization method includes: Acquire the sound field signal of the free space field; The covariance matrix is ​​calculated based on the sound field signal, and the covariance matrix is ​​fused and spliced ​​to generate a covariance tensor. The covariance tensor is input into an adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor. The spatial spectrum value is calculated based on the subspace feature vector and the preset scanning vector, and the sound source location of the free space field is determined by the spatial spectrum value. The adaptive convolutional attention neural network model includes a convolutional neural network module, a self-attention module, and a feedforward neural network module. The step of inputting the covariance tensor into the adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor includes: The covariance tensor is split into amplitude data and phase data; The amplitude data and the phase data are respectively input into the convolutional neural network module for feature encoding, and the first and second features of the covariance tensor are extracted. The first feature and the second feature are respectively input into the self-attention module to capture the global dependencies in the covariance matrix, thereby obtaining the third feature and the fourth feature of the covariance matrix; The third feature and the fourth feature are respectively input into the feedforward neural network module for nonlinear transformation and complexification to obtain the subspace feature vector of the covariance matrix; The step of calculating the spatial spectrum value based on the subspace feature vector and the preset scan vector, and determining the sound source location of the free space field using the spatial spectrum value, includes: Based on the region to be scanned in the free space field, determine a number of scanning points and the corresponding scanning vectors for each scanning point; Projection operations are performed based on the subspace feature vector and the scanning vector to obtain the spatial spectrum value of the scanning point; Based on the spatial spectrum value, a spatial spectrum distribution map is obtained, and the location of the sound source in the free space field is determined according to the peak position in the spatial spectrum distribution map.

2. The broadband sound source localization method as described in claim 1, characterized in that, The sound field signal includes acoustic signals and coordinate information, and the step of obtaining the sound field signal of the free space field includes: Set a microphone array with a preset number of channels, and move the microphone array to a preset measurement position in the free space field; Asynchronous measurement is performed using the microphone array to collect acoustic signals at the preset measurement location, thereby obtaining the coordinate information of the preset measurement location.

3. The broadband sound source localization method as described in claim 1, characterized in that, The steps of calculating the covariance matrix based on the sound field signal and fusing and splicing the covariance matrix to generate a covariance tensor include: The sound field signal is transformed to obtain the frequency domain signal of the sound field signal; Based on the frequency domain signal, calculate the covariance matrix of the frequency domain signal; The number of matrices and the dimension of the covariance matrix are determined, and the covariance matrix is ​​concatenated based on the number of matrices and the dimension of the matrix to generate the covariance tensor of the frequency domain signal.

4. The broadband sound source localization method as described in claim 3, characterized in that, The step of calculating the covariance matrix of the frequency domain signal based on the frequency domain signal includes: Perform conjugate transpose calculation on the frequency domain signal to generate the conjugate transpose of the frequency domain signal; Perform matrix multiplication on the frequency domain signal and its conjugate transpose to obtain the matrix product of the frequency domain signals; The expected value of the matrix product is calculated to obtain the covariance matrix of the frequency domain signal.

5. The broadband sound source localization method as described in claim 1, characterized in that, The method further includes: Obtain the synchronous measurement virtual array corresponding to the training signal obtained from asynchronous measurement; Based on the training signal of the synchronous measurement virtual array, a synchronous covariance matrix is ​​constructed. The synchronous covariance matrix is ​​then decomposed into features, and the feature vector corresponding to the largest feature value is extracted as a supervision label. The covariance tensor of asynchronous measurements is input into the adaptive convolutional attention neural network model. The adaptive convolutional attention neural network model is trained by minimizing the difference between the model's predicted output and the supervision label, resulting in the trained adaptive convolutional attention neural network model.

6. A broadband sound source localization device, characterized in that, The broadband sound source localization device includes: The sound field signal acquisition module is used to acquire the sound field signal of the free space field; The variance tensor generation module is used to calculate the covariance matrix based on the sound field signal, and to fuse and splice the covariance matrix to generate a covariance tensor. The feature vector generation module is used to input the covariance tensor into an adaptive convolutional attention neural network model to obtain the subspace feature vector of the covariance tensor. The sound source location determination module is used to calculate the spatial spectrum value based on the subspace feature vector and the preset scanning vector, and determine the sound source location of the free space field through the spatial spectrum value; The adaptive convolutional attention neural network model includes a convolutional neural network module, a self-attention module, and a feedforward neural network module. The feature vector generation module is further used to: split the covariance tensor into amplitude data and phase data. The amplitude data and the phase data are respectively input into the convolutional neural network module for feature encoding, and the first and second features of the covariance tensor are extracted. The first feature and the second feature are respectively input into the self-attention module to capture the global dependencies in the covariance matrix, thereby obtaining the third feature and the fourth feature of the covariance matrix; The third feature and the fourth feature are respectively input into the feedforward neural network module for nonlinear transformation and complexification to obtain the subspace feature vector of the covariance matrix; The sound source location determination module is further configured to: determine a number of scanning points and the scanning vectors corresponding to the scanning points based on the area to be scanned in the free space field; Projection operations are performed based on the subspace feature vector and the scanning vector to obtain the spatial spectrum value of the scanning point; Based on the spatial spectrum value, a spatial spectrum distribution map is obtained, and the location of the sound source in the free space field is determined according to the peak position in the spatial spectrum distribution map.

7. A broadband sound source localization device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the broadband sound source localization method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the broadband sound source localization method as described in any one of claims 1 to 5.

Images