Sound source localization method, device and equipment based on combined spectrum difference characteristics and medium

Abstract

The invention relates to the technical field of sound source localization, and provides a sound source localization method and device based on combined spectrum difference characteristics, equipment and a medium. According to the method, original waveform data sequences with preset lengths are sequentially intercepted according to the time sequence of data streams to serve as data frames, so that strict time sequence alignment is carried out on original collected data, then a Hanning window with low sidelobe level and linear phase response is adopted as a sliding window, and the phase of the data is kept undistorted in the frequency domain transformation process to the maximum extent. By setting a special combined difference calculation layer, difference signals of frame data of a plurality of channels in a frequency spectrum space are effectively extracted, and common-mode information among the data is abandoned, so that the data volume can be greatly compressed, a basis is provided for subsequent simplified calculation, and the method can be adapted to end-side AI equipment. And a non-uniform pooling method in a logarithm mode is adopted, high-frequency signals rich in sound source orientation characteristics are reserved, sparse sampling is carried out on low-frequency signals, and the balance of performance and efficiency is achieved.

Description

Technical Field

[0001] This application relates to the field of sound source localization technology, and in particular to a sound source localization method, apparatus, device and medium based on combined spectral difference characteristics. Background Technology

[0002] Sound source localization technology is a key technology that uses technical means to determine the location of a sound source in space. Its applications are wide-ranging, covering multiple fields including industry and civilian use. In the industrial sector, this technology can be used for equipment fault diagnosis, helping maintenance personnel quickly locate the specific location of abnormal sounds emanating from equipment, thereby improving maintenance efficiency and reducing equipment downtime. In the civilian sector, sound source localization technology plays an important role in virtual reality (VR) and augmented reality (AR). By simulating the propagation and localization of sound in three-dimensional space, it enables users to accurately perceive the direction and distance of the sound source, enhancing immersion and interactivity.

[0003] Traditional sound source localization techniques primarily rely on time delay estimation. The basic principle is to simulate the human auditory system, using the time difference between the arrival of sound signals at different sensors to calculate the sound source location. Specifically, based on the propagation characteristics of sound waves in air or solid materials and the time delay of the measured signal reaching each sensor, combined with the geometric arrangement of the sensor array, the direction of the sound source is determined. However, this method has several limitations. First, its noise resistance is weak; in complex noisy environments, localization accuracy is difficult to guarantee and it is easily affected by interference. Second, this method involves calculating the generalized cross-correlation coefficients of multiple time series; with multi-channel signal input, the computational load increases rapidly, leading to a decrease in processing speed and making it difficult to meet real-time requirements. Therefore, traditional sound source localization techniques face many challenges in practical applications, and an improved sound source localization technique is urgently needed to meet the higher requirements of different fields. Summary of the Invention

[0004] Therefore, it is necessary to provide a sound source localization method, device, equipment, and medium based on combined spectral difference features that can meet the sound source localization accuracy and real-time calculation requirements of edge AI devices, addressing the aforementioned technical problems.

[0005] A sound source localization method based on combined spectral difference features includes the following steps: The raw waveform data of each channel is acquired, and the raw waveform data sequence of preset length is extracted in the time order of the data stream as data frames. The Hanning window function is used to window each frame of data to obtain a windowed sequence. The windowed sequence is subjected to frequency domain transformation to obtain the frequency domain data of each channel, including frequency domain energy data and frequency domain phase data; The frequency domain data of each channel are combined and matched to identify the calculation unit of the combination difference. The difference calculation is performed on each pair of matched frequency domain data to construct a difference matrix. In the difference matrix, the frequency domain energy data and the frequency domain phase data each occupy half and are diagonally symmetrically distributed. The difference data in the difference matrix is ​​processed by nonlinear pooling; After depth modeling of the data processed by nonlinear pooling, the data is encapsulated and output to obtain sound source location data and sound source distance data.

[0006] In one embodiment, raw waveform data from each channel is acquired, and raw waveform data sequences of a preset length are sequentially extracted as data frames according to the time order of the data stream. A Hanning window function is then used to window each frame of data to obtain a windowed sequence, including: Collect raw waveform data from each channel, and extract raw waveform data sequences of preset lengths in the time order of the data stream as data frames. The Hanning window function is used to window each frame of data, resulting in a windowed sequence: ,

[0007] in, The length of the frame data; For the first The Hanning window function for each frame of data.

[0008] In one embodiment, the frame data length is 2. p , p Let be any real number.

[0009] In one embodiment, the windowed sequence is subjected to frequency domain transformation to obtain frequency domain data for each channel, including frequency domain energy data and frequency domain phase data, including: Perform a Fast Fourier Transform on the windowed sequence to obtain the frequency domain data for each channel, including frequency domain energy data and frequency domain phase data.

[0010] In one embodiment, the frequency domain data of each channel are combined and matched to identify the calculation unit for the combined difference. Difference calculation is then performed on each pair of matched frequency domain data to construct a difference matrix, including: By combining and matching the frequency domain data of each channel, the calculation unit for the combined difference is identified:

[0011] in, This represents the number of computational units for the combined difference. Number of channels; For each pair of matched frequency domain data, perform difference calculations to construct a difference matrix.

[0012] In one embodiment, the difference data in the difference matrix is ​​processed by nonlinear pooling, including: The difference data in the difference matrix is ​​processed using nonlinear pooling:

[0013] in, It is a linear space dimension; The dimension of the logarithmic space; The channel number is the sequence number; Number of channels; This is the result of a logarithmic mapping.

[0014] In one embodiment, the data after nonlinear pooling is depth-modeled and then encapsulated to obtain sound source location data and sound source distance data, including: A two-layer fully connected neural network is used to perform deep modeling on the data after nonlinear pooling. The data after depth modeling is encapsulated and output to obtain sound source location data and sound source distance data.

[0015] A sound source localization device based on combined spectral difference features, the device comprising: The preprocessing module is used to acquire the raw waveform data of each channel, extract the raw waveform data sequence of preset length in the time order of the data stream as data frames, and use the Hanning window function to window each frame of data to obtain the windowed sequence. The frequency domain transformation module is used to perform frequency domain transformation processing on the windowed sequence to obtain the frequency domain data of each channel, including frequency domain energy data and frequency domain phase data; The difference matrix construction module is used to combine and match the frequency domain data of each channel, identify the calculation unit for the combined difference, perform difference calculation on each pair of matched frequency domain data, and construct a difference matrix. In the difference matrix, frequency domain energy data and frequency domain phase data each occupy half and are diagonally symmetrically distributed. The pooling module is used to process the difference data in the difference matrix through non-linear pooling; The output module is used to encapsulate and output the data after nonlinear pooling processing, resulting in sound source location data and sound source distance data.

[0016] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: The raw waveform data of each channel is acquired, and the raw waveform data sequence of preset length is extracted in the time order of the data stream as data frames. The Hanning window function is used to window each frame of data to obtain a windowed sequence. The windowed sequence is subjected to frequency domain transformation to obtain the frequency domain data of each channel, including frequency domain energy data and frequency domain phase data; The frequency domain data of each channel are combined and matched to identify the calculation unit of the combination difference. The difference calculation is performed on each pair of matched frequency domain data to construct a difference matrix. In the difference matrix, the frequency domain energy data and the frequency domain phase data each occupy half and are diagonally symmetrically distributed. The difference data in the difference matrix is ​​processed by nonlinear pooling; After depth modeling of the data processed by nonlinear pooling, the data is encapsulated and output to obtain sound source location data and sound source distance data.

[0017] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: The raw waveform data of each channel is acquired, and the raw waveform data sequence of preset length is extracted in the time order of the data stream as data frames. The Hanning window function is used to window each frame of data to obtain a windowed sequence. The windowed sequence is subjected to frequency domain transformation to obtain the frequency domain data of each channel, including frequency domain energy data and frequency domain phase data; The frequency domain data of each channel are combined and matched to identify the calculation unit of the combination difference. The difference calculation is performed on each pair of matched frequency domain data to construct a difference matrix. In the difference matrix, the frequency domain energy data and the frequency domain phase data each occupy half and are diagonally symmetrically distributed. The difference data in the difference matrix is ​​processed by nonlinear pooling; After depth modeling of the data processed by nonlinear pooling, the data is encapsulated and output to obtain sound source location data and sound source distance data.

[0018] The aforementioned sound source localization method, apparatus, device, and medium based on combined spectral difference features achieve strict temporal alignment of the original acquired data by sequentially extracting pre-defined length sequences of original waveform data as data frames in the time order of the data stream. A Hanning window with low sidelobe levels and linear phase response is then used as a sliding window to maximize the preservation of phase integrity during frequency domain transformation. A dedicated combined differential calculation layer effectively extracts the difference signals in the spectral space of frame data from multiple channels while discarding common-mode information between data points, thus significantly compressing the data volume and providing a foundation for simplified subsequent calculations, making it adaptable to edge AI devices. Furthermore, a logarithmic non-uniform pooling method is employed to retain high-frequency signals rich in sound source directional features while sparsely sampling low-frequency signals, achieving a balance between performance and efficiency. Attached Figure Description

[0019] Figure 1 This is an application scenario diagram of a sound source localization method based on combined spectral difference features in one embodiment; Figure 2 This is a flowchart illustrating a sound source localization method based on combined spectral difference features in one embodiment. Figure 3 This is a neural network structure diagram of a sound source localization method based on combined spectral difference features in one embodiment; Figure 4 This is a schematic diagram illustrating the construction of the difference matrix in one embodiment; Figure 5 This is a schematic diagram of nonlinear pooling in one embodiment; Figure 6 This is a structural block diagram of a sound source localization device based on combined spectral difference features in one embodiment; Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] The sound source localization method based on combined spectral difference features provided in this application can be applied to, for example... Figure 1 In the application environment shown, the input signal processed by this invention originates from multiple channels of sensors 104, with each channel corresponding to one sensor. Sensors 104 acquire signals of air density and material pressure changes caused by the mechanical vibration of the target sound source 102 and convert them into analog electrical signals. After filtering out the DC component and performing digital-to-analog conversion, the analog electrical signals yield 16-bit precision PCM (Pulse Code Modulation) raw waveform data. .

[0022] In one embodiment, such as Figure 2 As shown, a sound source localization method based on combined spectral difference features is provided, and this method is applied to... Figure 1 Taking the sensor in the example, the explanation includes the following steps: Step 201: Collect the raw waveform data of each channel, extract the raw waveform data sequence of preset length in the time order of the data stream as data frames, and use the Hanning window function to window each frame of data to obtain the windowed sequence.

[0023] Step 202: Perform frequency domain transformation on the windowed sequence to obtain frequency domain data for each channel, including frequency domain energy data and frequency domain phase data.

[0024] Step 203: Combine and match the frequency domain data of each channel, confirm the calculation unit of the combination difference, perform difference calculation on each pair of matched frequency domain data, and construct a difference matrix. In the difference matrix, frequency domain energy data and frequency domain phase data each occupy half and are diagonally symmetrically distributed.

[0025] Step 104: Process the difference data in the difference matrix using nonlinear pooling.

[0026] Step 205: After depth modeling the data processed by nonlinear pooling, encapsulate and output the data to obtain the sound source orientation data and sound source distance data.

[0027] In the aforementioned sound source localization method based on combined spectral difference features, the original waveform data sequence of a preset length is sequentially extracted as data frames according to the time order of the data stream, thereby strictly aligning the original acquired data in time. A Hanning window with low sidelobe level and linear phase response is then used as a sliding window to maximize the preservation of phase distortion during frequency domain transformation. By setting up a dedicated combined differential calculation layer, the difference signals in the spectral space of frame data from multiple channels are effectively extracted, while common-mode information between data is discarded. This significantly compresses the data volume, providing a foundation for subsequent simplified calculations and making it adaptable to edge AI devices. Furthermore, a logarithmic non-uniform pooling method is employed to retain high-frequency signals rich in sound source directional features, while sparsely sampling low-frequency signals, achieving a balance between performance and efficiency.

[0028] In one embodiment, raw waveform data from each channel is acquired, and raw waveform data sequences of a preset length are sequentially extracted as data frames according to the time order of the data stream. A Hanning window function is then used to window each frame of data to obtain a windowed sequence, including: The raw waveform data of each channel is acquired, and the raw waveform data sequence of preset length is extracted sequentially in the time order of the data stream as a data frame.

[0029] The Hanning window function is used to window each frame of data, resulting in a windowed sequence: ,

[0030] in, The length of the frame data; For the first The Hanning window function for each frame of data.

[0031] In this embodiment, the Hanning window function is selected because the Hanning window is a centrally symmetric cosine window with good linear phase characteristics, which can ensure that the phase of the signal after processing by the window function is not distorted. At the same time, the fast sidelobe attenuation characteristics of the Hanning window make it perform well in suppressing noise and interference. When processing signals containing multiple frequency components, it can reduce the mutual interference between different frequency components and improve the accuracy of spectrum analysis.

[0032] In this embodiment, the real number sequence data of the Hanning window is as follows: [0.00000000e+00, 1.51774011e-04, 6.07003903e-04, 1.36541331e-03, 2.42654180e-03, 3.78974516e-03, 5.45419581e-03, 7.41888327e-03, 9.68261477e-03, 1.22440160e-02, 1.51015320e-02, 1.82534279e-02, 2.16977902e-02, 2.54325280e-02, 2.94553737e-02, 3.37638853e-02, 3.83554469e-02, 4.32272712e-02, 4.83764003e-02, 5.37997084e-02, 5.94939029e-02, 6.54555268e-02, 7.16809611e-02, 7.81664261e-02, 8.49079846e-02, 9.19015438e-02, 9.91428580e-02, 1.06627531e-01, 1.14351019e-01, 1.22308633e-01, 1.30495541e-01, 1.38906775e-01, 1.47537227e-01, 1.56381657e-01, 1.65434697e-01, 1.74690850e-01, 1.84144497e-01, 1.93789898e-01, 2.03621199e-01, 2.13632430e-01, 2.23817514e-01, 2.34170266e-01, 2.44684403e-01, 2.55353542e-01, 2.66171204e-01, 2.77130822e-01, 2.88225744e-01, 2.99449233e-01, 3.10794475e-01, 3.22254583e-01, 3.33822600e-01, 3.45491503e-01, 3.57254207e-01, 3.69103571e-01, 3.81032402e-01, 3.93033458e-01, 4.05099453e-01, 4.17223062e-01, 4.29396924e-01, 4.41613649e-01, 4.53865820e-01, 4.66145999e-01, 4.78446731e-01, 4.90760548e-01, 5.03079973e-01, 5.15397529e-01, 5.27705737e-01, 5.39997126e-01, 5.52264232e-01, 5.64499608e-01, 5.76695827e-01, 5.88845485e-01, 6.00941205e-01, 6.12975643e-01, 6.24941495e-01, 6.36831495e-01, 6.48638425e-01, 6.60355118e-01, 6.71974459e-01, 6.83489396e-01, 6.94892937e-01, 7.06178159e-01, 7.17338211e-01, 7.28366318e-01, 7.39255785e-01, 7.50000000e-01, 7.60592441e-01, 7.71026678e-01, 7.81296376e-01, 7.91395299e-01, 8.01317318e-01, 8.11056408e-01, 8.20606657e-01, 8.29962267e-01, 8.39117559e-01, 8.48066973e-01, 8.56805077e-01, 8.65326566e-01, 8.73626267e-01, 8.81699141e-01, 8.89540287e-01, 8.97144945e-01, 9.04508497e-01, 9.11626474e-01, 9.18494554e-01, 9.25108568e-01, 9.31464500e-01, 9.37558491e-01, 9.43386843e-01, 9.48946016e-01, 9.54232636e-01, 9.59243493e-01, 9.63975545e-01, 9.68425919e-01, 9.72591914e-01, 9.76471000e-01, 9.80060823e-01, 9.83359202e-01, 9.86364136e-01, 9.89073800e-01, 9.91486550e-01, 9.93600920e-01, 9.95415627e-01, 9.96929568e-01, 9.98141826e-01, 9.99051664e-01, 9.99658530e-01, 9.99962055e-01, 9.99962055e-01, 9.99658530e-01, 9.99051664e-01, 9.98141826e-01, 9.96929568e-01, 9.95415627e-01, 9.93600920e-01, 9.91486550e-01, 9.89073800e-01, 9.86364136e-01, 9.83359202e-01, 9.80060823e-01, 9.76471000e-01, 9.72591914e-01, 9.68425919e-01, 9.63975545e-01, 9.59243493e-01, 9.54232636e-01, 9.48946016e-01, 9.43386843e-01, 9.37558491e-01, 9.31464500e-01, 9.25108568e-01, 9.18494554e-01, 9.11626474e-01, 9.04508497e-01, 8.97144945e-01, 8.89540287e-01, 8.81699141e-01, 8.73626267e-01, 8.65326566e-01, 8.56805077e-01, 8.48066973e-01, 8.39117559e-01, 8.29962267e-01, 8.20606657e-01, 8.11056408e-01, 8.01317318e-01, 7.91395299e-01, 7.81296376e-01, 7.71026678e-01, 7.60592441e-01, 7.50000000e-01, 7.39255785e-01, 7.28366318e-01, 7.17338211e-01, 7.06178159e-01, 6.94892937e-01, 6.83489396e-01, 6.71974459e-01, 6.60355118e-01, 6.48638425e-01, 6.36831495e-01, 6.24941495e-01, 6.12975643e-01, 6.00941205e-01, 5.88845485e-01, 5.76695827e-01, 5.64499608e-01, 5.52264232e-01, 5.39997126e-01, 5.27705737e-01, 5.15397529e-01, 5.03079973e-01, 4.90760548e-01, 4.78446731e-01, 4.66145999e-01, 4.53865820e-01, 4.41613649e-01, 4.29396924e-01, 4.17223062e-01, 4.05099453e-01, 3.93033458e-01, 3.81032402e-01, 3.69103571e-01, 3.57254207e-01, 3.45491503e-01, 3.33822600e-01, 3.22254583e-01, 3.10794475e-01, 2.99449233e-01, 2.88225744e-01, 2.77130822e-01, 2.66171204e-01, 2.55353542e-01, 2.44684403e-01, 2.34170266e-01, 2.23817514e-01, 2.13632430e-01, 2.03621199e-01, 1.93789898e-01, 1.84144497e-01, 1.74690850e-01, 1.65434697e-01, 1.56381657e-01, 1.47537227e-01, 1.38906775e-01, 1.30495541e-01, 1.22308633e-01, 1.14351019e-01, 1.06627531e-01, 9.91428580e-02, 9.19015438e-02, 8.49079846e-02, 7.81664261e-02, 7.16809611e-02, 6.54555268e-02, 5.94939029e-02, 5.37997084e-02, 4.83764003e-02, 4.32272712e-02, 3.83554469e-02, 3.37638853e-02, 2.94553737e-02, 2.54325280e-02, 2.16977902e-02, 1.82534279e-02, 1.51015320e-02, 1.22440160e-02, 9.68261477e-03, 7.41888327e-03, 5.45419581e-03, 3.78974516e-03, 2.42654180e-03, 1.36541331e-03, 6.07003903e-04, 1.51774011e-04, 0.00000000e+00]) As can be seen from the parameters of the Hanning window above, the data at the beginning and end of the frame will inevitably decay to 0, and there will be no rigid truncation. The frame data after windowing is represented as follows.

[0033] In one embodiment, the frame data length is 2. p , p Let be any real number.

[0034] In this embodiment, the frame data length is set to a power of 2 to facilitate subsequent accelerated calculations. In this embodiment, the length of each frame data is fixed at 256, that is, the sampling frequency of the original data is set to 44.1kHz, and the time domain length is 5.8 milliseconds.

[0035] In one embodiment, the windowed sequence is subjected to frequency domain transformation to obtain frequency domain data for each channel, including frequency domain energy data and frequency domain phase data, including: Perform a Fast Fourier Transform on the windowed sequence to obtain the frequency domain data for each channel, including frequency domain energy data and frequency domain phase data.

[0036] In this embodiment, the classic Cooley-Tukey time-domain decimation algorithm is used to perform Fast Fourier Transform (FFT) calculations on the windowed sequence. The Cooley-Tukey time-domain decimation algorithm is a well-tested and mature algorithm. It cleverly utilizes a divide-and-conquer strategy to perform FFT calculations on the windowed sequence of length... N The Discrete Fourier Transform (DFT) is decomposed into smaller DFTs, significantly reducing computational complexity. In the actual computation process, the algorithm fully utilizes the characteristics of butterfly operations and twitch factors, completing the FFT calculation through iterative iterations. The butterfly operation, as the core computational unit, reduces unnecessary computation with its efficient computational structure; while the twitch factor establishes a close connection between the frequency and time domains, providing strong support for achieving efficient frequency domain transformation.

[0037] In this embodiment, the sequence obtained after performing a Fast Fourier Transform on the windowed sequence is denoted as . Each element of the sequence is a vector of dimension 128.

[0038] In one embodiment, reference Figure 4 The frequency domain data of each channel are combined and matched to determine the calculation unit for the combined difference. Difference calculations are then performed on each pair of matched frequency domain data to construct a difference matrix, including: By combining and matching the frequency domain data of each channel, the calculation unit for the combined difference is identified:

[0039] in, This represents the number of computational units for the combined difference. This represents the number of channels.

[0040] For each pair of matched frequency domain data, perform difference calculations to construct a difference matrix.

[0041] In this embodiment, the number of channels is 4, corresponding to 6 computational units for combined differential calculations, meaning there are 6 computational units for combined differential calculations. After confirming the computational units for combined differential calculations, differential calculations are performed on each pair of matched frequency domain data to construct a difference matrix, as follows: Figure 4 As shown, the frequency domain energy data and frequency domain phase data each occupy half of the difference matrix, and are distributed diagonally symmetrically.

[0042] In one embodiment, the difference data in the difference matrix is ​​processed by nonlinear pooling, including: The difference data in the difference matrix is ​​processed using nonlinear pooling:

[0043] in, It is a linear space dimension; The dimension of the logarithmic space; The channel number is the sequence number; Number of channels; This is the result of a logarithmic mapping.

[0044] In this embodiment, nonlinear pooling is used to process the difference data in the difference matrix, thereby reducing the network parameter size while maximizing the preservation of useful information. (Refer to...) Figure 5 By using nonlinear pooling, 128-dimensional vector difference data is mapped to a 32-dimensional logarithmic space.

[0045] In one embodiment, the data after nonlinear pooling is depth-modeled and then encapsulated to obtain sound source location data and sound source distance data, including: A two-layer fully connected neural network is used to perform deep modeling on the data after nonlinear pooling.

[0046] The data after depth modeling is encapsulated and output to obtain sound source location data and sound source distance data.

[0047] In this embodiment, a classic two-layer fully connected neural network is used as the hidden layer to perform deep modeling on the data after nonlinear pooling. The nodes in the first hidden layer are divided into two categories: H nodes (representing linear nodes) and K nodes (representing nonlinear nodes). K nodes execute functions such as Max and Min.

[0048] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0049] Reference Figure 3 The diagram shows the neural network structure of a sound source localization method based on combined spectral difference features. Only four channels are shown in the diagram; in reality, the number of channels can reach 16 or even more. In the neural network structure diagram, following the order of the signal forward propagation process, the layers are: input layer, preprocessing layer, frequency domain transformation layer, combined difference layer, pooling layer, fully connected layer, and output layer. Each layer corresponds to a specific calculation or processing step.

[0050] The circular symbols in the diagram represent data formats at specific stages of the neural network. These represent the input data for channels 1, 2, 3, and n, respectively. The data type is pulse code modulation (PCM) data with a sampling frequency of 44.1kHz, the data length is 256, and the data format is a 16-bit integer. symbol and symbols These represent the frequency domain energy data and frequency domain phase data of the corresponding channels, respectively. The data length is 128, and the data format is 32-bit floating-point numbers. symbol This represents six vector data points after pooling. Each vector data point can be divided into energy data and phase data, with a data length of 64 and a data format of 32-bit floating-point numbers. Therefore, it can be seen that... Figure 3 The data size input to the fully connected layer in the network structure shown is A number of floating-point numbers; symbol Represents the final output data, where Represents the location information of the sound source. This represents the distance information to the sound source, and the data format is 32-bit floating-point numbers.

[0051] In one embodiment, such as Figure 6 As shown, a sound source localization device based on combined spectral difference features is provided, including: a preprocessing module, a frequency domain transformation module, a difference matrix construction module, a pooling module, and an output module.

[0052] The preprocessing module 901 is used to acquire the raw waveform data of each channel, extract the raw waveform data sequence of preset length in the time order of the data stream as data frames, and use the Hanning window function to window each frame of data to obtain a windowed sequence.

[0053] The frequency domain transformation module 902 is used to perform frequency domain transformation processing on the windowed sequence to obtain the frequency domain data of each channel, including frequency domain energy data and frequency domain phase data.

[0054] The difference matrix construction module 903 is used to combine and match the frequency domain data of each channel, confirm the calculation unit of the combined difference, perform difference calculation on each pair of matched frequency domain data, and construct a difference matrix. In the difference matrix, the frequency domain energy data and the frequency domain phase data each occupy half and are diagonally symmetrically distributed.

[0055] Pooling module 904 is used to process the difference data in the difference matrix through non-linear pooling.

[0056] The output module 905 is used to encapsulate and output the data after nonlinear pooling processing, thereby obtaining the sound source orientation data and sound source distance data.

[0057] In one embodiment, raw waveform data from each channel is acquired, and raw waveform data sequences of a preset length are sequentially extracted as data frames according to the time order of the data stream. A Hanning window function is then used to window each frame of data to obtain a windowed sequence, including: The raw waveform data of each channel is acquired, and the raw waveform data sequence of preset length is extracted sequentially in the time order of the data stream as a data frame.

[0058] The Hanning window function is used to window each frame of data, resulting in a windowed sequence: ,

[0059] in, The length of the frame data; For the first The Hanning window function for each frame of data.

[0060] In one embodiment, the frame data length is 2. p , p Let be any real number.

[0061] In one embodiment, the windowed sequence is subjected to frequency domain transformation to obtain frequency domain data for each channel, including frequency domain energy data and frequency domain phase data, including: Perform a Fast Fourier Transform on the windowed sequence to obtain the frequency domain data for each channel, including frequency domain energy data and frequency domain phase data.

[0062] In one embodiment, reference Figure 4 The frequency domain data of each channel are combined and matched to determine the calculation unit for the combined difference. Difference calculations are then performed on each pair of matched frequency domain data to construct a difference matrix, including: By combining and matching the frequency domain data of each channel, the calculation unit for the combined difference is identified:

[0063] in, This represents the number of computational units for the combined difference. This represents the number of channels.

[0064] For each pair of matched frequency domain data, perform difference calculations to construct a difference matrix.

[0065] In one embodiment, the difference data in the difference matrix is ​​processed by nonlinear pooling, including: The difference data in the difference matrix is ​​processed using nonlinear pooling:

[0066] in, It is a linear space dimension; The dimension of the logarithmic space; The channel number is the sequence number; Number of channels; This is the result of a logarithmic mapping.

[0067] In one embodiment, the data after nonlinear pooling is depth-modeled and then encapsulated to obtain sound source location data and sound source distance data, including: A two-layer fully connected neural network is used to perform deep modeling on the data after nonlinear pooling.

[0068] The data after depth modeling is encapsulated and output to obtain sound source location data and sound source distance data.

[0069] Specific limitations regarding the sound source localization device based on combined spectral difference features can be found in the limitations of the sound source localization method based on combined spectral difference features mentioned above, and will not be repeated here. Each module in the aforementioned sound source localization device based on combined spectral difference features can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0070] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a sound source localization method based on combined spectral difference characteristics. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0071] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0072] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0073] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Step 201: Collect the raw waveform data of each channel, extract the raw waveform data sequence of preset length in the time order of the data stream as data frames, and use the Hanning window function to window each frame of data to obtain the windowed sequence.

[0074] Step 202: Perform frequency domain transformation on the windowed sequence to obtain frequency domain data for each channel, including frequency domain energy data and frequency domain phase data.

[0075] Step 203: Combine and match the frequency domain data of each channel, confirm the calculation unit of the combination difference, perform difference calculation on each pair of matched frequency domain data, and construct a difference matrix. In the difference matrix, frequency domain energy data and frequency domain phase data each occupy half and are diagonally symmetrically distributed.

[0076] Step 204: Process the difference data in the difference matrix using nonlinear pooling.

[0077] Step 205: After depth modeling the data processed by nonlinear pooling, encapsulate and output the data to obtain the sound source orientation data and sound source distance data.

[0078] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0079] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A sound source localization method based on combined spectral difference features, characterized in that, Includes the following steps: The raw waveform data of each channel is acquired, and the raw waveform data sequence of preset length is extracted in the time order of the data stream as data frames. The Hanning window function is used to window each frame of data to obtain a windowed sequence. The windowed sequence is subjected to frequency domain transformation to obtain the frequency domain data of each channel, including frequency domain energy data and frequency domain phase data; The frequency domain data of each channel are combined and matched to identify the calculation unit of the combination difference. The difference calculation is performed on each pair of matched frequency domain data to construct a difference matrix. In the difference matrix, the frequency domain energy data and the frequency domain phase data each occupy half and are diagonally symmetrically distributed. The difference data in the difference matrix is ​​processed by nonlinear pooling; After depth modeling of the data processed by nonlinear pooling, the data is encapsulated and output to obtain sound source location data and sound source distance data.

2. The method according to claim 1, characterized in that, The raw waveform data from each channel is acquired, and a preset length of raw waveform data sequence is sequentially extracted from the data stream in chronological order as data frames. The Hanning window function is then used to window each frame of data, resulting in a windowed sequence, including: Collect raw waveform data from each channel, and extract raw waveform data sequences of preset lengths in the time order of the data stream as data frames. The Hanning window function is used to window each frame of data, resulting in a windowed sequence: ， in, The length of the frame data; For the first The Hanning window function for each frame of data.

3. The method according to claim 2, characterized in that, The frame data length is 2 p , p Let be any real number.

4. The method according to claim 1, characterized in that, The windowed sequence is subjected to frequency domain transformation to obtain frequency domain data for each channel, including frequency domain energy data and frequency domain phase data, including: Perform a Fast Fourier Transform on the windowed sequence to obtain the frequency domain data for each channel, including frequency domain energy data and frequency domain phase data.

5. The method according to claim 1, characterized in that, The frequency domain data of each channel are combined and matched to identify the calculation unit for the combined difference. Difference calculations are then performed on each pair of matched frequency domain data to construct a difference matrix, including: By combining and matching the frequency domain data of each channel, the calculation unit for the combined difference is identified: in, This represents the number of computational units for the combined difference. Number of channels; For each pair of matched frequency domain data, perform difference calculations to construct a difference matrix.

6. The method according to claim 1, characterized in that, Nonlinear pooling is used to process the difference data in the difference matrix, including: The difference data in the difference matrix is ​​processed using nonlinear pooling: in, It is a linear space dimension; The dimension of the logarithmic space; The channel number is the sequence number; Number of channels; This is the result of a logarithmic mapping.

7. The method according to claim 1, characterized in that, After depth modeling and encapsulation of the data processed by nonlinear pooling, the sound source location data and sound source distance data are obtained, including: A two-layer fully connected neural network is used to perform deep modeling on the data after nonlinear pooling. The data after depth modeling is encapsulated and output to obtain sound source location data and sound source distance data.

8. A sound source localization device based on combined spectral difference characteristics, characterized in that, The device includes: The preprocessing module is used to acquire the raw waveform data of each channel, extract the raw waveform data sequence of preset length in the time order of the data stream as data frames, and use the Hanning window function to window each frame of data to obtain the windowed sequence. The frequency domain transformation module is used to perform frequency domain transformation processing on the windowed sequence to obtain the frequency domain data of each channel, including frequency domain energy data and frequency domain phase data; The difference matrix construction module is used to combine and match the frequency domain data of each channel, identify the calculation unit for the combined difference, perform difference calculation on each pair of matched frequency domain data, and construct a difference matrix. In the difference matrix, frequency domain energy data and frequency domain phase data each occupy half and are diagonally symmetrically distributed. The pooling module is used to process the difference data in the difference matrix through non-linear pooling; The output module is used to encapsulate and output the data after nonlinear pooling processing, resulting in sound source location data and sound source distance data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images