Sound source positioning method and device, equipment, storage medium and program product

Abstract

The invention discloses a sound source positioning method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining M pieces of channel data, the M pieces of channel data correspond to the audio signals of M channels collected by a microphone array, and M is a positive integer; performing framing windowing processing on the M channel data to obtain N frames of time domain signals, N being a positive integer; determining a time domain covariance vector based on the N frames of time domain signals; and determining the orientation of the sound source based on the time domain covariance vector through a sound source localization model.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a sound source localization method, device, equipment, storage medium and program product. Background Art

[0002] Sound source localization technology is a technology that uses the sound wave signals received by the sensor array to determine the location of the sound source. It is widely used in various fields such as security and industry. It has the characteristics of not being restricted by line of sight conditions, strong concealment, and not being restricted by visibility. However, the sound source localization method in the related technology is still easily affected by noise interference, and has problems such as poor robustness and high computational complexity, resulting in inaccurate positioning results. Summary of the invention

[0003] To solve the above technical problems, embodiments of the present invention provide a sound source localization method, apparatus, device, storage medium and program product.

[0004] The sound source localization method provided in the embodiment of the present application includes:

[0005] Acquire M channel data, where the M channel data correspond to the audio signals of M channels collected by the microphone array, where M is a positive integer;

[0006] Performing frame division and windowing processing on the M channel data to obtain N frames of time domain signals, where N is a positive integer;

[0007] Based on the N frames of time domain signals, determining a time domain covariance vector;

[0008] The direction of the sound source is determined based on the time domain covariance vector by a sound source localization model.

[0009] The sound source localization device provided in the embodiment of the present application includes:

[0010] A collection unit, used to obtain M channel data, where the M channel data correspond to the audio signals of the M channels collected by the microphone array, where M is a positive integer;

[0011] A processing unit is used to perform frame-by-frame windowing processing on the M channel data to obtain N frames of time domain signals, where N is a positive integer; and determine a time domain covariance vector based on the N frames of time domain signals;

[0012] A determination unit is used to determine the direction of the sound source based on the time domain covariance vector through a sound source localization model.

[0013] The processing device provided in the embodiment of the present application includes: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute any one of the above-mentioned sound source localization methods.

[0014] The computer-readable storage medium provided in the embodiments of the present application is used to store a computer program, wherein the computer program enables a computer to execute any one of the above-mentioned sound source localization methods.

[0015] The computer program product provided in the embodiments of the present application includes computer program instructions, which enable a computer to execute any one of the above-mentioned sound source localization methods.

[0016] In the technical solution of the embodiment of the present application, by acquiring M channel data, the M channel data is framed and windowed to obtain N frames of time domain signals, and based on the N frames of time domain signals, the time domain covariance vector is determined, so that the direction of the sound source is determined based on the time domain covariance vector through the sound source localization model, wherein the M channel data corresponds to the audio signals of the M channels collected by the microphone array. In this way, by segmenting the multi-channel data collected by the microphone array and selecting a suitable window function for windowing processing, it is possible to reduce spectrum leakage, improve the accuracy of spectrum analysis, and improve the signal-to-noise ratio of the signal to a certain extent, and by converting the obtained multi-frame time domain signals into time domain covariance vectors, it is possible to effectively utilize the time correlation in the signal, and by capturing and analyzing the phase and amplitude changes between the signals, the efficiency and accuracy of signal processing are effectively improved, so that the direction of the sound source can be accurately determined based on the time domain covariance vector while improving the robustness through the sound source localization model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a sound source localization method provided in an embodiment of the present application;

[0018] Figure 2 It is a specific flow chart of another sound source localization method provided by an embodiment of the present application;

[0019] Figure 3 is a structural schematic diagram of a sound source localization device provided in an embodiment of the present application;

[0020] Figure 4 It is a structural schematic diagram of a processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0023] Sound source localization technology is a technology that uses multiple microphones or acoustic sensors to measure sound signals at different locations in the environment, and obtains the arrival direction (including azimuth, pitch angle) and distance of the sound source point relative to the microphone through the measured sound signals. It is widely used in various fields such as security and industry, and the sound source localization technology has the characteristics of not being restricted by line of sight conditions, strong concealment, and not being restricted by visibility. Among them, the sound source localization methods in related technologies are mainly the following:

[0024] (1) Positioning method based on controllable beamforming with maximum output power: By adjusting the weight of the sensor array, the output power in a specific direction is maximized, thereby forming a beam pointing to the sound source.

[0025] (2) Positioning methods based on high-resolution spectrum estimation: such as the MUSIC algorithm, which uses the orthogonality of the signal subspace and the noise subspace to determine the direction of the sound source by searching for spectral peaks in the spatial spectrum.

[0026] (3) Positioning method based on Time Difference of Arrival (TDOA) estimation: The location of the sound source is estimated by measuring the time difference between the sound waves arriving at different sensors.

[0027] (4) Machine learning-based methods: such as deep learning, which improves positioning accuracy by learning the complex relationship between sound sources and the environment.

[0028] However, the above-mentioned sound source localization method still has the following problems:

[0029] (1) The first sound source localization method mentioned above is greatly affected by array calibration errors, and its performance may be degraded in a multi-sound source environment. At the same time, this method is sensitive to noise and reverberation, which may lead to inaccurate positioning.

[0030] (2) The second sound source localization method has strict assumptions on the signal and noise model. Moreover, in the case of low signal-to-noise ratio or coherent signal, the performance of this method may decline and the computational complexity is high.

[0031] (3) The third sound source localization method has high requirements for time synchronization. A small synchronization error may lead to a large positioning error. Moreover, in a multipath environment, TDOA estimation may be inaccurate due to reflection and refraction. In addition, this method is also sensitive to environmental noise and reverberation.

[0032] (4) Regarding the fourth sound source localization method mentioned above, it requires a large amount of labeled data to train the model. However, the generalization ability of the model may be affected by the distribution difference between the training data and the test data. Therefore, if the data distribution difference is large, it will affect the generalization ability and training accuracy of the model. In addition, this method may also have difficulties in explaining the decision-making process of the machine learning model.

[0033] In order to solve the above technical problems, the present application proposes a sound source localization method based on covariance matrix and deep neural network. By combining the time domain covariance matrix and deep neural network for sound source localization, combined with the good robustness and algorithm accuracy of the deep neural network and the low computational complexity of the time domain covariance matrix, it can solve the robustness problem of being easily affected by noise interference, the problem of poor positioning accuracy and the problem of high computational complexity in the sound source localization process.

[0034] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0035] The present application embodiment provides a sound source localization method. Figure 1 is a flow chart of the sound source localization method provided in the embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0036] Step 101: Obtain M channel data, where M is a positive integer.

[0037] In the embodiment of the present application, the M channel data corresponds to the audio signals of the M channels collected by the microphone array, which is an array composed of multiple microphones located at different positions in space and arranged in a certain shape, and is usually used for collecting and processing audio signals. Therefore, the M channel data can be collected and obtained by the microphone array. The microphone array is specifically an acoustic sensor array.

[0038] Here, the microphone array collects signals containing the location information of the sound source by spatially sampling the sound signal propagated by the sound source. According to the distance between the sound source and the microphone array, the array can be divided into a near-field model and a far-field model. In general, when the distance between the sound source and the central reference point of the microphone array is greater than the signal wavelength, it can be regarded as a far field, otherwise it is regarded as a near field. According to the topological structure of the microphone array, the array can be divided into a linear array, a planar array, a stereo array, etc. Among them, the linear array is a one-dimensional microphone array, whose array element centers are located on the same straight line, and the linear array can only obtain the horizontal direction angle information of the signal; the planar array is a two-dimensional microphone array, whose array element centers are distributed on a plane, and the planar array can obtain the horizontal direction angle and vertical direction angle information of the signal; the stereo array is a three-dimensional microphone array, whose array element centers are distributed in a three-dimensional space, and the stereo array can obtain the horizontal direction angle, vertical direction angle of the signal and the distance between the sound source and the central reference point of the microphone array.

[0039] Here, the microphone array can also be used to solve the problem of noise interference. By adopting technologies such as multi-channel reverberation reduction algorithm and multi-channel point interference suppression algorithm, signal processing is performed on the multi-channel data collected by the microphone array, which can effectively remove noise and interference to improve the accuracy of signal recognition.

[0040] Based on this, in some implementations, the implementation of step 101 may specifically include:

[0041] Select M microphones to form a microphone array, where the M microphones are distributed at different positions; the microphone array is used to collect sound signals transmitted by a sound source;

[0042] Get M channels of data from the microphone array.

[0043] Here, M microphones distributed at different positions in space are selected to form a microphone array, and each microphone collects an audio signal of one channel. Accordingly, the M microphones collect audio signals of M channels. The process of recording using a microphone array is as follows: when the recording device is turned on, the recording operation can be started, and the signal propagated by the sound source during recording is collected by the microphone array until the recording operation is completed. At this time, the recording device is turned off, and the audio signals of M channels collected by the M microphones are obtained from the microphone array, that is, M channel data.

[0044] Step 102: Perform frame division and windowing processing on the M channel data to obtain N frames of time domain signals, where N is a positive integer.

[0045] In an embodiment of the present application, after acquiring M channel data, in order to further improve the accuracy of signal recognition, the M channel data can be framed and windowed to obtain N frames of time domain signals, that is, by dividing the M channel data into a series of overlapping short-time frame signals, and selecting a suitable window function for each frame signal to perform windowing processing, N frames of time domain signals are obtained.

[0046] Here, the windowing process can reduce the sudden changes at both ends of the frame by multiplying the selected window function with the frame signal, making the frame edge transition smoothly, which helps to reduce spectrum leakage. Among them, commonly used window functions include Hamming window, Hanning window, rectangular window, etc. The specific window function selected depends on the specific application scenario and is not limited here.

[0047] Based on this, in some implementations, the implementation of step 102 may specifically include:

[0048] The M channel data are divided by a preset frame length and a preset frame shift to obtain N overlapping frame signals;

[0049] Windowing is performed on each frame signal of the N overlapping frame signals to obtain N frames of time domain signals.

[0050] Here, the frame length represents the number of sample points contained in each frame, that is, the duration of each frame signal, and the frame shift represents the overlapping part between two adjacent frames, that is, the time difference between the end of one frame and the beginning of the next frame. The frame length and frame shift are two key parameters in the signal framing process. The frame length determines how many sample points each frame contains, and the frame shift determines the degree of overlap between frames. The M channels are segmented by the preset frame length and the preset frame shift to obtain N overlapping frame signals, so that the duration of each frame signal in the N overlapping frame signals is the preset frame length, and the time difference between each frame signal is the preset frame shift. A window function is applied to each frame signal in the N overlapping frame signals to obtain a series of windowed time domain framed signals (N frame time domain signals).

[0051] Take a channel in multi-channel data as an example. Assuming that the signal of this channel is continuous, it needs to be segmented into a series of frames. The length of each frame (frame length) is usually 20 to 30 milliseconds, and there are overlapping parts between adjacent frames. The overlapping part is the frame shift, and its length is usually half or one-third of the frame length. For example, if the frame length is 25 milliseconds and the frame shift is 10 milliseconds, then for the signal of each channel, the first 25 millisecond segment will be extracted from the beginning of the signal as the first frame, and then the next 25 millisecond segment will be extracted 10 milliseconds later as the second frame, and so on, until all signals of the channel are completely segmented. After the segmentation is completed, a window function is selected and applied to each frame. The window function is a function that gradually decreases to zero at the beginning and end of the frame. By multiplying the window function with each frame, a smooth transition can be introduced at the edge of the frame, thereby reducing the interference of high-frequency components. For example, when using the Hamming window in the window function, the window function value of each frame is calculated and multiplied with the corresponding frame data to obtain the windowed frame data.

[0052] It should be noted that each channel data in the multi-channel data is independently subjected to frame segmentation and windowing processing, and each channel after processing will contain a series of windowed frame signals for further signal analysis.

[0053] Step 103: Determine a time domain covariance vector based on N frames of time domain signals.

[0054] In an embodiment of the present application, in order to further capture the linear correlation between different frame signals, after obtaining N frames of time domain signals, the time domain covariance matrix corresponding to each frame of the time domain signal can be calculated first, and then the calculated N time domain covariance matrices are vectorized to obtain a time domain covariance vector. Among them, the covariance matrix is ​​a symmetric matrix, each element of which is the covariance between each vector element.

[0055] Here, the eigenvalues ​​and eigenvectors of the time domain covariance matrix corresponding to each frame of the time domain signal represent the energy distribution of the signal in various directions in space. By calculating the eigenvalues ​​and eigenvectors of the time domain covariance matrix, some specific directions of the signal in space and the energy of the signal in these specific directions can be found.

[0056] Based on this, in some implementations, the implementation of step 103 may specifically include:

[0057] Based on N frames of time domain signals, determine N time domain covariance matrices;

[0058] A time domain covariance vector is determined based on the N time domain covariance matrices.

[0059] Based on this, in some implementations, each frame of the N-frame time domain signal has M channels and L sample points, where L is a positive integer; for determining N time domain covariance matrices based on the N-frame time domain signal, it may specifically include:

[0060] The signal matrix of each frame of time domain signal is composed of M channels as M row dimensions and L sample points as L column dimensions;

[0061] Based on the signal matrix of each frame of the time domain signal, a time domain covariance matrix of each frame of the time domain signal is determined.

[0062] Here, since the N frames of time domain signals are obtained by performing frame windowing processing on the M channel data, the frame windowing processing does not change the number of channels of each frame of the time domain signal, so each frame of the N frames of time domain signals has M channels, and correspondingly, each channel in the M channels has L sample points. Based on this, for each frame of the time domain signal, the MxL signal matrix corresponding to each frame of the time domain signal is formed with M channels as rows and L sample points as columns, and then the corresponding time domain covariance matrix is ​​calculated through the MxL signal matrix corresponding to each frame of the time domain signal.

[0063] Based on this, in some implementations, for a signal matrix based on each frame of the time domain signal, determining the time domain covariance matrix of each frame of the time domain signal may specifically include:

[0064] For each frame of the time domain signal, based on each element of each row in the signal matrix of the frame of the time domain signal, determine the average value of each row;

[0065] Subtract the mean value of the corresponding row from each element in each row to obtain the signal matrix after the mean is normalized to zero;

[0066] Based on the signal matrix after mean-zeroing, the time domain covariance matrix of the frame time domain signal is determined, each element in the time domain covariance matrix represents the covariance between the i-th channel and the j-th channel in the M channels, and i and j are positive integers less than or equal to M.

[0067] Here, first initialize the covariance matrix, that is, create a zero matrix of size M x M to store the covariance value. For the M x L ​​signal matrix corresponding to each frame of the time domain signal, calculate the mean of the sample points corresponding to each channel (each row) in the signal matrix. Then, in the signal matrix, subtract the mean of the sample points in the corresponding row from each element in each row to perform mean zeroing. Finally, in the signal matrix after mean zeroing, for each pair of channels (i, j), calculate the covariance value between the i-th channel and the j-th channel, and fill the calculated covariance value into the corresponding position in the above M x M zero matrix.

[0068] For example, if M is 3, it is necessary to calculate the covariance values ​​between the 1st channel and the 1st channel, the 1st channel and the 2nd channel, the 1st channel and the 3rd channel, the 2nd channel and the 1st channel, the 2nd channel and the 2nd channel, the 2nd channel and the 3rd channel, the 3rd channel and the 1st channel, the 3rd channel and the 2nd channel, and the 3rd channel and the 3rd channel respectively. Among them, the position of the covariance value between the first channel and the first channel in the 3x3 zero matrix is ​​(1,1), the position of the covariance value between the first channel and the second channel is (1,2), the position of the covariance value between the first channel and the third channel is (1,3), the position of the covariance value between the second channel and the first channel is (2,1), the position of the covariance value between the second channel and the second channel is (2,2), the position of the covariance value between the second channel and the third channel is (2,3), the position of the covariance value between the third channel and the first channel is (3,1), the position of the covariance value between the third channel and the second channel is (3,2), and the position of the covariance value between the third channel and the third channel is (3,3). It can be seen that the covariance matrix is ​​a symmetric matrix, and the element values ​​of its upper triangle and lower triangle correspond to the same.

[0069] Based on this, in some implementations, determining the time domain covariance vector based on N time domain covariance matrices may specifically include:

[0070] Vectorizing the lower triangular part or the upper triangular part of each of the N time-domain covariance matrices to obtain N time-domain covariance matrix vectors;

[0071] The N time-domain covariance matrix vectors are concatenated to obtain a time-domain covariance vector.

[0072] Here, after calculating N time domain covariance matrices, since each time domain covariance matrix is ​​a symmetric matrix, the element values ​​of the upper triangle and the lower triangle correspond to the same, the lower triangle or upper triangle part of each time domain covariance matrix can be vectorized to obtain the vector corresponding to each time domain covariance matrix, and the vectors corresponding to each time domain covariance matrix are spliced ​​to obtain the time domain covariance vector corresponding to N frames of time domain signals.

[0073] Step 104: Determine the direction of the sound source based on the time domain covariance vector using a sound source localization model.

[0074] In the embodiment of the present application, after obtaining the time domain covariance vector, it means that the correlation between different signals in the N-frame time domain signal can be determined, and the eigenvalues ​​and corresponding eigenvectors included in the time domain covariance vector can be used to estimate the main arrival direction of the signal. To further determine the specific direction of the sound source, the covariance vector can be input into the sound source localization model, and the predicted sound source direction can be output through the sound source localization model.

[0075] Here, the eigenvalues ​​included in the time domain covariance vector represent the energy distribution of the signal in different directions, and the largest eigenvalue usually corresponds to the maximum energy distribution of the signal in a certain direction. Accordingly, each eigenvalue can be used to represent the probability of the signal in each positioning direction. The eigenvector corresponding to the eigenvalue is used to determine the positioning direction of the signal, and the eigenvector corresponding to the largest eigenvalue usually points to the positioning direction with the largest signal energy distribution. Accordingly, the eigenvector corresponding to each eigenvalue can be used to represent the positioning direction to which the signal points under the corresponding probability.

[0076] Based on this, in some implementations, the orientation of the sound source includes a positioning direction angle; the implementation of step 104 may specifically include:

[0077] The time domain covariance vector is input into the sound source localization model, so that the sound source localization model outputs probability values ​​of S direction angles, where S is a positive integer;

[0078] The direction angle corresponding to the maximum probability value is selected from the probability values ​​of the S direction angles and is used as the positioning direction angle.

[0079] Here, the sound source localization model is a neural network model including an input layer, a hidden layer and an output layer. The time domain covariance vector is input into the sound source localization model. The input layer is responsible for receiving the input time domain covariance vector, and the hidden layer is responsible for processing and transforming the time domain covariance vector of the input layer. Through weighted connections and activation functions, the features in the time domain covariance vector are extracted and nonlinear transformation is performed. The output layer is responsible for outputting probability values ​​of S direction angles according to the processing results of the hidden layer. These probability values ​​represent the possibilities of different directions. Among them, the direction angle corresponding to the maximum probability value among the probability values ​​of the S direction angles can be used as the positioning direction angle corresponding to the sound source orientation.

[0080] It should be noted that the sound source localization model is a pre-trained neural network model. The following is a detailed training process of the sound source localization model. The training process is divided into three stages, specifically including:

[0081] Data preprocessing stage

[0082] (1) Signal acquisition and segmentation: Use appropriate sensors or signal receiving devices (such as microphone arrays) to collect time domain signals containing positioning angle information in different scenarios. These scenarios cover various possible environmental conditions and signal sources distributed in different locations. After collecting the above time domain signals, the continuous time domain signals are segmented according to preset time windows. The signals in each time window after segmentation will be used as an independent data sample for subsequent calculation and analysis.

[0083] (2) Time domain covariance vector calculation: For each data sample (signal in each time window), the corresponding time domain covariance vector is calculated. The time domain covariance vector reflects the correlation and statistical characteristics of the signal at different time points and between different signal channels. The relevant content of calculating the time domain covariance vector corresponding to the signal can be referred to the above embodiment process, and will not be repeated here.

[0084] (3) Data labeling and organization: Determine the true positioning angle corresponding to each data sample, which can be obtained through the known signal source position or precise measurement equipment. Take the time domain covariance vector corresponding to each data sample as the data feature and the true positioning angle corresponding to each data sample as the data label. Associate and organize the time domain covariance vector corresponding to each data sample with its corresponding true positioning angle to construct a labeled data set for subsequent neural network training and verification.

[0085] Neural network architecture design and construction phase

[0086] (1) Input layer design: Design the input layer of the neural network based on the characteristic dimension of the time domain covariance vector of the data sample. The number of nodes in the input layer should match the number of elements in the time domain covariance vector to ensure that the input layer can fully receive and process the time domain covariance vector data. The input layer can perform some simple preprocessing operations on the input data, such as normalization or standardization, to map the value corresponding to the time domain covariance vector to a suitable numerical range to improve the training efficiency and stability of the network.

[0087] (2) Hidden layer structure design: Design a series of hidden layers to learn and extract implicit features and patterns in the time domain covariance vector. The number of hidden layers and the number of neurons in each hidden layer can be adjusted accordingly according to the characteristics and complexity of the data. Common hidden layer types include fully connected layers, convolutional layers, and recurrent layers. For the processing of the time domain covariance vector, you can consider using fully connected layers to capture global feature relationships, or using convolutional layers to extract local feature patterns. For example, construct a deep neural network containing multiple fully connected layers, each fully connected layer contains a certain number of neurons, and use nonlinear activation functions such as the rectified linear unit (ReLU) function to increase the nonlinear expression ability of the network.

[0088] (3) Output layer design: The design of the output layer should match the output form of the positioning angle. Since the probability of the corresponding positioning angle needs to be output, the output layer usually uses a softmax activation function to convert the network output into a probability distribution. The number of nodes in the output layer should correspond to the possible positioning angle range. For example, if the positioning angle of the signal within a range of 360 degrees needs to be predicted, the output layer can contain 360 nodes, each of which represents the probability of a specific angle.

[0089] Neural network training phase

[0090] (1) Training data preparation: The annotated dataset is divided into training set, validation set and test set. The training set is used to train the parameters of the neural network, the validation set is used to monitor the network performance during the training process and adjust the model hyperparameters, and the test set is used to evaluate the generalization ability of the final network. The training set data can be augmented by data operations such as random rotation, translation, and adding noise to increase the diversity of the data and the robustness of the network.

[0091] (2) Training process: Use the training set data to train the neural network, take the time domain covariance vector corresponding to the data sample as input, and the corresponding true positioning angle as the target output result, and calculate the output (probability distribution) of the network through forward propagation. Then, based on the actual output results and target output results of the network, calculate the loss function, calculate the gradient information of the loss function through the back propagation algorithm, and use the optimization algorithm (such as stochastic gradient descent and its variants) to update the network parameters such as weights and biases according to the gradient information of the loss function to minimize the loss function, thereby improving the network's prediction accuracy of the signal positioning angle by adjusting the network parameters.

[0092] (3) Hyperparameter adjustment and model optimization: During the training process, some key hyperparameters need to be adjusted, such as learning rate, batch size, number of hidden layer neurons and number of layers, etc. The best hyperparameter combination can be found through the validation set. Use the validation set data to monitor and evaluate the network training process. After each training cycle, calculate the loss function and other evaluation indicators (such as accuracy, recall, etc.) on the validation set to determine whether the network is overfitting or underfitting. Among them, if the performance indicators of the network on the validation set no longer improve or begin to decline, it means that it may be necessary to adjust the network hyperparameters, add regularization terms (such as L2 regularization), or use random dropout (Dropout) and other techniques to prevent overfitting.

[0093] Based on the foregoing embodiments, in some implementations, after the positioning direction angle is determined, the positioning direction angle can be sent to the LED display array so that the LED display array can determine an LED display flag in combination with the positioning direction angle, wherein the LED display flag is used to trigger the LED display array to update the LED display status.

[0094] Here, the determined positioning direction angle can be sent to the LED device, and the LED display array in the LED display device can calculate the LED display mark according to the positioning direction angle, and update the display status in the LED device according to the LED display mark, thereby achieving the purpose of updating the LED display status according to different sound source directions.

[0095] It should be noted that the above-mentioned process of multi-channel data collection, sound source positioning and LED display status update is carried out in a cycle. Whenever new multi-channel data is generated, the new sound source orientation is calculated, the new positioning direction angle is determined, and the display status in the LED display device is updated according to the new positioning direction angle.

[0096] In the technical solution of the embodiment of the present application, by acquiring M channel data, the M channel data is framed and windowed to obtain N frames of time domain signals, and based on the N frames of time domain signals, the time domain covariance vector is determined, so that the direction of the sound source is determined based on the time domain covariance vector through the sound source localization model, wherein the M channel data corresponds to the audio signals of the M channels collected by the microphone array. In this way, by segmenting the multi-channel data collected by the microphone array and selecting a suitable window function for windowing processing, it is possible to reduce spectrum leakage, improve the accuracy of spectrum analysis, and improve the signal-to-noise ratio of the signal to a certain extent, and by converting the obtained multi-frame time domain signals into time domain covariance vectors, it is possible to effectively utilize the time correlation in the signal, and by capturing and analyzing the phase and amplitude changes between the signals, the efficiency and accuracy of signal processing are effectively improved, so that the direction of the sound source can be accurately determined based on the time domain covariance vector while improving the robustness through the sound source localization model.

[0097] The present application also provides a method for localizing a sound source. Figure 2 is a schematic diagram of a specific process of the sound source localization method provided in an embodiment of the present application, such as Figure 2 As shown, the process is divided into three stages, each of which includes the following steps:

[0098] Microphone array recording stage

[0099] Step 201: Turn on the recording device and perform the recording operation.

[0100] Step 202: Collect multi-channel data through a microphone array.

[0101] Step 203: Determine whether the recording operation is stopped, if stopped, go to step 204; otherwise return to step 202.

[0102] Step 204: Turn off the recording device.

[0103] Sound source localization stage

[0104] Step 205: The audio collection thread obtains multi-channel data from the recording device and stores it in the message queue. There are one or more threads when collecting multi-channel data.

[0105] Step 206: Obtain multi-channel data from the message queue.

[0106] Step 207: perform frame windowing processing on the multi-channel data, read 256 points of data from the multi-channel data each time and splice them with the data read last time to obtain 512 points of data, and then perform windowing processing on them until all data in the multi-channel data are processed to obtain multi-frame time domain signals. The window function of the windowing processing includes but is not limited to the Hanning window, the Hamming window, etc.

[0107] Step 208: Calculate the covariance matrix of each frame of the time domain signal and construct multiple covariance matrices.

[0108] Step 209: vectorize multiple covariance matrices to obtain covariance vectors. Due to the symmetry of the covariance matrix, only the lower triangular matrix in the covariance matrix can be retained, and the rows are arranged from top to bottom to form a 10-dimensional vector.

[0109] Step 210: Input the covariance vector into a deep neural network model. The input of the deep neural network model is a 10-dimensional vector, the output of the deep neural network model is a 36-dimensional vector, and the deep neural network model includes but is not limited to CNN, DNN, RNN, transformer, etc. In addition, the input and output results and network depth of the deep neural network model depend on the microphone array and the specific application scenario, which are not limited here.

[0110] Step 211: The neural network model outputs the probability of each positioning angle, and selects the one with the largest probability as the current positioning direction. Here, the output probability result can be smoothed, and the smoothing method includes but is not limited to mean smoothing, median smoothing, etc.

[0111] Step 212: Determine whether the sound source localization is completed. If completed, end the sound source localization process; otherwise, return to step 206.

[0112] LED display stage

[0113] Step 213: Store the determined current positioning direction in the message queue.

[0114] Step 214: Get the current positioning direction from the message queue.

[0115] Step 215: Calculate the LED display mark by combining the LED display array and the positioning direction angle of the current positioning direction.

[0116] Step 216: Update the LED display status according to the LED display mark.

[0117] Step 217: Determine whether the LED display status is updated. If so, end the update process; otherwise, return to step 214.

[0118] In the technical solution of the embodiment of the present application, by segmenting the multi-channel data collected by the microphone array and selecting a suitable window function for windowing processing, it is possible to reduce spectral leakage, improve the accuracy of spectral analysis, and improve the signal-to-noise ratio of the signal to a certain extent. In addition, the obtained multi-frame time domain signal is converted into a time domain covariance vector, which can effectively utilize the time correlation in the signal, capture and analyze the phase and amplitude changes between signals, and improve the efficiency and accuracy of signal processing, so that the direction of the sound source can be accurately determined based on the time domain covariance vector while improving the robustness through the sound source localization model.

[0119] The present application also provides a sound source localization device. Figure 3 is a schematic diagram of the structure of the sound source localization device provided in an embodiment of the present application, such as Figure 3 As shown, the device comprises:

[0120] The collection unit 301 is used to obtain M channel data, where the M channel data corresponds to the audio signals of M channels collected by the microphone array, and M is a positive integer.

[0121] The processing unit 302 is used to perform frame division and windowing processing on the M channel data to obtain N frames of time domain signals, where N is a positive integer; and determine the time domain covariance vector based on the N frames of time domain signals.

[0122] The determination unit 303 is configured to determine the direction of the sound source based on the time domain covariance vector by using a sound source localization model.

[0123] In some implementations, the collection unit 301 is specifically configured to select M microphones to form a microphone array, where the M microphones are distributed at different locations; the microphone array is configured to collect sound signals propagated by a sound source; and M channel data are obtained from the microphone array.

[0124] In some implementations, the processing unit 302 is specifically configured to segment the M channel data by a preset frame length and a preset frame shift to obtain N overlapping frame signals; and perform windowing processing on each frame signal of the N overlapping frame signals to obtain N frames of time domain signals.

[0125] In some implementations, the processing unit 302 is further specifically configured to determine N time domain covariance matrices based on N frames of time domain signals; and determine a time domain covariance vector based on the N time domain covariance matrices.

[0126] In some embodiments, each frame of N frames of time domain signals has M channels and L sample points, where L is a positive integer; the processing unit 302 is further specifically used to form a signal matrix of each frame of time domain signal with M channels as M row dimensions and L sample points as L column dimensions; based on the signal matrix of each frame of time domain signal, determine the time domain covariance matrix of each frame of time domain signal.

[0127] In some embodiments, the processing unit 302 is further specifically used to determine, for each frame of the time domain signal, the average value of each row based on each element in each row of the signal matrix of the frame of the time domain signal; subtract the average value of the corresponding row from each element in each row to obtain the signal matrix after mean-normalization; based on the signal matrix after mean-normalization, determine the time domain covariance matrix of the frame of the time domain signal, each element in the time domain covariance matrix represents the covariance between the i-th channel and the j-th channel among M channels, where i and j are positive integers less than or equal to M.

[0128] In some embodiments, the processing unit 302 is further specifically used to vectorize the lower triangular or upper triangular part of each of the N time domain covariance matrices to obtain N time domain covariance matrix vectors; and concatenate the N time domain covariance matrix vectors to obtain a time domain covariance vector.

[0129] In some embodiments, the direction of the sound source includes a positioning direction angle; the determination unit 303 is specifically used to input the time domain covariance vector into the sound source localization model, so that the sound source localization model outputs probability values ​​of S direction angles, S is a positive integer; the direction angle corresponding to the maximum probability value is selected from the probability values ​​of the S direction angles, and used as the positioning direction angle.

[0130] In some embodiments, the device further comprises:

[0131] The updating unit is used to send the positioning direction angle to the LED display array, so that the LED display array determines the LED display mark in combination with the positioning direction angle, and the LED display mark is used to trigger the LED display array to update the LED display state.

[0132] In the technical solution of the embodiment of the present application, by acquiring M channel data, the M channel data is framed and windowed to obtain N frames of time domain signals, and based on the N frames of time domain signals, the time domain covariance vector is determined, so that the direction of the sound source is determined based on the time domain covariance vector through the sound source localization model, wherein the M channel data corresponds to the audio signals of the M channels collected by the microphone array. In this way, by segmenting the multi-channel data collected by the microphone array and selecting a suitable window function for windowing processing, it is possible to reduce spectrum leakage, improve the accuracy of spectrum analysis, and improve the signal-to-noise ratio of the signal to a certain extent, and by converting the obtained multi-frame time domain signals into time domain covariance vectors, it is possible to effectively utilize the time correlation in the signal, and by capturing and analyzing the phase and amplitude changes between the signals, the efficiency and accuracy of signal processing are effectively improved, so that the direction of the sound source can be accurately determined based on the time domain covariance vector while improving the robustness through the sound source localization model.

[0133] Those skilled in the art should understand that Figure 3 The functions implemented by each unit in the sound source localization device shown can be understood by referring to the relevant description of the aforementioned method. Figure 3 The functions of each unit in the sound source localization device shown can be implemented by a program running on a processor, or by a specific logic circuit.

[0134] Figure 4 is a schematic diagram of the structure of a processing device provided in an embodiment of the present application. The processing device may be a terminal device or a network device. Figure 4 The processing device shown includes a processor 401, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0135] Alternatively, if Figure 4 As shown, the processing device may further include a memory 402. The processor 401 may call and run a computer program from the memory 402 to implement the method in the embodiment of the present application.

[0136] The memory 402 may be a separate device independent of the processor 401 , or may be integrated into the processor 401 .

[0137] Alternatively, if Figure 4As shown, the processing device may further include a transceiver 403, and the processor 401 may control the transceiver 403 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0138] The transceiver 403 may include a transmitter and a receiver. The transceiver 403 may further include an antenna, and the number of the antennas may be one or more.

[0139] The processing device may specifically be the sound source localization device of the embodiment of the present application, and the processing device may implement the corresponding processes implemented by the sound source localization device in each method of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0140] It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0141] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0142] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0143] The embodiment of the present application also provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to the processing device in the embodiment of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the sound source localization device in each method of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0144] The embodiment of the present application also provides a computer program product, including computer program instructions. The computer program product can be applied to the processing device in the embodiment of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the sound source localization device in each method of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0145] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0150] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory,) ROM, random access memory (RandomAccess Memory, RAM), disk or optical disk and other media that can store program codes.

[0151] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A sound source localization method, characterized in that: The method comprises: Acquire M channel data, where the M channel data correspond to the audio signals of M channels collected by the microphone array, where M is a positive integer; Performing frame division and windowing processing on the M channel data to obtain N frames of time domain signals, where N is a positive integer; Based on the N frames of time domain signals, determining a time domain covariance vector; The direction of the sound source is determined based on the time domain covariance vector by a sound source localization model.

2. The method according to claim 1, characterized in that The obtaining of M channel data includes: Selecting M microphones to form a microphone array, wherein the M microphones are distributed at different positions; the microphone array is used to collect sound signals transmitted by a sound source; The M channel data are acquired from the microphone array.

3. The method according to claim 2, characterized in that The performing frame dividing and windowing processing on the M channel data to obtain N frames of time domain signals includes: The M channel data are divided by a preset frame length and a preset frame shift to obtain N overlapping frame signals; Windowing is performed on each frame signal of the N overlapping frame signals to obtain the N frames of time domain signals.

4. The method according to claim 3, characterized in that The determining of the time domain covariance vector based on the N frames of time domain signals includes: Based on the N frames of time domain signals, determining N time domain covariance matrices; The time domain covariance vector is determined based on the N time domain covariance matrices.

5. The method according to claim 4, characterized in that Each of the N frames of time domain signals has M channels and L sample points, where L is a positive integer; and determining N time domain covariance matrices based on the N frames of time domain signals includes: The M channels are used as M row dimensions, and the L sample points are used as L column dimensions to form a signal matrix of each frame of the time domain signal; Based on the signal matrix of each frame of the time domain signal, a time domain covariance matrix of each frame of the time domain signal is determined.

6. The method according to claim 5, characterized in that The determining the time domain covariance matrix of each frame of the time domain signal based on the signal matrix of each frame of the time domain signal comprises: For each frame of the time domain signal, based on each element of each row in the signal matrix of the frame of the time domain signal, determine the average value of each row; Subtract the average value of the corresponding row from each element in each row to obtain a signal matrix after the mean value is normalized to zero; Based on the signal matrix after mean-zeroing, the time domain covariance matrix of the frame time domain signal is determined, each element in the time domain covariance matrix represents the covariance between the i-th channel and the j-th channel in the M channels, and i and j are positive integers less than or equal to M.

7. The method according to claim 6, characterized in that The determining the time domain covariance vector based on the N time domain covariance matrices includes: Vectorizing the lower triangular part or the upper triangular part of each of the N time-domain covariance matrices to obtain N time-domain covariance matrix vectors; The N time-domain covariance matrix vectors are concatenated to obtain the time-domain covariance vector.

8. The method according to claim 1, characterized in that: The direction of the sound source includes a positioning direction angle; and determining the direction of the sound source based on the time domain covariance vector by using a sound source localization model includes: Inputting the time domain covariance vector into the sound source localization model, so that the sound source localization model outputs probability values ​​of S direction angles, where S is a positive integer; The direction angle corresponding to the maximum probability value is selected from the probability values ​​of the S direction angles and used as the positioning direction angle.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: The positioning direction angle is sent to a light emitting diode (LED) display array, so that the LED display array determines an LED display flag in combination with the positioning direction angle, and the LED display flag is used to trigger the LED display array to update the LED display state.

10. A sound source localization device, characterized in that: The device comprises: A collection unit, used to obtain M channel data, where the M channel data correspond to the audio signals of the M channels collected by the microphone array, where M is a positive integer; A processing unit is used to perform frame division and windowing processing on the M channel data to obtain N frames of time domain signals, where N is a positive integer; and determine a time domain covariance vector based on the N frames of time domain signals; A determination unit is used to determine the direction of the sound source based on the time domain covariance vector through a sound source localization model.

11. A processing device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method as claimed in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer program instructions which cause a computer to execute the method as claimed in any one of claims 1 to 9.

Images