Array decoupling sound source localization method and device based on deep learning, and readable medium

Abstract

The invention discloses a formation decoupling sound source localization method and device based on deep learning and a readable medium, and the method comprises the steps: constructing and training a sound source localization model, and obtaining a trained sound source localization model; acquiring a first sound source signal received by a first microphone and a second sound source signal received by a second microphone in the microphone array; calculating generalized cross-correlation frequency domain representation between the first sound source signal and the second sound source signal based on the first sound source signal and the second sound source signal; obtaining a frequency domain feature based on the guide vector between the first microphone and the second microphone and the generalized cross-correlation frequency domain representation; determining an input feature based on the frequency domain feature, and inputting the input feature into a trained sound source localization model to obtain a candidate sound source angle and a confidence coefficient corresponding to the candidate sound source angle; and post-processing the candidate sound source angle to obtain a sound source positioning result. The invention solves the problems that the existing sound source localization method based on deep learning is large in calculated parameter quantity and cannot be applied to embedded equipment and the like.

Description

Technical Field

[0001] This invention relates to the field of sound source localization, and specifically to a method, apparatus, and readable medium for array-decoupled sound source localization based on deep learning. Background Technology

[0002] Sound source localization (SSL) is primarily used to provide the location information of a target sound source. In real-time audio and video conferencing, it is often desirable to focus on the speaker and enhance their voice. This requires audio signal processing techniques such as VT (voice-to-text) selection, speech enhancement, and speech separation. A good sound source localization result can provide important reference value for the aforementioned post-processing algorithms. Most current sound source localization algorithms rely on a fixed microphone array size and select relatively specific training data and features, making it difficult to achieve model uniformity. Therefore, how to provide a low-power, low-latency sound source localization method without relying on microphone array size is a current technical problem that needs to be solved. Summary of the Invention

[0003] The purpose of this application is to address the technical problem of how to provide a low-power, low-latency sound source localization method without relying on the microphone array size. This application proposes a deep learning-based array decoupling sound source localization method, device, readable medium, and computer program product.

[0004] In a first aspect, the present invention provides a deep learning-based array decoupling sound source localization method, comprising the following steps:

[0005] Construct and train a sound source localization model to obtain a trained sound source localization model;

[0006] Acquire the first sound source signal received by the first microphone and the second sound source signal received by the second microphone in the microphone array; calculate the generalized cross-correlation frequency domain representation between the first sound source signal and the second sound source signal based on the first sound source signal and the second sound source signal;

[0007] Frequency domain features are obtained based on the steering vector between the first and second microphones and the frequency domain representation of the generalized cross-correlation.

[0008] Input features are determined based on frequency domain features and input into the trained sound source localization model to obtain candidate sound source angles and their corresponding confidence scores; the candidate sound source angles are then post-processed to obtain the sound source localization results.

[0009] Preferably, a generalized cross-correlation frequency domain representation between the first and second sound source signals is calculated based on the first and second sound source signals, specifically including:

[0010] The frequency domain representation of the first sound source signal is obtained by performing Fast Fourier Transform on the first and second sound source signals respectively. Frequency domain representation of the second sound source signal ,in, Indicates frequency, Indicates the index of the first microphone. Indicates the index of the second microphone;

[0011] Multiplying the frequency domain representations of the first and second sound source signals by their conjugate complex numbers yields the generalized cross-correlation function between them, as shown in the following equation:

[0012] ;

[0013] in, This represents the generalized cross-correlation function between the first and second sound source signals. Represents the conjugate complex number;

[0014] By applying phase transform weighting to the generalized cross-correlation function between the first and second sound source signals, the frequency domain representation of the generalized cross-correlation between the first and second sound source signals is obtained, as shown in the following equation:

[0015] ;

[0016] in, The frequency domain representation of the generalized cross-correlation between the first and second sound source signals.

[0017] As a preferred option, the formula for calculating the frequency domain features is as follows:

[0018] ;

[0019] in, Represents frequency domain characteristics, This represents the steering vector between the first and second microphones. , This indicates the distance between the first and second microphones. This represents the angle of arrival of the sound source signal to the microphone array. This indicates the speed at which sound travels through the air. The symbol represents the imaginary unit, and M represents the total number of microphones in the microphone array. This represents matrix multiplication.

[0020] As a preferred option, the sound source localization model adopts a CRNN network structure. The sound source localization model includes a CNN unit, an LSTM unit, and a classification layer connected in sequence. The CNN unit is used to reduce the dimensionality of the input features and aggregate them to obtain the first intermediate features. The LSTM unit is used to capture the time dimension information of the first intermediate features and extract the second intermediate features. The classification layer is used to map the second intermediate features to all angles for classification to obtain the candidate sound source angles and their corresponding confidence scores.

[0021] Preferably, the input features are the imaginary parts of the frequency domain features, in order to reduce the number of parameters required to train the sound source localization model.

[0022] Preferably, the post-processing includes one or more of threshold filtering, time smoothing, and peak filtering. Threshold filtering filters out candidate sound source angles with confidence levels below a threshold. Time smoothing smooths the results obtained by repeated predictions using a trained sound source localization model at preset intervals to obtain smoothed results. Peak filtering selects candidate sound source angles that meet the peak height and peak interval requirements.

[0023] Secondly, the present invention provides a deep learning-based array decoupling sound source localization device, comprising:

[0024] The model building module is configured to build and train a sound source localization model to obtain a trained sound source localization model.

[0025] The generalized cross-correlation frequency domain representation calculation module is configured to acquire a first sound source signal received by a first microphone and a second sound source signal received by a second microphone in a microphone array; and calculate the generalized cross-correlation frequency domain representation between the first sound source signal and the second sound source signal based on the first sound source signal and the second sound source signal.

[0026] The frequency domain feature calculation module is configured to obtain frequency domain features based on the steering vector between the first microphone and the second microphone and the frequency domain representation of the generalized cross-correlation.

[0027] The prediction module is configured to determine input features based on frequency domain features, input them into the trained sound source localization model, obtain candidate sound source angles and their corresponding confidence scores, and then perform post-processing on the candidate sound source angles to obtain the sound source localization results.

[0028] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0029] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0030] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The deep learning-based array decoupling sound source localization method proposed in this invention can achieve decoupling from the microphone array size while satisfying the stability and accuracy of sound source localization. For microphone arrays with different array sizes, there is no need to retrain the sound source localization model, which can reduce development time and cost. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating a deep learning-based array decoupling sound source localization method according to an embodiment of this application.

[0035] Figure 2 This is a schematic diagram of a sound source localization model for a deep learning-based array decoupling sound source localization method according to an embodiment of this application.

[0036] Figure 3 This is a schematic diagram showing the arrival of the sound source signal at the microphone array in the array decoupling sound source localization method based on deep learning, as described in an embodiment of this application.

[0037] Figure 4 This is a schematic diagram of a deep learning-based array decoupling sound source localization device according to an embodiment of this application;

[0038] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0040] One of the key experiences in real-time audio and video conferencing is ensuring that the camera and audio processing link are always pointed at the speaker. This requires sound source localization to provide the spatial coordinates of the "speaker's angle," and its accuracy directly determines the subsequent image and audio processing quality. Currently, sound source localization methods include traditional algorithms and deep learning methods. Traditional algorithms include GCC, SRP, and MUSIC.

[0041] Against this backdrop, the inventors of this application discovered that traditional algorithms, based on theoretical assumptions, perform poorly in real acoustic environments. With the development of deep learning technology, it has demonstrated strong potential in the field of audio signal processing. Deep learning algorithms include CNNs and RNNs. For sound source localization tasks, deep learning methods have significant advantages in terms of high accuracy and robustness compared to traditional algorithms. However, for multi-channel time-varying audio signals, network structures such as RNNs and two-dimensional CNNs are generally required, resulting in excessively large model parameters, making them unsuitable for embedded devices. Furthermore, most deep learning algorithms rely on specific microphone array sizes and require the selection of corresponding training data and features, making model uniformity impossible. Different microphone arrays require retraining of corresponding models, significantly increasing development time. Therefore, providing a low-power, low-latency sound source localization method without relying on microphone array size is a current technical problem that needs to be solved.

[0042] To address the current technical problem of providing a low-power, low-latency sound source localization method without relying on microphone array size, this application provides a deep learning-based array-decoupled sound source localization method, comprising: constructing and training a sound source localization model to obtain a trained sound source localization model; acquiring a first sound source signal received by a first microphone and a second sound source signal received by a second microphone in a microphone array; calculating a generalized cross-correlation frequency domain representation between the first and second sound source signals based on the first and second sound source signals; obtaining frequency domain features based on the steering vector and the generalized cross-correlation frequency domain representation between the first and second microphones; determining input features based on the frequency domain features and inputting them into the trained sound source localization model to obtain candidate sound source angles and their corresponding confidence scores; and post-processing the candidate sound source angles to obtain the sound source localization result.

[0043] Figure 1This application illustrates an embodiment of a deep learning-based array decoupling sound source localization method, comprising the following steps:

[0044] S1. Construct and train a sound source localization model to obtain a trained sound source localization model.

[0045] In a specific embodiment, the sound source localization model adopts a CRNN network structure. The sound source localization model includes a CNN unit, an LSTM unit, and a classification layer connected in sequence. The CNN unit is used to reduce the dimensionality of the input features and aggregate them to obtain the first intermediate features. The LSTM unit is used to capture the time dimension information of the first intermediate features and extract the second intermediate features. The classification layer is used to map the second intermediate features to all angles for classification to obtain the candidate sound source angles and their corresponding confidence scores.

[0046] Specifically, the sound source localization model in this application primarily employs a CRNN network structure, treating sound source localization as a multi-classification task. The CNN unit is mainly used for dimensionality reduction and aggregation in the feature dimension, the LSTM unit is used to capture information in the temporal dimension, and the final classification layer uses a fully connected layer and a sigmoid activation function layer to map to all angle classifications. (Reference) Figure 2 The CNN unit comprises four sequentially connected CNN structures. Each CNN structure includes a sequentially connected convolutional module and a max pooling layer. Each convolutional module includes a 3×3 convolutional layer, a batch normalization (BN) layer, and a ReLU activation function layer. The BN and ReLU activation function layers primarily improve the model's generalization ability, facilitating training and preventing overfitting. The max pooling layer is placed after each convolutional module, mainly to reduce the dimensionality of the feature axis, extract key features, and maintain the time axis by padding. Finally, after passing through the four CNN structures, the first intermediate feature is obtained. The first intermediate feature is input into an LSTM unit to obtain the second intermediate feature. The second intermediate feature is input into a classification layer, where a fully connected layer performs feature mapping on the second intermediate feature. A sigmoid activation function layer outputs all candidate sound source angles and their corresponding confidence scores. In other embodiments, the sound source localization model can also use other CNN or RNN network structures.

[0047] S2, acquire the first sound source signal received by the first microphone and the second sound source signal received by the second microphone in the microphone array; calculate the generalized cross-correlation frequency domain representation between the first sound source signal and the second sound source signal based on the first sound source signal and the second sound source signal.

[0048] In a specific embodiment, the generalized cross-correlation frequency domain representation between the first sound source signal and the second sound source signal is calculated based on the first sound source signal and the second sound source signal, specifically including:

[0049] The frequency domain representation of the first sound source signal is obtained by performing Fast Fourier Transform on the first and second sound source signals respectively. Frequency domain representation of the second sound source signal ,in, Indicates frequency, Indicates the index of the first microphone. Indicates the index of the second microphone;

[0050] Multiplying the frequency domain representations of the first and second sound source signals by their conjugate complex numbers yields the generalized cross-correlation function between them, as shown in the following equation:

[0051] ;

[0052] in, This represents the generalized cross-correlation function between the first and second sound source signals. Represents the conjugate complex number;

[0053] By applying phase transform weighting to the generalized cross-correlation function between the first and second sound source signals, the frequency domain representation of the generalized cross-correlation between the first and second sound source signals is obtained, as shown in the following equation:

[0054] ;

[0055] in, The frequency domain representation of the generalized cross-correlation between the first and second sound source signals.

[0056] Specifically, the sound source signal acquired by the microphone is first subjected to a Fast Fourier Transform (FFT) to obtain the frequency domain representation of the sound source signal received by the microphone. The expression for the frequency domain representation of the sound source signal received by the microphone is as follows:

[0057] ;

[0058] X represents the frequency domain representation of the sound source signal, A represents the steering vector, S represents the sound source signal, N represents noise or other interference signals, and f represents the frequency. This indicates the angle of arrival of the sound source signal to the microphone array.

[0059] refer to Figure 3 The steering vector is a mathematical representation of the relative positions and phase information of a sound source signal arriving at multiple microphones. The direction of the sound source can be determined by calculating the distance from the sound source signal to each microphone. The steering vector is typically represented as a matrix, where its column vectors correspond to the phase difference of the sound source signal at different incident angles. For a linear array, it can be represented as:

[0060] ;

[0061] ;

[0062] in, This represents the time delay of the m-th microphone relative to the reference microphone. Let m represent the distance between the m-th microphone and the reference microphone, and C represent the speed of sound in the air.

[0063] The generalized cross-correlation function between the first and second sound source signals is calculated using their frequency domain representations. Due to noise, reverberation, and other interference, the correlation peak of the generalized cross-correlation function is weakened, and even spurious peaks may be introduced. To further highlight the main peak, weighting is performed in the frequency domain. Here, phase transform (PHAT) weighting is chosen to enhance the higher frequency components of the signal with higher SNR, suppress noise, and result in a generalized cross-correlation function with a sharp peak.

[0064] The weights used in PHAT weighting for:

[0065] .

[0066] S3, frequency domain features are calculated based on the generalized cross-correlation frequency domain representation between the first sound source signal and the second sound source signal, and the steering vector between the first microphone and the second microphone.

[0067] In a specific embodiment, the formula for calculating the frequency domain features is as follows:

[0068] ;

[0069] in, Represents frequency domain characteristics, This represents the steering vector between the first and second microphones. , This indicates the distance between the first and second microphones. This represents the angle of arrival of the sound source signal to the microphone array. This indicates the speed at which sound travels through the air. The symbol represents the imaginary unit, and M represents the total number of microphones in the microphone array. This represents matrix multiplication.

[0070] Specifically, the steering vector contains the phase information of the microphone array relative to the sound source, and the generalized cross-correlation function (GCC) can represent the correlation between two microphones. Taking a linear microphone array as an example: traversing the selected microphone combination and all possible sound source angles, multiplying the frequency domain representation of the GCC between any two microphones in the array with the steering vector yields the frequency domain features. The dimensions of the GCC are P, T, and F, and the dimensions of the steering vector A are P, F, and N. Here, P represents the number of microphone combinations, N represents the number of sound source angles, T represents the time dimension, and F represents the frequency dimension. For a linear microphone array, the pickup range is generally 0~180 degrees, and the selected angle can be determined according to different resolutions. The final frequency domain feature dimensions are: P, T, N. Its physical meaning can be understood as the correlation between different microphone combinations across all candidate sound source angles. Theoretically, the higher the correlation at a certain angle, the greater the probability that it is the target sound source angle. Of course, in reality, it is affected by noise, reverberation, and other interference, as well as the uncertainty of instantaneous information. Therefore, a neural network needs to perform long-term statistical learning to obtain a true and stable target angle.

[0071] S4. Based on the frequency domain features, the input features are determined and input into the trained sound source localization model to obtain the candidate sound source angles and their corresponding confidence levels. The candidate sound source angles are then post-processed to obtain the sound source localization results.

[0072] In a specific embodiment, the input feature is the imaginary part of the frequency domain feature, in order to reduce the number of parameters required to train the sound source localization model.

[0073] Specifically, since the frequency domain feature calculations mentioned above are performed in the frequency domain, they consume a lot of computational resources during actual inference, especially when there are many microphones and high angular resolution. They also increase the number of model parameters, making it difficult to apply the sound source localization model to embedded devices.

[0074] Due to two complex numbers and Multiplication between them can be expressed as:

[0075] ;

[0076] ;

[0077] in As the imaginary part of the result of multiplication As the real part of the result of multiplication It represents the imaginary unit.

[0078] Therefore, the frequency domain features can be regarded as the frequency domain representation of the reconstructed sound source signal. The calculation process in the frequency domain features is decomposed into imaginary and real parts according to the multiplication of the two complex numbers mentioned above. The imaginary part of the frequency domain features contains the main phase information. Therefore, the imaginary part of the frequency domain features can be used as the input feature of the sound source localization model. Specific experimental evaluation has also verified that the content of the imaginary part is sufficient for the sound source localization model to learn the required information. The key is that it saves half of the feature calculation performance, which makes a huge contribution to the application of the sound source localization model to embedded devices.

[0079] The trained sound source localization model is deployed. During the inference phase, the sound source signal collected by the microphone array is processed through steps S2 and S3 to obtain the corresponding frequency domain features. The imaginary part of the frequency domain features is input into the trained sound source localization model to obtain the candidate sound source angle and its corresponding confidence level.

[0080] In a specific embodiment, the post-processing process includes one or more of threshold filtering, time smoothing, and peak filtering. Threshold filtering filters out candidate sound source angles with confidence levels below a threshold. Time smoothing smooths the results obtained by repeated predictions using a trained sound source localization model at preset intervals to obtain smoothed results. Peak filtering selects candidate sound source angles that meet the peak height and peak interval requirements.

[0081] Specifically, threshold filtering, time smoothing, and peak filtering can be used for post-processing to further improve the accuracy and stability of sound source localization. The details are as follows:

[0082] (1) Threshold filtering: The confidence level output by the trained sound source localization model is a probability value between 0 and 1. Therefore, an appropriate confidence threshold can be set, generally 0.5, to filter out candidate sound source angles with low confidence to improve stability. The specific process is as follows: compare the confidence level of the candidate sound source angle with the confidence threshold, and filter out candidate sound source angles with confidence levels lower than the confidence threshold.

[0083] (2) Time smoothing: Select a preset time interval, such as 100ms, to perform one inference using the trained sound source localization model. This will yield prediction results at different times. Smooth the prediction results at different times and use the smoothed results to determine the direction of arrival (DOA), i.e., the sound source localization result. This can improve the model's performance and stability. The preset time interval selected during time smoothing should balance the delay. If the preset time interval for inference is too large, it will increase the delay of sound source localization. Specifically, the confidence of all candidate angles output by the trained sound source localization model at the current time is denoted as DOPMap, and the confidence of all candidate angles output by the trained sound source localization model at the previous time is denoted as DOPMap_last. DOPMap' = α×DOPMap + (1-α)×DOPMap_last; where α is the smoothing coefficient. The smoothed result DOPMap' is used to determine the DOA. Therefore, accuracy and delay can be balanced based on requirements.

[0084] (3) Peak filtering: When multiple sound sources need to be located at the same time, the peak height and the distance between peaks (peak interval) can be set. The angle of the candidate sound source that meets the peak height and peak interval can be used as the final sound source location result. Peak filtering can distinguish the location results of different sound sources.

[0085] This application provides a deep learning-based array decoupling sound source localization method that generates frequency domain features independent of array geometry online using "guide vector × GCC-PHAT". This allows the same set of CRNN weights to be directly used for microphone arrays with different numbers of microphones without retraining. It can achieve decoupling from the microphone array size while meeting the stability and accuracy of sound source localization. For microphone arrays with different array sizes, there is no need to retrain the sound source localization model, which can reduce development time and cost and meet the requirements of low power consumption and low latency conference terminals.

[0086] The array-based decoupling sound source localization method provided in this application optimizes the input features and structure of the sound source localization model, which can significantly reduce the number of parameters in the training process and the performance consumption in the inference process, enabling the sound source localization model to be applied to embedded devices.

[0087] The array decoupling sound source localization method based on deep learning provided in this application embodiment can achieve stable localization for different microphone array sizes, and the overall error can be controlled within a small range; moreover, the number of parameters trained in the sound source localization model is small, which can meet the deployment requirements of most embedded devices.

[0088] Further reference Figure 4As an implementation of the methods shown in the above figures, this application provides an embodiment of a deep learning-based array decoupling sound source localization device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0089] This application provides a deep learning-based array decoupling sound source localization device, including:

[0090] Model building module 1 is configured to build and train a sound source localization model to obtain a trained sound source localization model;

[0091] The generalized cross-correlation frequency domain representation calculation module 2 is configured to acquire the first sound source signal received by the first microphone and the second sound source signal received by the second microphone in the microphone array; and calculate the generalized cross-correlation frequency domain representation between the first sound source signal and the second sound source signal based on the first sound source signal and the second sound source signal.

[0092] Frequency domain feature calculation module 3 is configured to obtain frequency domain features based on the steering vector between the first microphone and the second microphone and the frequency domain representation of the generalized cross-correlation.

[0093] Prediction module 4 is configured to determine input features based on frequency domain features, input them into the trained sound source localization model, obtain candidate sound source angles and their corresponding confidence scores, and perform post-processing on the candidate sound source angles to obtain the sound source localization results.

[0094] This application provides a deep learning-based array decoupling sound source localization device. It applies a deep learning-based array decoupling sound source localization method, which generates frequency domain features independent of array geometry online through "guide vector × GCC-PHAT". This allows the same set of CRNN weights to be directly used for microphone arrays with different numbers of microphones without retraining. It can achieve decoupling from the microphone array size while meeting the stability and accuracy of sound source localization. For microphone arrays with different array sizes, there is no need to retrain the sound source localization model, which can reduce development time and cost and meet the requirements of low power consumption and low latency conference terminals. Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device of this embodiment includes a processor 501 and a memory 502; wherein the memory 502 is used to store computer execution instructions; and the processor 501 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0095] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.

[0096] When the memory 502 is set up independently, the electronic device also includes a bus 503 for connecting the memory 502 and the processor 501.

[0097] This application provides a deep learning-based array decoupling sound source localization electronic device. It employs a deep learning-based array decoupling sound source localization method, which generates frequency domain features independent of array geometry online using "guide vector × GCC-PHAT". This allows the same set of CRNN weights to be directly applied to microphone arrays with different numbers of microphones without retraining. While maintaining the stability and accuracy of sound source localization, it achieves decoupling from the microphone array size. For microphone arrays with different array sizes, there is no need to retrain the sound source localization model, reducing development time and cost, and meeting the requirements of low-power, low-latency conferencing terminals.

[0098] This application embodiment also provides a computer storage medium storing computer execution instructions, which, when executed by processor 501, implement the above method.

[0099] This application provides a deep learning-based array decoupling sound source localization computer storage medium. It implements a deep learning-based array decoupling sound source localization method, which generates frequency domain features independent of array geometry online through "guide vector × GCC-PHAT". This allows the same set of CRNN weights to be directly used for microphone arrays with different numbers of microphones without retraining. It achieves decoupling from the microphone array size while maintaining the stability and accuracy of sound source localization. For microphone arrays with different array sizes, there is no need to retrain the sound source localization model, reducing development time and cost, and meeting the requirements of low-power, low-latency conferencing terminals.

[0100] This application also provides a computer program product, including a computer program, which, when executed by processor 501, implements the above-described method.

[0101] This application provides a deep learning-based array decoupling sound source localization computer storage medium. It implements a deep learning-based array decoupling sound source localization method, which generates frequency domain features independent of array geometry online through "guide vector × GCC-PHAT". This allows the same set of CRNN weights to be directly used for microphone arrays with different numbers of microphones without retraining. It achieves decoupling from the microphone array size while maintaining the stability and accuracy of sound source localization. For microphone arrays with different array sizes, there is no need to retrain the sound source localization model, reducing development time and cost, and meeting the requirements of low-power, low-latency conferencing terminals.

[0102] The present application provides a deep learning-based array decoupling sound source localization method, device, readable medium, and computer program product that can achieve decoupling from the microphone array size while meeting the stability and accuracy of sound source localization. For microphone arrays with different array sizes, there is no need to retrain the sound source localization model, which can reduce development time and cost.

[0103] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0104] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0105] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0106] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 501 to execute some steps of the methods of the various embodiments of this application.

[0107] It should be understood that the processor 501 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 501 can be any conventional processor 501. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 501 executing the steps, or as a combination of hardware and software modules within the processor 501 executing the steps.

[0108] The memory 502 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0109] Bus 503 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 503 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 503 in the accompanying drawings of this application is not limited to only one bus 503 or one type of bus 503.

[0110] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0111] An exemplary storage medium is coupled to processor 501, enabling processor 501 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of processor 501. Processor 501 and storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, processor 501 and storage medium can exist as discrete components in an electronic device or host device.

[0112] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning-based array decoupling sound source localization method, characterized in that, Includes the following steps: Construct and train a sound source localization model to obtain a trained sound source localization model; Acquire a first sound source signal received by a first microphone and a second sound source signal received by a second microphone in a microphone array; calculate the generalized cross-correlation frequency domain representation between the first sound source signal and the second sound source signal based on the first sound source signal and the second sound source signal; Frequency domain features are obtained based on the steering vector between the first microphone and the second microphone and the generalized cross-correlation frequency domain representation; Based on the frequency domain features, the input features are determined and input into the trained sound source localization model to obtain candidate sound source angles and their corresponding confidence levels; the candidate sound source angles are then post-processed to obtain the sound source localization results.

2. The array-decoupled sound source localization method based on deep learning according to claim 1, characterized in that, The calculation of the generalized cross-correlation frequency domain representation between the first sound source signal and the second sound source signal based on the first sound source signal and the second sound source signal specifically includes: Based on the first and second sound source signals, a fast Fourier transform is performed respectively to obtain the frequency domain representation of the first sound source signal. Frequency domain representation of the second sound source signal ,in, Indicates frequency, Indicates the index of the first microphone. Indicates the index of the second microphone; Multiplying the frequency domain representations of the first and second sound source signals by their conjugate complex numbers yields the generalized cross-correlation function between them, as shown in the following equation: ； in, This represents the generalized cross-correlation function between the first and second sound source signals. Represents the conjugate complex number; The generalized cross-correlation function between the first and second sound source signals is subjected to phase transform weighting to obtain the frequency domain representation of the generalized cross-correlation between the first and second sound source signals, as shown in the following equation: ； in, The frequency domain representation of the generalized cross-correlation between the first and second sound source signals.

3. The array-based decoupling sound source localization method according to claim 2, characterized in that, The formula for calculating the frequency domain features is as follows: ； in, Represents frequency domain characteristics, This represents the steering vector between the first and second microphones. , This indicates the distance between the first and second microphones. This indicates the angle of arrival of the sound source signal to the microphone array. This indicates the speed at which sound travels through the air. The symbol represents the imaginary unit, and M represents the total number of microphones in the microphone array. This represents matrix multiplication.

4. The array-based decoupling sound source localization method according to claim 1, characterized in that, The sound source localization model adopts a CRNN network structure, which includes a CNN unit, an LSTM unit, and a classification layer connected in sequence. The CNN unit is used to reduce and aggregate the input features in terms of feature dimension to obtain a first intermediate feature. The LSTM unit is used to capture the time dimension information of the first intermediate feature and extract a second intermediate feature. The classification layer is used to map the second intermediate feature to all angles for classification to obtain candidate sound source angles and their corresponding confidence scores.

5. The array-based decoupling sound source localization method according to claim 1, characterized in that, The input feature is the imaginary part of the frequency domain feature, in order to reduce the number of parameters required for training the sound source localization model.

6. The array-based decoupling sound source localization method according to claim 2, characterized in that, The post-processing process includes one or more of threshold filtering, time smoothing, and peak filtering. The threshold filtering filters out candidate sound source angles with confidence levels below a threshold. The time smoothing process smooths the results obtained by repeated predictions using the trained sound source localization model at preset intervals to obtain a smoothed result. Candidate sound source angles that meet the peak height and peak spacing criteria are selected by peak value screening.

7. A deep learning-based array-decoupled sound source localization device, characterized in that, include: The model building module is configured to build and train a sound source localization model to obtain a trained sound source localization model. The generalized cross-correlation frequency domain representation calculation module is configured to acquire a first sound source signal received by a first microphone and a second sound source signal received by a second microphone in a microphone array; and calculate the generalized cross-correlation frequency domain representation between the first sound source signal and the second sound source signal based on the first sound source signal and the second sound source signal. The frequency domain feature calculation module is configured to obtain frequency domain features based on the steering vector between the first microphone and the second microphone and the generalized cross-correlation frequency domain representation; The prediction module is configured to determine input features based on the frequency domain features, input them into the trained sound source localization model, obtain candidate sound source angles and their corresponding confidence scores, and perform post-processing on the candidate sound source angles to obtain the sound source localization results.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images