Sound source localization method, device and storage medium

Abstract

The present application discloses a sound source localization method, device and storage medium. The sound source localization method comprises: calculating a first correlation coefficient based on a covariance matrix of a current calculation frequency point; and calculating a second correlation coefficient based on a distance between two microphones in a microphone pair and an angular frequency of the current calculation frequency point; comparing the magnitude relationship between the first correlation coefficient and the second correlation coefficient and a preset threshold value, respectively, to obtain a comparison result; taking the current calculation frequency point whose comparison result meets a preset condition as a valid frequency point, and using all valid frequency points to perform sound source localization, and the correlation coefficients between microphones can be considered in multiple dimensions, and the valid frequency points in the current time frame can be determined by combining multi-dimensional information, so as to improve the accuracy of the selected valid frequency points, reduce the influence of low-quality audio signals on sound source localization, and improve the accuracy of sound source localization.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a sound source localization method, device and storage medium. Background Art

[0002] Sound source localization is an information technology that uses the physical properties of audio signals, such as arrival time difference, phase difference, intensity difference, etc., to determine the direction of the sound source. This technology has a wide range of applications in many fields, such as environmental perception, video conferencing, machine interaction, etc. For example, in the field of environmental perception, sound source localization can be used to identify and locate the direction of abnormal sounds for recording or alarming; in the field of video conferencing, sound source localization technology can be used to determine the spatial position information of the speaker, and improve the communication efficiency of online meetings by linking with the camera; in the field of machine interaction, sound source localization technology can help machines understand and respond to instructions, improving the efficiency and accuracy of interaction.

[0003] However, since the application scenarios are generally more complex, especially in scenarios with reverberation and noise, the quality of the collected audio is poor, which will affect the accuracy of sound source localization. Summary of the invention

[0004] In order to solve the above technical problems, the present application at least provides a sound source localization method, device and storage medium.

[0005] The first aspect of the present application provides a sound source localization method, the method comprising: performing Fourier transform on an audio signal received by each microphone in a microphone array to obtain a frequency domain signal corresponding to each microphone; forming a microphone pair with any two microphones in the microphone array, and calculating a covariance matrix between current calculation frequency points in the frequency domain signals corresponding to the two microphones in the microphone pair; calculating a correlation parameter between the two microphones in the microphone pair based on the covariance matrix, and calculating a first correlation coefficient using the correlation parameter of each microphone pair; and calculating a diffuse field noise parameter based on a distance between the two microphones in the microphone pair and an angular frequency of the current calculation frequency point, and calculating a second correlation coefficient using the diffuse field noise parameter of each microphone pair; comparing the first correlation coefficient and the second correlation coefficient with a preset threshold value to obtain a comparison result; taking the current calculation frequency point whose comparison result meets the preset condition as a valid frequency point, and using all the valid frequency points to perform sound source localization.

[0006] In one embodiment, a first correlation coefficient is calculated using correlation parameters of each microphone pair, including: constructing a first correlation matrix based on the correlation parameters of each microphone pair; selecting the maximum eigenvalue of the first correlation matrix to obtain a first eigenvalue; and calculating the first correlation coefficient using the first eigenvalue; wherein the absolute value of the first eigenvalue is proportional to the first correlation coefficient.

[0007] In one embodiment, the second correlation coefficient is calculated using the diffuse field noise parameters of each microphone pair, including: constructing a second correlation matrix based on the diffuse field noise parameters of each microphone pair; selecting the maximum eigenvalue of the second correlation matrix to obtain a second eigenvalue; and calculating the second correlation coefficient using the first eigenvalue and the second eigenvalue; wherein the absolute value of the difference between the first eigenvalue and the second eigenvalue is proportional to the second correlation coefficient, and the first eigenvalue is inversely proportional to the second correlation coefficient.

[0008] In one embodiment, the method also includes: summing the correlation parameters of each microphone pair to obtain a summation result; combining the summation result and the diffuse field noise parameter to calculate a third correlation coefficient; comparing the first correlation coefficient and the second correlation coefficient with a preset threshold value to obtain a comparison result, including: comparing the first correlation coefficient, the second correlation coefficient and the third correlation coefficient with a preset threshold value to obtain a comparison result.

[0009] In one embodiment, the preset threshold includes a first threshold value, a second threshold value and a third threshold value; comparing the size relationship between the first correlation coefficient, the second correlation coefficient and the third correlation coefficient and the preset threshold value respectively to obtain a comparison result, including: comparing the size relationship between the first correlation coefficient and the first threshold value, comparing the size relationship between the second correlation coefficient and the second threshold value, and comparing the size relationship between the third correlation coefficient and the third threshold value to obtain a comparison result; taking the current calculated frequency point whose comparison result meets the preset conditions as the valid frequency point, including: selecting the current calculated frequency point whose first correlation coefficient is greater than the first threshold value, the second correlation coefficient is greater than the second threshold value, and the third correlation coefficient is greater than the third threshold value to obtain a valid frequency point.

[0010] In one embodiment, combining the summation result and the fringing field noise parameter to calculate the third correlation coefficient includes: using the following formula to calculate the third correlation coefficient:

[0011]

[0012] Among them, x represents the summation result, y represents the diffuse field noise parameter, and R{x} represents the real part of x.

[0013] In one embodiment, a first correlation coefficient is calculated using correlation parameters of each microphone pair, including: summing the correlation parameters of each microphone pair to obtain a summation result; combining the summation result and diffuse field noise parameters to calculate the first correlation coefficient; a second correlation coefficient is calculated using the diffuse field noise parameters of each microphone pair, including: constructing a first correlation matrix based on the correlation parameters of each microphone pair, selecting the maximum eigenvalue of the first correlation matrix to obtain a first eigenvalue; constructing a second correlation matrix based on the diffuse field noise parameters of each microphone pair, selecting the maximum eigenvalue of the second correlation matrix to obtain a second eigenvalue; and calculating the second correlation coefficient using the first eigenvalue and the second eigenvalue; wherein the absolute value of the difference between the first eigenvalue and the second eigenvalue is proportional to the second correlation coefficient, and the first eigenvalue is inversely proportional to the second correlation coefficient.

[0014] In one embodiment, a first correlation coefficient is calculated using correlation parameters of each microphone pair, including: constructing a first correlation matrix based on the correlation parameters of each microphone pair, selecting the maximum eigenvalue of the first correlation matrix to obtain a first eigenvalue; using the first eigenvalue to calculate the first correlation coefficient; wherein the absolute value of the first eigenvalue is proportional to the first correlation coefficient; a second correlation coefficient is calculated using diffuse field noise parameters of each microphone pair, including: summing the correlation parameters of each microphone pair to obtain a summation result; and calculating the second correlation coefficient by combining the summation result and the diffuse field noise parameters.

[0015] The second aspect of the present application provides a sound source localization device, which includes: an audio preprocessing module, which is used to perform Fourier transform on the audio signal received by each microphone in a microphone array to obtain frequency domain signals corresponding to each microphone; a covariance calculation module, which is used to form a microphone pair with any two microphones in the microphone array, and calculate the covariance matrix between the current calculation frequency points in the frequency domain signals corresponding to the two microphones in the microphone pair; a correlation coefficient calculation module, which is used to calculate the correlation parameters between the two microphones in the microphone pair based on the covariance matrix, and calculate the first correlation coefficient using the correlation parameters of each microphone pair; and, based on the distance between the two microphones in the microphone pair and the angular frequency of the current calculation frequency point, calculate the diffuse field noise parameters, and calculate the second correlation coefficient using the diffuse field noise parameters of each microphone pair; a threshold comparison module, which is used to compare the size relationship between the first correlation coefficient and the second correlation coefficient and a preset threshold value, respectively, to obtain a comparison result; and a positioning module, which is used to take the current calculation frequency point whose comparison result meets the preset conditions as the valid frequency point, and use all the valid frequency points to perform sound source localization.

[0016] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned sound source localization method.

[0017] A fourth aspect of the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned sound source localization method is implemented.

[0018] The above scheme performs Fourier transform on the audio signal received by each microphone in the microphone array to obtain the frequency domain signal corresponding to each microphone; any two microphones in the microphone array are combined into a microphone pair, and the covariance matrix between the current calculation frequency points in the frequency domain signals corresponding to the two microphones in the microphone pair is calculated; based on the covariance matrix, the correlation parameter between the two microphones in the microphone pair is calculated, and the first correlation coefficient is calculated using the correlation parameter of each microphone pair; and based on the distance between the two microphones in the microphone pair and the angular frequency of the current calculation frequency point, the diffuse field noise parameter is calculated, and the second correlation coefficient is calculated using the diffuse field noise parameter of each microphone pair; the first correlation coefficient and the second correlation coefficient are compared with the preset threshold value to obtain a comparison result; the current calculation frequency point whose comparison result meets the preset conditions is taken as the effective frequency point, and all the effective frequency points are used to locate the sound source, the correlation coefficients between the microphones can be considered in multiple dimensions, and the effective frequency points in the current time frame can be determined by combining the multi-dimensional information, which can improve the accuracy of the selected effective frequency points, reduce the influence of low-quality audio signals on sound source positioning, and improve the accuracy of sound source positioning.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0021] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0022] Figure 2 is a flow chart of a sound source localization method shown in an exemplary embodiment of the present application;

[0023] Figure 3 is a block diagram of a sound source localization device shown in an exemplary embodiment of the present application;

[0024] Figure 4 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0026] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0027] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0028] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] The sound source localization method provided in the embodiment of the present application is described below.

[0030] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are connected to each other in communication.

[0031] The number of the terminal 110 may be one or more. The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0032] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0033] In one example, the server 120 may perform sound source localization processing on the audio signal obtained from the terminal 110 to obtain a sound source localization result. Of course, the server 120 may store the sound source localization result locally, transmit it back to the terminal 110, or transmit it to other terminals 110.

[0034] In one example, a client of a target application is installed and running in the terminal 110. For example, the target application may be an application that provides a sound source localization function. The server 120 may be a background server of the target application, configured to provide background services for the client of the target application.

[0035] In the sound source localization method provided in the embodiment of the present application, the execution subject of each step can be the terminal 110, such as the client of the target application installed and running in the terminal 110, or the server 120, or the terminal 110 and the server 120 interact and cooperate to execute, that is, part of the steps of the method are executed by the terminal 110 and the other part of the steps are executed by the server 120.

[0036] See also Figure 2 , Figure 2 is a flowchart of a sound source localization method shown in an exemplary embodiment of the present application. The sound source localization method can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0037] like Figure 2 As shown, the sound source localization method at least includes steps S210 to S250, which are described in detail as follows:

[0038] Step S210: Perform Fourier transform on the audio signal received by each microphone in the microphone array to obtain a frequency domain signal corresponding to each microphone.

[0039] The microphone array consists of multiple microphones, each of which is used to collect audio in the environment to obtain an audio signal.

[0040] The audio signal received by each microphone is Fourier transformed to obtain the frequency domain signal corresponding to each microphone, so as to convert the audio signal from the time domain to the frequency domain.

[0041] Specifically, the audio signal is firstly subjected to frame division and windowing processing, such as dividing the audio signal x(l) of the microphone into frames according to a preset number of sample points per frame, and superimposing analysis windows of a preset number of points.

[0042] The audio signal after the frame and window processing is subjected to Fast Fourier Transform (FFT), which can be specifically expressed as the following formula 1:

[0043] (Formula 1)

[0044] in, , , H represents conjugate transpose calculation, k represents frequency point, l represents time frame, M represents the number of microphones in the microphone array, x represents audio signal, y represents frequency domain signal, and w represents angular frequency.

[0045] Of course, other methods may also be used to convert the audio signal from the time domain to the frequency domain, such as discrete time Fourier transform (DTFT), discrete Fourier transform (DFT), etc., which are not limited in this application.

[0046] Step S220: any two microphones in the microphone array are combined into a microphone pair, and the covariance matrix between the currently calculated frequency points in the frequency domain signals respectively corresponding to the two microphones in the microphone pair is calculated.

[0047] The covariance of each frequency point of the frequency domain signal corresponding to any two microphones is calculated.

[0048] For example, the microphone pair includes microphone i and microphone j, and the current calculation frequency point is k, then the calculation formula of the covariance matrix can be expressed as the following formula 2:

[0049] (Formula 2)

[0050] in, is the smoothing coefficient, such as the first-order smoothing coefficient. The basic idea of ​​first-order exponential smoothing is to give higher weights to the most recent observations, while the weights of distant observations decrease exponentially; y represents the frequency domain signal; is the covariance matrix of the frequency domain signals corresponding to microphone i and microphone j for frequency point k.

[0051] Step S230: Based on the covariance matrix, calculate the correlation parameters between the two microphones in the microphone pair, and use the correlation parameters of each microphone pair to calculate the first correlation coefficient; and, based on the distance between the two microphones in the microphone pair and the angular frequency of the current calculation frequency point, calculate the diffuse field noise parameters, and use the diffuse field noise parameters of each microphone pair to calculate the second correlation coefficient.

[0052] Based on the calculated covariance matrix, a correlation parameter between two microphones contained in the microphone pair is further obtained.

[0053] For example, the correlation parameter between microphone i and microphone j is proportional to the covariance matrix between microphone i and microphone j. Based on the covariance matrix, the correlation parameter between the two microphones included in the microphone pair is calculated.

[0054] For example, the correlation parameter calculation formula can be expressed as the following formula 3:

[0055] (Formula 3)

[0056] in, is the covariance matrix of the frequency domain signals corresponding to microphone i and microphone j for frequency point k; is the covariance matrix of the frequency domain signal corresponding to microphone i at frequency point k; is the covariance matrix of the frequency domain signal corresponding to microphone j at frequency point k; to a minimum value to prevent the denominator from being 0.

[0057] It can be seen from the above formula 3 that the correlation parameter between microphone i and microphone j is proportional to the covariance matrix between microphone i and microphone j, and is inversely proportional to the covariance matrix of microphone i and microphone j itself.

[0058] A first correlation coefficient is calculated according to the correlation parameter of each microphone pair.

[0059] In addition, diffuse field noise parameters are calculated based on the distance between two microphones in the microphone pair and the angular frequency of the current calculation frequency point.

[0060] The steady-state noise collected by the microphone array conforms to the diffuse field characteristics in space. The diffuse field noise is modeled and the diffuse field noise parameters are obtained.

[0061] Specifically, diffuse field noise parameters are obtained by modeling according to the distance between two microphones in the microphone pair and the angular frequency of the current calculation frequency point.

[0062] Taking the fringing field noise parameter obtained by modeling based on the sinc function as an example, the fringing field noise parameter can be expressed as the following formula 4:

[0063] (Formula 4)

[0064] in, is the angular frequency of the current calculated frequency point k, is the distance between microphone i and microphone j, c is the speed of sound propagation, is the diffuse field noise parameter.

[0065] It is understandable that, in addition to using the sinc function for modeling, other spectrum sampling functions may also be used for modeling, and this application does not limit this.

[0066] Step S240: Compare the magnitude relationships between the first correlation coefficient and the second correlation coefficient and the preset threshold value respectively to obtain a comparison result.

[0067] The preset threshold can be pre-set based on experience or can be flexibly set. For example, thresholds corresponding to different scene categories are pre-stored, the scene category of the audio collection scene corresponding to the microphone array is identified, the threshold matching the scene category is queried, and the threshold is compared with the first correlation coefficient and the second correlation coefficient; or, the audio quality of the audio signal collected by each microphone in the microphone array is identified, a threshold is set based on the audio quality of each microphone, and the threshold is compared with the first correlation coefficient and the second correlation coefficient.

[0068] The preset threshold may be a threshold value or a threshold interval, which is not limited in this application.

[0069] Step S250: taking the currently calculated frequency point whose comparison result meets the preset condition as the valid frequency point, and using all the valid frequency points to perform sound source localization.

[0070] It should be noted that the preset conditions may be pre-set based on experience, or may be flexibly set based on the scene category, audio quality, etc. of the audio acquisition scene.

[0071] For example, the preset threshold includes a first threshold value and a second threshold value. The first correlation coefficient is compared with the first threshold value, and the second correlation coefficient is compared with the second threshold value. If the comparison result shows that the first correlation coefficient is greater than the first threshold value, and the second correlation coefficient is greater than the second threshold value, then it is judged that the comparison result meets the preset condition; or, if the comparison result shows that the first correlation coefficient is greater than the first threshold value, or the second correlation coefficient is greater than the second threshold value, then it is judged that the comparison result meets the preset condition; or, the first correlation coefficient and the second correlation coefficient are weightedly summed, and if the weighted sum result is greater than the preset threshold, then it is judged that the comparison result meets the preset condition.

[0072] All frequency points whose comparison results meet the preset conditions are selected to obtain valid frequency points, and all valid frequency points are used to locate the sound source.

[0073] After obtaining the effective frequency points, the sound source orientation can be estimated based on the effective frequency points to obtain the sound source localization result. It should be noted that the specific implementation method of estimating the sound source orientation can refer to all the implementable methods in the prior art, and this application does not limit this.

[0074] For example, the estimation method of the sound source direction can refer to the following formulas 5 to 7:

[0075] (Formula 5)

[0076] (Formula 6)

[0077] (Formula 7)

[0078] in, is the angle to be traversed, is the direction vector composed of the traversal angle information, k represents the frequency point, is the noise subspace obtained by eigenvalue decomposition of the covariance matrix, is the index value corresponding to the maximum peak value of the current input sequence, Represents a set containing all valid frequency points.

[0079] By traversing each angle value and finding the maximum peak of P, the angle corresponding to the maximum peak is taken as the sound source angle.

[0080] The present application considers the correlation coefficients between microphones in multiple dimensions to determine the effective frequency points in the current time frame by combining multi-dimensional information, thereby improving the accuracy of the selected effective frequency points, reducing the impact of low-quality audio signals on sound source localization, and improving the accuracy of sound source localization.

[0081] Next, some embodiments of the present application are described in detail.

[0082] In some embodiments, the first correlation coefficient is calculated using the correlation parameters of each microphone pair, including: constructing a first correlation matrix based on the correlation parameters of each microphone pair; selecting the maximum eigenvalue of the first correlation matrix to obtain a first eigenvalue; using the first eigenvalue to calculate the first correlation coefficient; wherein the absolute value of the first eigenvalue is proportional to the first correlation coefficient.

[0083] The correlation parameters of each microphone pair are used as matrix elements to construct a first correlation matrix.

[0084] Exemplarily, assuming that the number of microphones is M, the constructed first correlation matrix is ​​expressed as the following formula 8:

[0085] (Formula 8)

[0086] Among them, the correlation parameter The calculation method of can refer to the above formula 3.

[0087] The maximum eigenvalue of the first correlation matrix is ​​selected to obtain the first eigenvalue. The absolute value of the first eigenvalue is proportional to the first correlation coefficient, and the first correlation coefficient is calculated using the first eigenvalue.

[0088] Optionally, the maximum eigenvalue of the selected first correlation matrix is ​​compensated, as shown in the following formula 9:

[0089] (Formula 9)

[0090] in, represents the maximum eigenvalue of the selected first correlation matrix, that is, the first eigenvalue, and M is the number of microphones.

[0091] According to the compensated value Calculate the first correlation coefficient, see the following formula 10 for details:

[0092] (Formula 10)

[0093] Calculate the first correlation coefficient of frequency point k .

[0094] In some embodiments, the second correlation coefficient is calculated using the diffuse field noise parameters of each microphone pair, including: constructing a second correlation matrix based on the diffuse field noise parameters of each microphone pair; selecting the maximum eigenvalue of the second correlation matrix to obtain a second eigenvalue; and calculating the second correlation coefficient using the first eigenvalue and the second eigenvalue; wherein the absolute value of the difference between the first eigenvalue and the second eigenvalue is directly proportional to the second correlation coefficient, and the first eigenvalue is inversely proportional to the second correlation coefficient.

[0095] The method for obtaining the first characteristic value is similar to that in the previous embodiment and will not be described in detail here.

[0096] The diffuse field noise parameters of each microphone pair are used as matrix elements to construct a second correlation matrix.

[0097] Exemplarily, assuming that the number of microphones is M, the constructed second correlation matrix is ​​expressed as the following formula 11:

[0098] (Formula 11)

[0099] Among them, the diffuse field noise parameter The calculation method of can refer to the above formula 4.

[0100] The maximum eigenvalue of the second correlation matrix is ​​selected to obtain the second eigenvalue. The second correlation coefficient is calculated based on the fact that the absolute value of the difference between the first eigenvalue and the second eigenvalue is proportional to the second correlation coefficient and the first eigenvalue is inversely proportional to the second correlation coefficient.

[0101] Optionally, the maximum eigenvalue of the selected second correlation matrix is ​​compensated, as shown in the following formula 12:

[0102] (Formula 12)

[0103] in, represents the maximum eigenvalue of the selected second correlation matrix, that is, the second eigenvalue, and M is the number of microphones.

[0104] According to the compensated value Calculate the second correlation coefficient, see the following formula 13 for details:

[0105] (Formula 13)

[0106] Calculate the second correlation coefficient of frequency point k .

[0107] In some implementations, based on the above embodiments, the method further includes: summing the correlation parameters of each microphone pair to obtain a summation result; combining the summation result and the diffuse field noise parameter to calculate a third correlation coefficient. Then, comparing the magnitude relationships between the first correlation coefficient, the second correlation coefficient, and the third correlation coefficient and a preset threshold value, respectively, to obtain a comparison result.

[0108] Exemplarily, the correlation parameters of each microphone pair are summed up and calculated, as shown in the following formula 14:

[0109] (Formula 14)

[0110] Among them, the correlation parameter The calculation method can be referred to in the above formula 3, where M is the number of microphones, to calculate the summation result .

[0111] Of course, the correlation parameters of each microphone pair may also be directly added to obtain a sum result, which is not limited in the present application.

[0112] Then, the third correlation coefficient is calculated using the following formula 15:

[0113] (Formula 15)

[0114] Among them, the diffuse field noise parameter The calculation method can refer to the above formula 4 to calculate the third correlation coefficient .

[0115] Among them, the summation result in formula 15 is As input x, diffuse field noise parameters The specific calculation method of the input y can be found in the following formula 16:

[0116] (Formula 16)

[0117] in, It means taking the real part of x.

[0118] It should be noted that Formula 16 is only a schematic illustration. In actual application scenarios, Formula 16 can be flexibly adjusted according to actual conditions, and this application does not limit this.

[0119] The first correlation coefficient, the second correlation coefficient and the third correlation coefficient calculated jointly are used to comprehensively judge whether the current calculated frequency point k can be used as a valid frequency point.

[0120] Exemplarily, the preset threshold value includes a first threshold value, a second threshold value and a third threshold value; comparing the size relationship between the first correlation coefficient, the second correlation coefficient and the third correlation coefficient and the preset threshold value, respectively, to obtain a comparison result, including: comparing the size relationship between the first correlation coefficient and the first threshold value, comparing the size relationship between the second correlation coefficient and the second threshold value, and comparing the size relationship between the third correlation coefficient and the third threshold value, respectively, to obtain a comparison result; selecting the current calculation frequency point whose comparison result meets the preset conditions to obtain a valid frequency point, including: selecting the current calculation frequency point whose first correlation coefficient is greater than the first threshold value, the second correlation coefficient is greater than the second threshold value, and the third correlation coefficient is greater than the third threshold value, to obtain a valid frequency point.

[0121] For example, the threshold judgment rule can be expressed as the following formula 17:

[0122] (Formula 17)

[0123] in, , , are the calculated first correlation coefficient, second correlation coefficient and third correlation coefficient respectively; thr1, thr2 and thr3 are the first threshold value, the second threshold value and the third threshold value respectively, and the values ​​can be set according to actual conditions, such as thr1 is 0.2, thr2 is 1, and thr3 is 1.25; N(l) represents the set of all valid frequency points selected in the current time frame l.

[0124] In some embodiments, the first correlation coefficient is calculated using the correlation parameters of each microphone pair, including: summing the correlation parameters of each microphone pair to obtain a summation result; combining the summation result and the diffuse field noise parameter to calculate the first correlation coefficient. And, the second correlation coefficient is calculated using the diffuse field noise parameter of each microphone pair, including: constructing a first correlation matrix based on the correlation parameters of each microphone pair, selecting the maximum eigenvalue of the first correlation matrix to obtain a first eigenvalue; constructing a second correlation matrix based on the diffuse field noise parameters of each microphone pair, selecting the maximum eigenvalue of the second correlation matrix to obtain a second eigenvalue; calculating the second correlation coefficient using the first eigenvalue and the second eigenvalue; wherein the absolute value of the difference between the first eigenvalue and the second eigenvalue is proportional to the second correlation coefficient, and the first eigenvalue is inversely proportional to the second correlation coefficient.

[0125] That is, the first correlation coefficient is calculated using the embodiment corresponding to formula 15, and the second correlation coefficient is calculated using the embodiment corresponding to formula 13.

[0126] In some embodiments, the first correlation coefficient is calculated using the correlation parameters of each microphone pair, including: constructing a first correlation matrix based on the correlation parameters of each microphone pair, selecting the maximum eigenvalue of the first correlation matrix to obtain a first eigenvalue; using the first eigenvalue to calculate the first correlation coefficient; wherein the absolute value of the first eigenvalue is proportional to the first correlation coefficient. And, the second correlation coefficient is calculated using the diffuse field noise parameters of each microphone pair, including: summing the correlation parameters of each microphone pair to obtain a summation result; combining the summation result and the diffuse field noise parameters to calculate the second correlation coefficient.

[0127] That is, the first correlation coefficient is calculated using the embodiment corresponding to formula 10, and the second correlation coefficient is calculated using the embodiment corresponding to formula 15.

[0128] In some embodiments, based on the covariance matrix, a correlation parameter between two microphones in a microphone pair is calculated, the correlation parameters of each microphone pair are summed to obtain a first correlation coefficient, and based on the distance between the two microphones in the microphone pair and the angular frequency of the current calculation frequency point, a diffuse field noise parameter is calculated, and the diffuse field noise parameter is used as the second correlation coefficient. Then, the time frame correlation coefficient is calculated using Formula 15, and the comparison result can be obtained by comparing the size relationship between the time frame correlation coefficient and a preset threshold.

[0129] In some embodiments, based on the covariance matrix, the correlation parameters between the two microphones in the microphone pair are calculated, and a first correlation matrix is ​​constructed based on the correlation parameters of each microphone pair; the maximum eigenvalue of the first correlation matrix is ​​selected to obtain the first eigenvalue; the first eigenvalue is used as the first correlation coefficient; a second correlation matrix is ​​constructed based on the diffuse field noise parameters of each microphone pair; the maximum eigenvalue of the second correlation matrix is ​​selected to obtain the second eigenvalue; the second correlation coefficient is calculated using the first correlation coefficient and the second eigenvalue. Then, it is only necessary to compare the magnitude relationship between the second correlation coefficient and the preset threshold value to obtain the comparison result.

[0130] In combination with the above embodiments, effective frequency points that are convenient for positioning are selected, and sound source positioning is performed according to the selected effective frequency points, thereby improving the anti-reverberation and anti-noise capabilities of sound source positioning.

[0131] The sound source localization method provided by the present application performs Fourier transform on the audio signal received by each microphone in the microphone array to obtain the frequency domain signal corresponding to each microphone; any two microphones in the microphone array form a microphone pair, and calculate the covariance matrix between the current calculation frequency points in the frequency domain signals corresponding to the two microphones in the microphone pair; based on the covariance matrix, calculate the correlation parameter between the two microphones in the microphone pair, and use the correlation parameter of each microphone pair to calculate the first correlation coefficient; and based on the distance between the two microphones in the microphone pair and the angular frequency of the current calculation frequency point, calculate the diffuse field noise parameter, and use the diffuse field noise parameter of each microphone pair to calculate the second correlation coefficient; compare the size relationship between the first correlation coefficient and the second correlation coefficient and the preset threshold value respectively to obtain a comparison result; take the current calculation frequency point whose comparison result meets the preset conditions as the effective frequency point, and use all the effective frequency points to perform sound source localization, and can consider the correlation coefficients between the microphones in multiple dimensions, so as to determine the effective frequency points in the current time frame by combining the multi-dimensional information, so as to improve the accuracy of the selected effective frequency points, reduce the influence of low-quality audio signals on sound source localization, and improve the accuracy of sound source localization.

[0132] Figure 3 FIG. 1 is a block diagram of a sound source localization device shown in an exemplary embodiment of the present application. Figure 3 As shown, the exemplary sound source localization device 300 includes: an audio preprocessing module 310, a covariance calculation module 320, a correlation coefficient calculation module 330, a threshold comparison module 340 and a localization module 350. Specifically:

[0133] The audio preprocessing module 310 is used to perform Fourier transform on the audio signal received by each microphone in the microphone array to obtain a frequency domain signal corresponding to each microphone;

[0134] A covariance calculation module 320 is used to form a microphone pair with any two microphones in the microphone array, and calculate the covariance matrix between the current calculation frequency points in the frequency domain signals corresponding to the two microphones in the microphone pair;

[0135] The correlation coefficient calculation module 330 is used to calculate the correlation parameter between the two microphones in the microphone pair based on the covariance matrix, and calculate the first correlation coefficient using the correlation parameter of each microphone pair; and calculate the diffuse field noise parameter based on the distance between the two microphones in the microphone pair and the angular frequency of the current calculation frequency point, and calculate the second correlation coefficient using the diffuse field noise parameter of each microphone pair;

[0136] A threshold comparison module 340 is used to compare the magnitude relationship between the first correlation coefficient and the second correlation coefficient and a preset threshold value, and obtain a comparison result;

[0137] The positioning module 350 is used to use the current calculated frequency point whose comparison result meets the preset conditions as the effective frequency point, and use all the effective frequency points to perform sound source positioning.

[0138] It should be noted that the sound source localization device provided in the above embodiment and the sound source localization method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the sound source localization device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0139] See also Figure 4 , Figure 4 4 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 400 includes a memory 401 and a processor 402, and the processor 402 is used to execute the program instructions stored in the memory 401 to implement the steps in any of the above-mentioned sound source positioning method embodiments. In a specific implementation scenario, the electronic device 400 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 400 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0140] Specifically, the processor 402 is used to control itself and the memory 401 to implement the steps in any of the above-mentioned sound source localization method embodiments. The processor 402 can also be called a central processing unit (CPU). The processor 402 may be an integrated circuit chip with signal processing capabilities. The processor 402 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 402 can be implemented by an integrated circuit chip.

[0141] See also Figure 5 , Figure 5 The computer-readable storage medium 500 stores program instructions 510 that can be executed by a processor, and the program instructions 510 are used to implement the steps in any of the above-mentioned sound source localization method embodiments.

[0142] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0143] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0144] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0145] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

Claims

1. A sound source localization method, characterized in that: include: Performing Fourier transform on the audio signal received by each microphone in the microphone array to obtain frequency domain signals corresponding to each microphone; Any two microphones in the microphone array are combined into a microphone pair, and a covariance matrix between currently calculated frequency points in the frequency domain signals respectively corresponding to the two microphones in the microphone pair is calculated; Based on the covariance matrix, a correlation parameter between two microphones in the microphone pair is calculated, and a first correlation coefficient is calculated using the correlation parameter of each microphone pair; and, based on the distance between two microphones in the microphone pair and the angular frequency of the current calculation frequency point, calculating a diffuse field noise parameter, and calculating a second correlation coefficient using the diffuse field noise parameter of each microphone pair; and, summing up the correlation parameters of each microphone pair to obtain a summation result, and combining the summation result with the diffuse field noise parameter to calculate a third correlation coefficient; Comparing the magnitude relationships between the third correlation coefficient, the first correlation coefficient, and the second correlation coefficient and a preset threshold value, respectively, to obtain a comparison result; The currently calculated frequency point whose comparison result meets the preset condition is taken as the effective frequency point, and all effective frequency points are used to perform sound source positioning.

2. The method according to claim 1, characterized in that The step of calculating the first correlation coefficient using the correlation parameter of each microphone pair includes: A first correlation matrix is ​​constructed based on the correlation parameters of each microphone pair; Selecting the maximum eigenvalue of the first correlation matrix to obtain a first eigenvalue; A first correlation coefficient is calculated using the first eigenvalue; wherein the absolute value of the first eigenvalue is proportional to the first correlation coefficient.

3. The method according to claim 2, characterized in that The step of calculating the second correlation coefficient using the diffuse field noise parameter of each microphone pair includes: A second correlation matrix is ​​constructed based on diffuse field noise parameters of each microphone pair; Selecting the maximum eigenvalue of the second correlation matrix to obtain a second eigenvalue; A second correlation coefficient is calculated using the first eigenvalue and the second eigenvalue; wherein the absolute value of the difference between the first eigenvalue and the second eigenvalue is proportional to the second correlation coefficient, and the first eigenvalue is inversely proportional to the second correlation coefficient.

4. The method according to claim 1, characterized in that: The preset threshold value includes a first threshold value, a second threshold value and a third threshold value; the comparison of the first correlation coefficient, the second correlation coefficient and the third correlation coefficient with the preset threshold value to obtain a comparison result includes: Respectively comparing the magnitude relationship between the first correlation coefficient and the first threshold value, comparing the magnitude relationship between the second correlation coefficient and the second threshold value, and comparing the magnitude relationship between the third correlation coefficient and the third threshold value to obtain a comparison result; The taking the current calculated frequency point whose comparison result satisfies the preset condition as the effective frequency point includes: Select the current calculation frequency point where the first correlation coefficient is greater than the first threshold value, the second correlation coefficient is greater than the second threshold value, and the third correlation coefficient is greater than the third threshold value to obtain a valid frequency point.

5. The method according to claim 1, characterized in that The combining the summation result and the fringing field noise parameter to calculate a third correlation coefficient comprises: The third correlation coefficient is calculated using the following formula: Among them, x represents the summation result, y represents the diffuse field noise parameter, and R{x} represents the real part of x.

6. The method according to claim 1, characterized in that The step of calculating the first correlation coefficient using the correlation parameter of each microphone pair includes: The correlation parameters of each microphone pair are summed to obtain a summation result; Combining the summation result and the diffuse field noise parameter, calculating a first correlation coefficient; The step of calculating the second correlation coefficient using the diffuse field noise parameter of each microphone pair includes: A first correlation matrix is ​​constructed based on the correlation parameters of each microphone pair, and a maximum eigenvalue of the first correlation matrix is ​​selected to obtain a first eigenvalue; A second correlation matrix is ​​constructed based on the diffuse field noise parameters of each microphone pair, and a maximum eigenvalue of the second correlation matrix is ​​selected to obtain a second eigenvalue; A second correlation coefficient is calculated using the first eigenvalue and the second eigenvalue; wherein the absolute value of the difference between the first eigenvalue and the second eigenvalue is proportional to the second correlation coefficient, and the first eigenvalue is inversely proportional to the second correlation coefficient.

7. The method according to claim 1, characterized in that The step of calculating the first correlation coefficient using the correlation parameter of each microphone pair includes: A first correlation matrix is ​​constructed based on the correlation parameters of each microphone pair, and a maximum eigenvalue of the first correlation matrix is ​​selected to obtain a first eigenvalue; Calculating a first correlation coefficient using the first eigenvalue; wherein the absolute value of the first eigenvalue is proportional to the first correlation coefficient; The step of calculating the second correlation coefficient using the diffuse field noise parameter of each microphone pair includes: Calculating the sum of the correlation parameters of each microphone pair to obtain a sum result; The second correlation coefficient is calculated by combining the summation result and the fringing field noise parameter.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, and the processor is used to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 7.

Images