Residual echo estimation method, apparatus, system, device, and storage medium

Abstract

Embodiments of the application disclose a residual echo estimation method, device, system, equipment and storage medium, wherein the method comprises the following steps: determining a reference signal of remote transmission and an error signal of the reference signal after linear echo cancellation by a main microphone; acquiring a reference microphone signal collected by a reference microphone; performing data analysis on energy of the error signal, energy of the reference signal and energy of the reference microphone signal to determine an estimated correlation coefficient between the reference signal and a residual echo to be estimated; and performing residual echo estimation according to the estimated correlation coefficient and the reference signal.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of audio and video communication, and particularly to a method, apparatus, system, device, and storage medium for residual echo estimation. Background Technology

[0002] In voice communication systems, the voice of a distant speaker is transmitted and played back through a near-end speaker. After being picked up by a near-end microphone, it is then eliminated by an echo cancellation system. Traditional methods for estimating residual echo rely on the correlation between the residual echo signal and the distant reference signal. This involves performing correlation analysis on the error signal after linear echo cancellation and the distant reference signal, calculating the correlation coefficient between them, and using this coefficient to determine the proportion of residual echo, thus deciding the degree of residual echo suppression.

[0003] However, the error signal is not only determined by the far-end reference signal, but also includes the near-end speech signal and noise signal, which makes the residual echo have the characteristics of rapid change and large dynamic range. Therefore, traditional residual echo estimation methods may lead to insufficient residual echo estimation, resulting in residual echo, or excessive residual echo estimation, resulting in near-end speech damage. Summary of the Invention

[0004] In view of this, embodiments of this application provide at least one residual echo estimation method, apparatus, system, device, and storage medium.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a residual echo estimation method applied to an acoustic acquisition system, the acoustic acquisition system including a sound-emitting unit, a main microphone, and a reference microphone, wherein the position of the reference microphone is no greater than a preset threshold from the position of the sound-emitting unit, and the method includes:

[0007] The process involves determining a reference signal transmitted from a remote location and an error signal after linear echo cancellation of the reference signal by the main microphone; acquiring a reference microphone signal collected by the reference microphone; performing data analysis on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal to determine the estimated correlation coefficient between the reference signal and the residual echo to be estimated; and estimating the residual echo based on the estimated correlation coefficient and the reference signal.

[0008] Secondly, embodiments of this application provide a residual echo estimation device applied to an acoustic acquisition system. The acoustic acquisition system includes a sound-emitting unit, a main microphone, and a reference microphone. The position of the reference microphone is no greater than a preset threshold from the position of the sound-emitting unit. The device includes:

[0009] The first determining module is used to determine the reference signal transmitted from the remote end, and the error signal after the main microphone performs linear echo cancellation on the reference signal;

[0010] The first acquisition module is used to acquire the reference microphone signal collected by the reference microphone;

[0011] The second determining module is used to perform data analysis on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal to determine the estimation correlation coefficient between the reference signal and the residual echo to be estimated.

[0012] The echo estimation module is used to perform residual echo estimation based on the estimated correlation coefficient and the reference signal.

[0013] Thirdly, embodiments of this application provide an acoustic acquisition system, including a sound-emitting unit, a main microphone, and a reference microphone, wherein the distance between the position of the reference microphone and the position of the sound-emitting unit is no greater than a preset threshold; the sound-emitting unit is used to play a reference signal from a distant location; the main microphone and the reference microphone are used to receive the reflected signal of the reference signal at a near location; and residual echo estimation is performed in the acoustic acquisition system according to the residual echo estimation method described in the first aspect.

[0014] Fourthly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0015] Fifthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0016] In this embodiment, firstly, a reference signal for remote transmission and an error signal after linear echo cancellation of the reference signal by the main microphone are determined; then, a reference microphone signal collected by the reference microphone is acquired; next, data analysis is performed on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal to determine the estimated correlation coefficient between the reference signal and the residual echo to be estimated; finally, residual echo estimation is performed based on the estimated correlation coefficient and the reference signal. Thus, by adding a reference microphone and using both the reference microphone and the main microphone to calculate the correlation parameter with the remote reference signal, both residual echo and near-end signal can be considered simultaneously, allowing for more accurate estimation of residual echo in principle. This enables adaptation to changes in near-end speech signals and noise, resulting in more precise residual echo estimation and ensuring good speech quality.

[0017] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 A schematic diagram of an optional process for the residual echo estimation method provided in an embodiment of this application;

[0020] Figure 2 A schematic diagram of an optional process for the residual echo estimation method provided in an embodiment of this application;

[0021] Figure 3 A schematic diagram of an optional process for the residual echo estimation method provided in an embodiment of this application;

[0022] Figure 4A This is a schematic diagram of the acoustic acquisition system provided in the embodiments of this application;

[0023] Figure 4B A logic flowchart for residual echo estimation provided in the embodiments of this application;

[0024] Figure 5 A schematic diagram of the signal model for residual echo estimation provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the composition structure of a residual echo estimation device provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to be limiting of this application.

[0031] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0032] Near end: The local end in the communication network during a voice call.

[0033] Remote end: The other end of the communication network during a voice call.

[0034] Near-end device: The communication device used by the speaker at the near end in a voice call. The near-end device is equipped with audio acquisition devices (such as microphones) and audio playback devices (such as speakers and receivers).

[0035] Remote device: The communication device used by the speaker at a remote end during a voice call. The remote device is equipped with audio acquisition devices (such as microphones) and audio playback devices (such as speakers and receivers).

[0036] Near-end speech signal: During a voice call, the speech signal of the speaker at the near end is captured by the audio acquisition device of the near-end device.

[0037] Remote voice signal: During a voice call, the speech of the speaker at the remote end is captured by the audio acquisition device of the remote device and transmitted to the near end device through the communication network.

[0038] Echo signal: During a voice call, the voice signal from the far end is played by the audio playback device of the near end device and then captured by the audio acquisition device of the near end device.

[0039] Echo cancellation: The process of filtering out echo signals from the audio acquisition signal acquired by the audio acquisition device of the near-end device.

[0040] Assuming the two devices conducting the voice call are A and B, from the perspective of user a (user 'a') of device A, device A is the near-end device, and B is the corresponding far-end device (the peer device). Similarly, from the perspective of user b (user 'b') of device B, device B is the near-end device, and A is the corresponding far-end device. When A is the near-end device, the voice signal of the local speaker (user 'a') captured by A's audio acquisition device is the near-end voice signal, and the voice signal sent by B to A (the voice signal generated by the peer speaker, user 'b') is the far-end voice signal, also known as the reference signal. The voice signal captured by A's audio acquisition device after the far-end voice signal is played by A's audio playback device is the echo signal. The process of eliminating the echo signal from the voice signal captured by A's audio acquisition device is called echo cancellation.

[0041] In voice communication systems, the voice of a distant speaker is transmitted and played back through a near-end speaker. After being picked up by a near-end microphone, it is then processed by an echo cancellation system. However, linear echo cancellation is often ineffective at completely eliminating echoes, and poor residual echo suppression can severely impact the listening experience. Traditional residual echo estimation uses the correlation between the residual echo signal and the distant reference signal. Specifically, it performs correlation analysis between the error signal after linear echo cancellation and the distant reference signal to calculate the correlation coefficient between the two signals. This unique correlation coefficient is used to determine the proportion of residual echo, thereby deciding the degree of residual echo suppression.

[0042] Existing residual echo estimation methods, which use a single correlation parameter calculated from the far-end reference signal and the error signal to determine residual echoes, have significant drawbacks, resulting in inaccurate estimations. This is because the error signal is not solely determined by the far-end reference signal but also includes the near-end speech signal and noise, leading to residual echoes exhibiting rapid changes and a large dynamic range. Consequently, traditional residual echo estimation methods can result in either underestimation of residual echoes, leading to echo persistence, or overestimation, causing near-end speech impairment.

[0043] This application provides a residual echo estimation method applied to an acoustic acquisition system. The acoustic acquisition system includes a sound-emitting unit, a main microphone, and a reference microphone. The distance between the reference microphone and the sound-emitting unit is no greater than a preset threshold. The sound-emitting unit can specifically be a loudspeaker. This method can be executed by a terminal system or a server.

[0044] It should be noted that the terminal system can be a terminal device that integrates audio playback devices, audio acquisition devices, and processors. The specific type of this terminal device is not limited in this application embodiment, as long as it is a device capable of voice calls, including but not limited to mobile phones, tablets, etc. The terminal system can also be a voice call system composed of independent audio playback devices, audio acquisition devices, and processors. For example, the terminal system can be a video conferencing system, which includes multiple audio acquisition devices (such as microphones), one or more audio playback devices (such as speakers), and a processor. The audio acquisition devices and audio playback devices can be distributed according to the actual needs of the meeting venue.

[0045] Understandably, when the terminal system is a terminal device integrating various components, the method can be executed by the processor integrated in the terminal device or by the server corresponding to the terminal device. When the terminal system is a voice call system composed of multiple independent devices, the method can be executed by the processor in the terminal system or by the server corresponding to the terminal system. In short, the method can be executed by either the terminal system or the server. Specifically, for the terminal system executing this voice call method, the terminal system is the near-end device in the current voice call, and the far-end device communicating with the terminal system is the remote device.

[0046] Figure 1 The following is a schematic diagram of an optional process for the residual echo estimation method provided in the embodiments of this application, such as... Figure 1 As shown, the method includes the following steps S110 to S140:

[0047] Step S110: Determine the reference signal for remote transmission and the error signal after the main microphone performs linear echo cancellation on the reference signal.

[0048] Here, the reference signal transmitted at the far end can be an audio signal transmitted from the far end that is not played by the near-end speaker. The reference signal is played by the speaker and reflected at the near end and received by the microphone, so that the reference signal can be obtained based on the far-end signal played by the near-end speaker.

[0049] The error signal is the difference between the signal acquired by the main microphone and the estimated echo signal, also known as the residual signal. The signal acquired by the main microphone includes near-end speech signals, echo signals, and ambient noise signals. In voice calls, echo signals and ambient noise signals need to be eliminated. The echo signal is generated due to the presence of far-end speech signals, and this echo signal can be the residual echo signal after linear echo cancellation of the acquired signal. It should be noted that the signal in the embodiments of this application can be an audio signal.

[0050] Step S120: Obtain the reference microphone signal collected by the reference microphone.

[0051] Here, the reference microphone signal acquired by the reference microphone may include, but is not limited to, the user's voice signal acquired by the reference microphone at the near end and the audio signal acquired by the reference microphone at the far end that is transmitted through the network link and played by the near end speaker.

[0052] Step S130: Perform data analysis on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal to determine the estimation correlation coefficient between the reference signal and the residual echo to be estimated.

[0053] Here, since the residual echo is generated by the reflection of the far-end reference signal through the near-end, it can be considered that the residual echo is strongly correlated with the far-end reference signal. Compared with the shortcomings of traditional residual echo estimation, which relies solely on the error signal of the far-end reference signal and the single main microphone to calculate the residual echo, the embodiments of this application use both the reference microphone and the main microphone to jointly calculate the correlation parameters with the reference signal.

[0054] It should be noted that the signal energy of each signal can reflect the magnitude of the speech signal it contains. In practical applications, the signal energy of a speech signal can be determined based on the signal amplitude or peak envelope of the speech signal.

[0055] Step S140: Perform residual echo estimation based on the estimated correlation coefficient and the reference signal.

[0056] Here, by making a certain assumption that the current residual echo and the predicted signal have a certain linear correlation, the residual echo can be estimated by estimating the correlation coefficient.

[0057] In some implementations, the estimated correlation coefficient is smoothed in the frequency domain; the smoothed estimated correlation coefficient is then used to filter out residual echoes from the reference signal.

[0058] In this embodiment, firstly, a reference signal for remote transmission and an error signal after linear echo cancellation of the reference signal by the main microphone are determined; then, a reference microphone signal collected by the reference microphone is acquired; next, data analysis is performed on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal to determine the estimated correlation coefficient between the reference signal and the residual echo to be estimated; finally, residual echo estimation is performed based on the estimated correlation coefficient and the reference signal. Thus, by adding a reference microphone and using both the reference microphone and the main microphone to calculate the correlation parameter with the remote reference signal, both residual echo and near-end signal can be considered simultaneously, allowing for more accurate estimation of residual echo in principle. This enables adaptation to changes in near-end speech signals and noise, resulting in more precise residual echo estimation and ensuring good speech quality.

[0059] based on Figure 1 , Figure 2 The following is a schematic diagram of an optional process for the residual echo estimation method provided in the embodiments of this application, such as... Figure 2 As shown, step S130 above may include steps S210 to S230 as follows:

[0060] Step S210: Determine the echo relative transfer function based on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal.

[0061] Here, the echo relative transfer function Γ(k,i) represents the correlation coefficient H2(k,i) between the echo signal from the reference microphone and the far-end reference signal, and the correlation coefficient between the residual echo from the main microphone and the far-end reference signal. The ratio.

[0062] Since the error signal e1(k,i) of the main microphone includes the first near-end signal s1(k,i) and residual echo... Simultaneously referencing the microphone signal y2(k,i), which includes the second near-end signal s2(k,i) and the echo signal d2(k,i), the expression for the echo relative transfer function Γ(k,i) can be obtained. Therefore, the cross-power spectral density between the reference signal x(k,i) and the error signal e1(k,i) of the main microphone can be calculated based on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal. And the cross-power spectral density between the reference signal x(k,i) and the reference microphone signal y2(k,i), i.e. Where G1(k,i) and G2(k,i) are the near-end speech signal transfer functions of the main microphone and the reference microphone, respectively, the correlation coefficient H2(k,i) and the correlation coefficient are then derived. The proportional relationship between them yields the echo relative transfer function Γ(k,i).

[0063] In some implementations, for near-field acoustic signals, due to Φ xs (k,i)≈0, the above step S210 can be implemented through the following process: determining the first cross-power spectral density between the energy of the error signal and the energy of the reference signal; determining the second cross-power spectral density between the energy of the reference microphone signal and the energy of the reference signal; and determining the echo relative transfer function based on the ratio of the first cross-power spectral density to the second cross-power spectral density.

[0064] Step S220: Determine the near-end signal relative transfer function based on the energy of the error signal and the energy of the reference microphone signal;

[0065] Here, the near-end signal relative transfer function Θ(k,i) represents the ratio of the reference microphone near-end speech transfer function G2(k,i) to the main microphone near-end speech transfer function G1(k,i). Therefore, the relationship between the second near-end signal and the first near-end signal can be expressed as s2(k,i)=Θ(k,i)·s1(k,i).

[0066] When the call is in near-end one-way mode, since there is no far-end voice signal, the reference signal x(k,i) = 0. Therefore, the error signal e1(k,i) = G1(k,i)·s(k,i) of the main microphone is obtained, and the reference microphone signal y2(k,i) = G2(k,i)·s(k,i). Based on the energy of the error signal and the energy of the reference microphone signal, the cross-power spectral density between the reference microphone signal y2(k,i) and the error signal e1(k,i) of the main microphone is calculated respectively. And the cross coefficients between the reference microphone signal y2(k,i) and the reference microphone signal y2(k,i). Then, the proportional relationship between the near-end speech transfer function G2(k,i) of the reference microphone and the near-end speech transfer function G1(k,i) of the main microphone is derived, and the echo relative transfer function Γ(k,i) is obtained.

[0067] In some implementations, step S220 above can be performed by the following process: determining a first cross-power spectral density between the energy of the error signal and the energy of the reference signal; determining a third cross-power spectral density between the energy of the error signal and the energy of the reference microphone signal; and determining the near-end signal relative transfer function based on the ratio of the second cross-power spectral density to the energy of the reference microphone signal.

[0068] Step S230: Using the echo relative transfer function and the near-end signal relative transfer function, determine the estimated correlation coefficient between the reference signal and the residual echo to be estimated.

[0069] Here, due to the signal energy of the reference microphone's acquired signal... Energy containing near-end signals and the energy of the echo signal Right now Meanwhile, the residual signal of the main microphone Includes proximal signals and residual echoes Right now By substituting the expressions for the echo relative transfer function and the near-end signal relative transfer function into the above relationships, the estimated residual echo can be obtained. The expression is then used to analyze the estimated correlation coefficient between the reference signal and the residual echo to be estimated.

[0070] In this embodiment of the application, based on the cross-power spectral density of the error signal and the reference signal... Cross-power spectral density of the reference signal and the reference microphone signal Calculate the echo relative transfer function; based on the cross-power spectral density of the error signal and the reference microphone signal. Signal energy of the reference microphone The process involves calculating the relative transfer function of the near-end signal; calculating the estimated correlation coefficient between the reference signal and the residual echo based on the relative transfer function of the echo and the relative transfer function of the near-end signal; and estimating the residual echo based on the estimated correlation coefficient between the reference signal and the reference signal. By using a reference microphone, both the residual echo and the near-end signal can be considered simultaneously, allowing for a more accurate estimation of the residual echo in principle. This improves the echo cancellation effect and further enhances the listening experience.

[0071] based on Figure 2 , Figure 3 The following is a schematic diagram of an optional process for the residual echo estimation method provided in the embodiments of this application, such as... Figure 3 As shown, step S230 above, "using the echo relative transfer function and the near-end signal relative transfer function to determine the estimated correlation coefficient between the reference signal and the residual echo," may include the following steps S310 to S340:

[0072] Step S310: Determine the second near-end signal energy of the reference microphone based on the near-end signal relative transfer function and the first near-end signal energy in the energy of the error signal.

[0073] Here, the first near-end signal is obtained by filtering the estimated residual echo. In implementation, the estimated residual echo is further filtered by a post-filter to obtain the output first near-end signal.

[0074] It should be noted that the second near-end signal energy of the reference microphone satisfies The square of the near-end signal relative transfer function is then multiplied by the first near-end signal energy to obtain the second near-end signal energy of the reference microphone.

[0075] Step S320: Determine the echo signal energy of the reference microphone based on the energy of the reference microphone signal and the energy of the second near-end signal of the reference microphone.

[0076] Here, due to the energy of the signal collected by the reference microphone Energy containing near-end signals and echo signal energy Right now Therefore, the echo signal energy of the reference microphone can be determined based on this principle.

[0077] Step S330: Determine the residual echo energy in the energy of the error signal based on the echo relative transfer function and the echo signal energy of the reference microphone.

[0078] Here, since the echo is generated by the reflection of the far-end reference signal through the near-end, it can be assumed that the residual echo energy of the main microphone is... Energy Φ of the reference signal at the far end xx (k,i) are strongly correlated, that is in This indicates the correlation between the residual echo and the reference signal.

[0079] Similarly, referencing microphone signal energy Energy Φ of the reference signal at the far end xx (k,i) is also strongly correlated, that is So it is possible Therefore, based on the echo relative transfer function and the echo signal energy of the reference microphone, the residual echo energy in the energy of the error signal is determined.

[0080] Step S340: Determine the estimated correlation coefficient between the reference signal and the residual echo based on the relationship between the energy of the reference signal and the energy of the residual echo.

[0081] Here, the estimated correlation coefficient between the reference signal and the residual echo is as described above.

[0082] In this embodiment, due to the presence of a reference microphone, the calculation of residual echo not only takes into account the echo residue in the error signal, but also the near-end speech signal in the error signal. Therefore, it is more accurate than simply using the correlation between the residual signal and the far-end reference signal.

[0083] The residual echo estimation method described above will be illustrated below with a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this application and does not constitute an improper limitation of this application.

[0084] This application provides a method for residual echo estimation, which is applied to novel acoustic structures. Figure 4A This is a schematic diagram of the acoustic acquisition system provided in the embodiments of this application, as shown below. Figure 4A As shown, the acoustic acquisition system includes a speaker 41, a main microphone 42, and a reference microphone 43. The reference microphone 43 is located near the speaker 41. In implementation, the speaker 41 plays a distant speech signal, and the main microphone 42 and reference microphone 43 acquire echoes while acquiring the speech signal. Because the reference microphone 43 is closer to the speaker 41, the acquired echo is relatively large. In this embodiment, due to the presence of the reference microphone 43, the calculation of residual echoes considers not only the echo residue in the error signal but also the near-end speech signal in the error signal, demonstrating structural innovation.

[0085] Figure 4B The logic flowchart for residual echo estimation provided in the embodiments of this application is as follows: Figure 4B As shown, the process includes the following steps S410 to S440:

[0086] Step S410: Obtain the main microphone error signal energy, the reference microphone signal energy, the cross-power spectral density of the error signal and the reference signal, the cross-power spectral density of the reference microphone signal and the reference signal, and the cross-power spectral density of the error signal and the reference microphone signal.

[0087] Step S420: Calculate the echo relative transfer function and the near-end signal relative transfer function based on the cross-power spectral density of the error signal and the reference signal, the cross-power spectral density of the reference microphone signal and the reference signal, and the cross-power spectral density of the error signal and the reference microphone signal.

[0088] Here, the echo relative transfer function is calculated based on the cross-power spectral density of the error signal and the reference signal, and the cross-power spectral density of the reference signal and the reference microphone signal; the near-end signal relative transfer function is calculated based on the cross-power spectral density of the error signal and the reference microphone signal, and the signal energy of the reference microphone signal.

[0089] It should be noted that, in practice, the above-mentioned cross-power spectral density can correspond to the energy in the aforementioned embodiments. In fact, the cross-power spectral density is the representation of energy in the frequency domain.

[0090] Step S430: Calculate the estimated correlation coefficient between the residual echo and the reference signal.

[0091] Step S440: Perform residual echo estimation based on the estimated correlation coefficient and the reference signal.

[0092] This application improves upon traditional residual echo estimation methods that rely solely on the error signal from a distant reference signal and a single main microphone to calculate residual echoes. By adding a reference microphone, both the reference and main microphones are used to calculate correlation parameters with the distant reference signal. Based on acoustic principles, a more reasonable correlation calculation method is employed, resulting in a more accurate calculation of the correlation between the residual echo and the distant reference signal. In principle, this application incorporates the influence of the near-end speech signal and noise, thereby providing a more accurate residual echo estimation.

[0093] Figure 5 This is a schematic diagram of the signal model for residual echo estimation provided in an embodiment of this application, as shown below. Figure 5 As shown, x(n) represents the far-end reference signal, which is played by the speaker 41 and received by the microphone after reflection at the near end. At the same time, the near-end speech signal s(n) is also received by the two microphones, and y1(n) and y2(n) represent the signals received by the main microphone 42 and the reference microphone 42, respectively. and These represent the acoustic echo paths estimated by the main microphone 42 and the reference microphone 43, respectively. and These represent the echoes estimated by the main microphone 42 and the reference microphone 43, respectively. e1(n) and e2(n) represent the error signals 43 of the main microphone 42 and the reference microphone, respectively.

[0094] The residual echo is calculated using the signal y2(n) received by the reference microphone 43 and the error signal e1(n) from the main microphone 42, based on the update filter 51. The error signal e1(n) from the main microphone 42 is then further filtered by the post-filter 52 (Echo Filtering) to obtain the output near-end speech signal.

[0095] It should be noted that the residual echo is generated by the reflection of the far-end reference signal x(n) from the near end. Therefore, it can be considered that the residual echo is strongly correlated with the far-end reference signal x(n), as expressed by the following formula (1):

[0096]

[0097] in, Φ represents the residual echo signal energy of the main microphone. xx (k,i) represents the signal energy of the distant reference signal x(n). This indicates the correlation between the two. H1(k,i) is the frequency domain representation of h1(n), representing the acoustic echo path from the actual far-end reference signal to the main microphone in the frequency domain. yes The frequency domain representation represents the estimated acoustic echo path from the far-end reference signal to the main microphone in the frequency domain. This represents the systematic error between the actual acoustic echo path and the estimated acoustic echo path. Here, k represents the frame number, and i represents the frequency.

[0098] Error signal energy of the main microphone Includes near-end signal energy and residual echo energy It can be expressed as the following formula (2):

[0099]

[0100] Reference microphone signal energy Includes near-end signal energy and echo signal energy It can be expressed as the following formula (3):

[0101]

[0102] Next, we will introduce two functions to handle residual echoes. Calculate the echo relative transfer function Γ(k,i) and the near-end signal relative transfer function Θ(k,i).

[0103] The echo relative transfer function Γ(k,i) represents the correlation coefficient H2(k,i) between the reference microphone echo and the far-end reference signal, and the correlation coefficient between the main microphone residual echo and the far-end reference signal. The ratio: Therefore, the echo d2(k,i) estimated by the reference microphone can be written as

[0104] Therefore, the error signal e1(k,i) of the main microphone and the reference microphone signal y2(k,i) can be written as formulas (4) and (5) respectively:

[0105]

[0106]

[0107] Where G1(k,i) and G2(k,i) are the near-end speech signal transfer functions of the main microphone and the reference microphone, respectively, s(k,i) is the frequency domain representation of the near-end speech signal s(n), and x(k,i) is the frequency domain representation of the far-end reference signal x(n).

[0108] Therefore, the cross-correlation coefficients between the reference signal x(k,i) and the error signal e1(k,i) of the main microphone and the reference microphone signal y2(k,i) are expressed as follows: (6) and (7) respectively:

[0109]

[0110]

[0111] Where, Φ xs (k,i) is the cross-power spectral density between the error signal e1(k,i) and the reference signal x(k,i), Φ xx (k,i) represents the signal energy of the reference signal.

[0112] Because for near-field sound signals, Φ xs (k,i)≈0, therefore, combining formulas (6) and (7), we can obtain the following formula (8):

[0113]

[0114] The near-end signal relative transfer function Θ(k,i) represents the ratio of the reference microphone near-end speech transfer function G2(k,i) to the main microphone near-end speech transfer function G1(k,i): Therefore, the near-end speech signal s2(k,i) can be written as s2(k,i)=Θ(k,i)·s1(k,i).

[0115] When the call is in near-end one-way mode, the far-end reference signal x(k,i) = 0, so formulas (4) and (5) can be simplified to the following formulas (9) and (10):

[0116] e1(k,i)=G1(k,i)·s(k,i) Formula (9);

[0117] y2(k,i)=G2(k,i)·s(k,i) Formula (10);

[0118] Therefore, the cross-power spectral density of the reference microphone signal y2(k,i) and the error signal e1(k,i) of the main microphone is... and the signal energy of the reference microphone signal y2(k,i) These are expressed as formulas (11) and (12), respectively:

[0119]

[0120]

[0121] Therefore, by combining formulas (11) and (12), the expression of the near-end signal relative transfer function Θ(k,i) can be obtained as follows: (13)

[0122]

[0123] Based on the echo relative transfer function Γ(k,i) expressed in formula (8) and the near-end signal relative transfer function Θ(k,i) expressed in formula (13), formula (3) can be rewritten to obtain the following formula (14):

[0124]

[0125] in, Let the signal energy of the near-end speech signal s1(n) be expressed as:

[0126] From formula (2), we can know the residual echo energy. Substituting into formula (14), we can obtain the estimated residual echo expression as formula (15):

[0127]

[0128] Because of the presence of the reference microphone, the calculation of residual echoes takes into account not only the echo residue in the error signal but also the near-end speech signal in the error signal. Therefore, it is more accurate than simply using the correlation between the error signal and the far-end reference signal.

[0129] Because the residual signal from the main microphone includes not only residual echoes but also near-end speech signals and noise, traditional residual echo estimation algorithms, which rely solely on the correlation parameter between the residual echo and the far-end reference signal, suffer from poor accuracy. This application's embodiment employs an innovative product structure that utilizes a reference microphone to simultaneously consider both residual echoes and near-end signals, theoretically allowing for a more accurate estimation of the residual echo. This improves echo cancellation effectiveness and further enhances the listening experience.

[0130] Based on the foregoing embodiments, this application provides a residual echo estimation device applied to an acoustic acquisition system. The acoustic acquisition system includes a sound-emitting unit, a main microphone, and a reference microphone. The position of the reference microphone is no greater than a preset threshold from the position of the sound-emitting unit. The device includes all the modules, as well as the sub-modules and units included in each module. It can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0131] Figure 6 This is a schematic diagram of the composition of a residual echo estimation device provided in an embodiment of this application, as shown below. Figure 6 As shown, the residual echo estimation device 600 includes: a first determining module 610, a first acquiring module 620, a second determining module 630, and an echo estimation module 640, wherein:

[0132] The first determining module 610 is used to determine the reference signal for remote transmission and the error signal after the main microphone performs linear echo cancellation on the reference signal;

[0133] The first acquisition module 620 is used to acquire the reference microphone signal collected by the reference microphone;

[0134] The second determining module 630 is used to perform data analysis on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal to determine the estimation correlation coefficient between the reference signal and the residual echo to be estimated.

[0135] The echo estimation module 640 is used to perform residual echo estimation based on the estimated correlation coefficient and the reference signal.

[0136] In some possible embodiments, the second determining module 630 includes: a first determining submodule, configured to determine an echo relative transfer function based on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal; a second determining submodule, configured to determine a near-end signal relative transfer function based on the energy of the error signal and the energy of the reference microphone signal; and a third determining submodule, configured to determine an estimation correlation coefficient between the reference signal and the residual echo to be estimated using the echo relative transfer function and the near-end signal relative transfer function.

[0137] In some possible embodiments, the first determining submodule includes: a first determining unit, configured to determine a first cross-power spectral density between the energy of the error signal and the energy of the reference signal; determine a second cross-power spectral density between the energy of the reference microphone signal and the energy of the reference signal; and determine the echo relative transfer function based on the ratio of the first cross-power spectral density to the second cross-power spectral density.

[0138] In some possible embodiments, the second determining submodule includes: a fourth determining unit for determining a first cross-power spectral density between the energy of the error signal and the energy of the reference signal; a fifth determining unit for determining a third cross-power spectral density between the energy of the error signal and the energy of the reference microphone signal; and a sixth determining unit for determining the near-end signal relative transfer function based on the ratio of the second cross-power spectral density to the energy of the reference microphone signal.

[0139] In some possible embodiments, the reference microphone signal includes a near-end speech signal and an echo signal. The third determining submodule includes: a seventh determining unit, configured to determine a second near-end signal energy of the reference microphone based on the near-end signal relative transfer function and a first near-end signal energy in the energy of the error signal; wherein the first near-end signal is obtained by filtering the estimated residual echo; an eighth determining unit, configured to determine the echo signal energy of the reference microphone based on the energy of the reference microphone signal and the second near-end signal energy of the reference microphone; a ninth determining unit, configured to determine the residual echo energy in the energy of the error signal based on the echo relative transfer function and the echo signal energy of the reference microphone; and a tenth determining unit, configured to determine an estimated correlation coefficient between the reference signal and the residual echo based on the relationship between the energy of the reference signal and the residual echo energy.

[0140] In some possible embodiments, the echo estimation module 640 includes: a smoothing submodule for frequency domain smoothing of the estimated correlation coefficient; and a filtering submodule for filtering residual echoes from the reference signal using the smoothed estimated correlation coefficient.

[0141] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0142] It should be noted that, in the embodiments of this application, if the above-mentioned residual echo estimation method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0143] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0144] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0145] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0146] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0147] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0148] It should be noted that, Figure 7 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 7 As shown, the hardware entity of the computer device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein:

[0149] Processor 701 typically controls the overall operation of computer device 700.

[0150] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.

[0151] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704.

[0152] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0153] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0155] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0157] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0158] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0159] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for estimating residual echoes, characterized in that, An acoustic acquisition system is applied, the acoustic acquisition system including a sound-generating unit, a main microphone, and a reference microphone, wherein the position of the reference microphone is no greater than a preset threshold from the position of the sound-generating unit, and the method includes: Determine the reference signal for remote transmission, and the error signal after the main microphone performs linear echo cancellation on the reference signal; Acquire the reference microphone signal collected by the reference microphone; Data analysis is performed on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal to determine the estimation correlation coefficient between the reference signal and the residual echo to be estimated. Residual echo estimation is performed based on the estimated correlation coefficient and the reference signal; The step of performing data analysis on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal to determine the estimation correlation coefficient between the reference signal and the residual echo to be estimated includes: The echo relative transfer function is determined based on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal. The near-end signal relative transfer function is determined based on the energy of the error signal and the energy of the reference microphone signal; The estimated correlation coefficient between the reference signal and the residual echo to be estimated is determined using the echo relative transfer function and the near-end signal relative transfer function.

2. The method according to claim 1, characterized in that, Determining the echo relative transfer function based on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal includes: Determine the first cross-power spectral density between the energy of the error signal and the energy of the reference signal; Determine the second cross-power spectral density between the energy of the reference microphone signal and the energy of the reference signal; The echo relative transfer function is determined based on the ratio of the first cross-power spectral density to the second cross-power spectral density.

3. The method according to claim 1, characterized in that, Determining the near-end signal relative transfer function based on the energy of the error signal and the energy of the reference microphone signal includes: Determine the first cross-power spectral density between the energy of the error signal and the energy of the reference signal; Determine the third cross-power spectral density between the energy of the error signal and the energy of the reference microphone signal; The near-end signal relative transfer function is determined based on the ratio of the second cross-power spectral density to the energy of the reference microphone signal.

4. The method according to claim 1, characterized in that, The reference microphone signal includes a near-end speech signal and an echo signal. Determining the estimated correlation coefficient between the reference signal and the residual echo using the echo relative transfer function and the near-end signal relative transfer function includes: The second near-end signal energy of the reference microphone is determined based on the near-end signal relative transfer function and the energy of the first near-end signal in the energy of the error signal; wherein the first near-end signal is obtained by filtering the estimated residual echo; The echo signal energy of the reference microphone is determined based on the energy of the reference microphone signal and the energy of the second near-end signal of the reference microphone; Based on the echo relative transfer function and the echo signal energy of the reference microphone, determine the residual echo energy in the energy of the error signal; Based on the relationship between the energy of the reference signal and the energy of the residual echo, the estimated correlation coefficient between the reference signal and the residual echo is determined.

5. The method according to any one of claims 1 to 4, characterized in that, The residual echo estimation based on the estimated correlation coefficient and the reference signal includes: The estimated correlation coefficient is smoothed in the frequency domain. The reference signal is filtered out by using the smoothed estimated correlation coefficient.

6. A residual echo estimation device, characterized in that, An acoustic acquisition system is used, the acoustic acquisition system including a sound-generating unit, a main microphone, and a reference microphone, wherein the position of the reference microphone is no greater than a preset threshold from the position of the sound-generating unit, and the device includes: The first determining module is used to determine the reference signal transmitted from the remote end, and the error signal after the main microphone performs linear echo cancellation on the reference signal; The first acquisition module is used to acquire the reference microphone signal collected by the reference microphone; The second determining module is used to perform data analysis on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal to determine the estimation correlation coefficient between the reference signal and the residual echo to be estimated. The echo estimation module is used to perform residual echo estimation based on the estimated correlation coefficient and the reference signal; The second determining module is further configured to: determine the echo relative transfer function based on the energy of the error signal, the energy of the reference signal, and the energy of the reference microphone signal; determine the near-end signal relative transfer function based on the energy of the error signal and the energy of the reference microphone signal; and determine the estimation correlation coefficient between the reference signal and the residual echo to be estimated using the echo relative transfer function and the near-end signal relative transfer function.

7. An acoustic acquisition system, comprising a sound-generating unit, a main microphone, and a reference microphone, wherein the distance between the position of the reference microphone and the position of the sound-generating unit is not greater than a preset threshold; The sound-emitting unit is used to play a reference signal from a distant location; The main microphone and the reference microphone are used to receive the reflected signal of the reference signal at the near end; on the acoustic acquisition system, residual echo estimation is performed according to the residual echo estimation method as described in any one of claims 1 to 5.

8. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5.

Images