Echo cancellation method and apparatus, computer storage medium and electronic device

By acquiring spatial sound transmission latency and adding timestamps for echo prediction signal alignment in multi-terminal collaborative audio and video calls, the echo cancellation failure problem caused by network jitter and latency in traditional echo cancellation technology is solved, achieving higher quality call results.

WO2024244259A1PCT designated stage expired Publication Date: 2024-12-05CHINA TELECOM CORP LTD GUANGDONG RESEARCH INSTITUTE +1
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2023/124823
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2023-10-16
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Traditional echo cancellation technology cannot effectively eliminate echoes in multi-terminal collaborative audio and video calls or conferences due to network jitter and latency, which causes echo signal distortion and errors, thus affecting call quality.

Method used

By acquiring the spatial sound transmission delay, adding playback and acquisition timestamps, and using echo prediction signals for time alignment, echoes can be accurately eliminated.

Benefits of technology

It enables more accurate echo cancellation in multi-device collaborative audio and video calls or conferences, reducing noise and echo and improving call quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023124823_05122024_PF_FP_ABST
    Figure CN2023124823_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of data processing, and relates to an echo cancellation method and apparatus, a storage medium and an electronic device. The method is applied to a first terminal in an echo cancellation system, and comprises: acquiring a spatial sound transmission delay; receiving an opposite-end audio signal, decoding and playing the opposite-end audio signal, adding a playing timestamp and an audio frame confirmation stamp to the opposite-end audio signal during playing, and determining an echo prediction signal according to the playing timestamp and the audio frame confirmation stamp; acquiring an acquisition audio signal and an acquisition timestamp corresponding to the acquisition audio signal; and according to the spatial sound transmission delay, the playing timestamp and the acquisition timestamp, aligning the echo prediction signal and the acquisition audio signal, and according to the aligned echo prediction signal, performing echo cancellation on the acquisition audio signal to obtain a target audio signal. The present application can ensure the accuracy of an echo prediction signal, improve the echo cancellation effect, and thus improve the call quality.
Need to check novelty before this filing date? Find Prior Art

Description

Echo cancellation method, device, computer storage medium and electronic equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202310615427.1, filed on May 26, 2023, entitled “Echo Cancellation Method, Device, Computer Storage Medium and Electronic Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of data processing technology, and in particular to an echo cancellation method, an echo cancellation device, a computer-readable storage medium, and an electronic device. Background Art

[0004] Echo cancellation can reduce echo, noise, and howling during audio and video calls and conferences, improving voice clarity and ensuring call / conference service quality. Multi-terminal collaboration allows terminals such as televisions and cameras to provide video calls or video conferences through mutual support.

[0005] Currently, traditional echo cancellation technology is commonly used to eliminate echoes, noise, and howling in audio and video calls and audio and video conferences. However, due to network jitter, the echo prediction signal is distorted. At the same time, due to network delay, the echo signal error is caused. The distortion of the echo prediction signal and the error of the echo signal can cause echo cancellation failure, making traditional echo cancellation technology unable to be directly applied to multi-end collaborative video calls or video conferences.

[0006] Summary of the Invention

[0007] According to a first aspect of the present application, an echo cancellation method is provided, which is applied to a first terminal in an echo cancellation system, including: obtaining a spatial sound transmission delay, wherein the spatial sound transmission delay is determined by time-aligning the first terminal and the second terminal; receiving an opposite-end audio signal, decoding and playing the opposite-end audio signal, adding a playback timestamp and an audio frame confirmation stamp to the opposite-end audio signal during playback, and determining an echo prediction signal based on the playback timestamp and the audio frame confirmation stamp; obtaining a collected audio signal and a collection timestamp corresponding to the collected audio signal, wherein the collected audio signal is obtained by the second terminal collecting an audio signal of a local user and an echo signal corresponding to the opposite-end audio signal; aligning the echo prediction signal and the collected audio signal based on the spatial sound transmission delay, the playback timestamp, and the collection timestamp, and performing echo cancellation on the collected audio signal based on the aligned echo prediction signal to obtain a target audio signal.

[0008] According to a second aspect of the present application, an echo cancellation device is provided, which is configured at a first terminal in an echo cancellation system, and includes: an acquisition module for acquiring a spatial sound transmission delay, where the spatial sound transmission delay is determined by the first terminal and the second terminal; a prediction module for receiving a peer audio signal, decoding and playing the peer audio signal, adding a playback timestamp and an audio frame confirmation stamp to the peer audio signal during playback, and determining an echo prediction signal based on the playback timestamp and the audio frame confirmation stamp; the acquisition module is further configured to acquire a collected audio signal and a collection timestamp corresponding to the collected audio signal, where the collected audio signal is acquired by the second terminal from an audio signal of a local user and an echo signal corresponding to the peer audio signal; and an cancellation module for aligning the echo prediction signal with the collected audio signal based on the spatial sound transmission delay, the playback timestamp, and the collection timestamp, and performing echo cancellation on the collected audio signal based on the aligned echo prediction signal to obtain a target audio signal.

[0009] According to a third aspect of the present application, an echo cancellation system is provided, comprising: a first terminal, configured to obtain a spatial sound transmission delay and an opposite-end audio signal, decode and play the opposite-end audio signal, add a playback timestamp and an audio frame confirmation stamp to the opposite-end audio signal during playback, and determine an echo prediction signal based on the playback timestamp and the audio frame confirmation stamp; after obtaining a collected audio signal and a collection timestamp corresponding to the collected audio signal, align the echo prediction signal and the collected audio signal based on the spatial sound transmission delay, the playback timestamp, and the collection timestamp, and perform echo cancellation on the collected audio signal based on the aligned echo prediction signal to obtain a target audio signal; a second terminal, connected to the first terminal, configured to cooperate with the first terminal to determine the spatial sound transmission delay, collect the audio signal of a local user and the echo signal corresponding to the opposite-end audio signal to generate the collected audio signal, and determine the collection timestamp corresponding to the collected audio signal.

[0010] According to a fourth aspect of the present application, a computer storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the above-mentioned echo cancellation method is implemented.

[0011] According to a fifth aspect of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-mentioned echo cancellation method by executing the executable instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG1 schematically shows a flow chart of a conventional echo cancellation method in the related art.

[0013] FIG2 schematically shows a system architecture diagram of an echo cancellation method applied in an embodiment of the present application.

[0014] FIG3 schematically shows an architecture diagram of an echo cancellation system in an embodiment of the present application.

[0015] FIG4 schematically shows a flow chart of the echo cancellation method in an embodiment of the present application.

[0016] FIG5 schematically shows a flow chart of a multi-terminal collaborative audio call in an embodiment of the present application.

[0017] FIG6 schematically shows a structural diagram of an echo cancellation device in an embodiment of the present application.

[0018] FIG7 schematically shows a block diagram of a computer system structure of an electronic device suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0019] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0020] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0021] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0022] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0023] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0024] In the relevant technologies in this field, for scenarios in which multi-terminal collaboration allows terminals such as televisions and cameras to provide audio and video calls or audio and video conferences through mutual assistance of capabilities, echoes, noises, and howling sounds are usually present during the call, affecting the call quality. For the existing abnormal audio signals, traditional echo cancellation methods are usually used to cancel the echoes. Figure 1 schematically shows a flow chart of the traditional echo cancellation method. As shown in Figure 1, in step S101, the first terminal receives the opposite-end audio signal sent by the opposite-end call terminal; in step S102, the opposite-end audio signal is decoded and played; in step S103, an echo prediction signal is determined based on the opposite-end audio signal; in step S104, the second terminal collects the current user's voice signal and echo signal to generate a collected audio signal; in step S105, the collected audio signal is sent to the first terminal; in step S106, the first terminal performs echo cancellation on the collected audio signal based on the echo prediction signal to obtain a target audio signal; in step S107, the target audio signal is encoded, packaged, and sent to the opposite-end call terminal.

[0025] However, traditional echo cancellation methods have corresponding problems, specifically in two aspects: First, network jitter between multi-end collaboration causes echo signal distortion: the played sound is inconsistent with the sound data input to the acoustic echo canceller, resulting in echo prediction signal distortion, which makes it difficult to meet the requirements of echo cancellation in the microphone input signal; second, network transmission delay between multi-end collaboration causes echo signal error: the transmission delay causes an uncertain time delay from sound playback to microphone capture, making it difficult to time-align the original echo cancellation model, and the echo prediction signal fails to align with the received signal, causing echo cancellation failure.

[0026] To address the technical issues in related technologies, the present invention proposes an echo cancellation method in an embodiment to accurately eliminate echoes, noise, and howling in received signals, thereby improving the call quality between terminals. Before describing the technical solutions in the embodiments of the present invention in detail, the technical terms that may be involved in the embodiments of the present invention are first explained and described.

[0027] (1) Echo: When sound waves propagate, they will be reflected at the interface when they encounter a large reflecting surface (such as the wall of a building, the inside of a mountain, etc.). People call the reflected sound waves that can be distinguished from the original sound an echo.

[0028] (2) Multi-terminal collaboration: refers to the way two or more devices connect and collaborate with each other to complete tasks.

[0029] After introducing the technical terms that may be involved in the embodiments of the present application, the echo cancellation method in the present application is described in detail.

[0030] FIG2 schematically shows a system architecture block diagram of a hardware system applying the technical solution of the present application.

[0031] As shown in FIG. 2 , the system architecture 200 may include a current call terminal 201 , a counterpart call terminal 202 , and a communication network 203 . Among them, the current call terminal 201 and / or the opposite call terminal 202 can be a multi-terminal collaborative call terminal. The multi-terminal collaborative call terminal varies according to different call scenarios. When the call scenario is a video call, the multi-terminal collaborative call terminal may include a smart TV, a smart camera with a voice acquisition function, or may include a display for displaying video images (for example: TV, etc.), a set-top box, a smart camera with a voice acquisition function, and of course other terminals; when the call scenario is a voice call, the multi-terminal collaborative call terminal may include a microphone with a voice acquisition function, a processor with data processing capabilities to achieve echo cancellation and a speaker with a voice playback function (for example: a speaker, etc.), or a smart TV and a microphone with a voice acquisition function, etc.; further, if the current call terminal 201 or the opposite call terminal 202 is a multi-terminal assisted call terminal, the other call terminal may be an electronic device including a smart phone, a tablet computer, a desktop computer, a smart car terminal, etc.; the communication network 203 is used to provide a data transmission path between the current call terminal 201 and the opposite call terminal 202.

[0032] In one embodiment of the present application, the current call terminal 201 is set as a multi-terminal collaborative call terminal, the TV + set-top box or smart TV in the current call terminal 201 is the first terminal, and the smart camera with voice collection function is the second terminal. First, the first terminal and the second terminal are initialized. The spatial sound transmission delay is obtained by time-aligning the first and second terminals. The first terminal then receives the peer audio signal sent by the peer call terminal 202, parses it, and plays it. During the playback process, a playback timestamp is added at a preset time interval. A determination is made as to whether the peer audio signal is missing. Based on the determination, an audio frame confirmation stamp is added. The playback timestamp and the audio frame confirmation stamp are then input into an echo prediction model to obtain an echo prediction signal. The second terminal then generates a collected audio signal using the audio signal of the user using the current call terminal and the echo generated by the first terminal playing the peer audio signal. Collection timestamps are added at the preset time intervals. Finally, after receiving the collected audio signal and the collection timestamp, the first terminal aligns the echo prediction signal and the collected audio signal based on the spatial sound transmission delay, the playback timestamp, and the collection timestamp. Echo cancellation is then performed on the collected audio signal based on the aligned echo prediction signal to obtain a target audio signal, which is the user audio signal after echo removal. The target audio signal is then encoded and sent to the peer call terminal 202 via the communication network 203.

[0033] The technical solution provided in the embodiment of the present application can be applied to the local call terminal 201 or the opposite call terminal 202.

[0034] The echo cancellation method in this application can be applied to any scenario involving multi-terminal collaborative calls, such as multi-terminal collaborative audio and video calls, multi-terminal collaborative audio and video conferences, etc. The following takes the multi-terminal collaborative video call scenario as an example and describes the echo cancellation method provided by this application in detail in combination with specific implementation methods.

[0035] Before describing the echo cancellation method, the echo cancellation system in this application is described first. It is worth noting that the echo cancellation system in this application is only for multi-terminal collaborative call terminals.

[0036] Figure 3 schematically shows an architectural diagram of an echo cancellation system. As shown in Figure 3, the echo cancellation system 300 includes a first terminal 301 and a second terminal 302. The first terminal 301 and the second terminal 302 are connected via a wireless network or a wired network. The first terminal 301 is used to play the audio signal of the opposite end and cancel the echo. The second terminal 302 is used to collect the user's audio signal and the echo generated when the first terminal 301 plays the audio signal of the opposite end.

[0037] Furthermore, the first terminal 301 includes a soft clock synchronization module 301-1, an audio decoding module 301-2, a playback module 301-3, a network module 301-4, an integrity check module 301-5, an echo signal prediction module 301-6, an audio alignment module 301-7 and an echo cancellation module 301-8, and the second terminal 302 includes a soft clock synchronization module 302-1, a sound receiving module 302-2, an audio encoding module 302-3 and a network module 302-4. Specifically: the soft clock synchronization module 302-1 is used for soft clock synchronization of multiple terminals when the terminals collaborate to ensure that the time of the terminals remains consistent; the audio decoding module 301-2 is used to decode the audio stream transmitted from the opposite call terminal and hand it over to the playback module 301-3 for playback; the playback module 301-3 is used to play the audio stream transmitted from the opposite call terminal and add a playback timestamp to the played content, and the playback timestamp is marked at a preset time interval during the playback process; the communication network module 301-4 is used to receive the audio stream transmitted from the opposite call terminal, and send the target audio signal after echo cancellation to the opposite call terminal, and receive the audio stream recorded by the second terminal 302; integrity check Module 301-5 is used to detect the integrity of the audio stream transmitted from the other call terminal and whether there is packet loss or errors; the echo signal prediction module 301-6 is used to generate an echo prediction signal based on the audio stream transmitted from the other call terminal and its integrity according to the echo signal generation algorithm; the audio alignment module 301-7 is used to calculate the spatial sound transmission delay during initialization and time-align the echo prediction signal with the audio signal collected by the second terminal 302 based on the spatial sound transmission delay, the collection timestamp, and the playback timestamp; the echo cancellation module 301-8 is used to, after time alignment, eliminate the echo from the collected audio signal by removing the echo prediction signal and output the ideal target audio signal. The function of the soft clock synchronization module 302-1 is the same as that of the soft clock synchronization module 301-1; the sound receiving module 302-2 is used to receive user voice input, but also collects environmental echoes; the audio encoding module 302-3 is used to encapsulate the user voice input into an audio stream and add an acquisition timestamp. The acquisition timestamp is the same as the playback timestamp and is set at the same preset time interval; the network module 302-4 is used to send the user's audio stream to the first terminal 301 for processing.

[0038] FIG4 shows a flow chart of an echo cancellation method, which is applied to a first terminal, which may be the first terminal 301 in FIG3 . As shown in FIG4 , the echo cancellation method includes:

[0039] Step S410: Acquire a spatial sound transmission delay, where the spatial sound transmission delay is determined by time-aligning the first terminal and the second terminal.

[0040] Step S420: receiving a peer audio signal, decoding and playing the peer audio signal, adding a play timestamp and an audio frame confirmation stamp to the peer audio signal during the play process, and determining an echo prediction signal based on the play timestamp and the audio frame confirmation stamp;

[0041] Step S430: Acquire a collected audio signal and a collection timestamp corresponding to the collected audio signal, wherein the collected audio signal is obtained by the second terminal collecting the audio signal of the local user and the echo signal corresponding to the audio signal of the remote user;

[0042] Step S440: Align the echo prediction signal and the collected audio signal according to the spatial sound transmission delay, the playback timestamp, and the collection timestamp, and perform echo cancellation on the collected audio signal according to the aligned echo prediction signal to obtain a target audio signal.

[0043] The echo cancellation method of the present application is applied to the first terminal in the echo cancellation system. The first terminal first obtains the spatial sound transmission delay, and then, after obtaining the audio signal of the other end, decodes the audio signal of the other end and plays it. During the playback process, a playback timestamp and an audio frame confirmation stamp are added to the audio signal of the other end, and the echo prediction signal is determined based on the playback timestamp and the audio frame confirmation stamp. Then, the collected audio signal and the corresponding collection timestamp sent by the second terminal are obtained. Finally, the echo prediction signal and the collected audio signal are aligned according to the spatial sound transmission delay, the playback timestamp and the collection timestamp, and the collected audio signal is echo-cancelled according to the aligned echo prediction signal to obtain the target audio signal. The echo cancellation method of the present application can make the echo prediction signal more accurately match the collected audio signal through time alignment, achieve more accurate echo cancellation, reduce noise and echo, and thus improve call quality.

[0044] Next, based on the architecture of the echo cancellation system shown in FIG3 , each step of the echo cancellation method shown in FIG4 is described in detail.

[0045] In step S410, a spatial sound transmission delay is obtained, where the spatial sound transmission delay is determined by time-aligning the first terminal and the second terminal.

[0046] In an exemplary embodiment of the present application, after receiving the audio signal from the other end, the first terminal parses and plays it. The played audio information from the other end will propagate in the air and produce an echo when encountering an obstacle. If the current user inputs voice through the second terminal, the second terminal will not only collect the user's audio signal, but also collect the echo signal corresponding to the audio signal from the other end. If the audio signal containing the echo signal is directly sent to the other end call terminal, the video call quality will be degraded and the other end user will not be able to clearly obtain the current user's voice content. Therefore, it is necessary to eliminate the echo signal in the audio signal collected by the second terminal to improve the quality of the audio signal sent to the other end call terminal. When performing echo cancellation, since the time when the echo signal is generated and the time when the second terminal collects the audio signal are not synchronized, there is a time delay between the two. Therefore, it is necessary to calculate the time delay from the generation of the echo signal to the time when it is collected by the second terminal, and then align the echo signal and the collected audio signal based on this time delay to ensure that the echo signal is accurately removed from the collected audio signal.

[0047] In an exemplary embodiment of the present application, the delay can be determined jointly by the first terminal and the second terminal. Since the delay is the delay caused by the propagation of the audio signal in the air, it is also called the spatial sound transmission delay. When determining the spatial sound transmission delay, the first terminal and the second terminal can first be soft-clock synchronized by the soft clock synchronization modules in the first terminal and the second terminal. For example, 1588v2 (IEEE P1588 DM2.2) technology can be used for soft clock synchronization to align the time of the first terminal and the second terminal. Then, the first terminal plays a specific sound, such as a "beep", at time t1, and the second terminal collects the specific sound and timestamps it with t2. Then, based on t1 and t2, the spatial sound transmission delay can be obtained as t'=t2-t1.

[0048] In step S420, the peer audio signal is received, decoded and played, a play timestamp and an audio frame confirmation stamp are added to the peer audio signal during the play process, and an echo prediction signal is determined based on the play timestamp and the audio frame confirmation stamp.

[0049] In an exemplary embodiment of the present application, during a video call, the first terminal, the second terminal, and the opposite terminal ensure that the current user and the opposite user have a voice call by sending and receiving audio signals. For the local call terminal (first terminal + second terminal), after receiving the opposite terminal audio signal, the first terminal can decode and play it. In order to achieve time alignment between the echo prediction signal and the audio signal collected by the second terminal, a playback timestamp can be added to the played opposite terminal audio signal during the playback process. In an embodiment of the present application, a playback timestamp can be added to the opposite terminal audio signal at a preset time interval. The preset time interval can be any value between 1s and 10s. The specific value can be set according to actual needs. If you want to improve the effect of echo cancellation, you can set the preset time interval to a smaller value such as 1s or 2s. If you want to improve the efficiency of echo cancellation, you can set the preset time interval to a larger value such as 9s or 10s. Of course, the preset time interval can also be set to other values. The embodiment of the present application does not specifically limit this.

[0050] Adding a play timestamp according to a preset time interval is equivalent to dividing the peer audio signal into multiple audio segments according to the preset time interval, and each audio segment corresponds to a play timestamp.

[0051] In an exemplary embodiment of the present application, while adding a playback timestamp, it is also necessary to verify the integrity of the audio signal on the other end through an integrity check module. Since the playback timestamp is added according to a preset time interval, each audio segment can be integrity checked accordingly to determine whether there is packet loss or error in the audio segment. When the audio segment is incomplete and there is packet loss or error, a first audio frame confirmation stamp is added to the audio segment. When the audio segment is complete and there is no packet loss or error, a second audio frame confirmation stamp is added to the audio segment. The second audio frame confirmation stamp is different from the first audio frame confirmation stamp. The first audio frame confirmation stamp and the second audio frame confirmation stamp are essentially an identifier used to mark whether the audio is complete. For example, 1 can be used as the first audio frame confirmation stamp and 0 can be used as the second audio frame confirmation stamp. Of course, other identifiers can also be used. The embodiment of the present application does not specifically limit this.

[0052] In an exemplary embodiment of the present application, when performing integrity check on each audio segment, when the audio segment is incomplete, there is packet loss or error, a first audio frame confirmation stamp can be added to the audio segment; when the audio segment is complete, there is no packet loss or error, no audio frame confirmation stamp is added to the audio segment.

[0053] In an exemplary embodiment of the present application, when playing the peer audio signal, the first terminal may also input the played audio segment and the corresponding audio frame confirmation stamp into the echo prediction model, and process the audio segment through the echo prediction model to output an echo prediction signal. Of course, the first terminal may also input the peer audio signal and the audio frame confirmation stamp into the echo prediction model for processing after adding the playback timestamp and audio frame confirmation stamp to the entire audio signal to obtain an echo prediction signal. The echo prediction model may use an adaptive filtering algorithm to process the audio segment. Furthermore, the adaptive filtering algorithm may be an LMS adaptive filtering algorithm, an RLS adaptive filtering algorithm, etc., which is not specifically limited in the embodiments of the present application.

[0054] Furthermore, when there is no audio frame confirmation stamp in the audio segment, only the audio segment or the peer audio signal can be input into the echo prediction model, and the audio segment is processed by the echo prediction model based on the adaptive filtering algorithm to obtain the corresponding echo prediction signal.

[0055] In step S430, a collected audio signal and a collection timestamp corresponding to the collected audio signal are obtained. The collected audio signal is obtained by the second terminal collecting the audio signal of the local user and the echo signal corresponding to the opposite-end audio signal.

[0056] In an exemplary embodiment of the present application, after the first terminal plays the audio signal of the other end, the voice audio signal of the current user can be collected through the sound receiving module in the second terminal. While the second terminal collects the audio signal of the local user, it may also collect the echo signal of the audio signal of the other end reflected by the obstacle. That is to say, the collected audio signal collected by the second terminal includes the user's audio signal and the echo signal corresponding to the audio signal of the other end.

[0057] In an exemplary embodiment of the present application, after the sound receiving module in the second terminal collects the user audio signal and the echo signal, the collected audio signal can be encapsulated by the audio encoding module to form an audio stream, and a collection timestamp can be added to the audio stream at a preset time interval. The preset time interval is the same as the preset time interval used when adding the playback timestamp. In this way, after the collected audio signal and the echo prediction signal are time-aligned, each audio segment can be perfectly matched to achieve accurate echo cancellation.

[0058] In an exemplary embodiment of the present application, after the second terminal completes encoding the collected audio signal and adding the collection timestamp, it can send the audio stream containing the collected audio signal and the collection timestamp to the first terminal, so that the first terminal can perform echo cancellation on the collected audio signal to obtain a user audio signal without echo.

[0059] In step S440, the echo prediction signal and the collected audio signal are aligned according to the spatial sound transmission delay, the playback timestamp, and the collection timestamp, and echo cancellation is performed on the collected audio signal according to the aligned echo prediction signal to obtain a target audio signal.

[0060] In an exemplary embodiment of the present application, after the first terminal obtains the audio stream and acquisition timestamp containing the collected audio signal, it can decode the audio stream to obtain the collected audio signal, and then time-align the echo prediction signal and the collected audio signal according to the spatial sound transmission delay, playback timestamp and acquisition timestamp, and then perform echo cancellation on the collected audio signal according to the aligned echo prediction signal to obtain the target audio signal.

[0061] Among them, when the echo prediction signal and the collected audio signal are time-aligned, since the echo prediction signal is predicted based on the audio signal of the other end played by the first terminal, the echo prediction signal corresponds to the playback timestamp, and the playback time of the audio signal and the collection time of the echo signal differ by the spatial sound transmission delay. That is, the playback timestamp T1+t' corresponding to the same audio clip = the collection timestamp T2. Therefore, if you want to time-align the echo prediction signal with the collected audio signal, you can update the playback timestamp of the echo prediction signal according to the spatial sound transmission delay, and then update the echo prediction signal according to the updated playback timestamp to obtain the echo prediction signal to be removed that is time-aligned with the collected audio signal.

[0062] Since there are multiple playback timestamps and multiple acquisition timestamps, the target echo prediction signal and the corresponding target acquisition audio signal can be determined based on the playback timestamp, spatial sound transmission delay and acquisition timestamp. It is worth noting that the target echo prediction signal and the target acquisition audio signal are not strictly matched in sequence. For example, when the target echo prediction signal is the echo prediction signal corresponding to the audio segment corresponding to the first playback timestamp in the audio signal of the other end, the target acquisition audio signal is not necessarily the audio segment corresponding to the first acquisition timestamp in the acquisition audio signal, but may be the audio segment corresponding to the second acquisition timestamp, and so on, depending on the time when the echo signal arrives at the second terminal. In addition, since the preset time intervals of the playback timestamp and the acquisition timestamp are set to the same, the target echo prediction signal and the target acquisition audio signal have the same length and can be perfectly matched. After determining the target echo prediction signal and the target collected audio signal, the playback timestamp of the target echo prediction signal can be updated according to the spatial sound transmission delay, and the target echo prediction signal can be updated according to the updated playback timestamp to obtain the echo prediction signal to be removed. In essence, the phase of the target echo prediction signal is updated. For example, the waveform of the target echo prediction signal is shifted back in time according to the spatial sound transmission delay to obtain the shifted echo prediction signal, which is the echo prediction signal to be removed that is time-aligned with the target collected audio signal.

[0063] In an exemplary embodiment of the present application, after obtaining the to-be-removed echo prediction signal, the to-be-removed echo prediction signal is removed from the target collected audio signal to obtain the target audio signal from which the echo has been removed. After completing echo cancellation for all audio segments, all target audio signals are sequentially concatenated and encoded to produce a final audio signal that can be sent to the other end of the call. This audio signal is the audio signal currently being input to the second end by the user.

[0064] It is worth noting that since there are slight differences between the echo prediction signal and the actual echo signal, the echo cancellation method in the embodiment of the present application only eliminates the echo signal in the collected audio signal to the greatest extent, rather than completely eliminating it. However, the target audio signal finally obtained contains very little echo signal, which is basically unrecognizable by the human ear. Therefore, the user at the other end hears only the voice of the user at the local end, without any echo, noise or howling.

[0065] The echo cancellation method in this application can be applied to any scenario involving multi-terminal collaborative calls that require echo cancellation, such as multi-terminal collaborative audio and video calls. Taking multi-terminal collaborative audio calls as an example, the local user uses a smart speaker and microphone to talk to the other user, who can use any call terminal, such as a smartphone, laptop, multi-terminal collaborative call terminal, etc.

[0066] When a local user is talking with a remote user, the multi-terminal collaborative call terminal used by the local user can play the audio signal of the remote user, perform echo cancellation on the audio signal collected by the local terminal, and send the echo-cancelled audio signal to the remote user, thereby improving the quality of the multi-terminal collaborative audio call.

[0067] Figure 5 schematically shows a flowchart of a multi-terminal collaborative audio call. As shown in Figure 5, in step S501, after the smart speaker and the microphone are connected through a wireless network, soft clock alignment is performed; in step S502, the smart speaker determines the spatial sound transmission delay based on the timestamp of the specific audio signal it sends and the timestamp of the specific audio signal collected by the microphone; in step S503, the smart speaker receives the audio signal sent by the opposite call terminal; in step S504, the audio signal is decoded and played, and a playback timestamp is added at preset time intervals during the playback process; in step S505, whether the audio signal is complete is determined; in step S506, when the audio signal is incomplete, a first audio frame confirmation stamp is added; when the audio signal is complete, no audio frame confirmation stamp is added; in step S507, the audio signal is and the first audio frame confirmation stamp are input into the echo prediction model, and the echo prediction signal is calculated based on the adaptive filtering algorithm; in step S508, the microphone collects the user voice and the echo in the environment to form a collected audio signal; in step S509, the collected audio signal is encoded, and the collection timestamp is added according to the preset time interval; in step S510, the collected audio signal and the collection timestamp are sent to the smart speaker; in step S511, the echo prediction signal and the collected audio signal are time-aligned according to the playback timestamp, the collection timestamp and the spatial sound transmission delay; in step S512, the aligned echo prediction signal is removed from the collected audio signal to obtain the target audio signal; in step S513, the target audio signal is encoded, and the encoded target audio signal is sent to the other call terminal.

[0068] The echo cancellation method in the embodiment of the present application is applied to the first terminal in the echo cancellation system. The first terminal first obtains the spatial sound transmission delay, then decodes the peer audio signal after obtaining the peer audio signal and plays it. During the playback process, a playback timestamp and an audio frame confirmation stamp are added to the peer audio signal, and an echo prediction signal is determined based on the playback timestamp and the audio frame confirmation stamp. Then, the collected audio signal and the corresponding collection timestamp sent by the second terminal are obtained. Finally, the echo prediction signal and the collected audio signal are aligned according to the spatial sound transmission delay, the playback timestamp and the collection timestamp, and the collected audio signal is echo cancelled based on the aligned echo prediction signal to obtain the target audio signal. In the echo cancellation method of the present application, when multiple terminals collaborate to conduct an audio or video call or audio or video conference, each collaborative terminal performs time synchronization and spatial echo measurement, and the sound playback terminal and the sound collection terminal synchronously generate confirmation stamps and timestamps when playing and collecting sound, which are used for the generation and time alignment of echo prediction signals; the echo prediction signal generated by this method can better meet the echo cancellation requirements under multi-terminal collaboration, reduce the failure of echo cancellation caused by network jitter and uncertain delay, achieve more accurate echo cancellation, and thus improve call quality.

[0069] The present application further provides an echo cancellation device. FIG6 shows a schematic structural diagram of the echo cancellation device. The echo cancellation device 600 is configured in a first terminal in an echo cancellation system. As shown in FIG6 , the echo cancellation device 600 may include an acquisition module 601, a prediction module 602, and a cancellation module 603. Specifically:

[0070] An acquisition module 601 is configured to acquire a spatial sound transmission delay, where the spatial sound transmission delay is determined by the first terminal and the second terminal;

[0071] The prediction module 602 is configured to receive a peer audio signal, decode and play the peer audio signal, add a play timestamp and an audio frame confirmation stamp to the peer audio signal during the play process, and determine an echo prediction signal based on the play timestamp and the audio frame confirmation stamp;

[0072] The acquisition module 601 is further configured to acquire a collected audio signal and a collection timestamp corresponding to the collected audio signal, wherein the collected audio signal is obtained by the second terminal collecting the audio signal of the local user and the echo signal corresponding to the audio signal of the remote user;

[0073] The cancellation module 603 is used to align the echo prediction signal and the collected audio signal according to the spatial sound transmission delay, the playback timestamp and the collection timestamp, and perform echo cancellation on the collected audio signal according to the aligned echo prediction signal to obtain a target audio signal.

[0074] In an exemplary embodiment of the present application, the acquisition module 601 is configured to: obtain a recording timestamp generated when the second terminal records a preset audio signal; and determine the spatial sound transmission delay based on the playback timestamp generated when playing the preset audio signal and the recording timestamp.

[0075] In an exemplary embodiment of the present application, the prediction module 602 is configured to: during the playback process, segment the audio signal of the other end according to a preset time interval, and determine the playback timestamp corresponding to each audio segment according to the playback time corresponding to each audio segment after segmentation; perform integrity check on the audio segment; when the audio segment is incomplete, add a first audio frame confirmation stamp to the audio segment; when the audio segment is complete, do not add an audio frame confirmation stamp to the audio segment or add a second audio frame confirmation stamp different from the first audio frame confirmation stamp.

[0076] In an exemplary embodiment of the present application, the prediction module 602 is configured to: input the playback timestamp and the audio frame confirmation stamp into the echo prediction model, and process the peer audio signal based on an adaptive filtering algorithm through the echo prediction model to obtain the echo prediction signal.

[0077] In an exemplary embodiment of the present application, the number of the playback timestamps and the acquisition timestamps is multiple, and the time length between two adjacent playback timestamps and the distance between two adjacent acquisition timestamps are the same; the elimination module 603 is configured to: determine the target echo prediction signal and the target acquired audio signal corresponding to the target echo prediction signal according to the playback timestamp, the spatial sound transmission delay and the acquisition timestamp; update the playback timestamp of the target echo prediction signal according to the spatial sound transmission delay, and update the target echo prediction signal according to the updated playback timestamp to obtain an echo prediction signal to be removed that is time-aligned with the target acquired audio signal; and remove the echo prediction signal to be removed from the target acquired audio signal to obtain the target audio signal.

[0078] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0079] Furthermore, although the steps of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0080] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0081] FIG7 schematically shows a block diagram of a computer system structure of an electronic device for implementing an embodiment of the present application. The electronic device may be provided in a terminal device 101 or a server 102 .

[0082] It should be noted that the computer system 700 of the electronic device shown in FIG7 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0083] As shown in Figure 7, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 into the random access memory (RAM) 703. Various programs and data required for system operation are also stored in the random access memory 703. The CPU 701, the read-only memory 702, and the random access memory 703 are connected to each other via a bus 704. An input / output interface 705 (i.e., an I / O interface) is also connected to the bus 704.

[0084] In some embodiments, the following components are connected to the input / output interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a local area network card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 710 as needed so that a computer program read therefrom can be installed into the storage section 708 as needed.

[0085] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709 and / or installed from a removable medium 711. When the computer program is executed by the central processing unit 701, the various functions defined in the system of the present application are performed.

[0086] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable medium or any combination of the two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0087] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0088] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0089] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.) or on a network, and includes a number of instructions to enable an electronic device to execute the method according to the embodiments of the present application.

[0090] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An echo cancellation method, applied to a first terminal in an echo cancellation system, comprising: Obtaining a spatial sound transmission delay, where the spatial sound transmission delay is determined by time-aligning the first terminal and the second terminal; Receive a peer audio signal, decode and play the peer audio signal, add a play timestamp and an audio frame confirmation stamp to the peer audio signal during the play process, and determine an echo prediction signal according to the play timestamp and the audio frame confirmation stamp; Acquire a collected audio signal and a collection timestamp corresponding to the collected audio signal, wherein the collected audio signal is obtained by the second terminal collecting an audio signal of a local user and an echo signal corresponding to the opposite end audio signal; The echo prediction signal and the collected audio signal are aligned according to the spatial sound transmission delay, the playback timestamp and the collection timestamp, and the collected audio signal is echo-cancelled according to the aligned echo prediction signal to obtain a target audio signal.

2. The method according to claim 1, wherein: Before obtaining the spatial sound transmission delay, the method further includes: Soft clock synchronization is performed on the first terminal and the second terminal to align the time of the first terminal and the second terminal.

3. The method according to claim 1 or 2, wherein: The obtaining of spatial sound transmission delay comprises: Obtaining a recording timestamp generated when the second terminal records a preset audio signal; The spatial sound transmission delay is determined according to the playback timestamp generated when the preset audio signal is played and the recording timestamp.

4. The method according to claim 1, wherein: The adding a play timestamp and an audio frame confirmation stamp to the peer audio signal during the play process includes: During the playback process, the peer audio signal is segmented according to a preset time interval, and a playback timestamp corresponding to each audio segment is determined according to the playback time corresponding to each audio segment after segmentation; Performing integrity check on the audio segment; When the audio segment is incomplete, adding a first audio frame confirmation stamp to the audio segment; When the audio segment is complete, no audio frame confirmation stamp is added to the audio segment or a second audio frame confirmation stamp different from the first audio frame confirmation stamp is added.

5. The method according to claim 1 or 4, wherein: The step of determining the echo prediction signal according to the playback timestamp and the audio frame confirmation stamp comprises: The playback timestamp and the audio frame confirmation stamp are input into an echo prediction model, and the peer audio signal is processed by the echo prediction model based on an adaptive filtering algorithm to obtain the echo prediction signal.

6. The method according to claim 1, wherein: The number of the playback timestamps and the acquisition timestamps is plural, and the time length between two adjacent playback timestamps and the distance between two adjacent acquisition timestamps are the same; The step of aligning the echo prediction signal and the collected audio signal according to the spatial sound transmission delay, the playback timestamp, and the collection timestamp, and performing echo cancellation on the collected audio signal according to the aligned echo prediction signal to obtain a target audio signal includes: Determine a target echo prediction signal and a target collected audio signal corresponding to the target echo prediction signal according to the playback timestamp, the spatial sound transmission delay and the collection timestamp; Updating the playback timestamp of the target echo prediction signal according to the spatial sound transmission delay, and updating the target echo prediction signal according to the updated playback timestamp to obtain the echo prediction signal to be removed that is time-aligned with the target collected audio signal; The to-be-removed echo prediction signal is removed from the target collected audio signal to obtain the target audio signal.

7. An echo cancellation device, configured in a first terminal of an echo cancellation system, comprising: An acquisition module, configured to acquire a spatial sound transmission delay, where the spatial sound transmission delay is determined by the first terminal and the second terminal; A prediction module, configured to receive a peer audio signal, decode and play the peer audio signal, add a play timestamp and an audio frame confirmation stamp to the peer audio signal during the play process, and determine an echo prediction signal according to the play timestamp and the audio frame confirmation stamp; The acquisition module is further used to acquire a collected audio signal and a collection timestamp corresponding to the collected audio signal, wherein the collected audio signal is obtained by the second terminal collecting an audio signal of a local user and an echo signal corresponding to the opposite end audio signal; The elimination module is used to align the echo prediction signal and the collected audio signal according to the spatial sound transmission delay, the playback timestamp and the collection timestamp, and to perform echo elimination on the collected audio signal according to the aligned echo prediction signal to obtain a target audio signal.

8. An echo cancellation system, comprising: The first terminal is used to obtain a spatial sound transmission delay and an opposite-end audio signal, decode and play the opposite-end audio signal, add a playback timestamp and an audio frame confirmation stamp to the opposite-end audio signal during playback, and determine an echo prediction signal according to the playback timestamp and the audio frame confirmation stamp; after obtaining a collected audio signal and a collection timestamp corresponding to the collected audio signal, align the echo prediction signal and the collected audio signal according to the spatial sound transmission delay, the playback timestamp and the collection timestamp, and perform echo cancellation on the collected audio signal according to the aligned echo prediction signal to obtain a target audio signal; The second terminal is connected to the first terminal, and is used to cooperate with the first terminal to determine the spatial sound transmission delay, collect the audio signal of the local user and the echo signal corresponding to the audio signal of the other end to generate the collected audio signal, and determine the collection timestamp corresponding to the collected audio signal.

9. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the echo cancellation method according to any one of claims 1 to 7.

10. An electronic device, comprising: processor; as well as A memory, used to store executable instructions of the controller; The processor is configured to execute the echo cancellation method according to any one of claims 1 to 7 by executing the executable instructions.

Citation Information

Patent Citations

  • Method, device and system for eliminating acoustic echoes, server and call communication device

    CN106470284A

  • Echo cancellation method and device, and call equipment

    CN106791244A

  • Echo cancellation method and system based on timestamp synchronization

    CN113077805A

  • Voice echo cancellation method and device, equipment and storage medium

    CN113823306A

  • Echo cancellation method, device and system, electronic equipment and storage medium

    CN115762550A