Far-end echo cancellation system and method

Abstract

The present invention discloses a far-end echo cancellation system and method, which comprises: a far-end audio device, a sound decoding and receiving module, a receiving-end smoothing module, a receiving-end frequency domain conversion module, a sound encoding and sending module, a transmitting-end frequency domain conversion module, a transmitting-end mixing module, a transmitting-end delay synchronization module, a transmitting-end synchronization generation module, a transmitting-end delay processing module, a receiving-end echo cancellation module, a receiving-end frequency domain restoration module, and a conference bridge module; the receiving-end echo cancellation module is used to receive and mute audio frequency domain slice data and send the mute audio frequency domain slice data and slice timestamps to the receiving-end frequency domain restoration module. The far-end echo cancellation system and method of the present invention use smooth delay control to remove network jitter to ensure that the output audio data is smooth in the time dimension, thereby improving the accuracy of echo cancellation.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio processing, and in particular relates to a far-end echo cancellation system and method. Background Art

[0002] With the development of internet technology, remote voice communication via electronic devices is becoming increasingly popular. It solves the communication issues caused by distance and is a promising interactive method with widespread application. Voice conferencing and video conferencing are particularly common scenarios for remote voice communication. However, echo can affect the quality of remote voice communication and the user experience. Summary of the Invention

[0003] The present invention provides a far-end echo cancellation system and method, which adopts the following technical solutions:

[0004] A far-end echo cancellation system comprises: a far-end audio device, a sound decoding and receiving module, a receiving-end smoothing module, a receiving-end frequency domain conversion module, a sound encoding and sending module, a transmitting-end frequency domain conversion module, a transmitting-end mixing module, a transmitting-end delay synchronization module, a transmitting-end synchronization generation module, a transmitting-end delay processing module, a receiving-end echo cancellation module, a receiving-end frequency domain restoration module, and a conference bridge module;

[0005] The remote audio device is used to receive the sound data from the sound encoding and sending module, decode it and play it using the local speaker. The microphone of the remote audio device receives the sound played by the local speaker and the actual voice of the user, and encodes the mixed sound and sends it to the sound decoding and receiving module.

[0006] The sound decoding and receiving module is used to receive the mixed audio coded data containing the user's actual voice and the sound played by the speaker from the remote audio device through the network, and decode the mixed audio coded data into PCM audio data and send it to the receiving end smoothing module;

[0007] The receiving end smoothing module is used to calculate the delay threshold estimate, take the absolute value of the difference between the delay threshold estimate and the current delay threshold, and if the absolute value is greater than the delay threshold adjustment threshold, modify the current delay threshold to the delay threshold estimate. The receiving end smoothing module caches the PCM audio data according to the current delay threshold. When the cumulative audio data of the receiving end smoothing module exceeds the current delay threshold, the PCM audio data is segmented according to a fixed time to obtain PCM audio segment data, and the PCM audio segment data and the segment timestamp are sent to the receiving end frequency domain conversion module;

[0008] The receiving-end frequency domain conversion module is used to receive PCM audio fragment data and fragment timestamps, convert the PCM audio fragment data from time domain to frequency domain through FFT fast Fourier transform, and obtain audio frequency domain fragment data. The receiving-end frequency domain conversion module sends the audio frequency domain fragment data and fragment timestamps to the transmitting-end delay synchronization module and the receiving-end echo cancellation module;

[0009] The sending end synchronization generation module is used to generate a synchronization identification ID and corresponding synchronization audio data, and send the synchronization audio data to the sending end mixing module, and receive the timestamp of the sending end mixing module as the synchronization timestamp, and finally send the synchronization identification ID and synchronization timestamp to the sending end delay synchronization module;

[0010] The transmitting end delay synchronization module is used to calculate the synchronization timestamp difference, start timestamp and output timestamp based on the received information, send the start timestamp and output timestamp to the transmitting end delay processing module, and send the synchronization timestamp difference to the receiving end echo cancellation module;

[0011] The transmitting end mixing module is used to receive the PCM audio data and timestamp of the conference bridge module. The transmitting end mixing module receives the synchronous audio data of the transmitting end synchronization generation module and mixes the data according to the time domain. The transmitting end mixing module determines whether the PCM audio data is silent data by detecting the energy value of the audio data of the conference bridge module. If it is silent data, the audio data of the transmitting end synchronization generation module replaces the PCM audio data of the conference bridge module to obtain the mixed audio data. The transmitting end mixing module sends the mixed audio data and timestamp to the sound encoding sending module, and then returns the timestamp to the transmitting end synchronization generation module;

[0012] The sound coding and sending module is used to receive the PCM audio data and timestamp from the sending end mixing module, encode the PCM audio data, and send the audio data to the remote audio device through the grid. The sound coding and sending module also re-slices the PCM audio data according to a fixed time and generates a new timestamp, and sends the PCM audio slice data and the slice timestamp to the sending end frequency domain conversion module;

[0013] The transmitting end frequency domain conversion module is used to receive PCM audio fragment data and fragment timestamps, convert the PCM audio fragment data from time domain to frequency domain through FFT fast Fourier transform, and obtain audio frequency domain fragment data. The transmitting end frequency domain conversion module sends the audio frequency domain fragment data and fragment timestamps to the transmitting end delay processing module;

[0014] The transmitting end delay processing module is used to receive the audio frequency domain fragmentation data and fragmentation timestamp from the transmitting end frequency domain conversion module, put the audio frequency domain fragmentation data and fragmentation timestamp into a first-in-first-out queue, receive the start timestamp from the transmitting end delay synchronization module, discard the audio frequency domain fragmentation data whose fragmentation timestamp is less than the start timestamp, receive the output timestamp from the transmitting end delay synchronization module, and send the audio frequency domain fragmentation data and the sending timestamp whose fragmentation timestamp is less than or equal to the output timestamp to the receiving end echo cancellation module;

[0015] The receiving-end echo cancellation module is used to receive and mute the audio frequency domain slice data and send the mute audio frequency domain slice data and the slice timestamp to the receiving-end frequency domain restoration module;

[0016] The receiving-end frequency domain restoration module is used to receive the audio frequency domain fragmentation data and fragmentation timestamps from the receiving-end echo cancellation module. The receiving-end frequency domain restoration module performs an IFFT (Fast Fourier Inverse Transform) on the audio frequency domain fragmentation data, converts the frequency domain sound into time domain PCM sound data, and then sends the PCM sound data to the conference bridge module.

[0017] The conference bridge module is used to receive the PCM sound data from the frequency domain restoration module, mix the PCM sound data, and send the mixed PCM sound data and timestamp to the sending mixing modules of other conference members.

[0018] Furthermore, the receiving end smoothing module adopts smooth delay control to ensure that its output audio data is smooth in the time dimension. The receiving end smoothing module uses the preset delay as the initial value of the delay gate limit, counts the arrival time intervals of adjacent audio packets, counts the time intervals of fixed-period audio data packets, and the delay jitter value is greater than the minimum value of the time interval of audio data packets greater than the threshold within the period. The delay jitter value is multiplied by the adjustment coefficient to obtain the delay threshold statistic value. The Kalman filter is used to filter the delay threshold statistic value to obtain the delay threshold estimation value, filter the mutation, and smooth the delay threshold estimation value.

[0019] Furthermore, the transmitting end synchronization generation module uses a random algorithm to select a frequency in the high frequency group, the medium frequency group, and the low frequency group, and generates a synchronization identification ID with a value range of 0-26.

[0020] Synchronous identification ID = high frequency group frequency number * 9 + medium frequency group frequency number * 3 + low frequency group frequency number. The transmitting end synchronization generation module generates fixed-length synchronous audio data according to the three frequencies of the synchronous identification ID.

[0021] Furthermore, the transmitting end delay synchronization module receives the audio frequency domain fragmentation data and fragmentation timestamp of the receiving end frequency domain conversion module and the synchronization identification ID and sending timestamp of the receiving end synchronization generation module. The transmitting end delay synchronization module divides the audio frequency domain fragmentation data into three blocks according to the high frequency group, the medium frequency group and the low frequency group, and processes the audio frequency domain data of the three blocks respectively. The transmitting end delay synchronization module calculates the square value of the energy value of each frequency of the audio frequency domain data, amplifies the difference in the energy value of each frequency, and calculates the average value of the energy value of each frequency in the block. If the same If the step identifier ID in the frequency domain corresponding to the block is greater than the threshold k, the synchronization block ID is considered to match in this area. When all three blocks match, it is considered that the synchronization identifier ID is found in the audio frequency domain fragment data. The sending end delay synchronization module subtracts the fragment timestamp from the sending timestamp and then subtracts the adjustment coefficient as the synchronization timestamp difference. The sending end delay synchronization module adds the fragment timestamp to the synchronization timestamp difference to obtain the start timestamp, and adds the synchronization timestamp difference to the fragment timestamp to obtain the output timestamp. The start timestamp and output timestamp are sent to the sending end delay processing module, and the synchronization timestamp difference is sent to the sending end delay processing module.

[0022] Furthermore, the receiving-end echo cancellation module receives the audio frequency domain slice data and the sending timestamp from the transmitting-end delay processing module, and saves the audio frequency domain slice data to the transmitting-end buffer according to the sending timestamp. The transmitting-end buffer adopts the latest fixed-duration audio frequency domain slice data and uses the sending timestamp as the latest timestamp of the transmitting-end buffer. The receiving-end echo cancellation module receives the synchronization timestamp difference from the transmitting-end delay synchronization module, receives the audio frequency domain slice data and the slice timestamp from the receiving-end frequency domain conversion module, and calculates the delay estimate.

[0023] Delay estimate = fragment timestamp + synchronization timestamp difference - latest timestamp in the sender's buffer.

[0024] The receiving-end echo cancellation module uses the delay estimate as the initial value for the estimated delay of the linear part of the echo in the NLMS algorithm, accelerating the convergence of the NLMS algorithm. It then performs near- and far-end non-causal non-linear echo cancellation based on the NLMS algorithm's estimated delay. Finally, it sends the echo-cancelled audio frequency domain fragment data and fragment timestamps to the receiving-end frequency domain restoration module.

[0025] A far-end echo cancellation method comprises the following steps:

[0026] The remote audio device receives the sound data from the sound encoding and sending module, decodes it, and plays it using the local speaker. The microphone of the remote audio device receives the sound played by the local speaker and the actual voice of the user, and encodes the mixed sound and sends it to the sound decoding and receiving module.

[0027] The sound decoding and receiving module receives the mixed audio coded data containing the user's actual voice and the sound played by the speaker from the remote audio device through the network, and decodes the mixed audio coded data into PCM audio data and sends it to the receiving end smoothing module;

[0028] The receiving end smoothing module calculates the delay threshold estimate, takes the absolute value of the difference between the delay threshold estimate and the current delay threshold, and if the absolute value is greater than the delay threshold adjustment threshold, changes the current delay threshold to the delay threshold estimate. The receiving end smoothing module caches the PCM audio data according to the current delay threshold. When the cumulative audio data of the receiving end smoothing module exceeds the current delay threshold, the PCM audio data is segmented according to a fixed time to obtain PCM audio segment data, and the PCM audio segment data and the segment timestamp are sent to the receiving end frequency domain conversion module;

[0029] The receiving-end frequency domain conversion module receives PCM audio fragment data and fragment timestamps, converts the PCM audio fragment data from time domain to frequency domain through FFT fast Fourier transform, and obtains audio frequency domain fragment data. The receiving-end frequency domain conversion module sends the audio frequency domain fragment data and fragment timestamps to the transmitting-end delay synchronization module and the receiving-end echo cancellation module;

[0030] The sending end synchronization generation module generates a synchronization identification ID and corresponding synchronization audio data, and sends the synchronization audio data to the sending end mixing module, and receives the timestamp of the sending end mixing module as the synchronization timestamp, and finally sends the synchronization identification ID and synchronization timestamp to the sending end delay synchronization module;

[0031] The transmitting end delay synchronization module calculates the synchronization timestamp difference, the start timestamp and the output timestamp based on the received information, sends the start timestamp and the output timestamp to the transmitting end delay processing module, and sends the synchronization timestamp difference to the receiving end echo cancellation module;

[0032] The sending end mixing module receives the PCM audio data and timestamp from the conference bridge module. The sending end mixing module receives the synchronous audio data from the sending end synchronization generation module and mixes the data in the time domain. The sending end mixing module determines whether the PCM audio data is silent data by detecting the energy value of the audio data from the conference bridge module. If it is silent data, the audio data from the sending end synchronization generation module replaces the PCM audio data of the conference bridge module to obtain mixed audio data. The sending end mixing module sends the mixed audio data and timestamp to the sound encoding sending module, and then returns the timestamp to the sending end synchronization generation module.

[0033] The sound coding and sending module receives the PCM audio data and timestamp from the transmitting mixing module, encodes the PCM audio data, and sends the audio data to the remote audio device through the grid. The sound coding and sending module also re-slices the PCM audio data according to a fixed time and generates a new timestamp, and sends the PCM audio slice data and the slice timestamp to the transmitting frequency domain conversion module.

[0034] The transmitting end frequency domain conversion module receives PCM audio fragment data and fragment timestamps, converts the PCM audio fragment data from time domain to frequency domain through FFT fast Fourier transform, and obtains audio frequency domain fragment data. The transmitting end frequency domain conversion module sends the audio frequency domain fragment data and fragment timestamps to the transmitting end delay processing module;

[0035] The transmitting end delay processing module receives the audio frequency domain fragmentation data and fragmentation timestamp from the transmitting end frequency domain conversion module, and puts the audio frequency domain fragmentation data and fragmentation timestamp into a first-in-first-out queue. The transmitting end delay processing module receives the start timestamp from the transmitting end delay synchronization module, and discards the audio frequency domain fragmentation data whose fragmentation timestamp is less than the start timestamp. The transmitting end delay processing module receives the output timestamp of the transmitting end delay synchronization module, and sends the audio frequency domain fragmentation data and the sending timestamp whose fragmentation timestamp is less than or equal to the output timestamp to the receiving end echo cancellation module.

[0036] The receiving end echo cancellation module receives the audio frequency domain slice data, mutes it, and sends the muted audio frequency domain slice data and the slice timestamp to the receiving end frequency domain restoration module;

[0037] The receiving-end frequency domain restoration module receives the audio frequency domain fragmentation data and fragmentation timestamps from the receiving-end echo cancellation module. The receiving-end frequency domain restoration module performs an IFFT (Fast Fourier Inverse Transform) on the audio frequency domain fragmentation data, converts the frequency domain sound into time domain PCM sound data, and then sends the PCM sound data to the conference bridge module.

[0038] The conference bridge module receives the PCM sound data from the frequency domain restoration module, performs sound mixing on the PCM sound data, and sends the mixed PCM sound data and timestamp to the sending mixing modules of other conference members.

[0039] Furthermore, the specific method for the receiving end smoothing module to calculate the delay threshold estimation value is:

[0040] The receiving end smoothing module adopts smooth delay control to ensure that its output audio data is smooth in the time dimension. The receiving end smoothing module uses the preset delay as the initial value of the delay gate limit, counts the arrival time intervals of adjacent audio packets, counts the time intervals of fixed-period audio data packets, and the delay jitter value is greater than the minimum value of the time interval of audio data packets greater than the threshold within the period. The delay jitter value is multiplied by the adjustment coefficient to obtain the delay threshold statistic value. The Kalman filter is used to filter the delay threshold statistic value to obtain the delay threshold estimation value, filter the mutation, and smooth the delay threshold estimation value.

[0041] Furthermore, the specific method for the transmitting end synchronization generation module to generate the synchronization identification ID and the corresponding synchronization audio data is:

[0042] The transmitting end synchronization generation module uses a random algorithm to select a frequency in the high frequency group, medium frequency group, and low frequency group, and generates a synchronization identification ID with a value range of 0-26.

[0043] Synchronous identification ID = high frequency group frequency number * 9 + medium frequency group frequency number * 3 + low frequency group frequency number. The transmitting end synchronization generation module generates fixed-length synchronous audio data according to the three frequencies of the synchronous identification ID.

[0044] Furthermore, the specific method for the transmitting end delay synchronization module to calculate the synchronization timestamp difference, the start timestamp and the output timestamp based on the received information is as follows:

[0045] The transmitting end delay synchronization module receives the audio frequency domain fragmentation data and fragmentation timestamp from the receiving end frequency domain conversion module and the synchronization identification ID and sending timestamp from the receiving end synchronization generation module. The transmitting end delay synchronization module divides the audio frequency domain fragmentation data into three blocks according to the high frequency group, the medium frequency group and the low frequency group, and processes the audio frequency domain data of the three blocks respectively. The transmitting end delay synchronization module calculates the square value of the energy value of each frequency of the audio frequency domain data, amplifies the difference in the energy value of each frequency, and calculates the average value of the energy value of each frequency in the block. If the synchronization ID is If the identification ID in the frequency domain corresponding to the block is greater than the threshold k, the synchronization block ID is deemed to match in this area. When all three blocks match, it is considered that the synchronization identification ID is found in the audio frequency domain fragment data. The sending end delay synchronization module subtracts the fragment timestamp from the sending timestamp and then subtracts the adjustment coefficient as the synchronization timestamp difference. The sending end delay synchronization module adds the fragment timestamp to the synchronization timestamp difference to obtain the start timestamp, and adds the fragment timestamp to the synchronization timestamp difference to obtain the output timestamp. The start timestamp and output timestamp are sent to the sending end delay processing module, and the synchronization timestamp difference is sent to the sending end delay processing module.

[0046] Furthermore, the specific method for the receiving end echo cancellation module to receive and cancel the audio frequency domain slice data is as follows:

[0047] The receiving-end echo cancellation module receives the audio frequency domain fragmentation data and the sending timestamp from the transmitting-end delay processing module, and saves the audio frequency domain fragmentation data to the transmitting-end buffer according to the sending timestamp. The transmitting-end buffer uses the latest fixed-duration audio frequency domain fragmentation data and uses the sending timestamp as the latest timestamp of the transmitting-end buffer. The receiving-end echo cancellation module receives the synchronization timestamp difference from the transmitting-end delay synchronization module, receives the audio frequency domain fragmentation data and the fragmentation timestamp from the receiving-end frequency domain conversion module, and calculates the delay estimate.

[0048] Delay estimate = fragment timestamp + synchronization timestamp difference - latest timestamp in the sender's buffer.

[0049] The receiving-end echo cancellation module uses the delay estimate as the initial value for the estimated delay of the linear part of the echo in the NLMS algorithm, accelerating the convergence of the NLMS algorithm. It then performs near- and far-end non-causal non-linear echo cancellation based on the NLMS algorithm's estimated delay. Finally, it sends the echo-cancelled audio frequency domain fragment data and fragment timestamps to the receiving-end frequency domain restoration module.

[0050] The benefit of the present invention lies in that the far-end echo cancellation system and method provided use smooth delay control to remove network jitter, ensuring that the output audio data is smooth in the time dimension and improving the accuracy of echo cancellation.

[0051] The present invention is also beneficial in that the far-end echo cancellation system and method provided use three-frequency synchronous audio of high, medium and low frequencies to preliminarily determine the far-end audio delay, allowing the echo cancellation algorithm to return from high delay to low delay, thereby improving the accuracy of echo cancellation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of a far-end echo cancellation system of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] like Figure 1 The present invention shows a far-end echo cancellation system, which comprises: a far-end audio device, a sound decoding and receiving module (1), a receiving-end smoothing module (2), a receiving-end frequency domain conversion module (3), a sound encoding and sending module (4), a transmitting-end frequency domain conversion module (5), a transmitting-end mixing module (6), a transmitting-end delay synchronization module (7), a transmitting-end synchronization generation module (8), a transmitting-end delay processing module (9), a receiving-end echo cancellation module (10), a receiving-end frequency domain restoration module (11) and a conference bridge module (12).

[0055] The remote audio device refers to an audio device that does not support local echo cancellation, and is used to receive the sound data from the sound encoding and sending module (4), decode it, and play it using a local speaker. The microphone of the remote audio device receives the sound played by the local speaker and the actual voice of the user, and sends the mixed sound code to the sound decoding and receiving module (1).

[0056] The sound decoding receiving module (1) is used to receive mixed audio coding data containing the user's actual voice and the sound played by the speaker from the remote audio device through the network, and decode the mixed audio coding data into PCM audio data and send it to the receiving end smoothing module (2).

[0057] The receiving end smoothing module (2) is used to perform de-jitter processing on the audio data on the network. It calculates the delay threshold estimation value, takes the difference between the delay threshold estimation value and the current delay threshold and obtains the absolute value. If the absolute value is greater than the adjustment threshold of the delay threshold (set as 10% in this application), the current delay threshold is modified to the delay threshold estimation value. The receiving end smoothing module (2) caches the PCM audio data according to the current delay threshold value. When the audio data accumulated by the receiving end smoothing module (2) exceeds the current delay threshold, the PCM audio data is segmented according to a fixed time (set as 4 milliseconds in this application) to obtain PCM audio segment data, and the PCM audio segment data and the segment time stamp are sent to the receiving end frequency domain conversion module (3).

[0058] Among them, because the far-end echo method is a delay-sensitive method, the receiving end smoothing module (2) adopts smooth delay control to ensure that its output audio data is smooth in the time dimension. The receiving end smoothing module (2) uses a preset longer delay (set as 200 milliseconds in this application) as the initial value of the delay gate limit, counts the arrival time intervals of adjacent audio packets, counts the time intervals of audio data packets in a fixed period (set as 10 seconds in this application), takes a delay jitter value greater than the minimum value of the time interval of audio data packets greater than a threshold value (set as 90% in this application) within the period, multiplies the delay jitter value by an adjustment coefficient (set as 3 in this application) to obtain a delay threshold statistic value, uses a Kalman filter to filter the delay threshold statistic value to obtain a delay threshold estimation value, filters mutations, and performs smoothing processing on the delay threshold estimation value.

[0059] The receiving-end frequency domain conversion module (3) is used to receive PCM audio fragment data and fragment time stamps, and convert the PCM audio fragment data from time domain to frequency domain through FFT fast Fourier transform to obtain audio frequency domain fragment data. The receiving-end frequency domain conversion module (3) sends the audio frequency domain fragment data and fragment time stamps to the transmitting-end delay synchronization module (7) and the receiving-end echo cancellation module (10).

[0060] The transmitting end synchronization generation module (8) is used to generate a synchronization identification ID and corresponding synchronization audio data, and send the synchronization audio data to the transmitting end mixing module (6), and receive the timestamp of the transmitting end mixing module (6) as a synchronization timestamp, and finally send the synchronization identification ID and the synchronization timestamp to the transmitting end delay synchronization module (7).

[0061] The synchronization ID consists of one frequency each from the high-frequency group, the medium-frequency group, and the low-frequency group. The high-frequency group, the medium-frequency group, and the low-frequency group each select three frequencies, numbered 0 / 1 / 2 from low to high. For example, the high-frequency group is 1615HZ / 1767HZ / 1832HZ, the medium-frequency group is 1313HZ / 1187HZ / 1082HZ, and the low-frequency group is 891HZ / 781HZ / 695HZ.

[0062] The transmitting end synchronization generation module (8) uses a random algorithm to select a frequency in the high frequency group, the medium frequency group, and the low frequency group, and generates a synchronization identification ID with a value range of 0-26.

[0063] Synchronous identification ID = high frequency group frequency number * 9 + medium frequency group frequency number * 3 + low frequency group frequency number, the transmitting end synchronization generation module (8) generates fixed-length synchronous audio data (set to 50 milliseconds in this application) according to the three frequencies of the synchronous identification ID.

[0064] The transmitting end delay synchronization module (7) is used to calculate the synchronization timestamp difference, the start timestamp and the output timestamp based on the received information, send the start timestamp and the output timestamp to the transmitting end delay processing module (9), and send the synchronization timestamp difference to the receiving end echo cancellation module (10).

[0065] Specifically, the transmitting end delay synchronization module (7) receives the audio frequency domain fragmentation data and fragmentation timestamp from the receiving end frequency domain conversion module (3) and the synchronization identification ID and sending timestamp from the transmitting end synchronization generation module (8). The transmitting end delay synchronization module (7) divides the audio frequency domain fragmentation data into three blocks according to the high frequency group, the medium frequency group and the low frequency group, and processes the audio frequency domain data of the three blocks respectively. The transmitting end delay synchronization module (7) calculates the square value of the energy value of each frequency of the audio frequency domain data, amplifies the difference in the energy value of each frequency, and calculates the average value of the energy value of each frequency in the block. If the synchronization identification ID in the frequency domain corresponding to the block is greater than the threshold k, it is determined that the synchronization block ID matches in the area. When all three blocks match, it is considered that the synchronization identification ID is found in the audio frequency domain fragmentation data. The transmitting end delay synchronization module (7) takes the value of the sending timestamp minus the fragmentation timestamp minus the adjustment coefficient as the synchronization timestamp difference. The adjustment coefficient is the synchronization identification ID signal tone when the PCM audio fragment data is segmented and reserved. In this application, the value is three times the duration of the PCM audio fragment data fragment. The sending end delay synchronization module (7) adds the fragment timestamp to the synchronization timestamp difference to obtain a start timestamp, adds the fragment timestamp to the synchronization timestamp difference to obtain an output timestamp, sends the start timestamp and the output timestamp to the sending end delay processing module (9), and sends the synchronization timestamp difference to the sending end delay processing module (9).

[0066] The transmitting end mixing module (6) is used to receive the PCM audio data and time stamp from the conference bridge module (12). The transmitting end mixing module (6) receives the synchronous audio data from the transmitting end synchronous generation module (8) and performs data mixing in the time domain. The transmitting end mixing module (6) determines whether the PCM audio data is silent data by detecting the energy value of the audio data from the conference bridge module (12).

[0067] The audio data energy value is calculated by taking the absolute value of the PCM audio data and then adding the accumulated value. The accumulated value is divided by the number of samples to get the average value of the PCM audio data samples. If the average value is less than the threshold, it is considered silent data.

[0068] If it is silent data, the audio data of the transmitting end synchronization generation module (8) replaces the PCM audio data of the conference bridge module (12) to obtain mixed audio data. The transmitting end mixing module (6) sends the mixed audio data and the timestamp to the sound coding sending module (4), and then returns the timestamp to the transmitting end synchronization generation module (8).

[0069] The sound coding and sending module (4) is used to receive the PCM audio data and timestamp from the transmitting end mixing module (6), encode the PCM audio data, and send the audio data to the remote audio device through the grid. The sound coding and sending module (4) also re-segments the PCM audio data according to a fixed time (set to 4 milliseconds in this application) and generates a new timestamp, and sends the PCM audio segment data and the segment timestamp to the transmitting end frequency domain conversion module (5).

[0070] The transmitting end frequency domain conversion module (5) is used to receive PCM audio slice data and slice time stamps, convert the PCM audio slice data from time domain to frequency domain through FFT fast Fourier transform, and obtain audio frequency domain slice data. The transmitting end frequency domain conversion module (5) sends the audio frequency domain slice data and slice time stamps to the transmitting end delay processing module (9).

[0071] The transmitting end delay processing module (9) is used to receive the audio frequency domain fragmentation data and fragmentation timestamp from the transmitting end frequency domain conversion module (5), put the audio frequency domain fragmentation data and fragmentation timestamp into a first-in-first-out queue, receive the start timestamp from the transmitting end delay synchronization module (7), discard the audio frequency domain fragmentation data whose fragmentation timestamp is less than the start timestamp, receive the output timestamp from the transmitting end delay synchronization module (7), and send the audio frequency domain fragmentation data whose fragmentation timestamp is less than or equal to the output timestamp and the sending timestamp to the receiving end echo cancellation module (10).

[0072] The receiving end echo cancellation module (10) is used to receive and mute the audio frequency domain slice data and send the mute audio frequency domain slice data and the slice time stamp to the receiving end frequency domain restoration module (11).

[0073] Specifically, the receiving end echo cancellation module (10) receives the audio frequency domain slice data and the sending timestamp from the transmitting end delay processing module (9), saves the audio frequency domain slice data to the transmitting end buffer area according to the sending timestamp, and the transmitting end buffer area uses the latest fixed-length audio frequency domain slice data (set as 128 milliseconds in this application) and uses the sending timestamp as the latest timestamp of the transmitting end buffer area. The receiving end echo cancellation module (10) receives the synchronization timestamp difference from the transmitting end delay synchronization module (7), receives the audio frequency domain slice data and the slice timestamp from the receiving end frequency domain conversion module (3), and calculates the delay estimate.

[0074] Delay estimate = fragment timestamp + synchronization timestamp difference - latest timestamp in the sender's buffer.

[0075] The receiving end echo cancellation module (10) uses the delay estimation as the initial value of the estimated delay of the linear part echo of the linear echo cancellation NLMS algorithm, advances the convergence of the NLMS algorithm, and performs far-end and near-end non-causal estimation non-linear echo cancellation based on the estimated delay of the NLMS algorithm. Finally, the audio frequency domain slice data and the slice timestamp after the echo cancellation are sent to the receiving end frequency domain restoration module (11).

[0076] The receiving-end frequency domain restoration module (11) is used to receive the audio frequency domain fragmentation data and fragmentation timestamps from the receiving-end echo cancellation module (10). The receiving-end frequency domain restoration module (11) performs IFFT fast Fourier inverse transform on the audio frequency domain fragmentation data, inversely converts the frequency domain sound into time domain PCM sound data, and then sends the PCM sound data to the conference bridge module (12).

[0077] The conference bridge module (12) is used to receive the PCM sound data from the frequency domain restoration module (11), perform sound mixing on the PCM sound data, and send the mixed PCM sound data and time stamp to the sending mixing modules (6) of other conference members.

[0078] The present application also discloses a far-end echo cancellation method for use in the aforementioned far-end echo cancellation system, comprising the following steps:

[0079] The remote audio device receives the sound data from the sound encoding and sending module (4), decodes it and plays it using the local speaker. The microphone of the remote audio device receives the sound played by the local speaker and the actual voice of the user, and sends the mixed sound code to the sound decoding and receiving module (1).

[0080] The sound decoding receiving module (1) receives mixed audio coding data including the user's actual voice and the sound played by the speaker from the remote audio device through the network, and decodes the mixed audio coding data into PCM audio data and sends it to the receiving end smoothing module (2).

[0081] The receiving end smoothing module (2) calculates the delay threshold estimation value, takes the difference between the delay threshold estimation value and the current delay threshold and obtains the absolute value. If the absolute value is greater than the adjustment threshold of the delay threshold (set as 10% in this application), the current delay threshold is modified to the delay threshold estimation value. The receiving end smoothing module (2) caches the PCM audio data according to the current delay threshold value. When the audio data accumulated by the receiving end smoothing module (2) exceeds the current delay threshold, the PCM audio data is segmented according to a fixed time (set as 4 milliseconds in this application) to obtain PCM audio segment data, and the PCM audio segment data and the segment timestamp are sent to the receiving end frequency domain conversion module (3).

[0082] The receiving end frequency domain conversion module (3) receives PCM audio fragment data and fragment timestamps, converts the PCM audio fragment data from time domain to frequency domain through FFT fast Fourier transform, and obtains audio frequency domain fragment data. The receiving end frequency domain conversion module (3) sends the audio frequency domain fragment data and fragment timestamps to the sending end delay synchronization module (7) and the receiving end echo cancellation module (10).

[0083] The sending end synchronization generation module (8) generates a synchronization identification ID and corresponding synchronization audio data, and sends the synchronization audio data to the sending end mixing module (6), and receives the timestamp of the sending end mixing module (6) as a synchronization timestamp, and finally sends the synchronization identification ID and the synchronization timestamp to the sending end delay synchronization module (7).

[0084] The transmitting end delay synchronization module (7) calculates the synchronization timestamp difference, the start timestamp and the output timestamp based on the received information, sends the start timestamp and the output timestamp to the transmitting end delay processing module (9), and sends the synchronization timestamp difference to the receiving end echo cancellation module (10).

[0085] The transmitting end mixing module (6) receives the PCM audio data and timestamp from the conference bridge module (12). The transmitting end mixing module (6) receives the synchronous audio data from the transmitting end synchronization generation module (8) and performs data mixing in the time domain. The transmitting end mixing module (6) determines whether the PCM audio data is silent data by detecting the energy value of the audio data from the conference bridge module (12). If it is silent data, the audio data from the transmitting end synchronization generation module (8) replaces the PCM audio data from the conference bridge module (12) to obtain mixed audio data. The transmitting end mixing module (6) sends the mixed audio data and timestamp to the sound coding sending module (4), and then returns the timestamp to the transmitting end synchronization generation module (8).

[0086] The sound coding and sending module (4) receives the PCM audio data and timestamp from the transmitting end mixing module (6), encodes the PCM audio data, and sends the audio data to the remote audio device through the grid. The sound coding and sending module (4) also re-segments the PCM audio data according to a fixed time (set to 4 milliseconds in this application) and generates a new timestamp, and sends the PCM audio segment data and the segment timestamp to the transmitting end frequency domain conversion module (5).

[0087] The transmitting end frequency domain conversion module (5) receives the PCM audio slice data and the slice time stamp, converts the PCM audio slice data from the time domain to the frequency domain through FFT fast Fourier transform, and obtains audio frequency domain slice data. The transmitting end frequency domain conversion module (5) sends the audio frequency domain slice data and the slice time stamp to the transmitting end delay processing module (9).

[0088] The transmitting end delay processing module (9) receives the audio frequency domain fragmentation data and fragmentation timestamp from the transmitting end frequency domain conversion module (5), and puts the audio frequency domain fragmentation data and fragmentation timestamp into a first-in-first-out queue. The transmitting end delay processing module (9) receives the start timestamp from the transmitting end delay synchronization module (7), and discards the audio frequency domain fragmentation data whose fragmentation timestamp is less than the start timestamp. The transmitting end delay processing module (9) receives the output timestamp from the transmitting end delay synchronization module (7), and sends the audio frequency domain fragmentation data whose fragmentation timestamp is less than or equal to the output timestamp and the sending timestamp to the receiving end echo cancellation module (10).

[0089] The receiving end echo cancellation module (10) receives and mutes the audio frequency domain slice data and sends the muted audio frequency domain slice data and the slice time stamp to the receiving end frequency domain restoration module (11).

[0090] The receiving end frequency domain restoration module (11) receives the audio frequency domain fragmentation data and fragmentation timestamps from the receiving end echo cancellation module (10), performs IFFT fast Fourier inverse transform on the audio frequency domain fragmentation data, inversely converts the frequency domain sound into time domain PCM sound data, and then sends the PCM sound data to the conference bridge module (12).

[0091] The conference bridge module (12) receives the PCM sound data from the frequency domain restoration module (11), performs sound mixing on the PCM sound data, and sends the mixed PCM sound data and a timestamp to the originating mixing modules (6) of other conference members.

[0092] As a preferred embodiment, the specific method for the receiving end smoothing module (2) to calculate the delay threshold estimation value is:

[0093] The receiving end smoothing module (2) adopts smooth delay control to ensure that its output audio data is smooth in the time dimension. The receiving end smoothing module (2) uses a preset longer delay (set as 200 milliseconds in this application) as the initial value of the delay gate limit, counts the time intervals between the arrival of adjacent audio packets, counts the time intervals of audio data packets in a fixed period (set as 10 seconds in this application), takes a delay jitter value greater than the minimum value of the time interval of audio data packets greater than a threshold value (set as 90% in this application) within the period, multiplies the delay jitter value by an adjustment coefficient (set as 3 in this application) to obtain a delay threshold statistic value, uses a Kalman filter to filter the delay threshold statistic value to obtain a delay threshold estimation value, filters the mutation, and performs smoothing processing on the delay threshold estimation value.

[0094] The synchronization identification ID is composed of one frequency each from the high frequency group, the medium frequency group, and the low frequency group. The high frequency group, the medium frequency group, and the low frequency group each select three frequencies and are numbered 0 / 1 / 2 from low to high frequency, such as 1615HZ / 1767HZ / 1832HZ for the high frequency group, 1313HZ / 1187HZ / 1082HZ for the medium frequency group, and 891HZ / 781HZ / 695HZ for the low frequency group. As a preferred embodiment, the specific method for the transmitting end synchronization generation module (8) to generate the synchronization identification ID and the corresponding synchronization audio data is as follows:

[0095] The transmitting end synchronization generation module (8) uses a random algorithm to select a frequency in the high frequency group, the medium frequency group, and the low frequency group, and generates a synchronization identification ID with a value range of 0-26.

[0096] Synchronous identification ID = high frequency group frequency number * 9 + medium frequency group frequency number * 3 + low frequency group frequency number, the transmitting end synchronization generation module (8) generates fixed-length synchronous audio data (set to 50 milliseconds in this application) according to the three frequencies of the synchronous identification ID.

[0097] As a preferred embodiment, the specific method for the transmitting end delay synchronization module (7) to calculate the synchronization timestamp difference, the start timestamp and the output timestamp according to the received information is as follows:

[0098] The transmitting end delay synchronization module (7) receives the audio frequency domain fragmentation data and fragmentation timestamp from the receiving end frequency domain conversion module (3) and the synchronization identification ID and sending timestamp from the transmitting end synchronization generation module (8). The transmitting end delay synchronization module (7) divides the audio frequency domain fragmentation data into three blocks according to the high frequency group, the medium frequency group and the low frequency group, and processes the audio frequency domain data of the three blocks respectively. The transmitting end delay synchronization module (7) calculates the square value of the energy value of each frequency of the audio frequency domain data, amplifies the difference of the energy value of each frequency, and calculates the average value of the energy value of each frequency in the block. If the synchronization If the ID in the frequency domain corresponding to the block is greater than the threshold k, the synchronization block ID is considered to match in the area. When all three blocks match, it is considered that the synchronization ID is found in the audio frequency domain fragment data. The transmitting end delay synchronization module (7) takes the value of the transmission timestamp minus the fragment timestamp and then minus the adjustment coefficient as the synchronization timestamp difference. The transmitting end delay synchronization module (7) adds the fragment timestamp to the synchronization timestamp difference to obtain the start timestamp. The fragment timestamp plus the synchronization timestamp difference to obtain the output timestamp. The start timestamp and the output timestamp are sent to the transmitting end delay processing module (9), and the synchronization timestamp difference is sent to the transmitting end delay processing module (9). Among them, the adjustment coefficient is the synchronization ID signal tone in the PCM audio fragment data is segmented and reserved. In this application, the value is three times the duration of the PCM audio fragment data fragment.

[0099] As a preferred embodiment, the specific method for the receiving end echo cancellation module (10) to receive and mute the audio frequency domain slice data is as follows:

[0100] The receiving end echo cancellation module (10) receives the audio frequency domain slice data and the sending timestamp from the transmitting end delay processing module (9), and saves the audio frequency domain slice data to the transmitting end buffer area according to the sending timestamp. The transmitting end buffer area uses the latest fixed-length audio frequency domain slice data (set as 128 milliseconds in this application), and uses the sending timestamp as the latest timestamp of the transmitting end buffer area. The receiving end echo cancellation module (10) receives the synchronization timestamp difference from the transmitting end delay synchronization module (7), receives the audio frequency domain slice data and the slice timestamp from the receiving end frequency domain conversion module (3), and calculates the delay estimate.

[0101] Delay estimate = fragment timestamp + synchronization timestamp difference - latest timestamp in the sender's buffer.

[0102] The receiving end echo cancellation module (10) uses the delay estimation as the initial value of the estimated delay of the linear part echo of the linear echo cancellation NLMS algorithm, advances the convergence of the NLMS algorithm, and performs far-end and near-end non-causal estimation non-linear echo cancellation based on the estimated delay of the NLMS algorithm. Finally, the audio frequency domain slice data and the slice timestamp after the echo cancellation are sent to the receiving end frequency domain restoration module (11).

[0103] The positioning output module (4) converts the received spatial coordinates into a geodetic coordinate system according to the WGS84 ellipsoid as the final output.

[0104] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the scope of protection of the present invention.

Claims

1. A far-end echo cancellation system, characterized in that: It includes: remote audio equipment, sound decoding and receiving module, receiving end smoothing module, receiving end frequency domain conversion module, sound coding and sending module, transmitting end frequency domain conversion module, transmitting end mixing module, transmitting end delay synchronization module, transmitting end synchronization generation module, transmitting end delay processing module, receiving end echo cancellation module, receiving end frequency domain restoration module and conference bridge module; The remote audio device is used to receive the sound data from the sound encoding and sending module, decode it and play it using the local speaker. The microphone of the remote audio device receives the sound played by the local speaker and the actual voice of the user, and encodes the mixed sound and sends it to the sound decoding and receiving module. The sound decoding and receiving module is used to receive the mixed audio coded data containing the user's actual voice and the sound played by the speaker from the remote audio device through the network, and decode the mixed audio coded data into PCM audio data and send it to the receiving end smoothing module; The receiving end smoothing module is used to calculate the delay threshold estimate, take the absolute value of the difference between the delay threshold estimate and the current delay threshold, and if the absolute value is greater than the delay threshold adjustment threshold, modify the current delay threshold to the delay threshold estimate. The receiving end smoothing module caches the PCM audio data according to the current delay threshold. When the cumulative audio data of the receiving end smoothing module exceeds the current delay threshold, the PCM audio data is segmented according to a fixed time to obtain PCM audio segment data, and the PCM audio segment data and the segment timestamp are sent to the receiving end frequency domain conversion module; The receiving-end frequency domain conversion module is used to receive PCM audio fragment data and fragment timestamps, convert the PCM audio fragment data from time domain to frequency domain through FFT fast Fourier transform, and obtain audio frequency domain fragment data. The receiving-end frequency domain conversion module sends the audio frequency domain fragment data and fragment timestamps to the transmitting-end delay synchronization module and the receiving-end echo cancellation module; The sending end synchronization generation module is used to generate a synchronization identification ID and corresponding synchronization audio data, and send the synchronization audio data to the sending end mixing module, and receive the timestamp of the sending end mixing module as the synchronization timestamp, and finally send the synchronization identification ID and synchronization timestamp to the sending end delay synchronization module; The transmitting end delay synchronization module is used to calculate the synchronization timestamp difference, start timestamp and output timestamp based on the received information, send the start timestamp and output timestamp to the transmitting end delay processing module, and send the synchronization timestamp difference to the receiving end echo cancellation module; The transmitting end mixing module is used to receive the PCM audio data and timestamp of the conference bridge module. The transmitting end mixing module receives the synchronous audio data of the transmitting end synchronization generation module and mixes the data according to the time domain. The transmitting end mixing module determines whether the PCM audio data is silent data by detecting the energy value of the audio data of the conference bridge module. If it is silent data, the audio data of the transmitting end synchronization generation module replaces the PCM audio data of the conference bridge module to obtain the mixed audio data. The transmitting end mixing module sends the mixed audio data and timestamp to the sound encoding sending module, and then returns the timestamp to the transmitting end synchronization generation module; The sound coding and sending module is used to receive the PCM audio data and timestamp from the sending end mixing module, encode the PCM audio data, and send the audio data to the remote audio device through the grid. The sound coding and sending module also re-slices the PCM audio data according to a fixed time and generates a new timestamp, and sends the PCM audio slice data and the slice timestamp to the sending end frequency domain conversion module; The transmitting end frequency domain conversion module is used to receive PCM audio fragment data and fragment timestamps, convert the PCM audio fragment data from time domain to frequency domain through FFT fast Fourier transform, and obtain audio frequency domain fragment data. The transmitting end frequency domain conversion module sends the audio frequency domain fragment data and fragment timestamps to the transmitting end delay processing module; The transmitting end delay processing module is used to receive the audio frequency domain fragmentation data and fragmentation timestamp from the transmitting end frequency domain conversion module, put the audio frequency domain fragmentation data and fragmentation timestamp into a first-in-first-out queue, receive the start timestamp from the transmitting end delay synchronization module, discard the audio frequency domain fragmentation data whose fragmentation timestamp is less than the start timestamp, receive the output timestamp from the transmitting end delay synchronization module, and send the audio frequency domain fragmentation data and the sending timestamp whose fragmentation timestamp is less than or equal to the output timestamp to the receiving end echo cancellation module; The receiving-end echo cancellation module is used to receive and mute the audio frequency domain slice data and send the mute audio frequency domain slice data and the slice timestamp to the receiving-end frequency domain restoration module; The receiving-end frequency domain restoration module is used to receive the audio frequency domain fragmentation data and fragmentation timestamps from the receiving-end echo cancellation module. The receiving-end frequency domain restoration module performs an IFFT (Fast Fourier Inverse Transform) on the audio frequency domain fragmentation data, converts the frequency domain sound into time domain PCM sound data, and then sends the PCM sound data to the conference bridge module. The conference bridge module is used to receive the PCM sound data from the frequency domain restoration module, mix the PCM sound data, and send the mixed PCM sound data and timestamp to the sending mixing modules of other conference members.

2. The far-end echo cancellation system according to claim 1, wherein: The receiving end smoothing module adopts smooth delay control to ensure that its output audio data is smooth in the time dimension. The receiving end smoothing module uses the preset delay as the initial value of the delay gate limit, counts the arrival time intervals of adjacent audio packets, counts the time intervals of fixed-period audio data packets, and the delay jitter value is greater than the minimum value of the time interval of audio data packets greater than the threshold within the period. The delay jitter value is multiplied by the adjustment coefficient to obtain the delay threshold statistic value. The Kalman filter is used to filter the delay threshold statistic value to obtain the delay threshold estimation value, filter the mutation, and smooth the delay threshold estimation value.

3. The far-end echo cancellation system according to claim 1, wherein: The transmitting end synchronization generation module uses a random algorithm to select a frequency in the high frequency group, medium frequency group, and low frequency group, and generates a synchronization identification ID with a value range of 0-26. Synchronous identification ID = high frequency group frequency number * 9 + medium frequency group frequency number * 3 + low frequency group frequency number. The transmitting end synchronization generation module generates fixed-length synchronous audio data according to the three frequencies of the synchronous identification ID.

4. The far-end echo cancellation system according to claim 3, wherein: The transmitting end delay synchronization module receives the audio frequency domain fragmentation data and fragmentation timestamp from the receiving end frequency domain conversion module and the synchronization identification ID and sending timestamp from the receiving end synchronization generation module. The transmitting end delay synchronization module divides the audio frequency domain fragmentation data into three blocks according to the high frequency group, the medium frequency group and the low frequency group, and processes the audio frequency domain data of the three blocks respectively. The transmitting end delay synchronization module calculates the square value of the energy value of each frequency of the audio frequency domain data, amplifies the difference in the energy value of each frequency, and calculates the average value of the energy value of each frequency in the block. If the synchronization ID is If the ID in the frequency domain corresponding to the block is greater than the threshold k, the synchronization block ID is considered to match in the block. When all three blocks match, it is considered that the synchronization identification ID is found in the audio frequency domain fragment data. The sending end delay synchronization module subtracts the fragment timestamp from the sending timestamp and then subtracts the adjustment coefficient as the synchronization timestamp difference. The sending end delay synchronization module adds the fragment timestamp to the synchronization timestamp difference to obtain the start timestamp, and adds the fragment timestamp to the synchronization timestamp difference to obtain the output timestamp. The start timestamp and output timestamp are sent to the sending end delay processing module, and the synchronization timestamp difference is sent to the sending end delay processing module.

5. The far-end echo cancellation system according to claim 4, wherein: The receiving-end echo cancellation module receives the audio frequency domain fragmentation data and the sending timestamp from the transmitting-end delay processing module, and saves the audio frequency domain fragmentation data to the transmitting-end buffer according to the sending timestamp. The transmitting-end buffer uses the latest fixed-duration audio frequency domain fragmentation data and uses the sending timestamp as the latest timestamp of the transmitting-end buffer. The receiving-end echo cancellation module receives the synchronization timestamp difference from the transmitting-end delay synchronization module, receives the audio frequency domain fragmentation data and the fragmentation timestamp from the receiving-end frequency domain conversion module, and calculates the delay estimate. Delay estimate = fragment timestamp + synchronization timestamp difference - latest timestamp of the sending end buffer, The receiving-end echo cancellation module uses the delay estimate as the initial value for the estimated delay of the linear part of the echo in the NLMS algorithm, accelerating the convergence of the NLMS algorithm. It then performs near- and far-end non-causal non-linear echo cancellation based on the NLMS algorithm's estimated delay. Finally, it sends the echo-cancelled audio frequency domain fragment data and fragment timestamps to the receiving-end frequency domain restoration module.

6. A far-end echo cancellation method, characterized in that: The following steps are involved: The remote audio device receives the sound data from the sound encoding and sending module, decodes it, and plays it using the local speaker. The microphone of the remote audio device receives the sound played by the local speaker and the actual voice of the user, and encodes the mixed sound and sends it to the sound decoding and receiving module. The sound decoding and receiving module receives the mixed audio coded data containing the user's actual voice and the sound played by the speaker from the remote audio device through the network, and decodes the mixed audio coded data into PCM audio data and sends it to the receiving end smoothing module; The receiving end smoothing module calculates the delay threshold estimate, takes the absolute value of the difference between the delay threshold estimate and the current delay threshold, and if the absolute value is greater than the delay threshold adjustment threshold, changes the current delay threshold to the delay threshold estimate. The receiving end smoothing module caches the PCM audio data according to the current delay threshold. When the cumulative audio data of the receiving end smoothing module exceeds the current delay threshold, the PCM audio data is segmented according to a fixed time to obtain PCM audio segment data, and the PCM audio segment data and the segment timestamp are sent to the receiving end frequency domain conversion module; The receiving-end frequency domain conversion module receives PCM audio fragment data and fragment timestamps, converts the PCM audio fragment data from time domain to frequency domain through FFT fast Fourier transform, and obtains audio frequency domain fragment data. The receiving-end frequency domain conversion module sends the audio frequency domain fragment data and fragment timestamps to the transmitting-end delay synchronization module and the receiving-end echo cancellation module; The sending end synchronization generation module generates a synchronization identification ID and corresponding synchronization audio data, and sends the synchronization audio data to the sending end mixing module, and receives the timestamp of the sending end mixing module as the synchronization timestamp, and finally sends the synchronization identification ID and synchronization timestamp to the sending end delay synchronization module; The transmitting end delay synchronization module calculates the synchronization timestamp difference, the start timestamp and the output timestamp based on the received information, sends the start timestamp and the output timestamp to the transmitting end delay processing module, and sends the synchronization timestamp difference to the receiving end echo cancellation module; The sending end mixing module receives the PCM audio data and timestamp from the conference bridge module. The sending end mixing module receives the synchronous audio data from the sending end synchronization generation module and mixes the data in the time domain. The sending end mixing module determines whether the PCM audio data is silent data by detecting the energy value of the audio data from the conference bridge module. If it is silent data, the audio data from the sending end synchronization generation module replaces the PCM audio data of the conference bridge module to obtain mixed audio data. The sending end mixing module sends the mixed audio data and timestamp to the sound encoding sending module, and then returns the timestamp to the sending end synchronization generation module. The sound coding and sending module receives the PCM audio data and timestamp from the transmitting mixing module, encodes the PCM audio data, and sends the audio data to the remote audio device through the grid. The sound coding and sending module also re-slices the PCM audio data according to a fixed time and generates a new timestamp, and sends the PCM audio slice data and the slice timestamp to the transmitting frequency domain conversion module. The transmitting end frequency domain conversion module receives PCM audio fragment data and fragment timestamps, converts the PCM audio fragment data from time domain to frequency domain through FFT fast Fourier transform, and obtains audio frequency domain fragment data. The transmitting end frequency domain conversion module sends the audio frequency domain fragment data and fragment timestamps to the transmitting end delay processing module; The transmitting end delay processing module receives the audio frequency domain fragmentation data and fragmentation timestamp from the transmitting end frequency domain conversion module, and puts the audio frequency domain fragmentation data and fragmentation timestamp into a first-in-first-out queue. The transmitting end delay processing module receives the start timestamp from the transmitting end delay synchronization module, and discards the audio frequency domain fragmentation data whose fragmentation timestamp is less than the start timestamp. The transmitting end delay processing module receives the output timestamp of the transmitting end delay synchronization module, and sends the audio frequency domain fragmentation data and the sending timestamp whose fragmentation timestamp is less than or equal to the output timestamp to the receiving end echo cancellation module. The receiving end echo cancellation module receives the audio frequency domain slice data, mutes it, and sends the muted audio frequency domain slice data and the slice timestamp to the receiving end frequency domain restoration module; The receiving-end frequency domain restoration module receives the audio frequency domain fragmentation data and fragmentation timestamps from the receiving-end echo cancellation module. The receiving-end frequency domain restoration module performs an IFFT (Fast Fourier Inverse Transform) on the audio frequency domain fragmentation data, converts the frequency domain sound into time domain PCM sound data, and then sends the PCM sound data to the conference bridge module. The conference bridge module receives the PCM sound data from the frequency domain restoration module, performs sound mixing on the PCM sound data, and sends the mixed PCM sound data and timestamp to the sending mixing modules of other conference members.

7. The far-end echo cancellation method according to claim 6, wherein: The specific method for the receiving end smoothing module to calculate the delay threshold estimation value is: The receiving end smoothing module adopts smooth delay control to ensure that its output audio data is smooth in the time dimension. The receiving end smoothing module uses the preset delay as the initial value of the delay gate limit, counts the arrival time intervals of adjacent audio packets, counts the time intervals of fixed-period audio data packets, and the delay jitter value is greater than the minimum value of the time interval of audio data packets greater than the threshold within the period. The delay jitter value is multiplied by the adjustment coefficient to obtain the delay threshold statistic value. The Kalman filter is used to filter the delay threshold statistic value to obtain the delay threshold estimation value, filter the mutation, and smooth the delay threshold estimation value.

8. The far-end echo cancellation method according to claim 6, wherein: The specific method for the transmitting end synchronization generation module to generate the synchronization identification ID and the corresponding synchronization audio data is as follows: The transmitting end synchronization generation module uses a random algorithm to select a frequency in the high frequency group, medium frequency group, and low frequency group, and generates a synchronization identification ID with a value range of 0-26. Synchronous identification ID = high frequency group frequency number * 9 + medium frequency group frequency number * 3 + low frequency group frequency number. The transmitting end synchronization generation module generates fixed-length synchronous audio data according to the three frequencies of the synchronous identification ID.

9. The far-end echo cancellation method according to claim 8, wherein: The specific method by which the transmitting end delay synchronization module calculates the synchronization timestamp difference, start timestamp, and output timestamp based on the received information is as follows: The transmitting end delay synchronization module receives the audio frequency domain fragmentation data and fragmentation timestamp from the receiving end frequency domain conversion module and the synchronization identification ID and sending timestamp from the receiving end synchronization generation module. The transmitting end delay synchronization module divides the audio frequency domain fragmentation data into three blocks according to the high frequency group, the medium frequency group and the low frequency group, and processes the audio frequency domain data of the three blocks respectively. The transmitting end delay synchronization module calculates the square value of the energy value of each frequency of the audio frequency domain data, amplifies the difference in the energy value of each frequency, and calculates the average value of the energy value of each frequency in the block. If the synchronization ID is If the ID in the frequency domain corresponding to the block is greater than the threshold k, the synchronization block ID is considered to match in the block. When all three blocks match, it is considered that the synchronization identification ID is found in the audio frequency domain fragment data. The sending end delay synchronization module subtracts the fragment timestamp from the sending timestamp and then subtracts the adjustment coefficient as the synchronization timestamp difference. The sending end delay synchronization module adds the fragment timestamp to the synchronization timestamp difference to obtain the start timestamp, and adds the fragment timestamp to the synchronization timestamp difference to obtain the output timestamp. The start timestamp and output timestamp are sent to the sending end delay processing module, and the synchronization timestamp difference is sent to the sending end delay processing module.

10. The far-end echo cancellation method according to claim 9, wherein: The specific method for the receiving end echo cancellation module to receive and cancel the audio frequency domain fragment data is as follows: The receiving-end echo cancellation module receives the audio frequency domain fragmentation data and the sending timestamp from the transmitting-end delay processing module, and saves the audio frequency domain fragmentation data to the transmitting-end buffer according to the sending timestamp. The transmitting-end buffer uses the latest fixed-duration audio frequency domain fragmentation data and uses the sending timestamp as the latest timestamp of the transmitting-end buffer. The receiving-end echo cancellation module receives the synchronization timestamp difference from the transmitting-end delay synchronization module, receives the audio frequency domain fragmentation data and the fragmentation timestamp from the receiving-end frequency domain conversion module, and calculates the delay estimate. Delay estimate = fragment timestamp + synchronization timestamp difference - latest timestamp in the sender's buffer. The receiving-end echo cancellation module uses the delay estimate as the initial value for the estimated delay of the linear part of the echo in the NLMS algorithm, accelerating the convergence of the NLMS algorithm. It then performs near- and far-end non-causal non-linear echo cancellation based on the NLMS algorithm's estimated delay. Finally, it sends the echo-cancelled audio frequency domain fragment data and fragment timestamps to the receiving-end frequency domain restoration module.

Images