Echo detection and alignment method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of audio processing, and discloses an echo detection and alignment method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a playing signal and a recording signal in real time, and carrying out the down-sampling of the playing signal and the recording signal, and obtaining a far-end signal and a near-end signal of a set frequency; based on the far-end signal and the near-end signal, a pre-constructed segmented normalized minimum mean square filter is adopted to carry out time domain adaptive detection matching echo time delay, and echo time delay offset is obtained; and carrying out statistics on the echo time delay offset by adopting a statistical histogram to obtain echo time delay, and carrying out time domain alignment on the playing signal and the recording signal based on the echo time delay. Whether the echo signal exists or not is detected with low complexity and high matching accuracy, the echo signal and the original signal are aligned in the time domain, the processing effect of the echo cancellation model in a high-delay scene can be remarkably improved, and the application scene of the echo cancellation model is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to an echo detection and alignment method, device, computer equipment and storage medium. Background Art

[0002] In real-time audio communication, the main reason for echo is that the voice of the other party is broadcast through the local speaker, captured by the local recording device and the audio signal is transmitted to the other party again, and broadcast from the speaker on the other side, generating echo. Figure 1 As shown. Echoes can disrupt the continuity and immediacy of voice communications, reduce communication efficiency and call quality. In order to reduce the interference of echoes on voice communications, communication systems can use echo suppression or echo cancellation algorithms, which can detect and suppress echo signals to provide clearer and cleaner speech audio.

[0003] In recent years, deep learning-based echo cancellation algorithms have made significant progress in echo filtering, especially in the nonlinear part of echo processing, which is better than traditional algorithms. Due to the limitations of computing power and model complexity, most deep learning algorithms have difficulty dealing with echoes with a delay of more than 100ms, which causes serious echo residues in some high-latency scenarios. Figure 2 As shown in the figure, the latency of the cloud desktop echo is 50ms-300ms higher than that of the local terminal due to network fluctuations. In addition, the local latency of some weak terminals is also over 200ms. The high latency limits the application of deep learning echo models in these scenarios, so an echo detection and alignment method is urgently needed to reduce signal latency. Summary of the invention

[0004] In view of this, the present invention provides an echo detection and alignment method, apparatus, computer device and storage medium to solve the problem of insufficient processing effect of high-latency signals in high-latency scenarios using a deep learning-based echo cancellation model.

[0005] In a first aspect, the present invention provides an echo detection and alignment method, the method comprising:

[0006] Acquire the playback signal and recording signal in real time, and downsample the playback signal and recording signal to obtain the far-end signal and near-end signal of the set frequency respectively;

[0007] Based on the far-end signal and the near-end signal, a pre-built piecewise normalized minimum mean square filter is used to perform time domain adaptive detection and matching of the echo delay to obtain the echo delay offset;

[0008] The echo delay offset is statistically analyzed using a statistical histogram to obtain the echo delay, and the playback signal and the recording signal are aligned in time domain based on the echo delay.

[0009] The echo detection and alignment method provided by the present invention models and analyzes the relationship between the microphone signal and the speaker signal, extracts the characteristics of the microphone playback signal and the speaker recording signal, aligns the echo signal with the original signal in the time domain through an adaptive filter (i.e., a piecewise normalized minimum mean square filter), reduces the delay of the two signals with lower complexity, and uses a normalized step size so that the matching process can change the step size according to the energy size of the input signal, converges faster when the input signal energy is high, and ensures that the parameters are not easy to diverge when the input signal energy is low, thereby meeting the real-time requirements on low-end terminals, thereby improving the performance of the deep learning echo cancellation model in high-latency scenarios, and solving the problem of insufficient processing effect of the deep learning-based echo cancellation model in high-latency scenarios when the signal has high latency.

[0010] In an optional implementation, downsampling the playback signal and the recording signal to obtain the far-end signal and the near-end signal of the set frequency respectively includes:

[0011] Intercepting a playback signal at a preset time interval from the playback signal, dividing the playback signal at the preset time interval according to a preset time unit to obtain a multi-frame playback signal, and downsampling the multi-frame playback signal to obtain a remote signal of a set frequency;

[0012] After downsampling the recorded signal, it is divided into blocks according to preset sampling points. The sampling points of two adjacent blocks are arbitrarily selected and low-pass filtered using short-time Fourier transform. After filtering out signals above the Nyquist frequency in the recorded signal, an inverse short-time Fourier transform is performed to obtain a time domain signal. The time domain signal is downsampled to obtain a near-end signal of the set frequency.

[0013] The echo detection and alignment method provided by the present invention intercepts the playback signal of a preset time interval from the playback signal, divides the playback signal of the preset time interval according to the preset time unit to obtain a multi-frame playback signal, downsamples the multi-frame playback signal to obtain a far-end signal of a set frequency, reduces the transmission rate of the playback signal, reduces the signal size, and reduces the amount of calculation, which is conducive to storage. After downsampling the recording signal, it is divided into blocks according to preset sampling points, and the sampling points of two adjacent blocks are arbitrarily selected to perform low-pass filtering using short-time Fourier transform, and the signal higher than the Nyquist frequency in the recording signal is filtered out and then inverse short-time Fourier transform is performed to obtain a time domain signal, and the time domain signal is downsampled to obtain a near-end signal of a set frequency, thereby reducing the aliasing effect in the recording signal and providing conditions for subsequent detection of echo delay.

[0014] In an optional implementation, the piecewise normalized minimum mean square filter is constructed in the following manner:

[0015] Set the number of block filters and the number of parameters, starting offset, internal offset and normalization step size for each block filter;

[0016] Each block filter is constructed based on the number of block filters, the set number of parameters of each block filter, the starting offset, the internal offset and the normalization step size to obtain a piecewise normalized minimum mean square filter.

[0017] In an optional implementation, based on the far-end signal and the near-end signal, a pre-built piecewise normalized minimum mean square filter is used to perform time-domain adaptive detection and matching of the echo delay, and the echo delay offset is obtained, which includes:

[0018] traversing the output of each block filter and updating the parameters of each block filter based on the far-end signal and the near-end signal;

[0019] Obtaining energy corresponding to the near-end signal and accuracy of the current blocking filter, and calculating a maximum internal offset of the corresponding blocking filter based on the energy corresponding to the near-end signal, the accuracy of the current blocking filter, and parameters of the current blocking filter;

[0020] Based on the preset deblocking filter accuracy threshold and the corresponding deblocking filter maximum internal offset, it is determined whether there is an optimal delay estimate, and the optimal delay estimate is used as the echo delay offset.

[0021] The echo detection and alignment method provided by the present invention uses multi-block filter matching processing, which can ensure that matching for longer delays is achieved while limiting the number of filter parameters and reducing computing power requirements, meet high delay requirements, and achieve usability in high delay scenarios; using a normalized step size, the matching process can achieve step size changes according to the energy size of the input signal, converge faster when the input signal energy is high, and when the input signal energy is low, ensure that the parameters of the block filter are not easy to diverge, and achieve detection of whether the echo signal exists with lower complexity and higher matching accuracy.

[0022] In an optional implementation, traversing the output of each deblocking filter based on the far-end signal and the near-end signal and updating the parameters of each deblocking filter includes:

[0023] Selecting a number of remote signal sampling points from the remote signal equal to the number of parameters as inputs of any block filter, and calculating an output of each block filter based on the starting offset, each remote signal sampling point and each parameter;

[0024] Selecting a near-end signal sampling point equal to the internal offset value from the near-end signal, and calculating an error of each deblocking filter based on the near-end signal sampling point and the output of each deblocking filter;

[0025] The energy value corresponding to the far-end signal sampling point is obtained, and the parameters of each block filter are updated based on the energy value, the error of each block filter and the preset smoothing coefficient.

[0026] The echo detection and alignment method provided by the present invention obtains the output and error of each block filter based on the preset parameters of each block filter and the near-end signal and the far-end signal, and finally achieves the purpose of updating the parameters of each block filter, thereby improving the quality of the near-end signal and the far-end signal and improving the processing capability of each block filter.

[0027] In an optional implementation, judging whether there is an optimal delay estimate value based on a preset deblocking filter accuracy threshold and a corresponding deblocking filter maximum internal offset, and using the optimal delay estimate value as the echo delay offset includes:

[0028] Obtaining the maximum accuracy among multiple blocking filters;

[0029] If the maximum accuracy is greater than or equal to the preset block filter accuracy threshold, it is judged that there is an echo signal in the input far-end signal, and the corresponding block filter optimal delay estimate is used as the echo delay offset; if the maximum accuracy is less than the preset block filter accuracy threshold, it is judged that there is no echo signal in the input far-end signal, that is, there is no optimal delay estimate for the corresponding block filter.

[0030] The echo detection and alignment method provided by the present invention determines whether there is an echo signal in the input far-end signal based on the relationship between the maximum accuracy of multiple block filters and the preset block filter accuracy threshold, thereby improving the accuracy of judging the echo signal and realizing the detection and delay estimation of the echo signal.

[0031] In an optional implementation manner, using a statistical histogram to collect statistics on the echo delay offset to obtain the echo delay includes:

[0032] A statistical histogram is used to continuously count the echo delay offset within a preset time period. If the echo delay offset continues to appear for more than a preset number of times, the echo delay offset is used as the optimal delay offset, and the echo delay is calculated based on the optimal delay offset and the set frequency.

[0033] The echo detection and alignment method provided by the present invention adopts a statistical histogram to continuously count the echo delay offset within a preset time period. If the echo delay offset continues to appear for more than a preset number of times, the echo delay offset is used as the optimal delay offset, and the echo delay is calculated based on the optimal delay offset and the set frequency, thereby ensuring the stable output of the echo delay offset and avoiding the situation where the output delay frequently jumps and causes unstable output.

[0034] In a second aspect, the present invention provides an echo detection and alignment device, the device comprising:

[0035] The signal acquisition processing module is used to acquire the playback signal and the recording signal in real time, and downsample the playback signal and the recording signal to obtain the far-end signal and the near-end signal of the set frequency respectively;

[0036] A detection and matching module is used to perform time-domain adaptive detection and matching of echo delay based on the far-end signal and the near-end signal using a pre-built piecewise normalized minimum mean square filter to obtain an echo delay offset;

[0037] The statistical histogram processing and signal alignment module is used to use the statistical histogram to calculate the echo delay offset to obtain the echo delay, and to align the playback signal and the recording signal in the time domain based on the echo delay.

[0038] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the echo detection and alignment method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the echo detection and alignment method of the first aspect or any corresponding embodiment thereof.

[0040] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the echo detection and alignment method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a schematic diagram of the echo generation process in real-time voice communication;

[0043] Figure 2 This is an example diagram of the time delay between the echo signal and the reference signal;

[0044] Figure 3 is a flow chart of an echo detection and alignment method according to an embodiment of the present invention;

[0045] Figure 4 is a flow chart of another echo detection and alignment method according to an embodiment of the present invention;

[0046] Figure 5 is a flow chart of another echo detection and alignment method according to an embodiment of the present invention;

[0047] Figure 6 is a flow chart of yet another echo detection and alignment method according to an embodiment of the present invention;

[0048] Figure 7 is a frequency domain response diagram of a low-pass filter according to an embodiment of the present invention;

[0049] Figure 8 is a schematic diagram of input and output of a segmented NLMS filter according to an embodiment of the present invention;

[0050] Fig. 9 is a comparison diagram of the effects before and after the alignment of the playback signal and the recording signal according to an embodiment of the present invention;

[0051] FIG10( a ) is a schematic diagram of a pre-processing effect of aligning a playback signal with a recording signal according to an embodiment of the present invention;

[0052] FIG10( b ) is a schematic diagram showing the effect of aligning the playback signal with the recording signal according to an embodiment of the present invention;

[0053] Fig.11 is a structural block diagram of an echo detection and alignment device according to an embodiment of the present invention;

[0054] Fig.12 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0056] According to an embodiment of the present invention, an embodiment of an echo detection and alignment method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0057] In this embodiment, an echo detection and alignment method is provided, which can be used in a computer terminal, such as a central processing unit, a server, etc. Figure 3 is a flow chart of an echo detection and alignment method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0058] Step S101 , acquiring a playback signal and a recording signal in real time, and downsampling the playback signal and the recording signal respectively to obtain a far-end signal and a near-end signal of a set frequency.

[0059] Specifically, in the audio communication process, the microphone signal is used as the playback signal and the speaker signal is used as the recording signal. Downsampling is a multi-rate digital signal processing technology used to reduce the sampling rate of the signal, thereby reducing the data transmission rate or data size. The downsampling factor (commonly represented by the symbol M) is generally an integer or rational number greater than 1, indicating that the sampling period becomes several times larger than the original, or equivalently, the sampling rate becomes a fraction of the original. After downsampling the playback signal and the recording signal, the far-end signal and the near-end signal of the set frequency are obtained.

[0060] Step S102: Based on the far-end signal and the near-end signal, a pre-built piecewise normalized minimum mean square filter is used to perform time-domain adaptive detection and matching of echo delay to obtain an echo delay offset.

[0061] Specifically, the piecewise normalized least mean squares (NLMS) is an improved LMS algorithm (Least Mean Squares Algorithm, adaptive filter algorithm). According to the product of the error signal and the far-end input signal in the least mean square algorithm, the square of the far-end input signal is normalized, and the LMS algorithm with a fixed step size factor is converted into an algorithm with a variable step size that varies according to the time-varying input signal.

[0062] Through an adaptive filter, namely a piecewise normalized minimum mean square filter, the far-end signal and the near-end signal are adaptively matched to determine whether there is an echo signal in the far-end signal input and obtain the echo delay offset, thereby reducing the delay of the two signals with lower complexity and meeting the real-time requirements on low-end terminals.

[0063] Step S103: Use a statistical histogram to collect statistics on the echo delay offset to obtain the echo delay, and align the playback signal and the recording signal in the time domain based on the echo delay.

[0064] Specifically, a statistical histogram is obtained by measuring a physical quantity multiple times, dividing the measurement results into several intervals, counting the frequency of each interval, and then drawing a rectangular graph with the measurement data as the horizontal axis and the frequency as the vertical axis to obtain a statistical histogram.

[0065] The echo delay offset is counted using a statistical histogram to obtain the echo delay offset with the highest frequency, and the echo delay is obtained based on the relationship between the echo delay offset and the number of sampling points of the far-end signal.

[0066] The echo detection and alignment method provided in this embodiment models and analyzes the relationship between the microphone signal and the speaker signal, extracts the features of the microphone playback signal and the speaker recording signal, aligns the echo signal with the original signal in the time domain through an adaptive filter (i.e., a piecewise normalized minimum mean square filter), reduces the delay of the two signals with lower complexity, and uses a normalized step size so that the matching process can change the step size according to the energy size of the input signal. It converges faster when the input signal energy is high, and ensures that the parameters are not easy to diverge when the input signal energy is low, thereby meeting the real-time requirements on low-end terminals, thereby improving the performance of the deep learning echo cancellation model in high-latency scenarios, and solving the problem of insufficient processing effect of the deep learning-based echo cancellation model in high-latency scenarios when the signal has high latency.

[0067] In this embodiment, an echo detection and alignment method is provided, which can be used in a computer terminal, such as a central processing unit, a server, etc. Figure 4 is a flow chart of an echo detection and alignment method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0068] Step S201 , acquiring a playback signal and a recording signal in real time, and downsampling the playback signal and the recording signal respectively to obtain a far-end signal and a near-end signal of a set frequency.

[0069] Specifically, the above step S201 includes:

[0070] Step S2011, intercepting a playback signal at a preset time interval from the playback signal, dividing the playback signal at the preset time interval according to a preset time unit to obtain a multi-frame playback signal, and downsampling the multi-frame playback signal to obtain a remote signal of a set frequency.

[0071] For example, 1s of playback signal data is obtained, with 10ms as one frame, that is, 1s of playback signal data includes 1000 frames of data, and the downsampling factor used is 4kHz (kilohertz), that is, 1s of playback signal data is downsampled to obtain 4000 sampling point data as the remote signal, and the remote signal is cached locally. The processing process of the playback signal in the playback channel is as follows: Figure 6 shown.

[0072] Step S2012, after downsampling the recorded signal, divide it into blocks with preset sampling points, arbitrarily select sampling points of two adjacent blocks to perform low-pass filtering using short-time Fourier transform, filter out signals above the Nyquist frequency in the recorded signal, and then perform inverse short-time Fourier transform to obtain a time domain signal, and downsample the time domain signal to obtain a near-end signal of the set frequency.

[0073] Short-time Fourier transform, abbreviated as STFT (Short-time Fourier Transform), is a variation of Fourier transform, usually used to describe changes in frequency domain and time domain, and is an important signal tool in time-frequency analysis. Inverse short-time Fourier transform, abbreviated as ISTFT (Inverse Short-time Fourier Transform), is the inverse transform of STFT.

[0074] For example, take the near-end data, downsample to 16kHz, divide it into blocks with 64 sampling points, take the current block and the previous block with a total of 128 sampling points for STFT processing, and perform a low-pass filter on the frequency domain data. Ideally, the frequency above the Nyquist frequency (Nyquist frequency refers to the minimum sampling frequency that needs to be defined in the signal sampling process to prevent signal aliasing) will be completely filtered out to avoid aliasing. The frequency response diagram of the low-pass filter is as follows: Figure 7 As shown, Figure 7 The horizontal axis Frequency is the sampling frequency, and the vertical axis Amplitude is the low-pass filter amplitude.

[0075] The low-pass filtered signal is processed by ISTFT to restore the time domain data, and is also downsampled to 4kHz to obtain a 4kHz near-end signal. Figure 6 shown.

[0076] Step S202: Based on the far-end signal and the near-end signal, a pre-built segmented normalized minimum mean square filter is used to perform time domain adaptive detection and matching of the echo delay to obtain the echo delay offset. Figure 3 Step S102 of the illustrated embodiment will not be described in detail here.

[0077] Step S203: Use a statistical histogram to collect statistics on the echo delay offset to obtain the echo delay, and align the playback signal and the recording signal in the time domain based on the echo delay. Figure 3 Step S103 of the illustrated embodiment will not be described in detail here.

[0078] The echo detection and alignment method provided in this embodiment intercepts the playback signal of a preset time interval from the playback signal, and divides the playback signal of the preset time interval according to the preset time unit to obtain multiple frames of playback signals, downsamples the multiple frames of playback signals to obtain a far-end signal of a set frequency, reduces the transmission rate of the playback signal, reduces the signal size, and reduces the amount of calculation, which is conducive to storage. After downsampling the recording signal, it is divided into blocks according to preset sampling points, and the sampling points of two adjacent blocks are arbitrarily selected to perform low-pass filtering using short-time Fourier transform, and the signal above the Nyquist frequency in the recording signal is filtered out and then inverse short-time Fourier transform is performed to obtain a time domain signal, and the time domain signal is downsampled to obtain a near-end signal of a set frequency, thereby reducing the aliasing effect in the recording signal and providing conditions for subsequent detection of echo delay.

[0079] In this embodiment, an echo detection and alignment method is provided, which can be used in a computer terminal, such as a central processing unit, a server, etc. Figure 5 is a flow chart of an echo detection and alignment method according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0080] Step S301 , acquiring a playback signal and a recording signal in real time, and downsampling the playback signal and the recording signal respectively to obtain a far-end signal and a near-end signal of a set frequency.

[0081] Step S302: Based on the far-end signal and the near-end signal, a pre-built piecewise normalized minimum mean square filter is used to perform time-domain adaptive detection and matching of echo delay to obtain an echo delay offset.

[0082] Specifically, the piecewise normalized minimum mean square filter is constructed in the following way:

[0083] The number of block filters and the number of parameters of each block filter, the starting offset, the internal offset and the normalization step size are set; based on the number of block filters, the set number of parameters of each block filter, the starting offset, the internal offset and the normalization step size, each block filter is constructed to obtain a piecewise normalized minimum mean square filter.

[0084] Exemplarily, in order to meet high delay matching, 10 block filters are set, the number of parameters of each block filter is 512, and the time domain starting offset of the block filter is 384, which can meet the accurate matching of the delay within 1s; the normalization step size of the block filter is step, and the internal offset is 40.

[0085] The above step S302 includes:

[0086] Step S3021: traverse the output of each deblocking filter based on the far-end signal and the near-end signal and update the parameters of each deblocking filter.

[0087] The input and output diagram of the segmented NLMS filter is as follows Figure 8 As shown, for the nth block filter, let the current filter parameter be h[n], the block filter output be s[n], the output of the current i-th near-end signal be s[n][i], the error be e[n][i], and the error energy sum of the block filter be E_sum[n].

[0088] In some optional implementations, the above step S3021 includes:

[0089] Step a1, selecting from the remote signal a number of remote signal sampling points equal to the number of parameters as inputs of any deblocking filter, and calculating the output of each deblocking filter based on the starting offset, each remote signal sampling point and each parameter.

[0090] Exemplarily, assume that 512 sampling points of the current far-end signal are used as the input x[n][i] of the nth block filter, where n represents the nth block filter and the starting offset of the input is n*384. For example, the input of the second block filter is 512 sampling points after the latest input signal is offset by 384 sampling points, and i represents the internal offset of the block filter. For example, x[0][1] represents the far-end signal of 512 sampling points after the latest offset by 1 sampling point.

[0091] Traversing each block filter n∈[0,10), each near-end signal sampling point i∈[0,40), the output calculation formula of each block filter is as follows:

[0092] s[n][i]=h[n]*x[n][i] (1).

[0093] For example, in formula (1), the value of i ranges from 0 to 39. When i=0, that is, when 512 far-end signal sampling points x[0][1] are input, the corresponding block filter parameter h[n] is also an array of 512. Each far-end signal sampling point is multiplied by the corresponding block filter parameter to obtain the output s[n][0].

[0094] Figure 8 There are 10 block filters in total, and the number of sampling points that the filter can cover is 384*9+512=3968; at a sampling frequency of 4kHz, the number of sampling points in 1s is 4000, and the delay value that can be matched is 3968 / 4000*1000=992ms.

[0095] Step a2: selecting a near-end signal sampling point equal to the internal offset value from the near-end signal, and calculating the error of each deblocking filter based on the near-end signal sampling point and the output of each deblocking filter.

[0096] Specifically, the internal offset is 40, and the error formula for calculating each block filter based on the near-end signal sampling point and the output of each block filter is as follows:

[0097] e[n][i]=y[i]-s[n][i](2);

[0098] The near-end signal of 40 sampling points after the current preprocessing is assumed to be the expected signal y, and y[i] represents the i-th sampling point of the near-end signal.

[0099] The error energy and calculation formula of a single blocking filter are as follows:

[0100] E_sum[n]=∑ i∈[0,39] e[n][i] (3).

[0101] Step a3, obtaining the energy value corresponding to the far-end signal sampling point, and updating the parameters of each block filter based on the energy value, the error of each block filter and the preset smoothing coefficient.

[0102] Specifically, assuming that the far-end signal energy value of the current deblocking filter input is X2_sum[n][i], the calculation formula is as follows:

[0103] X2_sum[n][i]=∑ m∈[0,512) x[n][i][m] (4);

[0104] E2_sum[n]=∑ i∈[0,40) e[n][i] 2 (5);

[0105] Then the mth parameter of the nth block filter is h[n][m] and the calculation formula is as follows:

[0106] h[n][m]=step′*(e[n][i]*x[n][i][m]) / X2_sum[n][i] (6);

[0107] Among them, step′ is the smoothing coefficient.

[0108] The echo detection and alignment method provided in this embodiment obtains the output and error of each block filter based on the preset parameters of each block filter and the near-end signal and the far-end signal, and finally achieves the purpose of updating the parameters of each block filter, thereby improving the quality of the near-end signal and the far-end signal and improving the processing capability of each block filter.

[0109] Step S3022: Obtain the energy corresponding to the near-end signal and the accuracy of the current blocking filter, and calculate the maximum internal offset of the corresponding blocking filter based on the energy corresponding to the near-end signal, the accuracy of the current blocking filter and the parameters of the current blocking filter.

[0110] Specifically, assuming that the energy of the current near-end signal is Y2_sum[n], the accuracy of the current deblocking filter is Acc[n], and the internal offset of the maximum energy value after the deblocking filter coefficient is updated is Max[n]. The calculation formulas are as follows:

[0111] Y2_sum[n]=∑ i∈[0,40) y[n][i] 2 (7);

[0112] Acc[n]=Y2 sum[n] -E2_sum[n] (8);

[0113] Max[n]=max(h[n][m] 2 ) (9).

[0114] Step S3023: judging whether there is an optimal delay estimate value based on a preset deblocking filter accuracy threshold and a corresponding deblocking filter maximum internal offset, and taking the optimal delay estimate value as the echo delay offset.

[0115] In some optional implementations, the above step S3023 includes:

[0116] Step b1, obtaining the maximum accuracy among multiple block filters.

[0117] Specifically, the options where the internal offset Max[n]<=2 and Max[n]>502 (block filter size512-10) of the maximum energy after the block filter coefficient is updated are eliminated, and the filter with the highest accuracy among the remaining block filters is selected (implying two conditions: when the error energies are equal, the matching value with larger near-end signal energy is given priority; similarly, when the near-end signal energies are equal, the matching value with smaller error energy is given priority), and the maximum accuracy max(Acc[n]) in the block filter is obtained.

[0118] Step b2: If the maximum accuracy is greater than or equal to the preset block filter accuracy threshold, it is determined that there is an echo signal in the input far-end signal, and the corresponding block filter optimal delay estimate is used as the echo delay offset; if the maximum accuracy is less than the preset block filter accuracy threshold, it is determined that there is no echo signal in the input far-end signal, that is, there is no optimal delay estimate for the corresponding block filter.

[0119] Specifically, the offset corresponding to the maximum accuracy is the internal offset of the current deblocking filter plus the offset of the pre-deblocking filter (number of deblocking filters * 24 * 16 = 384 * n), and the calculation formula is as follows:

[0120] Shift = 384 * n + Max[n] (10).

[0121] The preset block filter accuracy threshold is equal to Acc_th, also known as the accuracy threshold. If max(Acc[n]) >= Acc_th, the offset Shift of this match is sent as the echo delay offset to the histogram statistics; if max(Acc[n]) < Acc_th, it means there is no echo signal in the currently input remote signal, and no further processing is required subsequently.

[0122] The echo detection and alignment method provided in this embodiment determines whether there is an echo signal in the input remote signal based on the magnitude relationship between the maximum accuracy of multiple block filters and the preset block filter accuracy threshold, improving the accuracy of judging the echo signal and realizing the detection and delay estimation of the echo signal.

[0123] Step S303: Use a statistical histogram to statistically obtain the echo delay of the echo delay offset, and time-domain align the playback signal and the recording signal based on the echo delay.

[0124] Specifically, the above step S303 includes:

[0125] Step c: Continuously use a statistical histogram to statistically obtain the echo delay offset within a preset time period. If the echo delay offset continuously appears more than a preset number of times, the echo delay offset is used as the best delay offset, and the echo delay is calculated based on the best delay offset and the set frequency. In this embodiment, the preset time period is 2 s and the preset number of times is 20 times. The statistical histogram continuously statistically obtains the echo delay offset matched in the recent 2 s. If a certain echo delay offset continuously appears more than 20 times and appears the most times among all the matching values, let the echo delay offset be the best delay offset Best_Shift. Combining the matching sampling rate of 4 kHz, the current output echo delay calculation formula is:

[0126] Delay = Best_Shift / 4000 (11);

[0127] where Delay is the echo delay, and the unit is ms.

[0128] Align the playback signal and the recording signal according to the obtained echo delay, that is, adjust the playback signal according to the echo delay value until it is aligned with the recording signal. The comparison diagram of the effects before and after the alignment of the playback signal and the recording signal is as Fig. 9 shown Fig. 9The playback data corresponds to the playback signal, and the recording data corresponds to the recording signal. The alignment result of this embodiment is applied to the echo cancellation model. The pre-processing effect and post-processing effect of the alignment of the playback signal and the recording signal are shown in Figure 10 (a) and Figure 10 (b), respectively. After the playback signal and the recording signal are aligned, the echo cancellation effect of the echo cancellation model is obvious. This embodiment can be used in some scenarios with real-time voice calls, such as online meetings, video chats, cloud desktop voice processing, etc., to improve the effect of echo cancellation under high latency and improve the quality of voice calls. Echo detection and delay estimation are being integrated into corresponding cloud desktop products. The above embodiments have proved that the processing effect of echo cancellation is well improved and the user experience is improved. For the diversified terminals of the cloud desktop, similar effects of voice communication quality and echo processing are achieved, avoiding the phenomenon of extended processing time and obvious echo residue in some low-end local processing.

[0129] The echo detection and alignment method provided in this embodiment uses multi-block filter matching processing, which can ensure that the matching of longer delays is achieved while limiting the number of filter parameters and reducing the computing power requirements, meet the requirements of high delay, and achieve availability in high delay scenarios; the normalized step size is used to enable the matching process to change the step size according to the energy of the input signal, converge faster when the input signal energy is high, and ensure that the parameters of the block filter are not easy to diverge when the input signal energy is low, so as to detect whether the echo signal exists with lower complexity and higher matching accuracy. A statistical histogram is used to continuously count the echo delay offset within a preset time period. If the echo delay offset continues to appear for more than a preset number of times, the echo delay offset is used as the optimal delay offset, and the echo delay is calculated based on the optimal delay offset and the set frequency to ensure the stable output of the echo delay offset, and avoid the situation where the output delay frequently jumps and causes the output to be unstable. In this embodiment, an echo detection and alignment device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation methods, and the descriptions that have been made will not be repeated. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0130] This embodiment provides an echo detection and alignment device, such as Fig.11 As shown, including:

[0131] The signal acquisition processing module 1101 is used to acquire the playback signal and the recording signal in real time, and downsample the playback signal and the recording signal to obtain the far-end signal and the near-end signal of the set frequency.

[0132] The detection and matching module 1102 is used to perform time-domain adaptive detection and matching of echo delays based on the far-end signal and the near-end signal using a pre-built piecewise normalized minimum mean square filter to obtain an echo delay offset.

[0133] The statistical histogram processing and signal alignment module 1103 is used to obtain the echo delay by using the statistical histogram to count the echo delay offset, and to perform time domain alignment of the playback signal and the recording signal based on the echo delay.

[0134] In some optional implementations, the signal acquisition processing module 1101 includes:

[0135] The playback signal processing unit is used to intercept the playback signal of the preset time interval from the playback signal, divide the playback signal of the preset time interval according to the preset time unit to obtain a multi-frame playback signal, and downsample the multi-frame playback signal to obtain a remote signal of the set frequency.

[0136] The recording signal processing unit is used to downsample the recording signal and then divide it into blocks according to preset sampling points, arbitrarily select the sampling points of two adjacent blocks and use short-time Fourier transform to perform low-pass filtering, filter out the signal higher than the Nyquist frequency in the recording signal and then perform inverse short-time Fourier transform to obtain the time domain signal, and downsample the time domain signal to obtain the near-end signal of the set frequency.

[0137] In some optional implementations, the detection and matching module 1102 includes:

[0138] The calculation updating unit is used for traversing the output of each deblocking filter and updating the parameters of each deblocking filter based on the far-end signal and the near-end signal.

[0139] The maximum internal offset calculation unit is used to obtain the energy corresponding to the near-end signal and the accuracy of the current blocking filter, and calculate the maximum internal offset of the corresponding blocking filter based on the energy corresponding to the near-end signal, the accuracy of the current blocking filter and the parameters of the current blocking filter.

[0140] The echo delay offset calculation unit is used to determine whether there is an optimal delay estimate based on a preset deblock filter accuracy threshold and a corresponding deblock filter maximum internal offset, and use the optimal delay estimate as the echo delay offset.

[0141] In some optional implementations, the calculation update unit includes:

[0142] The block filter output calculation subunit is used to select remote signal sampling points equal to the number of parameters from the remote signal as inputs of any block filter, and calculate the output of each block filter based on the starting offset, each remote signal sampling point and each parameter.

[0143] The block filter error calculation subunit is used to select a near-end signal sampling point equal to the internal offset value from the near-end signal, and calculate the error of each block filter based on the near-end signal sampling point and the output of each block filter.

[0144] The parameter updating subunit is used to obtain the energy value corresponding to the sampling point of the remote signal, and update the parameters of each block filter based on the energy value, the error of each block filter and the preset smoothing coefficient.

[0145] In some optional implementations, the echo delay offset calculation unit includes:

[0146] The maximum accuracy obtaining subunit is used to obtain the maximum accuracy among multiple blocking filters.

[0147] The judgment subunit is used to judge whether there is an echo signal in the input far-end signal if the maximum accuracy is greater than or equal to the preset block filter accuracy threshold, and use the corresponding block filter optimal delay estimate as the echo delay offset; if the maximum accuracy is less than the preset block filter accuracy threshold, it is judged that there is no echo signal in the input far-end signal, that is, there is no optimal delay estimate for the corresponding block filter.

[0148] In some optional implementations, the statistical histogram processing and signal alignment module 1103 includes:

[0149] The statistical histogram processing unit is used to continuously count the echo delay offset within a preset time period using a statistical histogram. If the echo delay offset continues to appear for more than a preset number of times, the echo delay offset is used as the optimal delay offset, and the echo delay is calculated based on the optimal delay offset and the set frequency.

[0150] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0151] The echo detection and alignment device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0152] The embodiment of the present invention also provides a computer device having the above Fig.11 The echo detection and alignment device shown.

[0153] See also Fig.12 , Fig.12 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.12 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.12 A processor 10 is taken as an example.

[0154] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0155] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0156] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0157] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0158] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0159] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0160] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0161] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0162] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An echo detection and alignment method, characterized in that: The method comprises: Acquire a playback signal and a recording signal in real time, and downsample the playback signal and the recording signal to obtain a far-end signal and a near-end signal of a set frequency; Based on the far-end signal and the near-end signal, a pre-built piecewise normalized minimum mean square filter is used to perform time domain adaptive detection and matching of echo delay to obtain an echo delay offset; The echo delay offset is statistically analyzed using a statistical histogram to obtain the echo delay, and the playback signal and the recording signal are aligned in time domain based on the echo delay.

2. The method according to claim 1, characterized in that Downsampling the playback signal and the recording signal to obtain a far-end signal and a near-end signal of a set frequency includes: Intercepting a playback signal at a preset time interval from the playback signal, dividing the playback signal at the preset time interval according to a preset time unit to obtain a multi-frame playback signal, and downsampling the multi-frame playback signal to obtain a remote signal of a set frequency; After downsampling the recorded signal, it is divided into blocks according to preset sampling points. The sampling points of two adjacent blocks are arbitrarily selected and low-pass filtered using short-time Fourier transform. After filtering out signals above the Nyquist frequency in the recorded signal, an inverse short-time Fourier transform is performed to obtain a time domain signal. The time domain signal is downsampled to obtain a near-end signal of the set frequency.

3. The method according to claim 1, characterized in that The piecewise normalized minimum mean square filter is constructed in the following manner: Set the number of block filters and the number of parameters, starting offset, internal offset and normalization step size for each block filter; Each block filter is constructed based on the number of block filters, the set number of parameters of each block filter, the starting offset, the internal offset and the normalization step size to obtain a piecewise normalized minimum mean square filter.

4. The method according to claim 3, characterized in that Based on the far-end signal and the near-end signal, a pre-built piecewise normalized minimum mean square filter is used to perform time domain adaptive detection and matching of echo delay, and the echo delay offset is obtained, including: Traversing the output of each block filter based on the far-end signal and the near-end signal and updating the parameters of each block filter; Acquire the energy corresponding to the near-end signal and the accuracy of the current blocking filter, and calculate the maximum internal offset of the corresponding blocking filter based on the energy corresponding to the near-end signal, the accuracy of the current blocking filter and the parameters of the current blocking filter; Based on the preset deblocking filter accuracy threshold and the corresponding deblocking filter maximum internal offset, it is determined whether there is an optimal delay estimate, and the optimal delay estimate is used as the echo delay offset.

5. The method according to claim 4, characterized in that Traversing the output of each deblocking filter based on the far-end signal and the near-end signal and updating the parameters of each deblocking filter comprises: Selecting from the remote signal a number of remote signal sampling points equal to the number of the parameters as inputs of any block filter, and calculating an output of each block filter based on a starting offset, each remote signal sampling point and each parameter; Selecting a near-end signal sampling point equal to the internal offset value from the near-end signal, and calculating the error of each deblocking filter based on the near-end signal sampling point and the output of each deblocking filter; The energy value corresponding to the far-end signal sampling point is obtained, and the parameters of each block filter are updated based on the energy value, the error of each block filter and the preset smoothing coefficient.

6. The method according to claim 4, characterized in that The method of judging whether there is an optimal delay estimate value based on a preset deblocking filter accuracy threshold and a corresponding deblocking filter maximum internal offset, and using the optimal delay estimate value as the echo delay offset comprises: Obtaining the maximum accuracy among multiple blocking filters; If the maximum accuracy is greater than or equal to the preset block filter accuracy threshold, it is judged that there is an echo signal in the input far-end signal, and the corresponding block filter optimal delay estimate is used as the echo delay offset; if the maximum accuracy is less than the preset block filter accuracy threshold, it is judged that there is no echo signal in the input far-end signal, that is, there is no optimal delay estimate for the corresponding block filter.

7. The method according to claim 1, characterized in that The step of using a statistical histogram to perform statistics on the echo delay offset to obtain the echo delay comprises: A statistical histogram is used to continuously count the echo delay offset within a preset time period. If the echo delay offset continues to appear for more than a preset number of times, the echo delay offset is used as the optimal delay offset, and the echo delay is calculated based on the optimal delay offset and the set frequency.

8. An echo detection and alignment device, characterized in that: The device comprises: A signal acquisition processing module, used for acquiring a playback signal and a recording signal in real time, and downsampling the playback signal and the recording signal to obtain a far-end signal and a near-end signal of a set frequency respectively; A detection and matching module, configured to perform time-domain adaptive detection and matching of echo delays based on the far-end signal and the near-end signal using a pre-built piecewise normalized minimum mean square filter to obtain an echo delay offset; The statistical histogram processing and signal alignment module is used to use a statistical histogram to perform statistics on the echo delay offset to obtain the echo delay, and to perform time domain alignment on the playback signal and the recording signal based on the echo delay.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the echo detection and alignment method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the echo detection and alignment method according to any one of claims 1 to 7.

Images