Echo cancellation method and electronic equipment
By converting voice data to the frequency domain and using the echo cancellation model for processing, the problem of high complexity of the echo cancellation algorithm in the prior art is solved, and efficient and accurate echo cancellation effect is achieved, reducing algorithm complexity and spectrum leakage and edge effects are reduced.
Patent Information
- Application Number
- CN202411857054.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-16
- Publication Date
- 2025-04-15
AI Technical Summary
In the prior art, the echo cancellation algorithm has high complexity and the adaptive filter does not work, resulting in low echo cancellation efficiency. Especially in the case of double speaking, residual echo seriously affects the quality of remote meetings.
By converting speech data to the frequency domain, using a pre-trained echo cancellation model for processing, generating mask values and performing frequency and time domain conversion, echo cancellation is achieved, reducing dependence on remote signals and adaptive filters, and using GRU and gating mechanism to replace the multi-head attention mechanism to reduce complexity.
The efficiency and accuracy of echo cancellation are achieved, the complexity of the algorithm is reduced, the spectrum leakage and edge effects are reduced, and the quality of echo cancellation is improved.
Smart Images
Figure CN119832922A_ABST
Abstract
Description
Background Art
[0002] In teleconference and other call systems (for example, video conferencing), the sound played by the speaker is transmitted and collected by the microphone again, which is called echo. Echo seriously affects the quality of the meeting. With the development of speakerphone, more and more users will use speakerphone as a sound receiving device. Such devices usually have the function of echo cancellation. However, with the complexity of application scenarios, residual echo will exist in different scenarios. Especially when two people are talking, the residual echo will be more obvious. This kind of residual echo seriously affects the quality of teleconference and needs to be further suppressed.
[0003] In the prior art, most methods that combine traditional algorithms and deep learning are used to perform echo cancellation, that is, the far-end signal is first used as the input signal of the linear filter to estimate the echo signal, and then the signal obtained by the microphone is subtracted to estimate the echo signal and obtain the residual signal. The residual echo is then suppressed on the residual signal. However, this method can suppress the echo signal, but the adaptive filter in this method does not work at all, resulting in a high algorithm complexity of this method. Summary of the invention
[0004] The present application provides an echo cancellation method and electronic device, which are used to implement echo cancellation, reduce algorithm complexity, and improve the efficiency of echo cancellation.
[0005] In a first aspect, an embodiment of the present application provides an echo cancellation method, the method comprising:
[0006] Acquire voice data to be processed, wherein the voice data to be processed is obtained by collecting voice data in the conference system based on an audio collection device;
[0007] Converting the speech data to be processed into the frequency domain to obtain the speech frequency domain data to be processed;
[0008] Inputting the speech features in the speech frequency domain data to be processed into a pre-trained echo cancellation model to generate initial speech data after echo cancellation, and calculating the mask value corresponding to the initial speech data after echo cancellation;
[0009] Multiplying the mask value by the speech frequency domain data to be processed to obtain intermediate frequency domain speech data;
[0010] Converting the intermediate frequency domain speech data into the time domain to generate intermediate time domain speech data;
[0011] The intermediate time-domain speech data is synthesized to output target speech data after echo cancellation.
[0012] In the embodiment of the present application, the speech data to be processed is converted into the frequency domain to obtain the speech data to be processed; then the speech features of the speech data to be processed are input into the pre-trained echo cancellation model to obtain the initial speech data after echo cancellation; and according to the initial speech data after echo cancellation, the mask value corresponding to the initial speech data is obtained; then according to the mask value and the speech data to be processed, the intermediate frequency domain speech data is obtained; and the intermediate speech data is converted into the time domain to obtain the intermediate time domain speech data; the intermediate time domain speech data is synthesized with the previous frame of speech data to be processed to obtain the target speech data after echo cancellation. Therefore, in the embodiment of the present application, only the speech data collected by the audio acquisition device is needed for echo cancellation, and no far-end signal is needed, and no adaptive filter is needed, so that echo cancellation is realized, and the algorithm complexity is reduced and the efficiency is improved.
[0013] In one embodiment, before obtaining the voice data to be processed, the method further includes:
[0014] Obtain voice data collected by the audio collection device;
[0015] Segmenting the voice data according to a specified length to obtain intermediate voice data;
[0016] Adding a Ninghan window to the first intermediate speech data to generate windowed first intermediate speech data, wherein the first intermediate speech data is any one of the intermediate speech data;
[0017] The windowed first intermediate speech data is determined as the speech data to be processed.
[0018] In the embodiment of the present application, by adding a Ninghan window to the first intermediate speech data, the spectrum leakage of the first intermediate speech data and the edge effect caused by the framing are reduced, thereby ensuring the accuracy of the echo cancellation.
[0019] In one embodiment, before inputting the speech features in the speech-frequency domain data to be processed into a pre-trained echo cancellation model, the method further includes:
[0020] Feature extraction is performed on the speech frequency domain data to be processed to obtain speech features in the speech frequency domain data to be processed.
[0021] In one embodiment, the extracting features of the speech frequency domain data to be processed to obtain speech features in the speech frequency domain data to be processed includes:
[0022] Obtaining a compressed amplitude spectrum according to the speech frequency domain data to be processed;
[0023] The compressed amplitude spectrum is subjected to feature extraction using a plurality of feature extraction layers to obtain speech features.
[0024] In the embodiment of the present application, a compressed amplitude spectrum is first obtained according to the speech frequency domain data to be processed, and then a plurality of feature extraction layers are used to extract features from the compressed amplitude spectrum to obtain speech features. Thus, the problem of excessive difference between amplitude spectra of large and small speech is reduced in the embodiment of the present application, and local features can be extracted, thereby ensuring the accuracy of echo cancellation.
[0025] In one embodiment, the speech frequency domain data to be processed includes speech real part data and speech imaginary part data;
[0026] The step of obtaining a compressed amplitude spectrum according to the speech frequency domain data to be processed comprises:
[0027] The compressed amplitude spectrum is obtained by the following formula:
[0028] compspec = log(sqrt(a 2 +b 2 +τ));
[0029] Among them, compspec is the compressed amplitude spectrum, a is the real part data of the speech, b is the imaginary part data of the speech, and τ is a preset constant value.
[0030] In the embodiment of the present application, a compressed amplitude spectrum is obtained by taking the logarithm based on the amplitude spectrum of the speech frequency domain data, thereby reducing the range of amplitude sizes and improving the generalization ability of the model.
[0031] In one embodiment, the echo cancellation model is obtained by replacing the multi-head attention mechanism in the conformer model with a gated recurrent unit GRU and a gating mechanism.
[0032] The embodiment of the present application replaces the original multi-head attention mechanism of the conformer by using GRU and gating, tracks long-term characteristics through GRU, adds different weights to different parts of the input, and gives high weights to important parts, achieving the same effect as the attention mechanism. In addition, the model in the present application greatly reduces complexity compared to the conformer.
[0033] In one embodiment, the echo cancellation model includes a first feedforward module, a GRU gating mechanism module, a convolution module, a second feedforward module and a normalization module, and the first feedforward module, the GRU gating mechanism module, the convolution module, the second feedforward module and the normalization module are connected in sequence;
[0034] The step of inputting the speech features into a pre-trained echo cancellation model to generate initial speech data after echo cancellation includes:
[0035] Using the first feedforward module to extract the speech feature to obtain a first speech feature;
[0036] Using the GRU gating mechanism module to extract the first speech feature to obtain a second speech feature;
[0037] Using the convolution module to perform a convolution operation on the second speech feature to obtain a third speech feature;
[0038] Using the second feedforward module to perform feature extraction on the third speech feature to obtain a fourth speech feature;
[0039] The fourth speech feature is normalized by using the normalization module to generate the initial speech data after the echo is eliminated.
[0040] In one embodiment, the GRU gating mechanism module includes a conditional layer normalization layer, a first convolution activation layer, a GRU, a second convolution activation layer, a one-dimensional convolution layer, and a sigmoid activation function; and any one of the convolution activation layers includes a one-dimensional convolution layer and a RELU activation function;
[0041] The step of extracting the first speech feature by using the GRU gating mechanism module to obtain the second speech feature includes:
[0042] Using the conditional layer normalization layer to normalize the first speech feature, and output a first intermediate speech feature;
[0043] Using the first convolution activation layer to perform feature extraction on the first intermediate speech feature to obtain a second intermediate speech feature;
[0044] Processing the second intermediate speech feature using the GRU to generate a third intermediate speech feature;
[0045] Using the second convolution activation layer to extract the third intermediate speech feature, to obtain a fourth intermediate speech feature;
[0046] Using the one-dimensional convolution layer to perform a convolution operation on the fourth intermediate speech feature to obtain a fifth intermediate speech feature;
[0047] Performing a normalization operation on the fifth intermediate speech feature by using the sigmoid activation function, and outputting a sixth intermediate speech feature;
[0048] The sixth intermediate speech feature is multiplied by the first intermediate speech feature to obtain the second speech feature.
[0049] In one embodiment, the calculating the mask value corresponding to the initial voice data after echo cancellation includes:
[0050] Using a fully connected layer, a fully connected operation is performed on the initial voice data after echo elimination to obtain a fifth voice feature;
[0051] The fifth speech feature is normalized by using a sigmoid activation function to generate the mask value.
[0052] In the embodiment of the present application, a masking value is obtained by performing full connection and normalization operations on the initial speech data after echo cancellation, and feature combination and conversion are achieved through the full connection operation, so that more detailed features can be captured. In addition, the normalization operation is used to map the data of different features in the speech data on the same scale, thereby reducing the impact of data range differences on the algorithm and improving the quality of the speech after echo cancellation.
[0053] A second aspect of the present application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;
[0054] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:
[0055] Acquire voice data to be processed, wherein the voice data to be processed is obtained by collecting voice data in the conference system based on an audio collection device;
[0056] Converting the speech data to be processed into the frequency domain to obtain the speech frequency domain data to be processed;
[0057] Inputting the speech features in the speech frequency domain data to be processed into a pre-trained echo cancellation model to generate initial speech data after echo cancellation, and calculating the mask value corresponding to the initial speech data after echo cancellation;
[0058] Multiplying the mask value by the speech frequency domain data to be processed to obtain intermediate frequency domain speech data;
[0059] Converting the intermediate frequency domain speech data into the time domain to generate intermediate time domain speech data;
[0060] The intermediate time-domain speech data is synthesized to output target speech data after echo cancellation.
[0061] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0063] Figure 1 One of the flow charts of the echo cancellation method provided in the embodiment of the present application is exemplarily shown;
[0064] Figure 2 The following is a flow chart showing a method for determining voice data to be processed provided in an embodiment of the present application;
[0065] Figure 3 The flowchart of the present invention for extracting features from speech-frequency domain data to be processed is exemplarily shown;
[0066] Figure 4 The structural diagram of the feature extraction layer provided in the embodiment of the present application is exemplarily shown;
[0067] Figure 5 The structure diagram of the echo cancellation model provided by the embodiment of the present application is exemplarily shown;
[0068] Figure 6 The schematic diagram of the structure of the GRU gating mechanism module provided in the embodiment of the present application is exemplarily shown;
[0069] Figure 7 The second flowchart of the echo cancellation method provided in the embodiment of the present application is exemplarily shown;
[0070] Figure 8 The third flowchart of the echo cancellation method provided in the embodiment of the present application is exemplarily shown;
[0071] Fig. 9 A schematic diagram of an echo cancellation device provided in an embodiment of the present application is exemplarily shown;
[0072] Fig.10 The structural diagram of the electronic device provided by the embodiment of the present application is exemplified. DETAILED DESCRIPTION
[0073] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.
[0074] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.
[0075] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.
[0076] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device containing a series of components is not necessarily limited to those components clearly listed, but may include other components that are not clearly listed or inherent to these products or devices.
[0077] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.
[0078] The idea of the embodiments of the present application is summarized below.
[0079] In the prior art, most methods that combine traditional algorithms and deep learning are used to perform echo cancellation, that is, the far-end signal is first used as the input signal of the linear filter to estimate the echo signal, and then the signal obtained by the microphone is subtracted to estimate the echo signal and obtain the residual signal. The residual echo is then suppressed on the residual signal. However, this method can suppress the echo signal, but the adaptive filter in this method does not work at all, resulting in a high algorithm complexity of this method.
[0080] Therefore, the embodiment of the present application provides an echo cancellation method, which converts the speech data to be processed into the frequency domain to obtain the speech frequency domain data to be processed; then inputs the speech features into a pre-trained echo cancellation model to obtain the initial speech data after echo cancellation; and according to the initial speech data after echo cancellation, obtains the mask value corresponding to the initial speech data; then according to the mask value and the speech frequency domain data to be processed, obtains the intermediate frequency domain speech data; and converts the intermediate speech data into the time domain to obtain the intermediate time domain speech data; synthesizes the intermediate time domain speech data with the previous frame of speech data to be processed to obtain the target speech data after echo cancellation. Therefore, in the embodiment of the present application, only the speech data collected by the audio acquisition device is needed for echo cancellation, and no far-end signal is needed, and no adaptive filter is needed, so that echo cancellation is realized, and the algorithm complexity is reduced and the efficiency is improved.
[0081] The echo cancellation method in the present application is described below with reference to the accompanying drawings. Figure 1 FIG. 1 is a flow chart of an echo cancellation method, which may specifically include the following steps:
[0082] Step 101: Acquire voice data to be processed, wherein the voice data is obtained by collecting voice data in a conference system based on an audio collection device;
[0083] Next, the method for determining the voice data to be processed in the embodiment of the present application is introduced. Figure 2 As shown, it is a schematic diagram of a process for determining the voice data to be processed, which may specifically include the following steps:
[0084] Step 201: Acquire voice data collected by an audio collection device;
[0085] Step 202: Segment the voice data according to a specified length to obtain intermediate voice data;
[0086] It should be noted that the specified length in the embodiment of the present application can be set according to the specific actual situation, and the embodiment of the present application does not limit the specific value of the specified length.
[0087] Step 203: Add a Ninghan window to the first intermediate speech data to generate windowed first intermediate speech data, wherein the first intermediate speech data is any one of the intermediate speech data; wherein the windowed first intermediate speech data corresponding to the first intermediate speech data can be obtained by formula (1):
[0088] y(n)=x(n)*w(n)……(1);
[0089] Among them, y(n) is the intermediate speech data of the nth frame after windowing (the first intermediate speech data after windowing), x(n) is the intermediate speech data of the nth frame (the first intermediate speech data), and w(n) is the Ninghan window corresponding to the intermediate speech data of the nth frame.
[0090] In one embodiment, the Ning-Han window corresponding to the first intermediate speech data is obtained in the following manner:
[0091] According to the number of frames of the first intermediate speech data, a Ninghan window corresponding to the first intermediate speech data is obtained, wherein the Ninghan window corresponding to the first intermediate speech data can be obtained by formula (2):
[0092]
[0093] Among them, w(n) is the Ninghan window corresponding to the intermediate speech data of the nth frame (the first intermediate speech data), n is the number of frames corresponding to the first intermediate speech data, and N is the length of the pre-set window function.
[0094] Step 204: Determine the windowed first intermediate speech data as the speech data to be processed.
[0095] Therefore, in the embodiment of the present application, by adding a Ninghan window to the first intermediate speech data, the spectrum leakage of the first intermediate speech data and the edge effect caused by the framing are reduced, thereby ensuring the accuracy of the echo cancellation step.
[0096] Step 102: converting the speech data to be processed into the frequency domain to obtain the speech frequency domain data to be processed;
[0097] In one embodiment, step 102 may be specifically implemented as: performing Fourier transform on the speech data to be processed to obtain the speech frequency domain data to be processed.
[0098] The speech frequency domain data to be processed in the embodiment of the present application includes speech real part data and speech imaginary part data.
[0099] Step 103: inputting the speech features in the speech frequency domain data to be processed into a pre-trained echo cancellation model to generate initial speech data after echo cancellation;
[0100] Below, firstly, the method for determining the voice features in the speech frequency domain data to be processed in the embodiment of the present application is introduced. In one embodiment, the voice features in the speech frequency domain data to be processed are obtained by the following method:
[0101] Feature extraction is performed on the speech frequency domain data to be processed to obtain speech features in the speech frequency domain data to be processed.
[0102] Next, the feature extraction method in the embodiment of the present application is introduced. Figure 3 As shown, it is a schematic diagram of a process of extracting features from speech-frequency domain data to be processed, which may specifically include the following steps:
[0103] Step 301: Obtain a compressed amplitude spectrum according to the speech frequency domain data to be processed; wherein the compressed amplitude spectrum can be obtained by formula (3):
[0104] compspec = log(sqrt(a 2 +b 2 +τ))……(3);
[0105] Among them, compspec is the compressed amplitude spectrum, a is the real part data of the speech, b is the imaginary part data of the speech, and τ is a preset constant value.
[0106] Step 302: Use multiple feature extraction layers to extract features from the compressed amplitude spectrum to obtain speech features.
[0107] Therefore, in the embodiment of the present application, a compressed amplitude spectrum is first obtained according to the speech frequency domain data to be processed, and then the compressed amplitude spectrum is feature extracted using multiple feature extraction layers to obtain speech features. Therefore, the problem of excessive difference between the amplitude spectrum of large and small speech is reduced in the embodiment of the present application, and local features can be extracted, thereby ensuring the accuracy of echo cancellation.
[0108] Next, the feature extraction layer in the embodiment of the present application is introduced. Figure 4 As shown, any feature extraction layer in the embodiment of the present application includes a two-dimensional convolution layer, a normalization layer and a relu6 activation function layer. Figure 4 It can be seen that each feature extraction layer is connected in series. Next, the feature extraction method in step 302 is introduced in conjunction with the accompanying drawings. Among them:
[0109] For any feature extraction layer, a two-dimensional convolution layer is used to perform a convolution operation on the target data to obtain a first feature; the first feature is normalized using a standardization layer to obtain a second feature; the second feature is input into a relu6 activation function layer to obtain a target feature.
[0110] In the embodiment of the present application, the target feature output by the last feature extraction layer is determined as the speech feature.
[0111] Among them, if any one of the feature extraction layers is the first feature extraction layer, the target data is the compressed amplitude spectrum; if any one of the feature extraction layers is not the first feature extraction layer, the target data is the target feature output by the previous feature extraction layer.
[0112] It should be noted that the number of feature extraction layers in the embodiments of the present application is only used for illustration and does not limit the number of feature extraction layers in the embodiments of the present application. The number of feature extraction layers in the embodiments of the present application can be set according to specific actual conditions.
[0113] Finally, the echo cancellation model in the embodiment of the present application is described. The echo cancellation model in the embodiment of the present application is obtained by replacing the multi-head attention mechanism in the conformer model with the gated recurrent unit GRU and the gate mechanism.
[0114] Therefore, in the embodiment of the present application, the multi-head attention mechanism of the original conformer is replaced by GRU and gating, and the long-term characteristics are tracked by GRU. By adding different weights to different parts of the input and taking high weights for important parts, the same effect as the attention mechanism is achieved. In addition, the model in the present application greatly reduces the complexity compared to the conformer.
[0115] like Figure 5 As shown in the figure, it is a schematic diagram of the structure of the echo cancellation model. Figure 5 It can be seen that the echo cancellation model 500 includes a first feedforward module 501, a GRU gating mechanism module 502, a convolution module 503, a second feedforward module 504 and a normalization module 505, and the first feedforward module 501, the GRU gating mechanism module 502, the convolution module 503, the second feedforward module 504 and the normalization module 505 are connected in sequence. The content of step 104 in the embodiment of the present application is introduced below in conjunction with the accompanying drawings, wherein:
[0116] The first feedforward module 501 is used to perform feature extraction on the speech feature to obtain a first speech feature. The GRU gating mechanism module 502 is used to perform feature extraction on the first speech feature to obtain a second speech feature. The convolution module 503 is used to perform a convolution operation on the second speech feature to obtain a third speech feature. The second feedforward module 504 is used to perform feature extraction on the third speech feature to obtain a fourth speech feature. The normalization module 505 is used to perform normalization processing on the fourth speech feature to obtain the initial speech data after the echo cancellation.
[0117] The structure of the GRU gating mechanism module in the embodiment of the present application is introduced below. Figure 6 As shown in the figure, it is a schematic diagram of the structure of the GRU gating mechanism module. Figure 6It can be seen that the GRU gating mechanism module 502 includes a conditional layer normalization layer 5021, a first convolution activation layer 5022, a GRU 5023, a second convolution activation layer 5024, a one-dimensional convolution layer 5025 and a sigmoid activation function 5026; and any convolution activation layer includes a one-dimensional convolution layer and a RELU activation function. That is, the first convolution activation layer 5022 includes a one-dimensional convolution layer 5022a and a RELU activation function 5022b. The second convolution activation layer 5024 includes a one-dimensional convolution layer 5024a and a RELU activation function 5024b. Below, in conjunction with the accompanying drawings, the method of extracting the first speech feature using the GRU gating mechanism module in the embodiment of the present application is introduced:
[0118] The first speech feature is normalized by using the conditional layer normalization layer 601 to obtain a first intermediate speech feature; the first convolution activation layer 602 is used to extract the first intermediate speech feature to obtain a second intermediate speech feature; the second intermediate speech feature is processed by using the GRU 603 to obtain a third intermediate speech feature; the third intermediate speech feature is extracted by using the second convolution activation layer 604 to obtain a fourth intermediate speech feature; the fourth intermediate speech feature is convolved by using the one-dimensional convolution layer 605 to obtain a fifth intermediate speech feature; the fifth intermediate speech feature is normalized by using the sigmoid activation function 606 to obtain a sixth intermediate speech feature; the sixth intermediate speech feature is multiplied by the first intermediate speech feature to obtain the second speech feature.
[0119] The following is an introduction to the training method of the echo cancellation model in the embodiment of the present application:
[0120] (1) Data recording and cleaning: Connect Speakerphone to a personal computer, select Speakerphone as the corresponding device for the computer's input and output devices, use the computer's built-in WAV player to play a voice with a constantly changing amplitude, and then use the recording tool on the computer to record the processed sound. Normally, if there is no residual echo, the recorded sound should be a data close to silence. Otherwise, it is a residual echo. By cleaning the recorded sound according to its amplitude, retaining the data with an amplitude greater than a certain threshold, and directly clearing the others, the recording of the residual echo is completed. In order to cover as many scenes as possible, it is necessary to record in different conference rooms using a single device and multiple devices in cascade. The minimum amount of data required needs to reach 100+ hours. The area and material of the conference room must be covered to a certain extent, such as glass wall materials and brick materials.
[0121] (2) Data synthesis: The residual echo and clean speech are first normalized, and then randomly multiplied by a coefficient between 0.06125 and 2.0 to simulate residual echo and speech sizes of various sizes. The two speech are added to obtain their mixed signal, and then the mixed signal is normalized again to limit its amplitude to between -1 and 1 to obtain a mixed speech signal.
[0122] (3) dividing the mixed speech signal into speech data according to a specified length to obtain intermediate mixed speech data; adding a Ninghan window to the first intermediate mixed speech data to obtain first intermediate mixed speech data after windowing, wherein the first intermediate mixed speech data is any one of the intermediate mixed speech data; converting the first intermediate mixed speech data after windowing into the frequency domain to obtain first intermediate mixed frequency domain speech data; extracting features from the first intermediate mixed frequency domain speech data to obtain speech features; inputting the speech features into a pre-trained echo cancellation model to obtain initial speech data after echo cancellation; and obtaining a speech data corresponding to the initial speech data according to the initial speech data after echo cancellation. Corresponding mask value; obtaining intermediate frequency domain voice data according to the mask value and the speech frequency domain data to be processed; converting the intermediate frequency domain voice data into the time domain to obtain intermediate time domain voice data; synthesizing the intermediate time domain voice data to obtain target voice data after echo elimination; comparing the target voice data after echo elimination with the pure voice in the mixed voice data to obtain an error value, adjusting the model parameters of the echo elimination model based on the error value, and judging whether the number of training times of the echo elimination model is greater than a specified threshold, if not, returning to the step of inputting the voice features into the pre-trained echo elimination model, until the number of training times of the echo elimination model is greater than the specified threshold, then ending.
[0123] In the embodiment of the present application, the initial learning rate of the model is lr=0.001, the optimizer is Adam, the learning rate update scheduler=StepLR(optimizer, step_size=75, gamma=0.5), and the specified threshold is 300. However, the numerical value is not limited in the embodiment of the present application, and can be set according to the specific actual situation in the embodiment of the present application.
[0124] In the embodiment of the present application, the echo cancellation model can also be divided into a real voice echo cancellation model and an imaginary voice echo cancellation model. In one embodiment, step 103 can also be implemented as follows: inputting the speech feature into the real voice echo cancellation model to obtain the initial real voice data after elimination, and inputting the speech feature into the imaginary voice echo cancellation model to obtain the initial imaginary voice data after elimination. Then, according to the initial real voice data after elimination, a first mask value corresponding to the initial real voice data after elimination is obtained, and according to the initial imaginary voice data after elimination, a second mask value corresponding to the initial imaginary voice data after elimination is obtained; the first mask value and the second mask value are merged to obtain a target mask value. And after the target mask value is determined as the mask value, step 105 is executed.
[0125] In the embodiment of the present application, the first mask value corresponding to the initial real voice data after elimination and the second mask value corresponding to the initial imaginary voice data after elimination are determined in the same manner as the mask value determined in the subsequent step 104. For details, please refer to the process in step 104, which will not be repeated here in the embodiment of the present application.
[0126] Step 104: Calculate the mask value corresponding to the initial speech data after echo cancellation;
[0127] In one embodiment, step 104 can be specifically implemented as follows: using a fully connected layer to perform a fully connected operation on the initial voice data after echo elimination to obtain a fifth voice feature; using a sigmoid activation function to normalize the fifth voice feature to generate the mask value.
[0128] Therefore, by performing full connection and normalization operations on the initial speech data after echo cancellation, the masking value is obtained, and the combination and conversion of features are realized through the full connection operation, which can capture more detailed features, and the normalization operation makes the data of different features in the speech data mapped on the same scale, reducing the impact of data range differences on the algorithm and improving the quality of the speech after echo cancellation.
[0129] Step 105: multiplying the mask value by the speech frequency domain data to be processed to obtain intermediate frequency domain speech data;
[0130] Therefore, by multiplying the mask value with the speech frequency domain data to be processed to obtain the intermediate frequency domain speech data, the quality of the speech after echo cancellation is guaranteed.
[0131] Step 106: Convert the intermediate frequency domain voice data into the time domain to generate intermediate time domain voice data;
[0132] In the embodiment of the present application, the intermediate frequency domain speech data is subjected to an inverse Fourier transform to obtain intermediate time domain speech data.
[0133] Step 107: synthesize the intermediate time-domain speech data and output target speech data after echo cancellation.
[0134] In one embodiment, step 107 may be specifically implemented as follows: multiplying the intermediate time-domain speech data by a synthesis window corresponding to the intermediate time-domain speech data to obtain target speech data after echo cancellation.
[0135] The synthesis window corresponding to the intermediate time domain speech data is obtained by formula (4):
[0136]
[0137] Among them, syn(n) is the synthesis window corresponding to the middle time domain speech data of the nth frame, norm() represents the normalized window, and N is the length of the preset window function.
[0138] The normalized window can be expressed by formula (5):
[0139] norm(n)=w(n)*w(n)……(5);
[0140] Wherein, norm(n) is the normalization window added to the middle time domain speech data of the nth frame.
[0141] The present application embodiment also provides another echo cancellation method, such as Figure 7 FIG. 2 is a flow chart of the echo cancellation method, which may include the following steps:
[0142] Step 701: Acquire voice data to be processed, wherein the voice data to be processed is obtained by collecting voice data in a conference system based on an audio collection device;
[0143] The method of determining the voice data to be processed in the embodiment of the present application is the same as the method in step 101 described above, and the embodiment of the present application will not be repeated here.
[0144] Step 702: converting the speech data to be processed into the frequency domain to obtain the speech frequency domain data to be processed;
[0145] The method of converting the speech data to be processed into the frequency domain to obtain the speech frequency domain data to be processed in the embodiment of the present application is the same as the method in step 102 described above, and the embodiment of the present application will not be repeated here.
[0146] Step 703: Divide the speech frequency range data to be processed according to each preset frequency range to obtain sub-speech frequency range data corresponding to each frequency range;
[0147] In the embodiment of the present application, since each voice signal of the speech-frequency-range data to be processed has a corresponding frequency, the speech-frequency-range data to be processed can be divided according to each pre-set frequency range to obtain each sub-speech-frequency-range data in the embodiment of the present application.
[0148] Step 704: extracting features from the first sub-speech frequency domain data to obtain sub-speech features, wherein the first sub-speech frequency domain data is sub-speech frequency domain data corresponding to any frequency range;
[0149] The method of feature extraction in step 704 in the embodiment of the present application is the same as the method of feature extraction described above, and the embodiment of the present application will not be repeated here.
[0150] Step 705: inputting the sub-speech features into a pre-trained echo cancellation model to obtain the sub-speech features after echo cancellation;
[0151] In the embodiment of the present application, step 705 is the same as the method of inputting speech features into the pre-trained echo cancellation model in step 103 described above, and the embodiment of the present application will not be repeated here.
[0152] Step 706: Fusing the sub-speech features to obtain fused speech features;
[0153] In one embodiment, step 706 may be specifically implemented as: adding the sub-speech features to obtain the fused speech feature.
[0154] Step 707: input the fused speech features into a pre-trained echo cancellation model to obtain the fused speech features after echo cancellation;
[0155] In the embodiment of the present application, step 707 is the same as the method of inputting speech features into the pre-trained echo cancellation model in step 103 described above, and the embodiment of the present application will not be described in detail here.
[0156] Step 708: obtaining a mask value corresponding to the fused speech data after echo cancellation according to the fused speech features after echo cancellation;
[0157] Step 709: obtaining intermediate frequency domain speech data according to the mask value and the speech frequency domain data to be processed;
[0158] Step 710: converting the intermediate frequency domain speech data into the time domain to obtain intermediate time domain speech data;
[0159] Step 711: synthesize the intermediate time-domain speech data to obtain target speech data after echo cancellation.
[0160] Steps 708 to 711 in the embodiment of the present application are the same as steps 104 to 107 described above, and the implementation of the present application will not be repeated here.
[0161] In order to further understand the video loading method in the embodiment of the present application, Figure 8 FIG. 1 is a flow chart of an echo cancellation method, which may specifically include the following steps:
[0162] Step 801: Acquire voice data collected by an audio collection device;
[0163] Step 802: Segment the voice data according to the specified length to obtain intermediate voice data;
[0164] Step 803: adding a Ninghan window to the first intermediate speech data to obtain windowed first intermediate speech data, wherein the first intermediate speech data is any one of the intermediate speech data;
[0165] Step 804: determining the windowed first intermediate speech data as the speech data to be processed;
[0166] Step 805: converting the speech data to be processed into the frequency domain to obtain the speech frequency domain data to be processed;
[0167] Step 806: obtaining a compressed amplitude spectrum according to the speech frequency domain data to be processed;
[0168] Step 807: extracting features from the compressed amplitude spectrum using multiple feature extraction layers to obtain speech features;
[0169] Step 808: inputting the speech features into a pre-trained echo cancellation model to obtain initial speech data after echo cancellation, wherein the echo cancellation model is obtained by replacing the multi-head attention mechanism in the conformer model with a gated recurrent unit GRU and a gating mechanism;
[0170] Step 809: using a fully connected layer to perform a fully connected operation on the initial voice data after echo cancellation to obtain a fifth voice feature;
[0171] Step 810: normalizing the fifth speech feature using a sigmoid activation function to obtain the mask value;
[0172] Step 811: multiplying the mask value by the speech frequency domain data to be processed to obtain the intermediate frequency domain speech data;
[0173] Step 812: converting the intermediate frequency domain speech data into the time domain to obtain intermediate time domain speech data;
[0174] Step 813: synthesize the intermediate time-domain speech data to obtain target speech data after echo cancellation.
[0175] Based on the same inventive concept, the echo cancellation method disclosed above can also be implemented by an echo cancellation device. The effect of the echo cancellation device is similar to that of the above method, and will not be described in detail here.
[0176] Fig. 9 It is a schematic diagram of the structure of an echo cancellation device according to an embodiment of the present disclosure.
[0177] like Fig. 9 As shown, the echo cancellation device 900 of the present disclosure may include an acquisition module 910 , a frequency domain conversion module 920 , an echo cancellation module 930 , an intermediate data determination module 940 , a time domain conversion module 950 and a data synthesis module 960 .
[0178] An acquisition module 910 is used to acquire voice data to be processed, wherein the voice data is acquired based on voice data in the conference system collected by an audio collection device;
[0179] A frequency domain conversion module 920, configured to convert the speech data to be processed into the frequency domain to obtain speech data to be processed in the frequency domain;
[0180] The echo cancellation module 930 is used to input the speech features in the speech frequency domain data to be processed into a pre-trained echo cancellation model, generate initial speech data after echo cancellation, and calculate the mask value corresponding to the initial speech data after echo cancellation;
[0181] An intermediate data determination module 940 is used to multiply the mask value by the speech frequency domain data to be processed to obtain intermediate frequency domain speech data;
[0182] A time domain conversion module 950, configured to convert the intermediate frequency domain voice data into the time domain to generate intermediate time domain voice data;
[0183] The data synthesis module 960 is used to synthesize the intermediate time-domain speech data and output the target speech data after echo cancellation.
[0184] In one embodiment, the apparatus further comprises:
[0185] A windowing module 970 is used to obtain the voice data collected by the audio collection device before obtaining the voice data to be processed;
[0186] Segmenting the voice data according to a specified length to obtain intermediate voice data;
[0187] Adding a Ninghan window to the first intermediate speech data to generate windowed first intermediate speech data, wherein the first intermediate speech data is any one of the intermediate speech data;
[0188] The windowed first intermediate speech data is determined as the speech data to be processed.
[0189] In one embodiment, the apparatus further comprises:
[0190] The feature extraction module 980 is used to extract features from the speech-frequency-domain data to be processed before inputting the speech features in the speech-frequency-domain data to be processed into the pre-trained echo cancellation model to obtain the speech features in the speech-frequency-domain data to be processed.
[0191] In one embodiment, the speech frequency domain data to be processed includes speech real part data and speech imaginary part data;
[0192] The feature extraction module 980 is further used for:
[0193] The compressed amplitude spectrum is obtained by the following formula:
[0194] compspec = log(sqrt(a 2 +b 2 +τ));
[0195] Among them, compspec is the compressed amplitude spectrum, a is the real part data of the speech, b is the imaginary part data of the speech, and τ is a preset constant value.
[0196] In one embodiment, the echo cancellation model is obtained by replacing the multi-head attention mechanism in the conformer model with a gated recurrent unit GRU and a gating mechanism.
[0197] In one embodiment, the echo cancellation model includes a first feedforward module, a GRU gating mechanism module, a convolution module, a second feedforward module and a normalization module, and the first feedforward module, the GRU gating mechanism module, the convolution module, the second feedforward module and the normalization module are connected in sequence;
[0198] The echo cancellation module 930 is specifically used for:
[0199] Using the first feedforward module to extract the speech feature to obtain a first speech feature;
[0200] Using the GRU gating mechanism module to extract the first speech feature to obtain a second speech feature;
[0201] Using the convolution module to perform a convolution operation on the second speech feature to obtain a third speech feature;
[0202] Using the second feedforward module to perform feature extraction on the third speech feature to obtain a fourth speech feature;
[0203] The fourth speech feature is normalized by using the normalization module to generate the initial speech data after the echo is eliminated.
[0204] In one embodiment, the GRU gating mechanism module includes a conditional layer normalization layer, a first convolution activation layer, a GRU, a second convolution activation layer, a one-dimensional convolution layer, and a sigmoid activation function; and any one of the convolution activation layers includes a one-dimensional convolution layer and a RELU activation function;
[0205] The echo cancellation module 930 is further used for:
[0206] Using the conditional layer normalization layer to normalize the first speech feature, and output a first intermediate speech feature;
[0207] Using the first convolution activation layer to perform feature extraction on the first intermediate speech feature to obtain a second intermediate speech feature;
[0208] Processing the second intermediate speech feature using the GRU to generate a third intermediate speech feature;
[0209] Using the second convolution activation layer to extract the third intermediate speech feature, to obtain a fourth intermediate speech feature;
[0210] Using the one-dimensional convolution layer to perform a convolution operation on the fourth intermediate speech feature to obtain a fifth intermediate speech feature;
[0211] Performing a normalization operation on the fifth intermediate speech feature by using the sigmoid activation function, and outputting a sixth intermediate speech feature;
[0212] The sixth intermediate speech feature is multiplied by the first intermediate speech feature to obtain the second speech feature.
[0213] In one embodiment, the echo cancellation module 930 is further configured to:
[0214] Using a fully connected layer, a fully connected operation is performed on the initial voice data after echo elimination to obtain a fifth voice feature;
[0215] The fifth speech feature is normalized by using a sigmoid activation function to generate the mask value.
[0216] After introducing an echo cancellation method and device according to an exemplary embodiment of the present invention, next, an electronic device according to another exemplary embodiment of the present invention is introduced.
[0217] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".
[0218] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the echo cancellation method according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 1 Steps 101-107 shown in .
[0219] Refer to the following Fig.10 The electronic device 1000 according to this embodiment of the present invention will be described. Fig.10 The electronic device 1000 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0220] like Fig.10 As shown, the components of the electronic device 1000 may include but are not limited to: the above-mentioned at least one processor 1001, the above-mentioned at least one computer storage medium 1002, and a bus 1003 connecting different system components (including the computer storage medium 1002 and the processor 1001).
[0221] Bus 1003 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.
[0222] The computer storage medium 1002 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1021 and / or cache storage media 1022 , and may further include read-only computer storage media (ROM) 1023 .
[0223] The computer storage medium 1002 may also include a program / utility 1025 having a set (at least one) of program modules 1024, such program modules 1024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0224] The electronic device 1000 may also communicate with one or more external devices 1004 (e.g., keyboards, pointing devices, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or communicate with any device that enables the electronic device 1000 to communicate with one or more other AR devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 1005. In addition, the electronic device 1000 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1006. As shown, the network adapter 1006 communicates with other modules for the electronic device 1000 through a bus 1003. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0225] In some possible implementations, various aspects of an echo cancellation method provided by the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the echo cancellation method according to various exemplary embodiments of the present invention described above in this specification.
[0226] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. An echo cancellation method, characterized in that: The method comprises: Acquire voice data to be processed, wherein the voice data to be processed is obtained by collecting voice data in the conference system based on an audio collection device; Converting the speech data to be processed into the frequency domain to obtain the speech frequency domain data to be processed; Inputting the speech features in the speech frequency domain data to be processed into a pre-trained echo cancellation model to generate initial speech data after echo cancellation, and calculating the mask value corresponding to the initial speech data after echo cancellation; Multiplying the mask value by the speech frequency domain data to be processed to obtain intermediate frequency domain speech data; Converting the intermediate frequency domain speech data into the time domain to generate intermediate time domain speech data; The intermediate time-domain speech data is synthesized to output target speech data after echo cancellation.
2. The method according to claim 1, characterized in that Before obtaining the voice data to be processed, the method further includes: Obtain voice data collected by the audio collection device; Segmenting the voice data according to a specified length to obtain intermediate voice data; Adding a Ninghan window to the first intermediate speech data to generate windowed first intermediate speech data, wherein the first intermediate speech data is any one of the intermediate speech data; The windowed first intermediate speech data is determined as the speech data to be processed.
3. The method according to claim 1, characterized in that Before inputting the speech features in the speech-frequency domain data to be processed into the pre-trained echo cancellation model, the method further includes: Feature extraction is performed on the speech frequency domain data to be processed to obtain speech features in the speech frequency domain data to be processed.
4. The method according to claim 3, characterized in that The extracting features of the speech-frequency domain data to be processed to obtain speech features in the speech-frequency domain data to be processed includes: Obtaining a compressed amplitude spectrum according to the speech frequency domain data to be processed; The compressed amplitude spectrum is subjected to feature extraction using a plurality of feature extraction layers to obtain speech features.
5. The method according to claim 4, characterized in that The speech frequency domain data to be processed includes speech real part data and speech imaginary part data; The step of obtaining a compressed amplitude spectrum according to the speech frequency domain data to be processed comprises: The compressed amplitude spectrum is obtained by the following formula: compspec=log(sqrt(a 2 +b 2 +τ)); Among them, compspec is the compressed amplitude spectrum, a is the real part data of the speech, b is the imaginary part data of the speech, and τ is a preset constant value.
6. The method according to claim 1, characterized in that The echo cancellation model is obtained by replacing the multi-head attention mechanism in the conformer model with the gated recurrent unit GRU and the gating mechanism.
7. The method according to claim 1 or 6, characterized in that: The echo cancellation model includes a first feedforward module, a GRU gating mechanism module, a convolution module, a second feedforward module and a normalization module, and the first feedforward module, the GRU gating mechanism module, the convolution module, the second feedforward module and the normalization module are connected in sequence; The step of inputting the speech features into a pre-trained echo cancellation model to generate initial speech data after echo cancellation includes: Using the first feedforward module to extract the speech feature to obtain a first speech feature; Using the GRU gating mechanism module to extract the first speech feature to obtain a second speech feature; Using the convolution module to perform a convolution operation on the second speech feature to obtain a third speech feature; Using the second feedforward module to perform feature extraction on the third speech feature to obtain a fourth speech feature; The fourth speech feature is normalized by using the normalization module to generate the initial speech data after the echo is eliminated.
8. The method according to claim 7, characterized in that The GRU gating mechanism module includes a conditional layer normalization layer, a first convolution activation layer, a GRU, a second convolution activation layer, a one-dimensional convolution layer and a sigmoid activation function; and any convolution activation layer includes a one-dimensional convolution layer and a RELU activation function; The step of extracting the first speech feature by using the GRU gating mechanism module to obtain the second speech feature includes: Using the conditional layer normalization layer to normalize the first speech feature, and output a first intermediate speech feature; Using the first convolution activation layer to perform feature extraction on the first intermediate speech feature to obtain a second intermediate speech feature; Processing the second intermediate speech feature using the GRU to generate a third intermediate speech feature; Using the second convolution activation layer to extract the third intermediate speech feature, to obtain a fourth intermediate speech feature; Using the one-dimensional convolution layer to perform a convolution operation on the fourth intermediate speech feature to obtain a fifth intermediate speech feature; Performing a normalization operation on the fifth intermediate speech feature by using the sigmoid activation function, and outputting a sixth intermediate speech feature; The sixth intermediate speech feature is multiplied by the first intermediate speech feature to obtain the second speech feature.
9. The method according to claim 1, characterized in that: The calculating the mask value corresponding to the initial voice data after echo cancellation includes: Using a fully connected layer, a fully connected operation is performed on the initial voice data after echo elimination to obtain a fifth voice feature; The fifth speech feature is normalized by using a sigmoid activation function to generate the mask value.
10. An electronic device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: Acquire voice data to be processed, wherein the voice data to be processed is obtained by collecting voice data in the conference system based on an audio collection device; Converting the speech data to be processed into the frequency domain to obtain the speech frequency domain data to be processed; Inputting the speech features in the speech frequency domain data to be processed into a pre-trained echo cancellation model to generate initial speech data after echo cancellation, and calculating the mask value corresponding to the initial speech data after echo cancellation; Multiplying the mask value by the speech frequency domain data to be processed to obtain intermediate frequency domain speech data; Converting the intermediate frequency domain speech data into the time domain to generate intermediate time domain speech data; The intermediate time-domain speech data is synthesized to output target speech data after echo cancellation.