Echo cancellation method, device, equipment and medium

Abstract

The invention discloses an echo cancellation method and device, equipment and a medium, and belongs to the field of audio processing. The method comprises the following steps: acquiring a first sound signal, wherein the first sound signal is a sound signal acquired by a microphone of equipment; according to the first sound signal and a reference signal corresponding to the first sound signal, obtaining an echo linear component, the reference signal being a sound signal played by a loudspeaker of the device; removing an echo linear component in the first sound signal to obtain a filtered signal; based on the filtered signal, echo residual components are obtained through prediction of a neural network model, and the echo residual components indicate components except echo linear components in the echo; and based on the echo linear component and the echo residual component, eliminating the echo in the first sound signal to obtain a second sound signal. According to the method, the generalization performance of the neural network model can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing, and in particular to an echo cancellation method, device, equipment and medium. Background Art

[0002] Echo refers to the phenomenon of hearing one's own voice during a call. Specifically, during a call, the speaker of the near-end device will emit sound, which will be collected by the microphone of the near-end device. The sound collected by the near-end device will be played on the far-end device, allowing the user of the far-end device to hear his or her own voice.

[0003] To eliminate echoes, related technologies provide the following technical solutions: obtain the near-end sound signal collected by the microphone and the reference signal corresponding to the near-end sound signal played by the speaker, suppress the echo in the near-end sound signal through a deep neural network model, and obtain the sound after echo elimination.

[0004] However, echo cancellation through a deep neural network model depends on the effectiveness and coverage of training samples. If the actual application samples in the actual application scenario are not included in the training sample set and there is a certain gap between them and the training samples in the training sample set, the suppression effect of the deep neural network model will be reduced. In other words, the relevant technology has high requirements for the training sample set of the deep neural network model. Summary of the invention

[0005] The present application provides an echo cancellation method, device, equipment and medium, which improves the generalization performance of the neural network model. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for echo cancellation is provided, the method comprising:

[0007] Acquire a first sound signal, where the first sound signal is a sound signal collected by a microphone of a device;

[0008] obtaining an echo linear component according to the first sound signal and a reference signal corresponding to the first sound signal, wherein the reference signal is a sound signal played by a speaker of the device;

[0009] Removing the echo linear component from the first sound signal to obtain a filtered signal;

[0010] Based on the filtered signal, predicting an echo residual component through a neural network model, wherein the echo residual component indicates a component in the echo other than the echo linear component;

[0011] Based on the echo linear component and the echo residual component, the echo in the first sound signal is eliminated to obtain a second sound signal.

[0012] According to another aspect of the present application, an echo cancellation device is provided, the device comprising:

[0013] An acquisition module, configured to acquire a first sound signal, where the first sound signal is a sound signal collected by a microphone of a device;

[0014] a processing module, configured to obtain an echo linear component according to the first sound signal and a reference signal corresponding to the first sound signal, wherein the reference signal is a sound signal played by a speaker of the device;

[0015] The processing module is further used to remove the echo linear component in the first sound signal to obtain a filtered signal;

[0016] The processing module is further used to predict the echo residual component through the neural network model based on the filtered signal, and the echo residual component indicates the component in the echo except the echo linear component;

[0017] The processing module is further configured to eliminate the echo in the first sound signal based on the echo linear component and the echo residual component to obtain a second sound signal.

[0018] According to one aspect of the present application, a computer device is provided, the computer device comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above echo cancellation method.

[0019] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the above echo cancellation method.

[0020] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above method.

[0021] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0022] In the related art, the training samples of the neural network model contain two aspects of echo: linear echo and nonlinear echo. When the neural network model of the related art is generalized, the linear echo and nonlinear echo of the actual application samples need to be close to the training samples. The generalization ability of the training sample set of the related art is relatively weak.

[0023] In the present application, the linear component of the echo in the first sound signal is removed to obtain a filtered signal, and the filtered signal (linear filtered signal) is input into the neural network model to predict the echo residual component. When the neural network model of the present application is generalized, it is only necessary that the nonlinear echo of the actual application sample is close to the training sample. Therefore, the training sample set of the neural network model of the present application will have a stronger generalization ability.

[0024] Moreover, in the related art, the linear component of the echo is filtered by a linear filter, and the nonlinear echo is further filtered out based on the filtered sound signal. If the linear filter is unstable, the filtering of the nonlinear echo will have a large deviation, and there is a problem of residual echo of the nonlinear echo. In the present application, the residual echo component (nonlinear echo) is predicted by a neural network model, which can solve the problem of residual echo of the nonlinear echo, and the residual echo component predicted by the neural network model is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a schematic diagram of a first echo cancellation method of the related technology provided by an embodiment of the present application;

[0027] Figure 2 is a schematic diagram of a second echo cancellation method of the related art provided by an embodiment of the present application;

[0028] Figure 3 It is a schematic diagram of the principle provided by an embodiment of the present application;

[0029] Figure 4 It is a schematic diagram of another exemplary embodiment of the present application;

[0030] Figure 5 is a flow chart of an echo cancellation method provided by an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of the structure of a neural network model provided by an exemplary embodiment of the present application;

[0032] Figure 7 is a schematic diagram of the structure of a convolution unit and a convolution block provided by an exemplary embodiment of the present application;

[0033] Figure 8 is a structural schematic diagram of a neural network model provided by another exemplary embodiment of the present application;

[0034] Fig. 9 is a structural schematic diagram of a neural network model provided by another exemplary embodiment of the present application;

[0035] Fig.10 is a structural schematic diagram of a neural network model provided by another exemplary embodiment of the present application;

[0036] Fig.11 It is a schematic diagram of another exemplary embodiment of the present application;

[0037] Fig.12 is a flow chart of an echo cancellation method provided by an exemplary embodiment of the present application;

[0038] Fig.13 is a schematic diagram of calculating a first attenuation gain and a second attenuation gain provided by an exemplary embodiment of the present application;

[0039] Fig.14 is a schematic diagram of performing echo cancellation by a first attenuation gain or a second attenuation gain provided by an exemplary embodiment of the present application;

[0040] Fig.15 is a structural block diagram of an echo cancellation device provided by an embodiment of the present application;

[0041] Fig.16 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0043] First, the nouns involved in the embodiments of the present application are briefly introduced:

[0044] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0045] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0046] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0047] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0048] VoIP: A voice call method based on IP network. After the voice data is encoded and compressed through the voice compression algorithm, it is packaged according to the network transmission protocol standard, and the data packet is sent to the destination IP address through the IP network. The voice data packet is then parsed and decompressed to restore the original voice signal, thereby achieving the purpose of transmitting voice signals through the Internet.

[0049] Echo: refers to the phenomenon of hearing one's own voice during a call. Specifically, during a call, the speaker of the near-end device will emit sound, which is collected by the microphone of the near-end device. The sound collected by the near-end device will be played on the far-end device, allowing the user of the far-end device to hear his or her own voice. Echo greatly interferes with the call process. Most mobile terminals have a hardware-implemented acoustic echo cancellation unit (AEC). However, existing echo cancellation solutions still have shortcomings.

[0050] The related art provides the following two echo cancellation methods.

[0051] The first echo cancellation method, combined with reference Figure 1 , Figure 1 The echo cancellation framework is shown. In the process of echo cancellation, a reference signal 101 is obtained. The reference signal 101 is a signal collected by a far-end microphone. The reference signal 101 is played through a speaker S of a near-end device. After multiple feedbacks, the reference signal 101 forms an echo signal 1 which is collected by a microphone M of a near-end device. The microphone of the near-end device collects the echo signal and other sounds (such as human voice) 2. The microphone inputs the collected near-end sound signal 102 into an echo delay detection unit 103. The echo delay detection unit 103 also receives the input reference signal 101 and synchronously aligns the reference signal 101 and the near-end sound signal 102. The echo delay detection unit 103 offsets the reference signal 101 according to the delay between the reference signal 101 and the near-end sound signal 102 to obtain a playback delay signal 104.

[0052] The play delay signal 104 is input to the adaptive filtering unit 105 to obtain the echo linear component 106. The echo linear component 106 is subtracted from the near-end sound signal 102 to obtain the filtered signal 107, which is the signal after the linear echo is removed from the near-end sound signal 102. The filtered signal 107 is input to the nonlinear suppression unit 108 to suppress the nonlinear echo and output the sound signal 109 after the echo is eliminated.

[0053] The nonlinear suppression unit 108 further exponentially attenuates the filtered signal 107 according to the correlation between the reference signal 101, the near-end sound signal 102 and the filtered signal 107, as well as the attenuation curves at different frequencies. The effect of the nonlinear suppression unit 108 can be expressed by the formula: f(k) *f(k),e f(k) represents the kth frequency point of the filtered signal 107, f(k) represents the attenuation gain of the kth frequency point, and then the suppressed frequency domain signal is transformed into the time domain to obtain the sound signal 109 after echo cancellation.

[0054] Furthermore, the calculation formula of f(k) is expressed as follows:

[0055] f(k)=hnl(k) overdrive(k)*curve(k) ;

[0056] Where curve(k) is the preset suppression curve for each frequency point, and k represents the kth frequency point, that is, any frequency point. The frequency point is the number of the fixed frequency. For example, if the frequency interval is 200Hz, multiple frequencies such as 100Hz, 300Hz, 500Hz, and 700Hz are taken out according to the frequency interval of 200Hz, and each frequency is numbered, such as 1, 2, 3, 4...10, etc. These numbers of the fixed frequencies are the frequency points.

[0057] When performing echo cancellation, the cross-correlation value AvgCohXD between the reference signal 101 and the near-end sound signal 102, the cross-correlation value AvgCohDE between the near-end sound signal 102 and the filtered signal 107, and the cross-correlation value AvgCohYD between the echo linear component 106 and the near-end sound signal 102 are calculated.

[0058] If AvgCohXD(k) is less than threshold 1 and AvgCohDE(k) is greater than threshold 2, it means that the volume of reference signal 101 is low and it is processed as echo-free. At this time, hnl(k) in the formula takes the cross-correlation coefficient cohde(k) between the near-end sound signal 102 and the filtered signal 107 at each frequency point k. The suppression degree coefficient overdrive(k) takes the constant Coef1, indicating that the suppression degree is the minimum when there is no echo.

[0059] If AvgCohXD(k) is greater than threshold 1, or AvgCohDE(k) is less than threshold 2, it means that reference signal 101 exists and the signal is processed as echo. At this time, hnl(k) in the formula is 1-cohxd(k) of each frequency point k, cohxd(k) is the mutual correlation coefficient between frequency point k in reference signal 101 and near-end sound signal 102, and the suppression degree coefficient overdrive is calculated according to the following formula.

[0060]

[0061] Among them, W xd , W yd and W de They represent the weights of the mutual correlation coefficients cohxd(k), cohyd(k) and cohde(k) in the calculation of overdrive(k), and their values ​​are between 0 and 1.

[0062] The second echo cancellation method, combined with reference Figure 2, the echo is eliminated by the DTLN (Dual-Signal Transformation LSTM Network for Real-Time Noise Suppression) (DTLN specific reference paper https: / / arxiv.org / abs / 2005.07551) deep neural network. The DTLN deep neural network will output the echo-eliminated sound signal 213 according to the reference signal 201 and the near-end sound signal 202. In the first-level echo elimination, the reference signal 201 and the near-end sound signal 202 are first obtained. The reference signal 201 and the near-end sound signal 202 are segmented 203 respectively, and the segmented sound signals are transformed into the frequency domain through Fourier transform 204, and the frequency domain features are subjected to instantaneous layer normalization processing 205. The features obtained after the reference signal 201 and the near-end sound signal are subjected to feature splicing 206. The concatenated features are input into two long short-term memory units 207, a fully connected network 208 and an activation function 209, and the first-level attenuation gain features are output. The first-level attenuation gain features are multiplied by the frequency domain features of the near-end sound signal to obtain the first-level echo cancellation result.

[0063] In the second-level echo cancellation, the first-level echo cancellation result is subjected to inverse Fourier transform 210 to obtain time domain features. The obtained time domain features and the frequency domain features of the reference signal are subjected to one-dimensional convolution 211 and instantaneous layer normalization 205 respectively. The two obtained features are subjected to feature concatenation 206. The concatenated features are input into two long short-term memory units 207, a fully connected network 208 and an activation function 209, and the attenuation gain features of the second level are output. The attenuation gain features of the second level are multiplied by the time domain features of the first-level echo cancellation result (on which one-dimensional convolution 211 has been performed) to obtain the second-level echo cancellation result.

[0064] The second-stage echo cancellation result is subjected to one-dimensional convolution 211 and superposition and summation calculation 212 to obtain a final echo-cancelled sound signal 213 .

[0065] In the first echo cancellation method described above, the nonlinear suppression unit 108 performs an exponential attenuation suppression strategy based on the cross-correlation value among the reference signal 101 , the near-end sound signal 102 and the filtered signal 107 .

[0066] 1. The accuracy of the cross-correlation value depends on the state of the linear filter (adaptive filter unit 105). If the linear filter is in a stable state, the ideal linear component of the echo can be obtained. However, if the linear filter is in an unstable state, the linear component of the echo output by the linear filter will deviate from the true linear component of the echo (usually much smaller than the true linear component), which will cause the cross-correlation value to fail to represent the true result of the echo, and the suppression ability of the nonlinear suppression unit 108 will drop rapidly, and the suppression effect of the nonlinear suppression processing will be far from expected.

[0067] 2. It is difficult to achieve precise suppression with an exponential attenuation strategy. While suppressing the echo, the effective sound in the near-end sound signal will also be greatly damaged, and ultimately the sound heard by the far-end user will become intermittent.

[0068] 3. The nonlinear characteristics of the echo depend on the acoustic performance of the near-end device and the actual reflection conditions of the playback environment. There is no stable corresponding relationship between the nonlinear performance and the near-end sound signal and reference signal, and this nonlinear relationship cannot be accurately described by the formula.

[0069] Based on the above analysis, the first echo cancellation method is prone to problems such as residual echo and effective sound damage.

[0070] In the second echo cancellation method mentioned above, the echo cancellation method based on the deep neural network model is completely driven by data samples, and the goal is to train an inference model with the smallest loss function error. Its echo suppression performance is greatly affected by sample diversity, representativeness, sample distribution characteristics, etc. It has a good echo suppression effect on nonlinear echoes covered by the training samples, achieving better results than the first echo cancellation method, but it lacks the ability to suppress echoes that are not within the feature range of the training samples and have certain differences from the echo features of the training samples, resulting in missed echoes, and there will be false damage to effective echo-like sounds in the near-end sound signal.

[0071] Unlike the first echo cancellation method, the second echo cancellation method is a data-driven processing method that does not rely on precise formula descriptions. As long as the training samples can better cover the actual application scenarios, the second echo cancellation method can better suppress echoes and achieve a better level of effective sound fidelity. The performance of the second echo cancellation method mainly depends on the effectiveness and coverage of the training samples. However, if the actual application scenarios (including acoustic equipment, acoustic space environment, etc.) are different from the training sample set, that is, the actual application samples are not included in the training sample set, and there is a certain gap between the actual application samples and the training sample set feature range, the performance of the deep neural network model will be reduced, and the playback devices and playback environments in actual applications cannot be exhausted.

[0072] Based on the above analysis, the second echo cancellation method has higher requirements on the training sample set.

[0073] Based on the above analysis, the present application provides the following echo cancellation method.

[0074] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the first sound signal and reference signal involved in this application are obtained with full authorization.

[0075] Moreover, when it comes to relevant information, the relevant information processors will follow the principles of legality, legitimacy and necessity, clarify the purpose, method and scope of relevant information processing, obtain the consent of the relevant information subjects, and take necessary technical and organizational measures to ensure the security of relevant information.

[0076] Figure 3 A schematic diagram of a principle provided by an exemplary embodiment of the present application is shown. Figure 3 The echo cancellation framework 300 provided by the present application is shown. In the process of performing echo cancellation, a reference signal 301 is obtained. The reference signal 301 can be understood as the sound collected by the microphone M of the far-end device, or the reference signal 301 is the sound emitted by the speaker S of the near-end device. The reference signal 301 is played through the speaker S of the near-end device, and an echo signal 1 is formed during the propagation process and collected by the microphone of the near-end device. The microphone M of the near-end device also collects other sounds 2, which together with the echo signal 1 constitute a first sound signal 302.

[0077] Based on the first sound signal 302 and the reference signal 301, an echo linear component 303 is obtained, and the echo linear component 303 indicates a linear component in the echo of the first sound signal 302. The echo linear component 303 in the first sound signal 302 is removed to obtain a filtered signal 304. The filtered signal 304 is input into a neural network model 305, and an echo residual component 306 is predicted. The echo linear component 303, the echo residual component 306 and the first sound signal 302 are input into an echo cancellation unit 307, and the echo cancellation unit 307 cancels the echo in the first sound signal 302 and outputs a second sound signal 308.

[0078] Furthermore, Figure 4A more specific echo cancellation framework 400 is provided. In the process of performing echo cancellation, a reference signal 401 is obtained. The reference signal 401 is played through the speaker S of the near-end device, and an echo signal 1 is formed during the propagation process and collected by the microphone M of the near-end device. The microphone M of the near-end device also collects other sounds 2, which together with the echo signal 1 constitute a first sound signal 402. The first sound signal 402 and the reference signal 401 are input into an echo delay estimation unit 403, and a playback delay signal 404 is output. The echo delay estimation unit is used to align the first sound signal 402 and the reference signal 401 to obtain the echo delay, and the reference signal 401 is offset according to the echo delay to obtain the playback delay signal 404. The playback delay signal 404 is input into a linear filter 405 to obtain an echo linear component 406. The echo linear component 406 in the first sound signal 402 is removed to obtain a filtered signal 407. The filtered signal 407 is input into a neural network model 408 to predict an echo residual component 409. The echo linear component 406, the echo residual component 409 and the first sound signal 402 are input to the echo cancellation unit 410, and the echo cancellation unit 410 cancels the echo in the first sound signal 402 and outputs the second sound signal 411. That is, Figure 4 Compared to Figure 3 The echo cancellation framework further provides a process for generating a linear component of the echo.

[0079] Understandably, Figure 3 and Figure 4 The echo cancellation framework shown in the figure inputs the filtered signal into the neural network model, and the neural network model predicts the echo residual component. Compared with the first echo cancellation method mentioned above, the neural network model predicts the nonlinear echo (echo residual component). Even if the linear echo obtained in the previous step is inaccurate, the neural network model will not be greatly affected when predicting the nonlinear echo, and the subsequent step will also achieve better results in eliminating the nonlinear echo in the near-end sound signal (first sound signal).

[0080] Compared with the second echo cancellation method, the training samples of the neural network model in the second echo cancellation method (hereinafter referred to as the original neural network model) include two aspects of echo: linear echo and nonlinear echo. In the second echo cancellation method, the input of the original neural network model is the reference signal and the near-end sound signal. When the original neural network model is generalized, the linear echo and nonlinear echo of the actual application sample need to be close to the training sample. The generalization ability of the training sample set of the original neural network model is relatively weak. Figure 3 and Figure 4 In the echo cancellation framework shown, the input of the neural network model is the filtered signal. When the neural network model generalizes, it only needs the nonlinear echo of the actual application sample to be close to the training sample. Therefore, Figure 3 and Figure 4 The training sample set of the neural network model in will have stronger generalization ability.

[0081] Above Figure 3 and Figure 4 In the schematic diagram shown, the echo cancellation framework is executed by a computer device, and the computer device includes a terminal 3. The terminal 3 includes a smart phone, a smart TV, a wearable device, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto, and this application does not make any limitation here.

[0082] Figure 5 A flowchart of an echo cancellation method provided by an exemplary embodiment of the present application is shown. Figure 3 or Figure 4 The method is illustrated by an example of a computer device (also referred to as a near-end device in this document) executing the method, which includes:

[0083] Step 510, obtaining a first sound signal, where the first sound signal is a sound signal collected by a microphone of the device;

[0084] The first sound signal is a sound signal collected by the microphone of the near-end device. The first sound signal includes an echo signal and an effective sound. The echo signal refers to a signal collected by the microphone of the near-end device after multiple feedbacks of the reference signal. The reference signal refers to the sound played by the speaker of the near-end device, or the reference signal can be understood as the sound collected by the microphone of the far-end device. The effective sound refers to other sounds collected by the microphone of the near-end device except the echo signal.

[0085] Step 520, obtaining an echo linear component according to the first sound signal and a reference signal corresponding to the first sound signal, where the reference signal is a sound signal played by a speaker of the device;

[0086] After the near-end device obtains the first sound signal, it obtains a reference signal according to the first sound signal; obtains an echo delay according to the first sound signal and the reference signal; offsets the reference signal based on the echo delay to obtain a playback delay signal; and inputs the playback delay signal into a linear filter to obtain an echo linear component.

[0087] Specifically, the near-end device searches for a target playback frame whose spectrum similarity is the maximum value with respect to the spectrum similarity of the first sound signal from historical playback frames within a preset time range; and determines the target playback frame as a reference signal. Optionally, the preset time range is 1 second. Typically, the echo delay is 100-300 milliseconds.

[0088] Specifically, the linear filter is an adaptive filter. During the filtering process of the adaptive filter, the filter coefficients will be continuously and adaptively adjusted so that the output linear echo component approaches the real linear echo component in the first sound signal.

[0089] Step 530, removing the echo linear component in the first sound signal to obtain a filtered signal;

[0090] The first sound signal is subtracted from the echo linear component to obtain a filtered signal, which is also called a linear filtered signal.

[0091] Step 540, based on the filtered signal, predicting the echo residual component through the neural network model, where the echo residual component indicates the component in the echo except the echo linear component;

[0092] The filtered signal is input into the neural network model, and the neural network model predicts the echo residual component. The echo residual component indicates the component in the echo except the echo linear component. The echo residual component is also called nonlinear echo.

[0093] Step 550: Eliminate the echo in the first sound signal based on the echo linear component and the echo residual component to obtain a second sound signal.

[0094] Based on the echo linear component and the echo residual component, the echo in the first sound signal is eliminated. Optionally, the echo linear component and the echo residual component are summed to obtain an echo signal, the echo signal is subtracted from the first sound signal to obtain a residual sound signal, and the attenuation gain of each frequency point is obtained according to the ratio of the power spectrum of the echo signal to the power spectrum of the residual sound signal, and the first sound signal is suppressed by the attenuation gain of each frequency point to obtain a second sound signal. The specific echo suppression method will be described in detail below.

[0095] In summary, the above embodiment removes the linear component of the echo in the first sound signal to obtain a filtered signal, inputs the filtered signal (linear filtered signal) into the neural network model, and predicts the echo residual component. When the neural network model of the present application is generalized, it only requires that the nonlinear echo of the actual application sample is close to the training sample. Therefore, the training sample set of the neural network model of the present application will have a stronger generalization ability.

[0096] Furthermore, the problem of missing echoes can be solved by predicting the residual components of echoes through a neural network model. The present application actually provides a technology that integrates two related technologies and can absorb the advantages of both methods.

[0097] based on Figure 5 In the optional embodiment shown, step 540 will output the echo residual component through the neural network model based on the filtered signal. Specifically, there are two possible inputs to the neural network model.

[0098] The first possible neural network model will only input the filtered signal. Figure 6 , Figure 6 Only the filtered signal is input into the neural network model shown. The filtered signal is transformed into the frequency domain by Fourier transform 601 to obtain the power spectrum of the filtered signal. The power spectrum of the filtered signal is input into the first recurrent neural network to predict the echo gain, and the echo gain is multiplied by the power spectrum of the filtered signal to obtain the power spectrum of the echo residual component. The power spectrum of the echo residual component is transformed into the time domain by inverse Fourier transform 606 to obtain the echo residual component.

[0099] Figure 6 The first recurrent neural network shown includes a first convolution unit 602, two gated recurrent units (GRU) 603, a second convolution unit 604 and a fully connected layer 605 in a cascade relationship. Optionally, the gated recurrent unit can also be replaced by a long short-term memory (LSTM) unit. The structure of any one of the first convolution unit 602 and the second convolution unit 604 is as follows: Figure 7 As shown in part (A). Figure 7 Part (A) shows that a convolution unit 700 includes multiple convolution blocks 701. Figure 7 Part (B) shows that any one of the multiple convolution blocks 701 includes a first convolution layer 701-1 and a second convolution layer 701-2, and the input of the second convolution layer 701-2 includes the input and output of the first convolution layer 701-1.

[0100] In one embodiment, the power spectrum of the filtered signal is input into a first convolution unit, at least one gated recurrent unit, a second convolution unit, and a fully connected layer in a cascade relationship to predict the echo gain. Alternatively, the power spectrum of the filtered signal is input into a first convolution unit, at least one long short-term memory unit, a second convolution unit, and a fully connected layer in a cascade relationship to predict the echo gain.

[0101] In one embodiment, the convolution unit may be a one-dimensional convolution unit or a two-dimensional convolution unit.

[0102] Summary: When the gain of the kth frequency point is predicted by the first recurrent neural network, the prediction results of the frequency points before the kth frequency point will be referred to, making the prediction results more accurate. The input of the first recurrent neural network includes the filtered signal, and the linear component of the echo has been removed from the filtered signal. When the neural network model is generalized, only the nonlinear echo of the actual application sample is required to be close to the training sample. Therefore, the training sample set of the first recurrent neural network will have a stronger generalization ability.

[0103] The second possible neural network model will simultaneously input the filtered signal and the playback delay signal, where the playback delay signal is a signal obtained by offsetting the reference signal by the echo delay, which is the time distance between the reference signal and the first sound signal. Figure 4 , the play delay signal in the current embodiment is Figure 4 The playback delay signal 404 is shown.

[0104] Combined with reference Figure 8 , Figure 8 The neural network model shown is input with the filtered signal and the playback delay signal at the same time. The filtered signal is transformed into the frequency domain by Fourier transform 801 to obtain the power spectrum of the filtered signal. The playback delay signal is transformed into the frequency domain by Fourier transform 801 to obtain the power spectrum of the playback delay signal. The power spectrum of the filtered signal and the power spectrum of the playback delay signal are input into the second recurrent neural network, and the echo gain is predicted. The echo gain is multiplied by the power spectrum of the filtered signal to obtain the power spectrum of the echo residual component. The power spectrum of the echo residual component is transformed into the time domain by inverse Fourier transform 809 to obtain the echo residual component.

[0105] Figure 8 The second recurrent neural network shown includes a third convolution unit 802 and a fourth convolution unit 803, as well as a feature concatenation unit 804, a fifth convolution unit 805, two gated recurrent units 806, a sixth convolution unit 807 and a fully connected layer 808 in a cascade relationship. Optionally, the gated recurrent unit can also be replaced by a long short-term memory unit. The structure of any one of the third convolution unit 802, the fourth convolution unit 803, the fifth convolution unit 805 and the sixth convolution unit 807 is as follows: Figure 7 As shown in part (A). Figure 7 Part (A) shows that a convolution unit 700 includes multiple convolution blocks 701. Figure 7 Part (B) shows that any one of the multiple convolution blocks 701 includes a first convolution layer 701-1 and a second convolution layer 701-2, and the input of the second convolution layer 701-2 includes the input and output of the first convolution layer 701-1.

[0106] In one embodiment, the power spectrum of the filtered signal is input into the third convolution unit to obtain a first feature representation; the power spectrum of the playback delayed signal is input into the fourth convolution unit to obtain a second feature representation; the first feature representation and the second feature representation are concatenated to obtain a third feature representation; the third feature representation is input into a fifth convolution unit, at least one gated recurrent unit, a sixth convolution unit and a fully connected layer in a cascade relationship to predict an echo gain; or, the third feature representation is input into a fifth convolution unit, at least one long short-term memory unit, a sixth convolution unit and a fully connected layer in a cascade relationship to predict an echo gain.

[0107] In one embodiment, the convolution unit may be a one-dimensional convolution unit or a two-dimensional convolution unit.

[0108] Summary: When the gain of the kth frequency point is predicted by the second recurrent neural network, the prediction results of the frequency points before the kth frequency point will be referenced to make the prediction results more accurate. The input of the second recurrent neural network includes the filtered signal, and the linear component of the echo has been removed from the filtered signal. When the neural network model is generalized, only the nonlinear echo of the actual application sample is required to be close to the training sample. Therefore, the training sample set of the second recurrent neural network will have a stronger generalization ability.

[0109] based on Figure 5 In the optional embodiment shown, step 540 will output the echo residual component through the neural network model based on the filtered signal. Specifically, there are two possible inputs to the neural network model.

[0110] The third possible neural network model will only input the filtered signal. Fig. 9 In the first-level echo residual component prediction process, the filtered signal is segmented 901, the obtained sound segments are transformed into the frequency domain through Fourier transform 902, and the frequency domain features are subjected to instantaneous layer normalization 903. The normalized features are input into two long short-term memory units 904, a fully connected network 905 and an activation function 906, and the first-level echo gain features are output. The first-level echo gain features are multiplied by the frequency domain features of the filtered signal to obtain the first-level echo residual components.

[0111] In the second-level echo residual component prediction process, the first-level echo residual component is subjected to inverse Fourier transform 907 to obtain time domain features. The obtained time domain features are subjected to one-dimensional convolution 908 and instantaneous layer normalization 903. The obtained features are input into two long short-term memory units 904, a fully connected network 905 and an activation function 906, and the second-level echo gain features are output. The second-level echo gain features are multiplied by the time domain features of the first-level echo residual signal (one-dimensional convolution 908 has been performed) to obtain the second-level echo residual component. The second-level echo residual component is determined as the final echo residual component.

[0112] In summary, when the gain of the kth frequency point is predicted by the recurrent neural network (specifically the DTLN network), the prediction results of the frequency points before the kth frequency point will be referenced to make the prediction results more accurate. In addition, the DTLN network is divided into two levels of prediction, the first level predicts through frequency domain features, and the second level predicts through time domain features, and the final echo residual component is more accurate. Fig. 9The input of the neural network shown includes the filtered signal, the linear component of the echo has been removed from the filtered signal, and when the neural network model generalizes, it only requires the nonlinear echo of the actual application sample to be close to the training sample. Therefore, Fig. 9 The training sample set of the neural network shown will have stronger generalization ability.

[0113] The fourth possible neural network model will simultaneously input the filtered signal and the playback delay signal, where the playback delay signal is a signal obtained by offsetting the reference signal by the echo delay, which is the time distance between the reference signal and the first sound signal. Figure 4 , the play delay signal in the current embodiment is Figure 4 The playback delay signal 404 is shown.

[0114] Combined with reference Fig.10 , Fig.10 The neural network model shown in the figure simultaneously inputs the filtered signal and the playback delay signal, and outputs the echo residual component according to the filtered signal and the playback delay signal.

[0115] In the first-level echo residual component prediction process, the filtered signal and the playback delay signal are first obtained. The filtered signal and the playback delay signal are segmented 1001 respectively, the segmented sound signal is transformed into the frequency domain through Fourier transform 1002, and the frequency domain features are normalized at the instantaneous layer 1003. The features obtained after the filtered signal and the playback delay signal are processed as above are spliced ​​1004. The spliced ​​features are input into two long short-term memory units 1005, a fully connected network 1006 and an activation function 1007, and the first-level echo gain features are output. The first-level echo gain features are multiplied by the frequency domain features of the filtered signal to obtain the first-level echo residual component.

[0116] In the second-level echo residual component prediction process, the first-level echo residual component is subjected to inverse Fourier transform 1008 to obtain time domain features. The obtained time domain features and the frequency domain features of the playback delay signal are respectively subjected to one-dimensional convolution 1009 and instantaneous layer normalization processing 1003. The two obtained features are subjected to feature splicing 1004. The spliced ​​features are input into two long short-term memory units 1005, a fully connected network 1006 and an activation function 1007, and the second-level echo gain features are output. The second-level echo gain features are multiplied by the time domain features of the first-level echo residual component (one-dimensional convolution 1009 has been performed) to obtain the second-level echo residual component. The second-level echo residual component is determined as the final echo residual component.

[0117] In summary, when the gain of the kth frequency point is predicted by the recurrent neural network (specifically the DTLN network), the prediction results of the frequency points before the kth frequency point will be referenced to make the prediction results more accurate. In addition, the DTLN network is divided into two levels of prediction, the first level predicts through frequency domain features, and the second level predicts through time domain features, and the final echo residual component is more accurate. Fig.10 The input of the neural network shown includes the filtered signal, the linear component of the echo has been removed from the filtered signal, and when the neural network model generalizes, it only requires the nonlinear echo of the actual application sample to be close to the training sample. Therefore, Fig.10 The training sample set of the neural network shown will have stronger generalization ability.

[0118] Fig.11 A schematic diagram of an echo cancellation method provided by an exemplary embodiment of the present application is shown.

[0119] Fig.11 An echo cancellation framework 1100 is provided. In the process of performing echo cancellation, a reference signal 1101 is obtained. The reference signal 1101 is played through the speaker S of the near-end device, and an echo signal 1 is formed during the propagation process and collected by the microphone M of the near-end device. The microphone M of the near-end device also collects other sounds 2, which together with the echo signal 1 constitute a first sound signal 1102. The first sound signal 1102 and the reference signal 1101 are input into an echo delay estimation unit 1103, and a playback delay signal 1104 is output. The echo delay estimation unit is used to align the first sound signal 1102 and the reference signal 1101 to obtain an echo delay 1105, and the reference signal 1101 is offset according to the echo delay 1105 to obtain a playback delay signal 1104. The playback delay signal 1104 is input into a linear filter 1106 to obtain an echo linear component 1107. The echo linear component 1107 in the first sound signal 1102 is removed to obtain a filtered signal 1108. The filtered signal 1108 is input into the neural network model 1109 to predict the echo residual component 1110. The first sound signal 1102, the echo delay 1105, the echo linear component 1107 and the echo residual component 1110 are input into the echo cancellation unit 1111, and the echo cancellation unit 1111 cancels the echo in the first sound signal 1102 and outputs the second sound signal 1112. Fig.11 The echo cancellation framework 1100 in the embodiment is executed by the terminal 3 in the computer device.

[0120] Combined with reference Fig.12 , Fig.12 Shows Fig.11 The steps performed by the echo cancellation unit in Fig.12 The method steps shown can also be used to replace Figure 5 Step 550 in .

[0121] Step 1201, obtaining a first attenuation gain of each frequency point of the first sound signal based on an echo linear component, an echo residual component and a first sound signal;

[0122] In one embodiment, in conjunction with reference Fig.13 Part (A) of Fig.13 Part (A) shows that the echo linear component 1301 and the echo residual component 1302 are added to obtain an echo signal 1303; the echo signal 1303 is subtracted from the first sound signal 1304 to obtain a residual sound signal 1305; based on the power spectrum 1306 of the echo signal and the inverse 1307 of the power spectrum of the residual sound signal, the prior signal-to-echo ratio 1308 of each frequency point is calculated. Based on the prior signal-to-echo ratio of each frequency point, the first attenuation gain 1309 of each frequency point of the first sound signal is obtained. Optionally, the first attenuation gain of each frequency point of the first sound signal is obtained by the Wiener filter formula.

[0123] Specifically, the sound signal after echo elimination of the previous frame is obtained; the ratio of the power spectrum of the sound signal after echo elimination of the previous frame to the power spectrum of the echo signal of the current frame is calculated to obtain the first prior signal-to-echo ratio of each frequency point; the ratio of the power spectrum of the residual sound signal of the current frame to the power spectrum of the echo signal of the current frame is calculated to obtain the second prior signal-to-echo ratio of each frequency point; for any frequency point, the first prior signal-to-echo ratio and the second prior signal-to-echo ratio are weightedly calculated to obtain the prior signal-to-echo ratio of the frequency point (prior signal-to-echo power ratio).

[0124] Schematically, it is expressed by the formula:

[0125]

[0126] Among them, SER prio (i,k) represents the prior signal-to-response ratio of the kth frequency point in the i-th frame, E out (i-1,k) represents the power of the kth frequency point of the sound signal after echo cancellation of the i-1th frame, Eecho′(i,k) represents the power of the kth frequency point of the echo signal of the i-th frame, and Eerr(i,k) represents the power of the kth frequency point of the residual sound signal of the i-th frame. a is the smoothing coefficient, which ranges from 0.8 to 1.

[0127] Specifically, for any frequency point, the a priori signal-to-return ratio of the frequency point is added by one and used as the denominator; the a priori signal-to-return ratio of the frequency point is used as the numerator; and the ratio of the numerator to the denominator is used as the first attenuation gain of the frequency point.

[0128] Schematically, it is expressed by the formula:

[0129]

[0130] Among them, g(i,k) represents the first attenuation gain of the kth frequency point in the i-th frame, SER prio (i, k) is the prior signal-to-echo ratio of the kth frequency point of the i-th frame. It can be understood that the above formula for calculating the first attenuation gain is a non-Wiener filtering method, that is, the linear echo obtained based on the linear filter and the nonlinear echo predicted based on the neural network model are then suppressed through the Wiener filtering formula to suppress the first sound signal.

[0131] Step 1202, obtaining an echo delay between a reference signal and a first sound signal;

[0132] In one embodiment, a target playback frame whose spectrum similarity is the maximum with respect to the spectrum similarity of the first sound signal is searched from historical playback frames within a preset time range; the target playback frame is determined as a reference signal; and the time distance between the target playback frame and the first sound signal is determined as the echo delay. Optionally, the preset time range is 1 second. Typically, the echo delay is 100-300 milliseconds.

[0133] Step 1203, when the echo delay is greater than a preset threshold value, for any frequency point of the first sound signal, multiply the first attenuation gain of the frequency point by the power of the frequency point to obtain an attenuation result of the frequency point; determine the attenuation result of each frequency point of the first sound signal; transform the attenuation result of each frequency point into the time domain to obtain a second sound signal;

[0134] The echo delay is greater than the preset threshold value, that is, the echo delay exceeds the reasonable threshold value, which indicates that the linear filter is in an unstable state. At this time, the echo residual component (nonlinear echo) predicted by the neural network model is used to suppress the first sound signal.

[0135] Combined with reference Fig.14 , Fig.14 It shows whether the echo delay is greater than a preset threshold value. If yes, the following steps are performed: The first attenuation gain 1401 of a frequency point and the power 1402 of the frequency point are multiplied to obtain the attenuation result 1403 of the frequency point. The attenuation result 1404 of each frequency point is determined.

[0136] Step 1204, when the echo delay is not greater than a preset threshold value, obtain the second attenuation gain of each frequency point of the first sound signal based on the echo linear component and the first sound signal; for any frequency point of the first sound signal, multiply the larger value of the first attenuation gain and the second attenuation gain of the frequency point by the power of the frequency point to obtain the attenuation result of the frequency point; determine the attenuation result of each frequency point of the first sound signal; transform the attenuation result of each frequency point into the time domain to obtain the second sound signal.

[0137] The echo delay is not greater than the preset threshold value, that is, the echo delay does not exceed the reasonable threshold value. At this time, it indicates that the linear filter is in a stable state. At this time, the echo residual component (nonlinear echo) predicted by the neural network model can be used to suppress the first sound signal. The nonlinear echo predicted by the neural network model may be greater than the actual nonlinear echo. If the nonlinear echo predicted by the neural network model is directly used, the effective sound in the first sound signal may be damaged. Therefore, if the linear filter is in a stable state, it is necessary to further determine whether the first attenuation gain calculated based on the nonlinear echo predicted by the neural network model is too small. If it is too small, other methods are used for echo suppression. Too small means that the effective sound in the first sound signal will be damaged. Optionally, the preset threshold value is the most recent M frames, and M is 20-40.

[0138] In one embodiment, the maximum echo of the first sound signal is estimated first, and the second attenuation gain required to suppress the maximum echo is calculated. If the second attenuation gain is greater than the first attenuation gain, the second attenuation gain is used. If the second attenuation gain is not greater than the first attenuation gain, the first attenuation gain is used. The first attenuation gain is calculated based on the nonlinear echo predicted by the neural network model. The attenuation gain is a number between 0 and 1. The larger the attenuation gain value, the weaker the suppression effect, and the smaller the attenuation gain value, the stronger the suppression.

[0139] Specifically, combined with reference Fig.13 Part (B) of Fig.13 Part (B) shows that the power 1310 of each frequency point of the echo linear component is multiplied by the preset gain value 1311 to obtain the power spectrum 1312 of the maximum echo signal, and the maximum echo signal represents the estimated maximum echo; the power spectrum 1312 of the maximum echo signal is multiplied by the inverse 1313 of the power spectrum of the first sound signal (i.e., divided by the power spectrum of the first sound signal) to obtain the maximum echo proportion 1314 of each frequency point; for any frequency point of the first sound signal, the maximum echo proportion 1315 of the frequency point is subtracted from one to obtain the second attenuation gain 1316 of the frequency point; and the second attenuation gain 1317 of each frequency point of the first sound signal is determined.

[0140] Schematically, it is expressed by the formula:

[0141]

[0142] Among them, g limit(i,k) represents the second attenuation gain of the kth frequency point of the i-th frame, Ey(i,k) represents the power of the kth frequency point of the linear component of the echo of the i-th frame, G represents the preset gain value, G is the amplification factor of Ey(i,k), and the value of G can be 1-10. G is used to amplify the predicted filtered signal to ensure that some sudden large echoes can also be detected and suppressed. Ey(i,k)*G represents the upper limit power spectrum of the echo, and Ed(i,k) represents the power of the kth frequency point of the first sound signal of the i-th frame, that is, it represents the power spectrum of the first sound signal of the i-th frame.

[0143] Combined with reference Fig.14 , Fig.14 It shows whether the echo delay is greater than a preset threshold value. If not, the following steps are performed. For a first attenuation gain 1401 of a frequency point and a second attenuation gain 1405 of a frequency point, a larger value 1406 is taken. The larger value 1406 is multiplied by the power 1402 of the frequency point to obtain the attenuation result 1403 of the frequency point. The attenuation results 1404 of each frequency point are determined.

[0144] In summary, Fig.12 In the provided embodiment, the first attenuation gain is calculated by the nonlinear echo (echo residual component) predicted by the neural network model. If the echo delay is greater than the preset threshold value (i.e., the linear filter is in an unstable state), the first attenuation gain is directly used for echo suppression. At this time, the echo residual problem of the first echo cancellation method in the related art can be solved.

[0145] If the echo delay is not greater than the preset threshold value (i.e., the linear filter is in a stable state), the first attenuation gain needs to be judged. If the first attenuation gain is greater than the second attenuation gain, the first attenuation gain is used for suppression. If the first attenuation gain is not greater than the second attenuation gain, the second attenuation gain is used for suppression. The second attenuation gain is calculated based on the estimated maximum echo. The above embodiment can protect the valid speech in the first sound signal and solve the problem that the neural network model in the second echo cancellation method of the related art is prone to accidentally damage the valid sound.

[0146] Fig.15 : is a structural block diagram of an echo cancellation device provided by an exemplary embodiment of the present application, the device comprising:

[0147] An acquisition module 1501 is used to acquire a first sound signal, where the first sound signal is a sound signal collected by a microphone of a device;

[0148] A processing module 1502 is used to obtain an echo linear component according to the first sound signal and a reference signal corresponding to the first sound signal, where the reference signal is a sound signal played by a speaker of the device;

[0149] The processing module 1502 is further used to remove the echo linear component in the first sound signal to obtain a filtered signal;

[0150] The processing module 1502 is further used to predict the echo residual component through the neural network model based on the filtered signal, where the echo residual component indicates the component in the echo except the echo linear component;

[0151] The processing module 1502 is further configured to eliminate the echo in the first sound signal based on the echo linear component and the echo residual component to obtain the second sound signal.

[0152] In an optional embodiment, the processing module 1502 is also used to transform the filtered signal into the frequency domain to obtain a power spectrum of the filtered signal; input the power spectrum of the filtered signal into the first recurrent neural network to predict the echo gain; multiply the echo gain by the power spectrum of the filtered signal to obtain a power spectrum of the echo residual component; transform the power spectrum of the echo residual component into the time domain to obtain the echo residual component.

[0153] In an optional embodiment, the processing module 1502 is further used to input the power spectrum of the filtered signal into a first convolution unit, at least one gated recurrent unit, a second convolution unit and a fully connected layer in a cascade relationship to predict the echo gain; or input the power spectrum of the filtered signal into a first convolution unit, at least one long short-term memory unit, a second convolution unit and a fully connected layer in a cascade relationship to predict the echo gain;

[0154] Among them, any one of the first convolution unit and the second convolution unit includes multiple convolution blocks, any one of the multiple convolution blocks includes a first convolution layer and a second convolution layer, and the input of the second convolution layer includes the input and output of the first convolution layer.

[0155] In an optional embodiment, the processing module 1502 is also used to transform the filtered signal into the frequency domain to obtain a power spectrum of the filtered signal; and transform the playback delay signal into the frequency domain to obtain a power spectrum of the playback delay signal, where the playback delay signal is a signal obtained by offsetting the reference signal by the echo delay, and the echo delay is the time distance between the reference signal and the first sound signal; input the power spectrum of the filtered signal and the power spectrum of the playback delay signal into the second recurrent neural network to predict the echo gain; multiply the echo gain by the power spectrum of the filtered signal to obtain a power spectrum of the echo residual component; and transform the power spectrum of the echo residual component to the time domain to obtain the echo residual component.

[0156] In an optional embodiment, the processing module 1502 is further used to input the power spectrum of the filtered signal into the third convolution unit to obtain a first feature representation; input the power spectrum of the playback delay signal into the fourth convolution unit to obtain a second feature representation; concatenate the first feature representation and the second feature representation to obtain a third feature representation; input the third feature representation into a fifth convolution unit, at least one gated recurrent unit, a sixth convolution unit, and a fully connected layer in a cascade relationship to predict an echo gain; or input the third feature representation into a fifth convolution unit, at least one long short-term memory unit, a sixth convolution unit, and a fully connected layer in a cascade relationship to predict an echo gain;

[0157] Among them, any one of the third convolution unit, the fourth convolution unit, the fifth convolution unit and the sixth convolution unit includes multiple convolution blocks, any one of the multiple convolution blocks includes a first convolution layer and a second convolution layer, and the input of the second convolution layer includes the input and output of the first convolution layer.

[0158] In an optional embodiment, the processing module 1502 is also used to obtain a reference signal based on the first sound signal; obtain an echo delay based on the first sound signal and the reference signal; offset the reference signal based on the echo delay to obtain a playback delay signal; input the playback delay signal into a linear filter to obtain an echo linear component.

[0159] In an optional embodiment, the processing module 1502 is further used to search for a target playback frame whose spectrum similarity is the maximum value with respect to the spectrum similarity of the first sound signal from historical playback frames within a preset time range; determine the target playback frame as a reference signal; and determine the time distance between the target playback frame and the first sound signal as the echo delay.

[0160] In an optional embodiment, the processing module 1502 is also used to obtain a first attenuation gain for each frequency point of the first sound signal based on the echo linear component, the echo residual component and the first sound signal; obtain the echo delay between the reference signal and the first sound signal; when the echo delay is greater than a preset threshold value, for any frequency point of the first sound signal, multiply the first attenuation gain of the frequency point by the power of the frequency point to obtain an attenuation result of the frequency point; determine the attenuation result of each frequency point of the first sound signal; and transform the attenuation result of each frequency point to the time domain to obtain a second sound signal.

[0161] In an optional embodiment, the processing module 1502 is also used to add the echo linear component and the echo residual component to obtain an echo signal; subtract the echo signal from the first sound signal to obtain a residual sound signal; calculate the prior signal-to-echo ratio of each frequency point based on the power spectrum of the echo signal and the power spectrum of the residual sound signal; and obtain the first attenuation gain of each frequency point of the first sound signal based on the prior signal-to-echo ratio of each frequency point.

[0162] In an optional embodiment, the processing module 1502 is also used to obtain the sound signal after echo elimination of the previous frame; calculate the ratio of the power spectrum of the sound signal after echo elimination of the previous frame to the power spectrum of the echo signal of the current frame to obtain the first prior signal-to-echo ratio of each frequency point; calculate the ratio of the power spectrum of the residual sound signal of the current frame to the power spectrum of the echo signal of the current frame to obtain the second prior signal-to-echo ratio of each frequency point; for any frequency point, perform weighted calculation on the first prior signal-to-echo ratio and the second prior signal-to-echo ratio to obtain the prior signal-to-echo ratio of the frequency point.

[0163] In an optional embodiment, the processing module 1502 is further used to, for any frequency point, add one to the prior signal-to-return ratio of the frequency point as the denominator; use the prior signal-to-return ratio of the frequency point as the numerator; and use the ratio of the numerator to the denominator as the first attenuation gain of the frequency point.

[0164] In an optional embodiment, the processing module 1502 is also used to obtain a second attenuation gain for each frequency point of the first sound signal based on the echo linear component and the first sound signal when the echo delay is not greater than a preset threshold value; for any frequency point of the first sound signal, multiply the larger value of the first attenuation gain and the second attenuation gain of the frequency point by the power of the frequency point to obtain the attenuation result of the frequency point; determine the attenuation result of each frequency point of the first sound signal; and transform the attenuation result of each frequency point to the time domain to obtain the second sound signal.

[0165] In an optional embodiment, the processing module 1502 is also used to multiply each frequency point of the power spectrum of the echo linear component by a preset gain value to obtain a power spectrum of the maximum echo signal, where the maximum echo signal represents the estimated maximum echo; divide the power spectrum of the maximum echo signal by the power spectrum of the first sound signal to obtain the maximum echo ratio of each frequency point; for any frequency point of the first sound signal, subtract the maximum echo ratio of the frequency point from one to obtain a second attenuation gain of the frequency point; and determine the second attenuation gain of each frequency point of the first sound signal.

[0166] To sum up, in the related art, the training samples of the neural network model contain two aspects of echo: linear echo and nonlinear echo. When the neural network model of the related art is generalized, the linear echo and nonlinear echo of the actual application samples need to be close to the training samples. The generalization ability of the training sample set of the related art is relatively weak.

[0167] In the present application, the linear component of the echo in the first sound signal is removed to obtain a filtered signal, and the filtered signal (linear filtered signal) is input into the neural network model to predict the echo residual component. When the neural network model of the present application is generalized, it is only necessary that the nonlinear echo of the actual application sample is close to the training sample. Therefore, the training sample set of the neural network model of the present application will have a stronger generalization ability.

[0168] Furthermore, the problem of missing echoes can be solved by predicting the residual components of echoes through a neural network model. The present application actually provides a technology that integrates two related technologies and can absorb the advantages of both methods.

[0169] Fig.16 The block diagram of the structure of a computer device 1600 provided by an exemplary embodiment of the present application is shown. The computer device 1600 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The computer device 1600 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0170] Typically, the computer device 1600 includes a processor 1601 and a memory 1602 .

[0171] The processor 1601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0172] The memory 1602 may include one or more computer-readable storage media, which may be non-transitory. The memory 1602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1602 is used to store at least one instruction, which is used to be executed by the processor 1601 to implement the echo cancellation method provided in the method embodiment of the present application.

[0173] In some embodiments, the computer device 1600 may further optionally include: a peripheral device interface 1603 and at least one peripheral device. The processor 1601, the memory 1602 and the peripheral device interface 1603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1603 via a bus, a signal line or a circuit board. For example, the peripheral device may include: at least one of a radio frequency circuit 1604, a display screen 1605, a camera assembly 1606, an audio circuit 1607 and a power supply 1608.

[0174] The peripheral device interface 1603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1601 and the memory 1602. In some embodiments, the processor 1601, the memory 1602, and the peripheral device interface 1603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1601, the memory 1602, and the peripheral device interface 1603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0175] The radio frequency circuit 1604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1604 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1604 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1604 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0176] The display screen 1605 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1605 is a touch display screen, the display screen 1605 also has the ability to collect touch signals on the surface or above the surface of the display screen 1605. The touch signal can be input to the processor 1601 as a control signal for processing. At this time, the display screen 1605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1605 can be one, arranged on the front panel of the computer device 1600; in other embodiments, the display screen 1605 can be at least two, respectively arranged on different surfaces of the computer device 1600 or in a folding design; in other embodiments, the display screen 1605 can be a flexible display screen, arranged on a curved surface or a folding surface of the computer device 1600. Even, the display screen 1605 can also be arranged as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 1605 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0177] The camera assembly 1606 is used to capture images or videos. Optionally, the camera assembly 1606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0178] The audio circuit 1607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1601 for processing, or input them into the radio frequency circuit 1604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the computer device 1600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1601 or the radio frequency circuit 1604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1607 may also include a headphone jack.

[0179] The power supply 1608 is used to power various components in the computer device 1600. The power supply 1608 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1608 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0180] In some embodiments, the computer device 1600 further includes one or more sensors 1609 , including but not limited to: an acceleration sensor 1610 , a gyroscope sensor 1611 , a pressure sensor 1612 , an optical sensor 1613 , and a proximity sensor 1614 .

[0181] The acceleration sensor 1610 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the computer device 1600. For example, the acceleration sensor 1610 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1601 can control the display screen 1605 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1610. The acceleration sensor 1610 can also be used to collect motion data of games or users.

[0182] The gyro sensor 1611 can detect the body direction and rotation angle of the computer device 1600, and the gyro sensor 1611 can cooperate with the acceleration sensor 1610 to collect the user's 3D actions on the computer device 1600. The processor 1601 can implement the following functions based on the data collected by the gyro sensor 1611: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0183] The pressure sensor 1612 can be set on the side frame of the computer device 1600 and / or the lower layer of the display screen 1605. When the pressure sensor 1612 is set on the side frame of the computer device 1600, it can detect the user's grip signal of the computer device 1600, and the processor 1601 performs left and right hand recognition or shortcut operations according to the grip signal collected by the pressure sensor 1612. When the pressure sensor 1612 is set on the lower layer of the display screen 1605, the processor 1601 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1605. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0184] The optical sensor 1613 is used to collect the ambient light intensity. In one embodiment, the processor 1601 can control the display brightness of the display screen 1605 according to the ambient light intensity collected by the optical sensor 1613. For example, when the ambient light intensity is high, the display brightness of the display screen 1605 is increased; when the ambient light intensity is low, the display brightness of the display screen 1605 is reduced. In another embodiment, the processor 1601 can also dynamically adjust the shooting parameters of the camera component 1606 according to the ambient light intensity collected by the optical sensor 1613.

[0185] The proximity sensor 1614, also called a distance sensor, is usually disposed on the front panel of the computer device 1600. The proximity sensor 1614 is used to collect the distance between the user and the front of the computer device 1600. In one embodiment, when the proximity sensor 1614 detects that the distance between the user and the front of the computer device 1600 is gradually decreasing, the processor 1601 controls the display screen 1605 to switch from the screen-on state to the screen-off state; when the proximity sensor 1614 detects that the distance between the user and the front of the computer device 1600 is gradually increasing, the processor 1601 controls the display screen 1605 to switch from the screen-off state to the screen-on state.

[0186] Those skilled in the art will understand that Fig.16 The structure shown in the figure does not constitute a limitation on the computer device 1600, and the computer device 1600 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0187] The present application also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the echo cancellation method provided by the above method embodiment.

[0188] The present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the echo cancellation method provided by the above method embodiment.

[0189] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0190] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0191] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An echo cancellation method, characterized in that: The method comprises: Acquire a first sound signal, where the first sound signal is a sound signal collected by a microphone of a device; obtaining an echo linear component according to the first sound signal and a reference signal corresponding to the first sound signal, wherein the reference signal is a sound signal played by a speaker of the device; Removing the echo linear component from the first sound signal to obtain a filtered signal; Based on the filtered signal, predicting an echo residual component through a neural network model, wherein the echo residual component indicates a component in the echo other than the echo linear component; Based on the echo linear component and the echo residual component, the echo in the first sound signal is eliminated to obtain a second sound signal.

2. The method according to claim 1, characterized in that The method of predicting the echo residual component based on the filtered signal by a neural network model includes: Transforming the filtered signal into the frequency domain to obtain a power spectrum of the filtered signal; Inputting the power spectrum of the filtered signal into a first recurrent neural network to predict the echo gain; Multiplying the echo gain by the filtered signal power spectrum to obtain an echo residual component power spectrum; The echo residual component power spectrum is transformed into the time domain to obtain the echo residual component.

3. The method according to claim 2, characterized in that The step of inputting the filtered signal power spectrum into a first recurrent neural network to predict an echo gain comprises: Inputting the power spectrum of the filtered signal into a first convolution unit, at least one gated recurrent unit, a second convolution unit and a fully connected layer in a cascade relationship, and predicting the echo gain; or, Inputting the power spectrum of the filtered signal into a first convolution unit, at least one long short-term memory unit, a second convolution unit and a fully connected layer in a cascade relationship, and predicting the echo gain; Among them, any one of the first convolution unit and the second convolution unit includes multiple convolution blocks, any one of the multiple convolution blocks includes a first convolution layer and a second convolution layer, and the input of the second convolution layer includes the input and output of the first convolution layer.

4. The method according to claim 1, characterized in that The method of predicting the echo residual component based on the filtered signal by a neural network model includes: transforming the filtered signal into the frequency domain to obtain a power spectrum of the filtered signal; and transforming the playback delay signal into the frequency domain to obtain a power spectrum of the playback delay signal, wherein the playback delay signal is a signal obtained by offsetting the reference signal by an echo delay, wherein the echo delay is a time distance between the reference signal and the first sound signal; Inputting the power spectrum of the filtered signal and the power spectrum of the playback delayed signal into a second recurrent neural network to predict the echo gain; Multiplying the echo gain by the filtered signal power spectrum to obtain an echo residual component power spectrum; The echo residual component power spectrum is transformed into the time domain to obtain the echo residual component.

5. The method according to claim 4, characterized in that The step of inputting the filtered signal power spectrum and the playback delay signal power spectrum into a second recurrent neural network to predict an echo gain includes: Inputting the power spectrum of the filtered signal into a third convolution unit to obtain a first feature representation; Inputting the power spectrum of the playback delay signal into a fourth convolution unit to obtain a second feature representation; Concatenating the first feature representation and the second feature representation to obtain a third feature representation; Input the third feature representation into a fifth convolution unit, at least one gated recurrent unit, a sixth convolution unit, and a fully connected layer in a cascade relationship, and predict the echo gain; or input the third feature representation into a fifth convolution unit, at least one long short-term memory unit, a sixth convolution unit, and a fully connected layer in a cascade relationship, and predict the echo gain; Among them, any one of the third convolution unit, the fourth convolution unit, the fifth convolution unit and the sixth convolution unit includes multiple convolution blocks, any one of the multiple convolution blocks includes a first convolution layer and a second convolution layer, and the input of the second convolution layer includes the input and output of the first convolution layer.

6. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining the echo linear component according to the first sound signal and a reference signal corresponding to the first sound signal includes: Acquire the reference signal according to the first sound signal; Obtaining an echo delay according to the first sound signal and the reference signal; Based on the echo delay, the reference signal is offset to obtain a playback delay signal; The playback delay signal is input into a linear filter to obtain an echo linear component.

7. The method according to claim 6, characterized in that The acquiring the reference signal according to the first sound signal comprises: Searching for a target playback frame whose frequency spectrum similarity is the maximum value with respect to the frequency spectrum similarity of the first sound signal from historical playback frames within a preset time range; determining the target playback frame as the reference signal; The obtaining the echo delay according to the first sound signal and the reference signal includes: The time distance between the target playback frame and the first sound signal is determined as the echo delay.

8. The method according to any one of claims 1 to 5, characterized in that: The step of eliminating the echo in the first sound signal based on the echo linear component and the echo residual component to obtain the second sound signal includes: Obtaining a first attenuation gain of each frequency point of the first sound signal based on the echo linear component, the echo residual component and the first sound signal; Acquire an echo delay between the reference signal and the first sound signal; When the echo delay is greater than a preset threshold value, for any frequency point of the first sound signal, multiply the first attenuation gain of the frequency point by the power of the frequency point to obtain the attenuation result of the frequency point; determine the attenuation results of each frequency point of the first sound signal; transform the attenuation results of each frequency point into the time domain to obtain the second sound signal.

9. The method according to claim 8, characterized in that The obtaining, based on the echo linear component, the echo residual component and the first sound signal, a first attenuation gain of each frequency point of the first sound signal comprises: Adding the echo linear component and the echo residual component to obtain an echo signal; Subtracting the echo signal from the first sound signal to obtain a residual sound signal; Based on the power spectrum of the echo signal and the power spectrum of the residual sound signal, calculating the a priori signal-to-echo ratio of each frequency point; Based on the a priori signal-to-return ratio of each frequency point, a first attenuation gain of each frequency point of the first sound signal is obtained.

10. The method according to claim 9, characterized in that The calculating the a priori signal-to-echo ratio of each frequency point based on the power spectrum of the echo signal and the power spectrum of the residual sound signal includes: Get the sound signal after echo cancellation of the previous frame; Calculate the ratio of the power spectrum of the sound signal after echo cancellation of the previous frame to the power spectrum of the echo signal of the current frame to obtain a first a priori signal-to-echo ratio of each frequency point; Calculating the ratio of the power spectrum of the residual sound signal of the current frame to the power spectrum of the echo signal of the current frame to obtain a second a priori signal-to-echo ratio of each frequency point; For any frequency point, weighted calculation is performed on the first a priori signal-to-return ratio and the second a priori signal-to-return ratio to obtain the a priori signal-to-return ratio of the frequency point.

11. The method according to claim 9, characterized in that The obtaining, based on the priori signal-to-return ratio of each frequency point, a first attenuation gain of each frequency point of the first sound signal comprises: For any frequency point, add one to the a priori signal-to-return ratio of the frequency point and use it as the denominator; The a priori signal-to-response ratio of the frequency point is used as a numerator; The ratio of the numerator to the denominator is used as the first attenuation gain of the frequency point.

12. The method according to claim 8, characterized in that The method further comprises: When the echo delay is not greater than the preset threshold value, obtaining a second attenuation gain of each frequency point of the first sound signal based on the echo linear component and the first sound signal; For any frequency point of the first sound signal, multiply a larger value of the first attenuation gain and the second attenuation gain of the frequency point by the power of the frequency point to obtain an attenuation result of the frequency point; Determine the attenuation result of each frequency point of the first sound signal; The attenuation results of the frequency points are transformed into the time domain to obtain the second sound signal.

13. The method according to claim 12, characterized in that The obtaining, based on the echo linear component and the first sound signal, a second attenuation gain of each frequency point of the first sound signal comprises: Multiply each frequency point of the power spectrum of the linear component of the echo by a preset gain value to obtain a power spectrum of a maximum echo signal, where the maximum echo signal represents an estimated maximum echo; Dividing the power spectrum of the maximum echo signal by the power spectrum of the first sound signal to obtain the maximum echo proportion of each frequency point; For any frequency point of the first sound signal, subtract the maximum echo proportion of the frequency point from one to obtain a second attenuation gain of the frequency point; A second attenuation gain of each frequency point of the first sound signal is determined.

14. An echo cancellation device, characterized in that: The device comprises: An acquisition module, configured to acquire a first sound signal, where the first sound signal is a sound signal collected by a microphone of a device; a processing module, configured to obtain an echo linear component according to the first sound signal and a reference signal corresponding to the first sound signal, wherein the reference signal is a sound signal played by a speaker of the device; The processing module is further used to remove the echo linear component in the first sound signal to obtain a filtered signal; The processing module is further used to predict the echo residual component through the neural network model based on the filtered signal, and the echo residual component indicates the component in the echo except the echo linear component; The processing module is further configured to eliminate the echo in the first sound signal based on the echo linear component and the echo residual component to obtain a second sound signal.

15. A computer device, characterized in that: The computer device comprises: a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the echo cancellation method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the echo cancellation method according to any one of claims 1 to 13.

17. A computer program product, characterized in that The computer program product stores a computer program, and the computer program is loaded and executed by a processor to implement the echo cancellation method according to any one of claims 1 to 13.

Images