Method for simulating far-end signal into echo signal
By simulating the remote signal into an approximate echo signal and transmitting it to the neural network, the existing deep learning methods have solved the problem of large network burden in echo cancellation and urgent demand for training data, and the effect of improving network performance and reducing burden is achieved.
Patent Information
- Application Number
- CN202311381539.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-24
- Publication Date
- 2025-04-25
AI Technical Summary
Existing deep learning methods rely on complex neural networks and diversified training data in echo cancellation, resulting in a large network burden, strict performance tests, and urgent demand for high-quality training data.
By simulating the distal signal in the training data as an approximate echo signal and transmitting it with the proximal microphone signal into the neural network, the process of the neural network predicting the echo path is eliminated, thereby reducing the burden on the network.
This method greatly reduces the burden on neural networks and improves network performance, so that using smaller models can achieve the same effect as traditional models, and is more friendly to devices with limited computing power.
Smart Images
Figure CN119889337A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of audio processing, and in particular relates to a method for simulating a far-end signal as an echo signal. Background Art
[0002] In the prior art, in the traditional echo cancellation algorithm, the echo in the near-end microphone signal needs to be eliminated with the help of the far-end signal, but because the echo also contains indirect echo, the effect of the traditional method is limited. In recent years, with the development of deep learning technology, the echo cancellation algorithm based on deep learning has become more mature. However, the deep learning method relies on high-quality training data.
[0003] Existing deep learning methods directly pass the far-end signal and the near-end microphone signal into the neural network, allowing the neural network to learn the correlation between the far-end signal and the echo signal in the near-end microphone signal, which places high demands on the performance of the neural network. Therefore, the disadvantage of existing deep learning methods is that more diverse training data and more complex neural networks are required to achieve better results, which puts more pressure on the performance of the neural network. Therefore, the acquisition of high-quality and diverse training data has become an increasingly urgent need.
[0004] In addition, commonly used technical terms include:
[0005] Neural network: A neural network is a mathematical model or computational model that mimics the structure and function of a biological neural network. A neural network is a large number of artificial neurons connected to perform calculations. Modern neural networks are a nonlinear statistical data modeling tool that is often used to model complex relationships between inputs and outputs.
[0006] Time domain: describes how a signal changes over time.
[0007] Frequency domain: describes how a signal changes with frequency.
[0008] Near-end microphone signal: refers to the signal received by the near-end microphone during a call, which can be divided into three situations.
[0009] 1. When the near-end person and the far-end person speak at the same time: The near-end microphone signal contains the near-end voice, background noise and echo.
[0010] 2. When only the near-end person is speaking: The near-end microphone signal contains the near-end voice and background noise.
[0011] 3. When only the far-end person is speaking: The near-end microphone signal contains background noise and echo.
[0012] Far-end signal: refers to the signal transmitted by the other party during a call, which is also equal to the signal played by the near-end speaker. There are two situations.
[0013] 1. When the near-end and far-end speakers are speaking at the same time or only the far-end speaker is speaking: the far-end signal includes the far-end speaker’s voice and the far-end background noise.
[0014] 2. When only the near-end person is speaking: the far-end signal is empty or contains only far-end background noise.
[0015] Echo: divided into direct echo (linear echo) and indirect echo (non-linear echo).
[0016] Direct echo (linear echo): The echo obtained after the near-end speaker plays the far-end signal and is directly collected by the near-end microphone; the direct echo is not affected by the environment and has a lot to do with the distance and position from the speaker to the microphone. Therefore, the direct echo is a linear signal.
[0017] Indirect echo (non-linear echo): After the near-end speaker plays the far-end signal, the voice signal is reflected by the complex and changeable wall and then collected by the near-end microphone. The size of the indirect echo is related to factors such as the room environment, the placement of objects, and the wall attraction coefficient. Therefore, the indirect echo is a non-linear signal. Summary of the invention
[0018] In order to solve the above problems, the purpose of this application is: This method simulates the far-end signal in the training data as an approximate echo signal. In this way, the approximate echo signal (far-end signal) and the near-end microphone signal are then transmitted to the neural network. Because the echo signals in the far-end signal and the near-end microphone signal are already similar at this time, the process of predicting the echo path of the neural network is omitted, which can greatly reduce the burden of the neural network, thereby improving the network performance.
[0019] Specifically, the present invention provides a method for simulating a far-end signal as an echo signal, the method comprising the following steps:
[0020] S1.Manufacture standard audio, including:
[0021] First, create a full-band standard audio, with the amplitude changes of each frequency band being the same;
[0022] S2. Play and record standard audio on the target device, including:
[0023] Use the target device to play and record standard audio. Use the target device to play and record in an environment below 20 decibels. The recorded audio can be considered as an echo.
[0024] S3. Generate a fitting function, including:
[0025] A fitting function is generated by the amplitude of the standard audio and the amplitude of the recorded echo. This function represents the relationship between the amplitude of the audio played by the current device and the amplitude of the recorded echo in each frequency band.
[0026] S4. Simulate the far-end signal as an echo signal, including: passing the amplitude x of the far-end signal into the fitted function func to calculate the corresponding y value, where the y value is the amplitude of the simulated echo signal, thereby obtaining an approximate echo signal.
[0027] In the step S1, each frequency band of the standard audio includes 150 Hz as a frequency band.
[0028] In the step S1, manufacturing the standard audio further comprises:
[0029] A full-band standard audio is produced through Python's wave library. Assume that the sampling rate is 16kHz. The standard audio is divided into two seconds, and the amplitude changes of each segment are the same; the frequency of the first segment is 0HZ~150HZ, the second segment is 50HZ~200HZ, and so on. The last segment is 7850HZ~8000HZ.
[0030] The standard audio is not unique, and the frequency range, the degree of amplitude change and the length of each audio segment can be changed according to actual conditions.
[0031] In step S2, after changing the new target device, it is necessary to re-record and regenerate a new fitting function. This method is not universal for all devices because the echo paths of different devices are different, so the standard audio needs to be replayed and recorded after changing the device.
[0032] In step S3, the amplitude of the standard audio in different frequency bands is equal, but the amplitude of each frequency band of the recorded echo may vary to different degrees; the amplitude ratio of each frequency band is obtained by fitting a function.
[0033] The amplitude ratio of each frequency band is obtained by fitting a function. Assuming there are 128 frequency bands, there will be 128 corresponding fitting functions.
[0034] In the step S4, simulating the far-end signal as an echo signal further comprises:
[0035] First, the remote signal is converted into frequency domain data through Fourier transform using the torch.stft() function in the python library;
[0036] In the frequency domain, the amplitude of each frequency band of the far-end signal is multiplied by the fitting function, and the data obtained at this time is the echo signal in the frequency domain;
[0037] Then, through the inverse Fourier transform, the echo signal in the time domain is obtained using the torch.fft.irfft() function in the python library.
[0038] Therefore, the advantages of this application are:
[0039] 1. The traditional training method is to directly use the far-end signal and near-end microphone signal in the data set to pass into the neural network, and use the neural network to estimate the correlation between the far-end signal and the echo signal in the microphone signal. However, this method pre-processes the data first and simulates the far-end signal as an approximate echo signal, which eliminates the process of the neural network predicting the echo path, greatly reduces the burden of the neural network, and improves performance.
[0040] 2. This method is an optimization method for training data, which eliminates the process of using neural networks to predict echo paths, allowing the use of smaller models to achieve the same effect as traditional models; it is more friendly to devices with limited computing power. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0042] Figure 1 It is a flow chart of the present application method.
[0043] Figure 2 It is a standard audio time domain diagram.
[0044] Figure 3 It is a standard audio frequency domain diagram.
[0045] Figure 4 It is the time domain diagram of the recorded audio.
[0046] Figure 5 It is the frequency domain diagram of the recorded audio. DETAILED DESCRIPTION
[0047] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.
[0048] This application proposes a method for simulating a far-end signal as an echo signal. Figure 1 The flowchart for the implementation of this method is as follows:
[0049] Step S1. Producing standard audio:
[0050] It is necessary to produce a full-band standard audio with the same amplitude change in each frequency band; for example, to produce a time domain diagram of a standard audio with a sampling rate of 16kHz as shown in the attached figure. Figure 2 As shown in the attached spectrum Figure 3As shown. This standard audio is divided into two seconds, and the amplitude change of each segment is the same. The frequency of the first segment is 0HZ~150HZ, the second segment is 50HZ~200HZ, and so on, and the last segment is 7850HZ~8000HZ. The standard audio is not unique, and the range of the frequency band, the degree of amplitude change and the length of each audio segment can be changed according to actual conditions.
[0051] Step S2. Use the target device to play and record the standard audio: generate standard audio frequency domain data and recorded audio frequency domain data;
[0052] Use the target device to play and record standard audio. In a relatively quiet environment below 20 decibels, use the target device to play and record and save the audio. The recorded audio is the echo. For example, if the standard audio is played for 1 minute, a 1-minute echo will be recorded. However, this method is not applicable to all devices because the echo paths of different devices are different. Therefore, you need to play and record the standard audio again after changing the device.
[0053] Step S3. Generate fitting function:
[0054] Generate a fitting function through the amplitude of the standard audio and the amplitude of the recorded echo, and set the fitting function to a quadratic equation func: y = ax 2 +bx+c, x is the amplitude of each frequency band of the standard audio, y is the amplitude of each frequency band of the echo; through the python library function curve_fit(func,x,y), after passing in the three parameters of func, x, and y, the result returned by the curve_fit function is y=ax 2 +bx+c, the three values of a, b, and c are fitted, and the function func is obtained. This function represents the relationship between the amplitude of the audio played by the current device and the recorded echo in each frequency band;
[0055] Note that the func here is the same function, but when used as the parameter func, there are no values of a, b, and c, while the a, b, and c of the returned func have definite values.
[0056] The amplitude of standard audio is equal in different frequency bands, but the amplitude of recorded echo in each frequency band will vary to different degrees: Figure 4 , Attachment Figure 5 As shown in the figure. The amplitude of standard audio is equal in different frequency bands, but the amplitude of each frequency band of the recorded echo will vary to varying degrees; the amplitude ratio of each frequency band is obtained by fitting the function. For example, if there are 128 frequency bands, there will be 128 corresponding fitting functions.
[0057] After replacing a new target device, it is necessary to re-record and regenerate a new fitting function, and re-execute steps S2 and S3. In this method, steps S2 and S3 are not universal for all devices because the echo paths of different devices are different. Therefore, the standard audio must be replayed and recorded after changing the device.
[0058] Step S4. Simulate the far-end signal to be processed as an echo signal: Simulate any far-end signal as an echo signal, including: passing the amplitude x of the far-end signal into the fitted function func to calculate the corresponding y value, which is the amplitude of the simulated echo signal.
[0059] It further includes: first converting the far-end signal into frequency domain data through Fourier transform, and implementing Fourier transform through the torch.stft() function in the python library; in the frequency domain, multiplying the amplitude of each frequency band of the far-end signal by the fitting function, and the specific calculation process is the same as the above step S4. At this time, the data obtained is the amplitude of the echo signal in the frequency domain; then through inverse Fourier transform, the torch.fft.irfft() function in the python library implements inverse Fourier transform to obtain the echo signal in the time domain.
[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for simulating a far-end signal as an echo signal, characterized in that: The method comprises the following steps: S1. Manufacture standard audio, including: First, create a full-band standard audio, with the amplitude changes of each frequency band being the same; S2. Play and record standard audio on the target device, including: Use the target device to play and record standard audio. Use the target device to play and record and save in an environment below 20 decibels. The recorded audio is the echo. S3. Generate a fitting function, including: Generate a fitting function through the amplitude of the standard audio and the amplitude of the recorded echo, and set the fitting function to a quadratic equation func: y = ax 2 +bx+c, x is the amplitude of each frequency band of the standard audio, y is the amplitude of each frequency band of the echo; through the python library function curve_fit(func,x,y), after passing in the three parameters func, x, and y, the result returned by the curve_fit function is func: y=ax 2 +bx+c, the three values of a, b, and c are fitted, and the function func is obtained. This function represents the relationship between the amplitude of the audio played by the current device and the recorded echo in each frequency band; S4. Simulate any remote signal as an echo signal, including: passing the amplitude x of the remote signal into the fitted function func to calculate the corresponding y value, where the y value is the amplitude of the simulated echo signal.
2. A method for simulating a far-end signal as an echo signal according to claim 1, characterized in that: In the step S1, each frequency band of the standard audio includes 150 Hz as a frequency band.
3. The method for simulating a far-end signal as an echo signal according to claim 1, characterized in that: In the step S1, manufacturing the standard audio further comprises: A full-band standard audio is produced through Python's wave library. Assume that the sampling rate is 16kHz. The standard audio is divided into two seconds, and the amplitude changes of each segment are the same; the frequency of the first segment is 0HZ~150HZ, the second segment is 50HZ~200HZ, and so on. The last segment is 7850HZ~8000HZ.
4. A method for simulating a far-end signal as an echo signal according to claim 3, characterized in that: The standard audio is not unique, and the frequency range, the degree of amplitude change and the length of each audio segment can be changed according to actual conditions.
5. The method for simulating a far-end signal as an echo signal according to claim 1, characterized in that: After replacing a new target device, it is necessary to re-record and regenerate a new fitting function, and re-execute steps S2 and S3. In this method, steps S2 and S3 are not universal for all devices because the echo paths of different devices are different. Therefore, the standard audio must be replayed and recorded after changing the device.
6. The method for simulating a far-end signal as an echo signal according to claim 1, characterized in that: In step S3, the amplitude of the standard audio in different frequency bands is equal, but the amplitude of each frequency band of the recorded echo may vary to different degrees; the amplitude ratio of each frequency band is obtained by fitting a function.
7. The method for simulating a far-end signal as an echo signal according to claim 1, characterized in that: The amplitude ratio of each frequency band is obtained by fitting a function. Assuming there are 128 frequency bands, there will be 128 corresponding fitting functions.
8. The method for simulating a far-end signal as an echo signal according to claim 1, characterized in that: In the step S4, simulating the far-end signal as an echo signal further comprises: first converting the far-end signal into frequency domain data by Fourier transform, and implementing Fourier transform by torch.stft() function in Python library; In the frequency domain, multiply the amplitude of each frequency band of the remote signal by the fitting function. Assume that in the 1000 Hz frequency band, the fitting function func is y = x 2 +0.5x+0.01, the amplitude of the far-end signal is 0.2, that is, x is 0.2, and y=0.15 is obtained by substituting it into the formula, that is, the amplitude of the echo signal simulated in the 1000HZ frequency band is 0.15, and the same is true for other frequency bands; the data obtained at this time is the amplitude of the echo signal in the frequency domain; Then, through the inverse Fourier transform, the torch.fft.irfft() function in the python library implements the inverse Fourier transform and obtains the echo signal in the time domain.