Audio processing method, terminal and storage medium

Abstract

The invention discloses an audio processing method, a terminal and a storage medium, and belongs to the technical field of audio. The method comprises: acquiring an original audio signal and a reference signal; the original audio signal is a mixed signal with a nonlinear echo signal, a linear echo signal and a pure voice signal, and the reference signal is a signal from an opposite-end microphone in a call scene or a background sound signal played by a playing device in a voice recognition scene. A linear relation exists between the reference signal and the linear echo signal, the linear echo signal is eliminated from the original audio signal through linear filtering, and a candidate audio signal is obtained. The relation between the reference signal and the nonlinear echo signal is learned through a generative network model, the generative network model continuously adds noise into a pure voice signal through a forward diffusion process to generate a candidate audio signal, and the backward diffusion process is an inverse process; the candidate audio signals are de-noised through the back diffusion process to output pure voice signals, and the audio processing effect is improved.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to an audio processing method, terminal, and storage medium. Background Art

[0002] Acoustic Echo Cancellation (AEC) algorithms are commonly used in audio processing and audio communications, such as voice calls and human-computer interaction scenarios. During a voice call, AEC algorithms can preemptively eliminate echoes at the near end before transmitting them to the far end. This prevents the far-end speaker from hearing their own voice (i.e., echo), resolving the issue of either speaker hearing their own voice during a call. During human-computer interaction, AEC algorithms can preemptively eliminate sounds played by smart devices, preventing them from interfering with speech recognition, such as music or speech.

[0003] Currently, due to limitations in the device's hardware, such as acoustic cavity design and speaker quality, speech signals are not processed in a completely linear manner. Consequently, the echo signal includes not only linear but also nonlinear echo signals. The AEC algorithm is designed to eliminate linear echoes but not nonlinear ones. Nonlinear echo signals remain in the AEC output signal, so residual echo cancellation (REC) is often required.

[0004] In related technologies, deep learning methods can be used to learn the output signal of the AEC, estimate the demarcation point between the nonlinear echo signal and the clean speech signal, and then distinguish the nonlinear echo signal from the clean speech signal based on this demarcation point. However, this approach can cause excessive allocation of the clean speech signal to the nonlinear echo signal, resulting in over-cancellation, or excessive allocation of the nonlinear echo signal to the clean speech signal, resulting in incomplete speech processing, resulting in poor audio processing performance. Summary of the Invention

[0005] This application provides an audio processing method, terminal, and storage medium to improve audio processing effects. The technical solution is as follows:

[0006] In a first aspect, an audio processing method is provided, comprising: obtaining an original audio signal and a reference signal; the original audio signal is a mixed signal comprising a nonlinear echo signal, a linear echo signal, and a clean speech signal; the clean speech signal is the final signal after the linear echo signal and the nonlinear echo signal have been eliminated. The reference signal is a signal from a microphone on the other end in a call scenario, or a background sound signal played by a playback device in a speech recognition scenario. There is a linear relationship between the reference signal and the linear echo signal; the linear echo signal can be eliminated from the original audio signal through linear filtering to obtain a candidate audio signal; the candidate audio signal is a signal with a residual nonlinear echo signal; that is, the candidate audio signal is a mixed signal comprising a nonlinear echo signal and a clean speech signal. The relationship between the reference signal and the nonlinear echo signal is learned using a generative network model. The generative network model generates the candidate audio signal by continuously adding noise to the clean speech signal through a forward diffusion process. The backward diffusion process is the inverse of the forward diffusion process. When eliminating residual nonlinear echo signals from a candidate audio signal, the candidate audio signal and a reference signal are fed into a generative network model. The model then uses a back-diffusion process to continuously remove noise from the candidate audio signal based on the reference signal, thereby generating a clean speech signal. Unlike related approaches that use a cutoff point estimation, this approach does not simply separate the clean speech signal from the nonlinear echo signal based on the cutoff point. Instead, it uses a back-diffusion process to denoise the candidate audio signal and output a clean speech signal. This reduces the probability of over-cancellation and under-processed speech, improving audio processing performance.

[0007] In the embodiment of the present application, the audio processing method can be applied to call scenarios and audio recognition scenarios. In a call scenario, after the voice of speaker A at the other end (corresponding to the microphone signal) is transmitted to the local end B, it is played through the speaker of the local device. After a series of acoustic reflections, it is collected by the microphone of the local device, and combined with the microphone signal of the local end B (corresponding to the pure voice signal that is ultimately desired), it is transmitted to the other end A. As a result, speaker A at the other end can hear his or her own voice again in a very short time. There is a linear relationship between the microphone signal from the other end and the linear echo signal, and the linear echo signal in the original audio signal can be eliminated by estimating the echo path.

[0008] In speech recognition scenarios, the sound played through the device's speakers (corresponding to the background sound signal) undergoes a series of acoustic reflections before being picked up by the device's microphone. This, combined with the microphone signal of the user's actual request (e.g., powering off, switching on), (corresponding to the desired pure speech signal), is then transmitted to the device. This can result in the device being unable to correctly recognize the microphone signal from the user. The background sound signal from the device and the linear echo signal are linearly related. This can be eliminated from the original audio signal by estimating the echo path to produce a candidate audio signal.

[0009] Due to the nonlinear characteristics of acoustic devices, the microphone signal from the other end, played through the local speaker, or the background sound signal from the playback device, played through the speaker, undergoes a series of acoustic reflections, generating not only linear but also nonlinear echo signals. Therefore, it is necessary to eliminate nonlinear echo signals. In this approach, a large language model is applied to the audio processing process. A generative network model is pre-trained using a forward diffusion process. Noise is continuously added to the clean speech signal to generate candidate audio signals. This training is iterative until the generative network model can predict the noise added to the candidate audio signals. The trained generative network model then performs the reverse of the forward diffusion process (i.e., backward diffusion). By continuously predicting the noise signal in the candidate audio signals, the noise is removed from the candidate audio signals to generate the clean speech signal. This approach does not rely on an estimated cutoff point and eliminates the need to separate the clean speech signal from the nonlinear echo signal. This reduces the probability of over-elimination and incomplete speech processing, thereby improving audio processing performance.

[0010] In one possible implementation, the generative network model includes an attention layer and a diffusion model. The attention layer determines an attention vector based on a reference signal. The candidate audio signal and the attention vector are then fed into the diffusion model, where a back-diffusion process is used to generate a clean speech signal.

[0011] In an embodiment of the present application, the attention layer can implement an attention mechanism (attention), and can weight different positions according to the importance of different positions in the input sequence data, so as to better capture the information of the sequence data. The candidate audio signal is a mixed signal with a nonlinear echo signal and a pure speech signal. The weight of the reference signal for the pure speech signal in the candidate audio signal is learned by the attention layer, which can be understood as extracting information in the reference signal that is associated with the predicted pure speech signal. The learned attention vector and the candidate audio signal are then input into the diffusion model. The diffusion model continuously predicts the noise signal in the candidate audio signal based on the attention vector, and then denoises the candidate audio signal to generate a pure speech signal.

[0012] By adding an attention layer to learn the influence of the reference signal on the pure speech signal in the candidate audio signal, and inputting the attention vector learned by the attention layer into the diffusion model, the diffusion model generates a more accurate pure speech signal through the reverse diffusion process, thereby improving the audio processing effect.

[0013] In one possible implementation, the generative network model includes an attention layer. The above-mentioned method for obtaining the attention vector can include the following two examples. Example 1: Input a reference signal into the attention layer, and use the attention layer to learn the weight of the reference signal relative to the clean speech signal in the candidate audio signal, and output an attention vector. Example 2: First, perform voiceprint extraction on the reference signal to obtain voiceprint features; then input the voiceprint features into the attention layer, and use the attention layer to learn the weight of the voiceprint features relative to the clean speech signal in the candidate audio signal, and output an attention vector.

[0014] In an embodiment of the present application, in one method, a reference signal is directly input into the attention layer, and the attention layer learns the weight of the reference signal for the pure speech signal in the candidate audio signal, and outputs an attention vector. In another method, the reference signal is first subjected to voiceprint extraction to obtain a voiceprint feature. The voiceprint feature is a high-dimensional vector that can better characterize the speaker's voice identification than the reference signal itself. The voiceprint feature is input into the attention layer, and the attention layer learns the weight of the voiceprint feature for the pure speech signal in the candidate audio signal, and outputs an attention vector. By introducing the voiceprint feature method, the generative network model can better utilize the voiceprint feature to complete the elimination of nonlinear echo signals, thereby improving the audio processing effect.

[0015] In one possible implementation, the reverse diffusion process implemented by the above-mentioned diffusion model is as follows: the candidate audio signal and the attention vector are input into the diffusion model, and in the reverse prediction process of the first time step, the noise corresponding to the noise added in the forward diffusion process is added, and the first predicted speech signal is output after the denoising process of the first time step; the noise corresponding to the noise added in the reverse prediction process of the second time step is added, and the second predicted speech signal is output after the denoising process of the second time step; until in the reverse prediction process of the last time step, the noise corresponding to the noise added in the forward diffusion process is added, and the pure speech signal is output after the denoising process of the last time step.

[0016] In the embodiments of the present application, the output of the diffusion model at the previous time step is taken as the input of the next time step. For example, the first predicted speech signal output at the first time step is taken as the input of the second time step. The time step is used to describe the time relationship between data points in sequence data, and can describe the time sequence of different events or behaviors. The step length between adjacent time steps can be appropriately set by those skilled in the art according to actual conditions.

[0017] The reverse diffusion process is the inverse process of the forward diffusion process. The forward diffusion process is a process of adding noise, that is, adding noise to the original data to obtain noise data. The reverse prediction process is a process of removing noise, that is, subtracting noise from noise data, that is, superimposing the noise on the noise data. The reverse prediction process is essentially also a process of adding noise. The candidate audio information is subtracted from the noise to obtain the original data. Both the forward diffusion process and the reverse diffusion process require noise. Therefore, after the diffusion model is trained, the noise corresponding to the noise added in the forward diffusion process at each time step also needs to be added in the reverse prediction process.

[0018] For example, the number of time steps of the forward diffusion process and the reverse diffusion process is the same. If noise is added to each time step in the forward diffusion process, noise is also added to each time step in the reverse prediction process. If the noise is divided according to a preset rule and then added to each time step in the forward diffusion process, the noise is also divided according to the preset rule and then added to each time step in the reverse prediction process. If noise is randomly added to each time step in the forward diffusion process, noise is also randomly added to each time step in the reverse prediction process. The embodiments of the present application do not limit this, as long as the noise corresponding to the noise added in the forward diffusion process at each time step is added in the reverse prediction process.

[0019] In the embodiments of the present application, the signal after removing noise at the previous time step in the diffusion model is taken as the input of the next time step, and noise removal is performed again. The noise signal in the candidate audio signal is continuously removed through multiple time steps to generate a pure speech signal. The pure speech signal and the nonlinear echo signal do not need to be divided, and the probability of over-removal and incomplete processing of the speech is reduced, thereby improving the audio processing effect.

[0020] In a possible implementation, when performing linear filtering on the original audio signal, the linear adaptive filter can be used to perform linear filtering on the original audio signal according to the reference signal. Based on this, after obtaining the pure speech signal, the audio processing method can further include: determining a filter coefficient according to the pure speech signal; adjusting a parameter of the linear adaptive filter according to the filter coefficient; and the adjusted linear adaptive filter is used for the next linear filtering process.

[0021] In the related art, a linear adaptive filter is used to linearly filter an original audio signal according to a reference signal to obtain a candidate filtered signal, which is considered as the final required signal, and then the parameters of the linear adaptive filter are adjusted using the candidate audio signal. Since the candidate filtered signal is a mixed signal including a nonlinear echo signal and a clean speech signal, the filter coefficients used to update the linear adaptive filter are not accurate. Further, since the related art uses a method of finding a demarcation point between the nonlinear echo signal and the clean speech signal for elimination, the elimination effect is poor, and thus, even if the linear adaptive filter is updated using the signal after nonlinear echo elimination in the related art, the adjustment process of the linear adaptive filter is not appropriate.

[0022] In the embodiments of the present application, the parameters of the linear adaptive filter are updated using the clean speech signal after the linear echo signal and the nonlinear echo signal are eliminated, and the updated linear adaptive filter is used for the next linear filtering process, which can improve the accuracy of the next linear filtering result.

[0023] In a possible implementation, the audio processing method can be applied to a speech recognition scenario. In the speech recognition scenario, after obtaining the clean speech signal, the audio processing method can further include: performing speech recognition on the clean speech signal to obtain a speech recognition result; and responding or processing according to the speech recognition result.

[0024] In the speech recognition scenario, the audio processing method can be applied to a playback device controlled or woken up by speech, such as a smart speaker, a smart TV, a sweeping robot, a smart butler, a voice assistant, and the like.

[0025] The playback device performs the audio processing method provided in the embodiments of the present application, linearly eliminates the linear echo signal and the nonlinear echo signal from the original audio signal according to the background sound signal to obtain a clean speech signal, and then performs speech recognition on the clean speech signal to obtain a speech recognition result, and responds to the speech recognition result by responding (for example, shutting down, answering "Hello, I'm here") or related processing (for example, switching to the next song, switching to the song mentioned by the user).

[0026] In one possible implementation, the initial generative network model includes an initial attention layer and an initial diffusion model. The audio processing method also includes a training process for the generative network model (i.e., a forward diffusion process). The training process includes: obtaining clean speech signal samples, reference signal samples, and noise signal samples. The reference signal samples are signal samples from the other end microphone in a call scenario or background sound signal samples played by the playback device in a speech recognition scenario. The noise signal samples include white noise samples and nonlinear echo signal samples. The weights of the reference signal samples are learned by the initial attention layer to obtain attention vector samples. The clean speech signal samples, attention vector samples, and noise signal samples are input into the initial diffusion model, and the audio signal samples are generated by the initial diffusion model through a forward diffusion process. Then, according to the loss function, the weights of the reference signal learned by the initial attention layer and the parameters of the initial diffusion model are adjusted until the deviation between the noise predicted by the initial generative network model based on the audio signal samples and the noise signal samples meets the preset conditions, thereby obtaining a generative network model.

[0027] In an embodiment of the present application, the weights of the reference signal samples are learned by the initial attention layer to obtain attention vector samples. The attention vector samples are then also used as noise and input into the initial diffusion model together with the pure voice signal samples and the noise signal samples. That is, during the forward diffusion process, noise is continuously added to the pure voice signal samples to generate audio signal samples. In this way, combined with the loss function, continuous iterative training is performed to update the weights of the reference signal learned by the initial attention layer and the parameters of the initial diffusion model until the initial generative network model can predict the noise added to the audio signal samples. In other words, training is continued until the deviation between the noise predicted by the initial generative network model based on the audio signal samples and the noise signal samples is small, indicating that the initial generative network model has converged, and a trained generative network model is obtained. The generative network model can eliminate nonlinear echo signals of the candidate audio signal to obtain a pure voice signal, thereby improving the audio processing effect.

[0028] The diffusion model is a type of generative network that gradually obfuscates the original data by adding noise, then learns to reverse the noise process to reconstruct the original data. Diffusion models have the characteristic of diverse outputs. To enable the diffusion model to generate a final clean speech signal, rather than a diverse set of clean speech signals, the noise signal samples added during the forward diffusion process also include nonlinear echo signal samples. This allows the diffusion model to be trained, allowing the reverse diffusion process to eliminate the nonlinear echo signal and generate a clean speech signal.

[0029] In one possible implementation, the forward diffusion process implemented by the above-mentioned diffusion model is as follows: the clean speech signal samples and the attention vector samples are input into the initial diffusion model, and in the forward prediction process of the first time step, the noise signal samples are added, and the first noisy signal is output after noise processing; the noise signal samples are continued to be added in the forward prediction process of the second time step, and the second noisy signal is output after noise processing; until in the forward prediction process of the last time step, the noise signal samples are added, and the audio signal samples are output after noise processing.

[0030] In this embodiment of the present application, the output of the initial diffusion model at the previous time step serves as the input for the next time step. For example, the first noisy signal output at the first time step serves as the input for the second time step. During the forward diffusion process, noise needs to be added at each time step.

[0031] For example, the noise may be added to each time step separately, or the noise may be divided according to a preset rule and then added to each time step separately, or the noise may be randomly added to each time step. This embodiment of the present application is not limited to this, as long as the noise corresponding to the noise added in the forward prediction process is added in the reverse prediction process.

[0032] In this embodiment, the signal after adding noise in the diffusion model at the previous time step serves as the input for the next time step, where noise is again superimposed. Over multiple time steps, noise signal samples are continuously added to the clean speech signal samples to generate audio signal samples. Training is then completed through a continuous iterative process, resulting in a trained generative network model. This generative network model is used to eliminate nonlinear echo signals from candidate audio signals, resulting in a clean speech signal and improving audio processing performance.

[0033] In one possible implementation, the initial generative network model includes an initial attention layer. The aforementioned method for obtaining an attention vector sample may include the following two examples. Example 1: Input a reference signal sample into the initial attention layer, learn the weights of the reference signal sample through the initial attention layer, and output an attention vector sample. Example 2: Perform voiceprint extraction on the reference signal sample to obtain a voiceprint feature sample; then input the voiceprint feature sample into the initial attention layer, learn the weights of the reference signal sample through the initial attention layer, and output an attention vector sample.

[0034] In an embodiment of the present application, in one method, the reference signal sample is directly input into the initial attention layer, the weight of the reference signal sample is learned through the initial attention layer, and the attention vector sample is output. In another method, the reference signal sample is first subjected to voiceprint extraction to obtain a voiceprint feature sample. The voiceprint feature sample is a high-dimensional vector that can better characterize the speaker's voice identification than the reference signal sample itself. The voiceprint feature sample is input into the initial attention layer, the weight of the voiceprint feature sample is learned through the initial attention layer, and the attention vector sample is output. By introducing the method of voiceprint feature samples, the initial generative network model can be better trained. The trained generative network model can then better utilize the voiceprint features to eliminate nonlinear echo signals and improve the audio processing effect.

[0035] In one possible implementation, the generative network model training process also includes acquiring nonlinear echo signal samples. Audio samples are acquired during a silent call or background music playback scenario. The audio samples are mixed signals consisting of linear and nonlinear echo signal samples. The audio samples are linearly filtered using a linear adaptive filter based on a reference signal sample to obtain nonlinear echo signal samples.

[0036] In the embodiments of the present application, the audio processing method can be applied to call scenarios and audio recognition scenarios. In a call scenario, when the local end is muted, the voice of speaker A on the other end (corresponding to the reference signal sample, i.e., the signal sample of the microphone) is transmitted to the local end B, played back through the speaker of the local device, and after a series of acoustic reflections, is collected by the microphone of the local device and transmitted to the other end A. The audio sample obtained when the local end is muted is a mixed signal including linear echo signal samples and nonlinear echo signal samples. The audio sample is then linearly filtered according to the reference signal sample using a linear adaptive filter to obtain a nonlinear echo signal sample.

[0037] In speech recognition scenarios, when the user is silent (not speaking), the sound played through the playback device's speakers (corresponding to the reference signal sample, i.e., the background sound signal sample) undergoes a series of acoustic reflections before being collected by the playback device's microphone and transmitted back to the playback device. The audio sample obtained when the user is silent is a mixed signal consisting of linear echo signal samples and nonlinear echo signal samples. A linear adaptive filter is then used to linearly filter the audio sample based on the reference signal sample to produce a nonlinear echo signal sample.

[0038] In an embodiment of the present application, in a call scenario and a voice recognition scenario, audio samples can be obtained by muting the local end and the user. The audio samples do not include pure voice signal samples. After linearly filtering the audio samples according to the reference signal samples using a linear adaptive filter, nonlinear echo signal samples can be obtained without going through complex extraction steps, thereby reducing the complexity of obtaining nonlinear echo signal samples and increasing the diversity of acquisition methods.

[0039] In a second aspect, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method as described in the first aspect or any one of the implementations of the first aspect is implemented.

[0040] In a third aspect, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method involved in the first aspect or any one of the implementations of the first aspect.

[0041] In a fourth aspect, an embodiment of the present application provides a chip system, which can be applied to a terminal. The chip system includes one or more processors, which are used to call computer instructions to enable the terminal to execute the method involved in the first aspect or any one of the implementations of the first aspect.

[0042] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method according to the first aspect or any one of the implementations of the first aspect.

[0043] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a schematic diagram of a call scenario provided by an embodiment of the present application;

[0045] Figure 2 is a schematic diagram of a speech recognition scenario provided by an embodiment of the present application;

[0046] Figure 3 is a schematic diagram of an echo cancellation process provided by an embodiment of the present application;

[0047] Figure 4 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0048] Figure 5This is a block diagram of a software system of a terminal provided in an embodiment of the present application;

[0049] Figure 6 is a schematic diagram of a diffusion model provided in an embodiment of the present application;

[0050] Figure 7 This is a schematic diagram of a training process of a generative network model provided in an embodiment of the present application;

[0051] Figure 8 is a schematic diagram of a forward diffusion process provided in an embodiment of the present application;

[0052] Figure 9 is a schematic diagram of another forward diffusion process provided in an embodiment of the present application;

[0053] Figure 10 This is a flowchart of an audio processing method provided by an embodiment of the present application;

[0054] Figure 11 is a schematic diagram of a reverse diffusion process provided in an embodiment of the present application;

[0055] Figure 12 is a schematic diagram of another reverse diffusion process provided in an embodiment of the present application;

[0056] Figure 13 This is a schematic diagram of adjusting a filter coefficient provided in an embodiment of the present application;

[0057] Figure 14 This is a schematic diagram of a device interaction process provided by an embodiment of the present application;

[0058] Figure 15 This is a schematic diagram of another device interaction process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0060] It should be understood that the “multiple” mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, words such as “first” and “second” are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as “first” and “second” do not limit the quantity and execution order, and words such as “first” and “second” do not necessarily limit them to be different.

[0061] The audio processing method provided in the embodiments of the present application is applied to echo cancellation in scenarios such as call scenarios or voice recognition scenarios. The following examples are given for call scenarios and voice recognition scenarios respectively.

[0062] In a call scenario, Figure 1 As shown, Figure 1 This is a schematic diagram of a call scenario provided by an embodiment of the present application. For example, the far-end speaker is the other end of the call and the near-end speaker is the active end. After the far-end speaker's voice is transmitted to the near-end device, it is played back by the speaker of the near-end device (e.g., a receiver or speaker). After a series of acoustic reflections, it is picked up by the microphone of the near-end device and transmitted back to the far-end device. Acoustic echo will cause the far-end speaker to hear their own previous voice again for a short period of time.

[0063] In speech recognition scenarios, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a speech recognition scenario provided by an embodiment of the present application. For example, the playback device is smart speaker A. Smart speaker A is playing music, and the user asks smart speaker A to switch to another song: "Little A, Little A, play the next song." The sound played by the playback device through the speaker (such as an earpiece or speaker) undergoes a series of acoustic reflections before being picked up by the device's microphone and transmitted to the playback device. This can result in the playback device being unable to correctly recognize the user's actual request.

[0064] Traditional echo cancellation algorithms (i.e., AEC algorithms) estimate the echo path from the speaker to the microphone using a linear filter. The linear echo signal can be understood as: X = S × RIR. Here, X represents the signal picked up by the microphone of the near-end device after the speaker of the near-end device plays the voice of the far-end speaker, undergoes a series of acoustic reflections, S represents the voice of the far-end speaker, and RIR represents the echo path. These three are linearly related. In other words, using air as the medium, the echo is reflected and superimposed multiple times in the air before being sampled by the microphone. Therefore, the linear echo cancellation component can be modeled using a linear filter to eliminate the linear echo.

[0065] Since the nonlinear echo cancellation part is related to the device hardware, it is impossible to perform effective estimation through the traditional echo cancellation algorithm. In the related art, an echo cancellation algorithm based on the traditional linear filter model combined with the neural network (NN) model is proposed. The linear echo is estimated by the normalized least mean square algorithm (NLMS) or Kalman filter (kalman), and after eliminating the linear echo signal, the candidate audio signal is obtained, and then the nonlinear echo formed by the device is estimated by the NN model. The NN model directly estimates the nonlinear echo based on the candidate audio signal (which is a mixed signal including a pure voice signal and a nonlinear echo signal), which will cause over-elimination or unclear voice processing.

[0066] For example, as mentioned above Figure 1 As shown in Figure 1, the far-end speech signal x(n) is broadcast from the near-end device's speaker and propagates through space (Room Impulse Response) before being picked up by the near-end device's microphone, i.e., y(n). Simultaneously, the near-end speech signal s(n) also enters the near-end device's microphone. The microphone picks up the superposition of the two signals, y(n) and s(n), i.e., d(n). The linear adaptive filter uses the estimated echo path Process x(n) to obtain the estimated echo signal d(n) and Input linear superposition processor, d(n) and The difference is taken as the error e(n). The error e(n) is the signal after the linear echo signal is eliminated, that is, the candidate audio signal. Then, the error e(n) is subjected to nonlinear residual echo cancellation, and the resulting signal is played through the far-end speaker. The error e(n) is passed to the linear adaptive filter, and the filter coefficients are iteratively updated to estimate the echo path. Approximate the true echo path w(n).

[0067] Among them, the above Figure 1Double Talk Detect (DTD) in the system is used to detect call states, including near-end single talk, far-end single talk, and double talk. Different call states correspond to different parameter indicators. In the near-end single talk scenario, the iterative update of the linear adaptive filter parameters needs to keep the output consistent with the microphone signal to the greatest extent possible. In the far-end single talk scenario, the iterative update of the adaptive filter parameters needs to suppress the echo to the greatest extent possible. In the double talk scenario, the iterative update of the linear adaptive filter parameters needs to suppress the echo as much as possible while retaining the near-end speech. The linear adaptive filter is used to eliminate the impact of the echo path on x(n), with the goal of minimizing the error e(n).

[0068] Based on the above Figure 1 ,like Figure 3 As shown, Figure 3 This is a schematic diagram of echo cancellation provided by an embodiment of the present application. Figure 3 The original audio signal in corresponds to the above Figure 1 The estimated echo signal corresponds to the d(n) in Figure 1 The linear adaptive filter output in The linear superposition processor performs the calculation of d(n) and Processing is performed to obtain the error e(n). The error e(n) is the signal after the linear echo signal is eliminated, that is, the candidate audio signal. The error e(n) is input into a nonlinear processor for nonlinear residual echo cancellation to obtain the clean speech signal s(n). If the near-end is silent, the resulting signal is silence.

[0069] Among them, based on the causes of communication echo, echo can be divided into acoustic echo (Acoustic Echo) and circuit echo (Line Echo). Correspondingly, echo cancellation technology includes acoustic echo cancellation (AEC) and circuit echo cancellation (Line Echo Cancellation, LEC). Acoustic echo is caused by the sound of the speaker being fed back to the microphone multiple times, that is, y(n), and circuit echo is caused by the matching coupling of physical electronic circuits. Based on this, the above Figure 3 The linear superposition processor can also process the circuit echo signal, that is, the linear superposition processor can process d(n), And the circuit echo signal is processed to obtain the error e(n).

[0070] It should be noted that the above Figure 3 The echo cancellation method shown in the same way is applicable to Figure 2 , Figure 3The original audio signal in the example is a mixture of an echo signal and the user's switching request. A linear adaptive filter is used to estimate the echo signal based on the background sound output from the playback device. A linear superposition processor processes the original audio signal and the echo signal output by the linear adaptive filter to obtain a signal after eliminating the linear echo signal, i.e., the candidate audio signal. The candidate audio signal is input into a nonlinear processor for nonlinear residual echo cancellation to obtain a pure speech signal (i.e., the user's switching request). If the user is silent, a silent signal is obtained.

[0071] Related echo cancellation methods use signal processing to estimate linear echoes. The signal after eliminating the linear echo from the original audio signal is called the "error signal" or "linear echo-removed signal," or candidate audio signal. The candidate audio signal is a mixture of a pure speech signal and a nonlinear echo signal. The nonlinear echo signal is then predicted using a reference signal, but this prediction is inaccurate. Consequently, related residual echo cancellation schemes are detrimental to the speech spectrum. At high volume levels, the nonlinearity of the device significantly degrades the near-field target voice, significantly impacting both the listening experience during phone calls and the performance of speech recognition.

[0072] To address the problem of over-elimination or incomplete speech processing in the aforementioned related technologies, resulting in poor audio processing performance, an embodiment of the present application provides an audio processing method. The method first obtains an original audio signal and a reference signal. The original audio signal is a mixture of a nonlinear echo signal, a linear echo signal, and a clean speech signal. The clean speech signal is the final signal after the linear and nonlinear echo signals have been eliminated. The reference signal is the signal from the other end's microphone in a call scenario or the background sound signal played by the playback device in a speech recognition scenario. There is a linear relationship between the reference signal and the linear echo signal. Then, the linear echo signal is eliminated from the original audio signal through linear filtering to obtain a candidate audio signal. The candidate audio signal is a signal with residual nonlinear echo signals; in other words, the candidate audio signal is a mixture of a nonlinear echo signal and a clean speech signal. The relationship between the reference signal and the nonlinear echo signal is learned using a generative network model. The generative network model generates the candidate audio signal by continuously adding noise to the clean speech signal through a forward diffusion process. The backward diffusion process is the inverse of the forward diffusion process. When eliminating the residual nonlinear echo signal of the candidate audio signal, the candidate audio signal and the reference signal are input into the generative network model. The generative network model undergoes a reverse diffusion process and continuously removes noise from the candidate audio signal according to the reference signal, thereby generating a pure speech signal.

[0073] Unlike related art approaches that estimate a cutoff point, this approach doesn't simply separate clean speech signals from nonlinear echo signals based on the cutoff point. This reduces the likelihood of over-cancellation and incomplete speech processing. In other words, while related art neural network models identify the cutoff point in the data, this approach learns the data distribution and uses a back-diffusion process to denoise candidate audio signals, outputting a clean speech signal, improving audio processing performance.

[0074] Furthermore, this application utilizes a diffusion model to perform multiple denoising operations on the error signal (i.e., the candidate audio signal) to generate a clean speech signal. This differs from the related art, which uses an echo cancellation algorithm to estimate the residual echo portion to separate the clean speech signal from the residual echo portion. Therefore, this solution also improves the anti-interference ability of the echo.

[0075] It should be noted that the specific algorithm of the audio processing method will be described in the following Figure 6-Figure 13 The embodiments are described in detail in the illustrated embodiment, and the embodiments of the present application are not described in detail here.

[0076] The audio processing method provided in the embodiments of the present application can be applied to terminals with call functions, such as tablets, desktop computers, laptops, PDAs, mobile phones, tablet computers, wireless terminal devices, communication devices, and other electronic devices. It can also be applied to playback devices that are controlled or awakened by voice, such as smart speakers, smart TVs, robot vacuums, smart housekeepers, voice assistants, and other smart devices.

[0077] Before explaining the audio processing method provided in the embodiment of the present application in detail, the terminal involved in the embodiment of the present application is first explained.

[0078] Figure 4 This is a schematic diagram of the structure of a terminal provided by an embodiment of the present application. Figure 4 The terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc.

[0079] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0080] The processor 110 may include one or more processing units, for example, an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0081] The controller may be the nerve center and command center of the terminal 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0082] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0083] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0084] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely illustrative and does not constitute a structural limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0085] The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0086] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0087] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the terminal 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0088] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied on the terminal 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0089] In some embodiments, antenna 1 and mobile communication module 150 of terminal 100 are coupled, and antenna 2 and wireless communication module 160 are coupled, so that terminal 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0090] Terminal 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0091] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), or the like. In some embodiments, the terminal 100 can include one or N display screens 194, where N is an integer greater than 1.

[0092] The digital signal processor is configured to process digital signals. In addition to processing digital image signals, the digital signal processor can also process other digital signals. For example, when the terminal 100 selects a frequency point, the digital signal processor is configured to perform Fourier transform on the frequency point energy, and the like.

[0093] The video codec is configured to compress or decompress digital videos. The terminal 100 can support one or more video codecs. In this way, the terminal 100 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and the like.

[0094] The NPU is a neural-network (NN) computing processor. By drawing on the structure of a biological neural network, such as the transmission mode between human brain neurons, the NPU can quickly process input information and continuously self-learn. Through the NPU, the terminal 100 can implement intelligent cognitive applications, such as image recognition, face recognition, voice recognition, text understanding, and the like.

[0095] The external memory interface 120 can be configured to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, music, video, and the like files can be saved in the external memory card.

[0096] The internal memory 121 can be used to store computer executable program codes including instructions. The processor 110 performs various functional applications and data processing of the terminal 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. The data storage area can store data (such as audio data, a phone book, etc.) created by the terminal 100 during use, and the like. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0097] The terminal 100 can realize audio functions such as music playing, recording, etc. through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc.

[0098] The audio module 170 is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or part of the functions of the audio module 170 can be disposed in the processor 110.

[0099] The speaker 170A, also known as a "loudspeaker", is used to convert an audio electrical signal into a sound signal. The terminal 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0100] The receiver 170B, also known as a "earpiece", is used to convert an audio electrical signal into a sound signal. When the terminal 100 answers a call or a voice message, the voice can be heard by placing the receiver 170B close to the ear.

[0101] The microphone 170C, also known as a "microphone", "sound transducer", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can make a sound by placing the mouth close to the microphone 170C, and input the sound signal into the microphone 170C. The terminal 100 can be provided with at least one microphone 170C. In other embodiments, the terminal 100 can be provided with two microphones 170C, in addition to collecting sound signals, it can also realize a noise reduction function. In other embodiments, the terminal 100 can also be provided with three, four or more microphones 170C, to realize the collection of sound signals, noise reduction, and can also identify the source of the sound, realize the function of directional recording, etc.

[0102] A touch sensor, also known as a "touch panel," can be provided on display screen 194. The touch sensor and display screen 194 form a touch screen, also known as a "touch screen." The touch sensor is used to detect touch operations applied to or near the touch sensor. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided via display screen 194. In other embodiments, the touch sensor can also be provided on the surface of terminal 100, in a location different from that of display screen 194.

[0103] Next, the software system of the terminal 100 will be described.

[0104] The software system of the terminal 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to exemplify the software system of the terminal 100.

[0105] Figure 5 This is a block diagram of a software system of a terminal 100 provided in an embodiment of the present application. Figure 5 The layered architecture divides software into several layers, each with distinct roles and divisions of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the hardware abstraction layer (HAL), and the kernel layer. In the layered architecture, the kernel layer and the hardware layer communicate with each other.

[0106] Understandably, Figure 5 As just one example, the layers divided in the terminal 100 are not limited to Figure 5 The layers shown, for example, between the application framework layer and the HAL layer, may also include the Android runtime and system libraries layers.

[0107] The application layer can include a series of application packages. Figure 5As shown, the application package can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message. In a call scenario, the user launches a call application in the application layer of the terminal. The terminal can be an electronic device such as a mobile phone or tablet. The call application can be a phone application, real-time chat software, or social networking software with voice capabilities. In a voice recognition scenario, the user launches a playback application in the application layer of the terminal. The terminal can be a smart device with playback and human-computer interaction functions, such as a smart speaker, smart TV, robot vacuum, smart housekeeper, or voice assistant. The playback application can be music, news, novels, sports, and other applications.

[0108] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes predefined functions. In a call scenario, the call application in the application layer uses the API of the application framework layer to call the audio processing algorithm in the HAL layer to execute audio processing methods, and calls the audio driver in the kernel layer to activate the microphone and speaker in the hardware layer.

[0109] like Figure 5 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. Similarly, the call application and the playback application may also call the content provider, resource manager, notification manager, window manager, view system, etc. according to actual business needs, and the embodiments of the present application do not impose any restrictions on this.

[0110] The Hardware Abstraction Layer (HAL) is used to abstract the hardware. For example, it can encapsulate the drivers in the driver layer and provide a calling interface to the application framework layer, shielding the implementation details of the underlying hardware. Among them, the hardware abstraction layer can include the hardware abstraction layer of the calling application and the hardware device abstraction layer of the playback application. The hardware abstraction layer of the calling application can be connected to the algorithm library to call the audio processing algorithms in the algorithm library. The hardware abstraction layer of the playback application can be connected to the algorithm library to call the audio processing algorithms and speech recognition algorithms in the algorithm library.

[0111] The kernel layer is the layer between hardware and software, used to provide drivers for different hardware devices. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0112] The hardware layer includes hardware modules that can be driven, such as microphones and speakers.

[0113] In this application, by calling the hardware interface in the application framework layer, the connection between the application layer above the application framework layer, the HAL layer below the application framework layer, the kernel layer and the hardware layer can be realized to realize the sending and receiving of audio signals.

[0114] It should be noted that in the voice call scenario, the audio processing algorithm can be set in the algorithm library of the HAL layer, and connected to the application layer where the call application is located through the application framework layer, which can improve the signal processing rate.

[0115] In speech recognition scenarios, the audio processing algorithm and speech recognition algorithm can be set up at the application layer, that is, both the audio processing algorithm and the call application are located at the application layer. Data exchange between the application layer and the hardware layer can pass through the hardware layer, the HAL layer, and then reach the application layer. The audio processing algorithm is then executed at the application layer to process the audio signal, improving the timeliness of signal processing and allowing the playback application to respond or perform relevant processing in a timely manner. It is understandable that in speech recognition scenarios, the audio processing algorithm can also be integrated into the playback application.

[0116] Furthermore, the audio processing algorithm in the speech recognition scenario can also be set in the algorithm library of the HAL layer, and connected to the application layer where the playback application is located through the application framework layer, which can improve the signal processing rate. This application does not limit the layer where the algorithm in the speech recognition scenario is located.

[0117] The following describes the workflow of the software and hardware of the terminal 100 in conjunction with a call scenario.

[0118] The call application in the application layer can be displayed as an icon on the screen of terminal 100. When the user clicks the call application icon to trigger it, terminal 100 starts running the call application. When the call application runs on terminal 100, it calls the audio driver in the kernel layer to activate the microphone and speaker in the terminal 100 hardware layer. The microphone collects the original audio signal, and the speaker plays the reference signal. Furthermore, when the call application runs on terminal 100, it calls the call application hardware abstraction layer in the HAL layer through the application programming interface in the application framework layer to invoke the audio processing algorithm to perform audio processing based on the original audio signal and the reference signal.

[0119] In the following, in conjunction with a speech recognition scenario, the workflow of the software and hardware of the terminal 100 is explained by taking the example that the audio processing algorithm and the speech recognition algorithm can be set at the application layer.

[0120] The user can wake up the playback application in terminal 100 by speaking, or by pressing or clicking a control button to start terminal 100, which then begins running the playback application. When the playback application runs on terminal 100, it invokes the audio driver in the kernel layer to activate the microphone and speaker in the terminal 100 hardware layer. The microphone captures the original audio signal, and the speaker plays a reference signal. The original audio signal and reference signal are transmitted to the application layer via the HAL layer, where speech recognition is implemented using the audio processing and speech recognition algorithms in the application layer.

[0121] The executor of the audio processing method provided in the embodiment of the present application can be the above-mentioned terminal, or it can be a functional module and / or functional entity in the terminal that can implement the audio processing method, and the present application solution can be implemented through hardware and / or software. The specific implementation can be determined according to actual usage requirements and is not limited by the embodiment of the present application.

[0122] The embodiment of the present application eliminates the nonlinear echo signal in the original audio signal through a generative network model. The training process of the generative network model (ie, the forward diffusion process) is described in detail below.

[0123] The generative network model includes the attention layer and the diffusion model. To facilitate understanding of the generation process of the generative network model, the diffusion model is first introduced. Figure 6 As shown, Figure 6 Schematic diagram of the diffusion model provided in the embodiment of the present application.

[0124] The diffusion model is a type of generative network, which is divided into a forward diffusion process (also called a forward noise addition process) and a backward diffusion process (also called a reverse diffusion process, a backward diffusion process or a reverse denoising process).

[0125] The forward diffusion process is the process of adding Gaussian noise to the original image step by step. With each addition of noise, the image will lose some information. After performing enough noise addition operations, the image will converge to the distribution of the noise, that is, Gaussian distribution. Assume that x represents the image, the subscript t represents the time step, and x0 represents the original image. The larger t is, the more noise is contained in the corresponding image. In the forward diffusion process, the expression q(X t |X t-1 ) and the noise variance β at time step t t The process of adding noise can be seen as the state X in the previous step. t-1 Multiply by a variance β t The coefficient of correlation, then add the mean zero and variance β t Gaussian distribution.

[0126] In the forward diffusion process, from X t-1 to X t The process of X t-1 Add variance as β t Normally distributed noise, thus obtaining X t Generate X t When X t-1 and β t Decide not to rely on anything other than X t-1 Therefore, the forward diffusion process follows the Markov process (Markov chain), and the state X of any time step can be derived. t The analytical expression, that is, the image X of a certain step can be calculated through the original image X0 t The noise added each time in the forward diffusion process is known, that is, q(X t |X t-1 ) is known, therefore, q(X t |X0) is also known, the forward diffusion process is q(X t |X0) and q(X t |X t-1 ) derivation.

[0127] The back diffusion process is the process of restoring the original image in the noise. Under given conditions, according to X t Get X t-1 The process, that is, q(X t-1 |X t ), but since the data set is not the full data set, the reverse process of accuracy cannot be obtained, so a network p needs to be trained. θ (X t-1 |X t ) to approximate this process. Although we do not know q(X t-1 |X t ), that is, q(X t-1 |X t ) is unknown, but q(X t-1 |X t , X0) can be expressed as q(X t |X t-1 ) and q(X t |X0) indicates that q(X t-1 |X t , X0) is known, so we can use q(X t-1 |X t , X0) to guide p θ (X t-1 |X t ) training process (i.e., forward diffusion process), the reverse diffusion process is q(X t-1 |Xt , derivation of X0).

[0128] above Figure 6 In the diffusion model shown, the noise addition process is from X0-X T , gradually add Gaussian noise, and finally become pure Gaussian noise, the noise adding process q(X t |X t-1 ) can be modeled by the Bayesian formula. In the denoising process, the neural network is used to denoise the denoising process q(X t-1 |X t ) modeling, and obtain a reasonable distribution that conforms to the data distribution by training the model for denoising.

[0129] What the diffusion model predicts is the noise residual, and the “distance” between the noise distribution predicted in the reverse diffusion process and the noise distribution applied in the forward diffusion process is required to be the minimum, that is, the optimization goal is that the estimated noise is close to the real noise. If the variables generated in the middle are regarded as latent variables, then the diffusion model is actually a model containing T latent variables. Therefore, the diffusion model can be regarded as a deeper variational autoencoder (Variational Auto Encoder, VAE), which is a combination of variational inference (variational inference) and autoencoder (Auto-encoder), and is an unsupervised generation model. The loss function of VAE can use variational inference to obtain a variational lower bound (variationallower bound). In this way, the diffusion model in the embodiment of the present application can use a variational lower bound (use variationallower bound) or an evidence lower bound (Evidence Lower Bound, ELBO) to help optimize the parameters of the model, thereby improving the performance and stability of the model. The parameters of the attention layer and diffusion model in the generative network model are updated by minimizing the loss function, and the training is iterated until the diffusion model converges or the preset number of training times is reached to obtain the trained generative network model; the loss function includes the deviation loss between the predicted noise and random noise output by the diffusion model.

[0130] Based on the above Figure 6 The diffusion models provided, such as Figure 7 As shown, Figure 7 This is a schematic diagram of a training process of a generative network model provided in an embodiment of the present application. The method includes the following steps:

[0131] S101. Acquire an audio sample in a silent call scenario or a silent background sound scenario at the local end. The audio sample is a mixed signal including a linear echo signal sample and a nonlinear echo signal sample.

[0132] S102 : Perform linear filtering on the audio sample according to the reference signal sample using a linear adaptive filter to obtain a nonlinear echo signal sample.

[0133] In the call scenario, as mentioned above Figure 1 As shown in the figure, when the near-end is muted, the near-end speech signal s(n) is not collected. After the far-end speech signal x(n) is transmitted to the near-end, it is broadcast from the speaker of the near-end device and propagates through space. It is collected by the microphone of the near-end device, and the microphone collects y(n). y(n) is a mixed signal including linear echo signals and nonlinear echo signals. The linear adaptive filter uses the estimated echo path Process y(n) to obtain the estimated echo signal y(n) and Input linear superposition processor, y(n) and The difference is the signal after the linear echo signal is eliminated, that is, the linear adaptive filter is used to linearly filter the audio sample according to the reference signal sample to obtain a nonlinear echo signal sample.

[0134] In the speech recognition scenario, as mentioned above Figure 2 As shown in the figure, when the user is muted (the user is not speaking), the background sound played by smart speaker A undergoes a series of acoustic reflections, is collected by the microphone of smart speaker A, and is transmitted to smart speaker A. When the user is muted, the signal collected by smart speaker A is a mixed signal including a linear echo signal and a nonlinear echo signal. The linear adaptive filter uses the estimated echo path to process the signal collected by smart speaker A to obtain an estimated echo signal. The signal collected by smart speaker A and the estimated echo signal are input into the linear superposition processor to obtain the signal after the linear echo signal is eliminated. That is, by using the linear adaptive filter to linearly filter the audio sample according to the reference signal sample, a nonlinear echo signal sample can be obtained.

[0135] The above-mentioned linear adaptive filter can implement linear filtering through the normalized least mean square algorithm (NLMS) or Kalman filtering, which is not limited in the embodiment of the present application.

[0136] In an embodiment of the present application, in a call scenario and a voice recognition scenario, audio samples can be obtained by muting the local end and the user. The audio samples do not include pure voice signal samples. After linearly filtering the audio samples according to the reference signal samples using a linear adaptive filter, nonlinear echo signal samples can be obtained without going through complex extraction steps, thereby reducing the complexity of obtaining nonlinear echo signal samples and increasing the diversity of acquisition methods.

[0137] S103. Determine an attention vector sample through the initial attention layer in the initial generative network model according to the obtained reference signal sample.

[0138] The initial generative network model includes an initial attention layer and an initial diffusion model. The initial attention layer implements the attention mechanism. The initial attention layer learns the weights of reference signal samples relative to the clean speech signal samples in the audio sample. This can be understood as extracting information from the reference signal samples that is relevant to predicting the clean speech signal.

[0139] like Figure 8 As shown, Figure 8 This is a schematic diagram of a forward diffusion process provided by an embodiment of the present application. A reference signal sample is input into the initial attention layer, and the initial attention layer outputs an attention vector sample. The attention vector sample is input into the initial diffusion model.

[0140] In an embodiment of the present application, reference signal samples can be extracted using a voiceprint extraction algorithm, a voiceprint extraction model, or a voiceprint classification model. Voiceprint feature extraction algorithms include, but are not limited to, methods based on Mel-Frequency Cepstrum (MFCC), linear predictive coding (LPC), and higher-order cepstral coefficients (LPCC). The voiceprint extraction model and the voiceprint classification model can be neural network models that have been trained based on a large amount of speech signals. Typically, after the speaker's voiceprint features (embedding) are extracted at the bottleneck layer of the voiceprint classification model, voiceprint verification and voiceprint recognition can proceed.

[0141] A voiceprint embedding is a high-dimensional vector that represents the speaker's voice identity. Each person's voice is unique. Even for the same person, when they call themselves, the sound captured by the far-end microphone will differ from the actual voice. In other words, the reference signal x(n) and the near-end speech signal s(n) for the same person will also be different. Therefore, compared to the signal itself, the voiceprint embedding can better reflect the speaker's voice identity.

[0142] like Figure 9 As shown, Figure 9 This is a schematic diagram of another forward diffusion process provided by an embodiment of the present application. This process can first use the voiceprint model to extract the voiceprint of the reference signal sample to obtain a voiceprint feature sample, then input the extracted voiceprint feature sample into the initial attention layer, and output the attention vector sample through the initial attention layer.

[0143] In this embodiment, voiceprint features are extracted from reference signal samples to obtain a voiceprint embedding. During the forward diffusion process, this voiceprint embedding is input into an initial generative network model and, combined with a clean speech signal sample, generates an audio signal sample through a normal diffusion process. Compared to the reference signal sample itself, the voiceprint embedding better reflects the speaker's vocal identity. By introducing the voiceprint method, the training accuracy of the initial generative network model is improved.

[0144] S104. Input the obtained clean speech signal samples and attention vector samples into the initial diffusion model, and in the forward prediction process of the first time step, output a first noisy signal after noise addition processing based on the clean speech signal samples, the attention vector samples and the noise signal samples, wherein the noise signal samples include nonlinear echo signal samples and the obtained white noise samples.

[0145] As mentioned above Figure 8 and Figure 9 As shown, the attention vector sample and the clean speech signal sample are input into the initial diffusion model. Among them, the clean speech signal sample corresponds to Figure 6 In the diffusion model of X0, the pure speech signal sample corresponds to Figure 1 s(n) in, i.e., clean near-end vocals; the reference signal sample corresponds to Figure 1 After the forward diffusion process, the audio signal samples are generated. The audio signal samples correspond to Figure 6 X in the diffusion model T , the audio signal samples correspond to Figure 1 e(n), that is, the signal after the linear echo is removed, the signal after the linear echo is removed is a mixed signal including a nonlinear echo signal and a pure speech signal.

[0146] above Figure 8 and Figure 9 The forward diffusion process of the initial diffusion model corresponds to the above Figure 6 The diffusion model from X0 to X T The first time step is the forward prediction process from X0 to X1. According to the pure speech signal sample (corresponding to X0), the attention vector sample and the noise signal sample, the above Figure 8 and Figure 9 In the figure, white noise samples are used to represent noise signal samples, and after noise processing, the first noise signal is output. The first noise signal is Figure 6 X1 in the diffusion model, Figure 6 X1 is not shown in FIG.

[0147] S105 , continuing the forward prediction process at the second time step, outputting a second noisy signal after performing noise processing based on the first noisy signal and the noise signal sample.

[0148] above Figure 8 and Figure 9 The second time step of the initial diffusion model corresponds to the above Figure 6 In the forward prediction process from X1 to X2, the first noise signal and the noise signal sample are processed and the second noise signal is output. The second noise signal is Figure 6 X2 in the diffusion model, Figure 6 X2 is not shown.

[0149] S106 , until in the forward prediction process of the last time step, according to the noise-added signal and the noise signal sample output at the previous time step of the last forward prediction, the audio signal sample is output after noise addition processing.

[0150] above Figure 8 and Figure 9 The last time step (T time step) of the initial diffusion model in corresponds to the above Figure 6 From X T-1 to X T The forward prediction process is to output the audio signal sample (corresponding to X) after the noise processing according to the noise signal and noise signal sample output at the last time step of the last forward prediction. T ). The last time step of the last forward prediction is the T-1 time step. The noise signal output according to the last time step of the last forward prediction is Figure 6 X in the diffusion model T-1 , Figure 6 X is not shown T-1 .

[0151] It should be noted that the noise added to each time step is different. For example, the noise signal samples can be divided according to preset rules and added to each time step respectively, or noise can be randomly added to each time step. The specific adjustment can be appropriately made by the initial diffusion model's own mechanism, and this embodiment of the present application does not limit this.

[0152] S107. According to the loss function, the weights of the reference signals learned by the initial attention layer and the parameters of the initial diffusion model are adjusted until the deviation between the noise predicted by the initial generative network model based on the audio signal sample and the noise signal sample meets the preset conditions, thereby obtaining a generative network model.

[0153] In an embodiment of the present application, an initial attention layer is added to learn the weights of the initial reference signal samples for the audio signal samples, and the learned attention vector samples are input into the initial diffusion model to improve the training accuracy of the initial generative network model.

[0154] The initial diffusion model has the characteristic of diversified output. Therefore, during training, not only white noise samples but also nonlinear echo signal samples are added to train the diffusion model. After training, the diffusion model generates the desired pure speech signal through the reverse diffusion process, rather than diversified pure speech signals, thereby improving the audio processing effect.

[0155] In the initial training phase of the generative network model, the audio signal samples are finally obtained by continuously adding noise, corresponding to Figure 6 The final white noise X T During training, T-step noise is randomly added to allow the initial diffusion model to predict the noise in the current signal. The initial generative network model is iteratively trained using a loss mechanism defined by the Bayesian method, so that the noise predicted by the initial diffusion model based on the current signal is close to the actual noise added. During inference, the noise at each time step is predicted and random white noise is added to generate the final expected result. The noise added during training includes white noise and nonlinear echo signals; during inference, white noise is added, and the nonlinear echo signal is obtained through prediction.

[0156] During the training process, the loss value is calculated based on the loss function, the noise predicted by the initial diffusion model based on the audio signal samples, and the noise signal samples. The weights of the reference signal learned by the initial attention layer and the parameters of the initial diffusion model are adjusted according to the loss value until the initial diffusion model converges to obtain a generative network model.

[0157] In an embodiment of the present application, the weights of the reference signal samples are learned through the initial attention layer, and the obtained attention vector samples can be used as part of the noise and input into the initial diffusion model together with the clean speech signal samples and the noise signal samples. During the forward diffusion process, audio signal samples are generated by continuously adding noise to the clean speech signal samples.

[0158] The training process of the initial diffusion model involves iterating the diffusion process. The initial diffusion model begins with the addition of noise and is gradually refined through multiple steps until the final audio signal sample is output. At each time step, the initial diffusion model estimates how to transform the current noisy signal into a denoised, pure speech signal sample. This, combined with a loss function, continuously iteratively trains the initial generative network model, updating the weights of the reference signal learned by the initial attention layer and the parameters of the initial diffusion model until the deviation between the noise predicted by the initial generative network model based on the current noisy signal and the noise signal sample is small, indicating that the initial generative network model has converged and the trained generative network model is obtained. This generative network model can eliminate nonlinear echo signals from candidate audio signals to obtain a pure speech signal, improving audio processing performance.

[0159] Next, the audio processing method involved in the embodiment of the present application is described in detail in conjunction with the above-mentioned trained generative network model. Figure 10 As shown, Figure 10 This is a flowchart of an audio processing method provided by an embodiment of the present application. The audio processing method can be applied to call scenarios and speech recognition scenarios. The method includes the following steps:

[0160] S201. Obtain an original audio signal and a reference signal, where the original audio signal is a mixed signal including a nonlinear echo signal, a linear echo signal, and a pure voice signal, and the reference signal is a microphone signal from the other end in a call scenario or a background sound signal played by a playback device in a voice recognition scenario.

[0161] In the call scenario, as mentioned above Figure 1 As shown, the far-end speech signal x(n) is the reference signal. The far-end speech signal x(n) is broadcast from the near-end device's speaker and, after propagating through space, is picked up by the near-end device's microphone, resulting in y(n). y(n) includes both linear and nonlinear echo signals. Simultaneously, the near-end speech signal s(n) also enters the near-end device's microphone. The microphone picks up the sum of the two signals, d(n), which includes both y(n) and s(n). d(n) is the original audio signal.

[0162] In the speech recognition scenario, as mentioned above Figure 2As shown, the background sound played by smart speaker A is the reference signal. Smart speaker A plays the background sound through its speaker. After a series of acoustic reflections, the background sound is picked up by the microphone of the playback device. At the same time, the user asks smart speaker A to switch to another song: "Xiao A, Xiao A, play the next song," which is also picked up by the microphone of the playback device. The microphone picks up a superposition of two signals: the echo signal of the background sound (which includes both linear and nonlinear echo signals) and the pure speech signal expressed by the user. The mixed signal of the background sound echo signal and the pure speech signal is the original audio signal.

[0163] S202 : Perform linear filtering on the original audio signal according to the reference signal using a linear adaptive filter to obtain a candidate audio signal.

[0164] In the call scenario, as mentioned above Figure 1 As shown, the linear adaptive filter uses the estimated echo path Process y(n) to obtain the estimated echo signal y(n) and Input linear superposition processor, y(n) and The difference is the signal after the linear echo signal is eliminated. That is, the original audio signal is linearly filtered according to the reference signal using a linear adaptive filter to obtain a candidate audio signal, in which the nonlinear echo signal remains.

[0165] In the speech recognition scenario, as mentioned above Figure 2 As shown, the linear adaptive filter uses the estimated echo path to process the signal collected by smart speaker A to obtain an estimated echo signal. The signal collected by smart speaker A and the estimated echo signal are input into a linear superposition processor to obtain a signal after the linear echo signal is eliminated. In other words, the linear adaptive filter linearly filters the original audio signal based on the reference signal to obtain a candidate audio signal, which contains a residual nonlinear echo signal.

[0166] S203. Determine an attention vector through the attention layer in the generative network model according to the reference signal.

[0167] The generative network model includes an attention layer and a diffusion model. The attention layer is used to implement the attention mechanism. The attention layer learns the information in the reference signal that is relevant to the predicted clean speech signal.

[0168] like Figure 11 As shown, Figure 11 This is a schematic diagram of a reverse diffusion process provided by an embodiment of the present application. A reference signal is input into the attention layer, and the attention layer outputs an attention vector. The attention vector is input into the diffusion model.

[0169] In the embodiment of the present application, the reference signal can be extracted by a voiceprint extraction algorithm, a voiceprint extraction model or a voiceprint classification model. Figure 12 As shown, Figure 12 This is a schematic diagram of another reverse diffusion process provided by an embodiment of the present application. The voiceprint model is used to extract the voiceprint of the reference signal to obtain the voiceprint feature; the voiceprint feature is then input into the attention layer, and the attention layer outputs the attention vector.

[0170] In this embodiment, a voiceprint embedding is obtained by extracting voiceprint features from a reference signal. During the back-diffusion process, this voiceprint embedding is input into a generative network model and combined with the candidate audio signal to generate a pure speech signal through back-diffusion. Compared to the reference signal itself, the voiceprint embedding better reflects the speaker's vocal identity. By incorporating the voiceprint method, the generative network model can better utilize the voiceprint to eliminate nonlinear echo signals, improving audio processing and enhancing audio quality.

[0171] S204. Input the candidate audio signal and the attention vector into the diffusion model, and in the reverse prediction process of the first time step, output the first predicted speech signal after denoising based on the candidate audio signal, the attention vector, and the noise corresponding to the noise added during the forward diffusion process.

[0172] As mentioned above Figure 11 and Figure 12 As shown, the attention vector and the candidate audio signal are input into the diffusion model. Among them, the candidate audio signal corresponds to Figure 6 X in the diffusion model T , the candidate audio signal corresponds to Figure 1 e(n), that is, the signal after the linear echo is removed. The signal after the linear echo is removed is a mixed signal including the nonlinear echo signal and the pure speech signal. The reference signal corresponds to Figure 1 After the back diffusion process, the x(n) in the image is converted into a pure speech signal. The pure speech signal corresponds to Figure 6 In the diffusion model of X0, the pure speech signal corresponds to Figure 1 s(n) in , i.e., the clean near-end vocals.

[0173] above Figure 11 and Figure 12 The reverse diffusion process of the diffusion model corresponds to the above Figure 6 The diffusion model from X T The first time step is from X T to X T-1 The reverse prediction process, according to the candidate audio signal (corresponding to X T), attention vector and the noise corresponding to the noise added during the forward diffusion process, the above Figure 11 and Figure 12 In the forward diffusion process, white noise is used to represent the noise corresponding to the noise added, and the first predicted speech signal is output after denoising. The first predicted speech signal is Figure 6 X in the diffusion model T-1 , Figure 6 X is not shown T-1 .

[0174] S205 , continuing the reverse prediction process at the second time step, outputting a second predicted speech signal after denoising based on the first predicted speech signal and the noise corresponding to the noise added during the forward diffusion process.

[0175] above Figure 11 and Figure 12 The second time step of the diffusion model corresponds to the above Figure 6 From X T-1 to X T-2 The reverse prediction process is to output the second predicted speech signal after denoising based on the first predicted speech signal and the noise corresponding to the noise added in the forward diffusion process. The second predicted speech signal is Figure 6 X in the diffusion model T-2 , Figure 6 X is not shown T-2 .

[0176] S206, until in the reverse prediction process of the last time step, according to the predicted speech signal output in the last time step of the last reverse prediction and the corresponding noise when adding noise in the forward diffusion process, a pure speech signal is output after denoising.

[0177] above Figure 11 and Figure 12 The last time step (T time step) of the diffusion model corresponds to the above Figure 6 In the reverse prediction process from X1 to X0, the predicted speech signal output at the last time step of the last reverse prediction and the corresponding noise when adding noise during the forward diffusion process are denoised and the pure speech signal (corresponding to X0) is output after denoising. The last time step of the last reverse prediction is the T-1 time step, and the predicted speech signal output at the last time step of the last reverse prediction is Figure 6 X1 in the diffusion model, Figure 6 X1 is not shown in FIG.

[0178] It should be noted that the reverse diffusion process essentially performs denoising on each time step. During denoising, noise needs to be added, and the added noise is subtracted from the predicted speech signal output at the previous time step. The noise added at each time step during the reverse diffusion process corresponds to the noise added during the forward diffusion process. That is, the type of noise added during the reverse diffusion process is the same as the type of noise added during the forward diffusion process, and both can be white noise. The difference is that the forward diffusion process not only adds white noise but also a nonlinear echo signal; whereas the reverse diffusion process only adds white noise, and the nonlinear echo signal is predicted. The specific rules for adding noise during the reverse diffusion process can be appropriately adjusted by the diffusion model's own mechanisms, and this is not limited in this embodiment of the present application.

[0179] S207: Determine a filter coefficient according to the clean speech signal.

[0180] S208 , adjusting the parameters of the linear adaptive filter according to the filter coefficient; the adjusted linear adaptive filter is used in the next linear filtering process.

[0181] Next, the adjustment process of the filter coefficients provided in the embodiment of the present application is described. Figure 13 As shown, Figure 13 This is a schematic diagram of adjusting a filter coefficient provided in an embodiment of the present application. The original audio signal and a reference signal are input into a linear adaptive filter, which linearly filters the original audio signal based on the reference signal and outputs a candidate audio signal. The candidate audio signal and the reference signal are input into a generative network model, which eliminates nonlinear echo signals and outputs a clean speech signal. The clean speech signal is used to determine the filter coefficients, and the parameters of the linear adaptive filter are adjusted based on the filter coefficients. The adjusted linear adaptive filter is used in the next linear filtering process.

[0182] For example, taking the linear adaptive filter as a Kalman filter as an example, three formulas of the Kalman filter include: prior estimation: Posteriori Estimate: Kalman Gain:

[0183] in, represents the prior state estimate at time k, represents the posterior state estimate at time k-1, u k-1 represents the input at time k-1, A is the transition matrix from the state at time k-1 to the state at time k, and B is the matrix from input to state. kRepresents the observation value, and H represents the transition matrix from state to measurement. represents the error covariance matrix of the prior state estimate at time k, R represents the covariance of the measurement noise, and R can be understood as the filter coefficient.

[0184] When using the Kalman filter to eliminate the linear echo signal in the original audio signal, the estimated acoustic echo path corresponding to the far-end reference signal x(n) and the Kalman filter is used. Perform convolution to obtain the estimated echo signal The estimated echo signal is then subtracted from the signal d(n) collected by the near-end microphone The linear adaptive filter minimizes the mean square error of the error signal e(n) by random gradient descent, and updates R to estimate the acoustic echo path. Approximate the true echo path w(n).

[0185] In related technologies, such as Figure 1 As shown, the linear echo is estimated by the linear adaptive filter. After the linear echo is eliminated from the original audio information, the error e(n) is obtained. The error e(n) is used to update the parameters of the linear adaptive filter so that the estimated echo path of the linear adaptive filter is Approximate the true echo path w(n). However, the error e(n) after eliminating the linear echo includes not only the pure speech signal, but also the nonlinear echo signal, which is itself a noisy signal. The solution of using the error e(n) to update the parameters of the linear adaptive filter is not accurate. Furthermore, since the related art eliminates the nonlinear echo signal by finding the dividing point between the nonlinear echo signal and the pure speech signal, the elimination effect is poor. In other words, the linear echo elimination and the nonlinear echo elimination in the related art are processed in series. Therefore, even if the related art uses the signal after nonlinear echo elimination to update the linear adaptive filter, the filter coefficient R used to update the parameters of the linear adaptive filter is not accurate, and the update process of the linear adaptive filter is not appropriate.

[0186] In an embodiment of the present application, a clean speech signal s(n), that is, a signal after eliminating the nonlinear echo signal in the error e(n), is used to determine the filter coefficient, and the parameters of the linear adaptive filter are updated according to the filter coefficient. The updated linear adaptive filter is used to perform the next linear filtering process, which can improve the accuracy of the next linear filtering result. In other words, the near-end human voice signal (i.e., the clean speech signal) obtained in the embodiment of the present application has a complete harmonic structure and is very suitable for updating the parameters of the linear adaptive filter, making the signal processing part more complete and further improving the linear filtering effect.

[0187] For the speech recognition scenario, after S206, the audio processing method further includes the following steps: performing speech recognition on the clean speech signal to obtain a speech recognition result; and responding or processing according to the speech recognition result.

[0188] In an embodiment of the present application, the audio processing method can also be applied to playback devices that are controlled or switched by voice, such as smart speakers, smart TVs, sweeping robots, smart housekeepers, voice assistants and other smart devices.

[0189] Exemplarily, when the playback device plays background sound (for example, music, TV sound, etc.), the user has the need to control the device to shut down or switch. For example, the user says "Xiao A, Xiao A, shut down", and the playback device needs to recognize the human voice in the background sound. Among them, "Xiao A" is a keyword customized by the playback device, and this embodiment of the present application is not limited to this. The reference signal is the background sound played by the playback device. The playback device executes the audio processing method provided by the embodiment of the present application, and eliminates the linear echo signal and the nonlinear echo signal of the original audio signal according to the background sound signal to obtain a pure voice signal, and then performs voice recognition on the pure voice signal to obtain a voice recognition result. In response to the voice recognition result, a response (for example, shut down, answer "Hello, I'm here") or related processing (for example, switch to the next song, switch to the song mentioned by the user) is performed.

[0190] In an embodiment of the present application, the parameters of the linear adaptive filter are updated using the pure speech signal after the linear echo signal and the nonlinear echo signal are eliminated. The updated linear adaptive filter is used to perform the next linear filtering process, which can improve the accuracy of the next linear filtering result.

[0191] Next, combining the above Figure 6-Figure 13 , the application of the audio processing method provided in the embodiment of the present application in the call scenario is illustrated by way of example. Figure 14 As shown, Figure 14 This is a schematic diagram of a device interaction process provided by an embodiment of the present application; the method is applied to the audio processing module of the terminal, and the near end represents the call end and the far end represents the call end as an example for explanation. The audio processing module corresponds to the above Figure 5 The audio processing algorithm in the algorithm library shown includes the following steps:

[0192] S301: The user starts a local call application.

[0193] For example, a user can click an application icon of a nearby call application to start the nearby call application. The call application can be a phone application, a real-time chat software, a social software with voice function, etc.

[0194] S302: In response to a user's start-up operation, a local call application is started.

[0195] S303: The local call application establishes a communication connection with the remote call application.

[0196] The near-end call application establishes a communication connection with the far-end call application through the wireless communication network, and both parties can receive and send audio signals.

[0197] S304: The local call application sends a call instruction 1 to the microphone.

[0198] Instruction 1 is used to call the microphone. After the local call application is started, the microphone can be called by default for voice collection.

[0199] S305: The local call application sends a call instruction 2 to the speaker.

[0200] Instruction 2 is used to call the speaker. After the local call application is started, the speaker can be called by default for voice playback.

[0201] S306: The far-end call application sends a far-end speech signal to the near-end call application.

[0202] The far-end speech signal can be collected by a far-end call application using a far-end microphone (not shown in the figure). The far-end speech signal corresponds to the reference signal.

[0203] S307: The loudspeaker plays the far-end speaking signal.

[0204] After the local call application receives the far-end speaking signal, it calls the speaker to play it. After the far-end speaking signal is played back to the local end, an echo signal is generated.

[0205] S308: The loudspeaker sends a far-end speaking signal to the audio processing module.

[0206] S309: While the loudspeaker plays the far-end speech signal, the user speaks, and the microphone collects the original audio signal, which includes the user's speech signal and the echo signal generated by the far-end speech signal.

[0207] The echo signal generated by the far-end speech signal includes a linear echo signal and a nonlinear echo signal.

[0208] S310: The microphone sends an original audio signal to the audio processing module.

[0209] S311 , the audio processing module performs linear echo and nonlinear echo cancellation on the original audio signal according to the far-end speech signal to obtain a pure speech signal.

[0210] The echo signal cancellation process and the technical effects that can be achieved in this example can be found in the above Figure 6-Figure 13 The description in , will not be repeated here.

[0211] S312: The audio processing module sends a clean voice signal to the near-end call application.

[0212] S313: The local call application sends a clean voice signal to the remote call application.

[0213] In an embodiment of the present application, during uplink transmission in a call scenario (i.e., the near-end sends a signal to the far-end), the audio processing module in the near-end performs echo cancellation (including linear echo and nonlinear echo) on the original audio signal collected by the microphone before sending it to the far-end, thereby improving the call quality at the far-end. Correspondingly, during downlink transmission in a call scenario (i.e., the far-end sends a signal to the near-end), the audio processing module in the far-end also performs echo cancellation (including linear echo and nonlinear echo) on the original audio signal collected by the far-end microphone before sending it to the near-end, thereby improving the call quality at the near-end.

[0214] Next, combining the above Figure 6-Figure 13 , the application of the audio processing method provided in the embodiment of the present application in the speech recognition scenario is illustrated. Figure 15 As shown, Figure 15 This is a schematic diagram of another device interaction process provided by an embodiment of the present application; the method is applied to the audio processing module of a terminal (e.g., a playback device) to execute an audio processing algorithm and a speech recognition algorithm, and the method includes the following steps:

[0215] S401: The user starts a playback application.

[0216] For example, the user may wake up the playback application in the playback device through voice, or the user may start the playback application in the playback device by pressing or clicking a control button.

[0217] S402: In response to a user's start operation, the playback application starts.

[0218] S403: The playback application sends a call instruction 1 to the microphone.

[0219] Call instruction 1 is used to call the microphone. After the playback application is started, the microphone can be called by default for voice collection.

[0220] S404: The playback application sends a call instruction 2 to the speaker.

[0221] Call instruction 2 is used to call the speaker. After the playback application is started, the speaker can be called by default for voice playback.

[0222] S405: The loudspeaker plays a background sound signal.

[0223] The background sound signal corresponds to the reference signal and generates an echo signal after the background sound signal is played.

[0224] S406: Send a background sound signal.

[0225] S407 : While the loudspeaker plays the background sound signal, the user speaks, and the microphone collects the original audio signal, which includes the user's speaking signal and the echo signal generated by the background sound signal.

[0226] The echo signal generated by the background sound signal includes a linear echo signal and a nonlinear echo signal.

[0227] S408: The microphone sends the original audio signal to the audio processing module.

[0228] S409: The audio processing module performs linear echo and nonlinear echo cancellation on the original audio signal according to the background sound signal to obtain a pure speech signal; and performs speech recognition on the pure speech signal to obtain a speech recognition result.

[0229] The echo signal cancellation process and speech recognition process in this example and the technical effects that can be achieved by each can be found in the above Figure 6-Figure 13 The description in , will not be repeated here.

[0230] S410: The audio processing module sends the speech recognition result to the playback application.

[0231] S411: The playback application responds or processes according to the voice recognition result.

[0232] The playback application responds or processes according to the voice recognition results, for example, playing the next song, playing the song the user wants, or shutting down the phone.

[0233] In the speech recognition scenario, in the embodiments of the present application, the audio processing module in the playback device performs echo cancellation (including linear and nonlinear echo cancellation) on the original audio signal collected by the microphone, thereby improving the accuracy of the audio processing results. Speech recognition is then performed based on the clean speech signal after echo cancellation, which can further improve the accuracy of the speech recognition results.

[0234] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.

[0235] The present application also provides a terminal, which includes a chip, and the chip is used to read and execute computer programs or instructions stored in a memory, so that the methods in each embodiment are executed.

[0236] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a terminal, the terminal executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0237] This embodiment further provides a computer program product, wherein the computer-readable storage medium stores program code. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0238] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0239] Among them, the terminal, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0240] The embodiments of the present application do not specifically limit the specific structure of the execution subject of the method provided in the embodiments of the present application; as long as the program containing the code of the method provided in the embodiments of the present application can be run to perform video processing according to the method provided in the embodiments of the present application, the execution subject of the method provided in the embodiments of the present application can be a terminal, or a functional module in the terminal that can call and execute the program.

[0241] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0242] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0243] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0244] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, in essence, or the part of the prior art that contributes to the technical solutions, or part of the technical solutions, can be embodied in the form of a computer software product stored in a storage medium, the computer software product includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The storage medium described above can include, but is not limited to: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, and various program code storage media.

[0245] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An audio processing method, characterized in that: The method comprises: Obtaining an original audio signal and a reference signal, where the original audio signal is a mixed signal including a nonlinear echo signal, a linear echo signal, and a clean speech signal, and the reference signal is a microphone signal from the other end in a call scenario or a background sound signal played by a playback device in a speech recognition scenario; performing linear filtering on the original audio signal according to the reference signal to obtain a candidate audio signal, wherein the nonlinear echo signal remains in the candidate audio signal; The candidate audio signal is denoised according to the reference signal through a generative network model, and the clean speech signal is output after a reverse diffusion process.

2. The method according to claim 1, wherein The step of performing denoising on the candidate audio signal according to the reference signal using a generative network model and outputting the clean speech signal after a reverse diffusion process includes: Determining an attention vector through an attention layer in the generative network model according to the reference signal; The candidate audio signal and the attention vector are input into a diffusion model in the generative network model, and the clean speech signal is generated through a reverse diffusion process by the diffusion model.

3. The method according to claim 2, wherein The determining, according to the reference signal, an attention vector by an attention layer in the generative network model, comprises: Inputting the reference signal into an attention layer in the generative network model, and outputting the attention vector through the attention layer; or, Perform voiceprint extraction on the reference signal to obtain a voiceprint feature; input the voiceprint feature into the attention layer in the generative network model, and output the attention vector through the attention layer.

4. The method according to claim 2 or 3, wherein: Inputting the candidate audio signal and the attention vector into the diffusion model in the generative network model, and generating the clean speech signal through a reverse diffusion process by the diffusion model, comprises: Inputting the candidate audio signal and the attention vector into the diffusion model, and in a backward prediction process at a first time step, outputting a first predicted speech signal after denoising based on the candidate audio signal, the attention vector, and noise corresponding to noise addition during the forward diffusion process; Continuing in the reverse prediction process at the second time step, outputting a second predicted speech signal after denoising based on the first predicted speech signal and the noise corresponding to the noise added during the forward diffusion process; Until in the reverse prediction process of the last time step, the predicted speech signal outputted in the last time step of the last reverse prediction and the corresponding noise when adding noise in the forward diffusion process are denoised and the pure speech signal is outputted.

5. The method according to any one of claims 1 to 4, characterized in that The linear filtering of the original audio signal according to the reference signal to obtain a candidate audio signal includes: performing linear filtering on the original audio signal according to the reference signal using a linear adaptive filter to obtain the candidate audio signal; After performing denoising on the candidate audio signal and the reference signal through a back diffusion process using a generative network model to obtain the clean speech signal, the method further includes: Determining a filter coefficient according to the clean speech signal; The parameters of the linear adaptive filter are adjusted according to the filter coefficient; the adjusted linear adaptive filter is used for the next linear filtering process.

6. The method according to any one of claims 1 to 5, wherein: The method is applied to speech recognition scenarios; After performing denoising on the candidate audio signal according to the reference signal using a generative network model and outputting the clean speech signal after a reverse diffusion process, the method further includes: Performing speech recognition on the clean speech signal to obtain a speech recognition result; Respond or process according to the speech recognition result.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: Acquire a clean speech signal sample, a reference signal sample, and a noise signal sample; the noise signal sample includes a white noise sample and a nonlinear echo signal sample; Determining an attention vector sample through an initial attention layer in an initial generative network model according to the reference signal sample; Inputting the clean speech signal sample, the attention vector sample, and the noise signal sample into an initial diffusion model in an initial generative network model, and generating an audio signal sample through a forward diffusion process using the initial diffusion model; According to the loss function, the weight of the reference signal learned by the initial attention layer and the parameters of the initial diffusion model are adjusted until the deviation between the noise predicted by the initial generative network model based on the audio signal sample and the noise signal sample meets a preset condition, thereby obtaining the generative network model.

8. The method according to claim 7, wherein The step of inputting the clean speech signal sample, the attention vector sample, and the noise signal sample into an initial diffusion model in an initial generative network model, and generating an audio signal sample through a forward diffusion process using the initial diffusion model, comprises: Inputting the clean speech signal sample and the attention vector sample into the initial diffusion model, and in a forward prediction process at a first time step, outputting a first noisy signal after performing a noise addition process based on the clean speech signal sample, the attention vector sample, and the noise signal sample; Continuing the forward prediction process in the second time step, outputting a second noisy signal after performing noise processing based on the first noisy signal and the noise signal sample; Until in the forward prediction process of the last time step, the audio signal sample is output after noise addition processing based on the noise signal output in the previous time step of the last forward prediction and the noise signal sample.

9. The method according to claim 7, wherein The determining, according to the reference signal sample, an attention vector sample by an initial attention layer in an initial generative network model, comprises: Inputting the reference signal sample into the initial attention layer in the initial generative network model, and outputting the attention vector sample through the initial attention layer; or, Performing voiceprint extraction on the reference signal sample to obtain a voiceprint feature sample; The voiceprint feature sample is input into the initial attention layer in the initial generative network model, and the attention vector sample is output through the initial attention layer.

10. The method according to any one of claims 7 to 9, characterized in that The method further comprises: In a silent call scenario or a silent background sound scenario, obtaining an audio sample; the audio sample is a mixed signal including a linear echo signal sample and a nonlinear echo signal sample; The audio sample is linearly filtered according to the reference signal sample by a linear adaptive filter to obtain the nonlinear echo signal sample.

11. A terminal, characterized in that: The terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 10 when executed by the processor.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 10.

13. A computer program product comprising instructions, characterized in that When the method is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 10.

Images