An audio signal processing method and related device

Abstract

The application provides an audio signal processing method and related equipment, applied to the field of audio. The method comprises the following steps: training a diffusion model by taking a sample audio signal and sample noise information of the sample audio signal as conditions to obtain a noise reduction model; obtaining noise information in a to-be-processed audio signal; inputting the noise information and the audio signal into the noise reduction model to obtain a noise-reduced speech signal. The method has better generalization performance and better noise reduction effect; accordingly, from the perspective of hearing, the noise-reduced speech signal provided by the audio signal processing method has less noise, so that the quality and intelligibility of the speech signal are higher, and the user experience is better. In some embodiments, the noise reduction model outputs a probability distribution of the audio signal, and the probability distribution of the audio signal is sampled to obtain the noise-reduced speech signal. The time-frequency points of the noise-removed part can be restored while the time-frequency point distribution of the speech part is also recovered.

Description

An audio signal processing method and related equipment Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an audio signal processing method and related equipment. Background Technology

[0002] Terminal devices are commonly used in scenarios that require the acquisition of voice signals, such as recording audio and video, making voice calls (e.g., audio and video calls), or voice recognition (e.g., voice input or voice interaction). The acquired voice signals usually contain a large amount of environmental noise, which seriously affects the quality of the voice signal.

[0003] Currently, numerous speech denoising methods have been proposed to address this problem. These methods train denoising models based on large noise training sets, and the trained denoising models can effectively remove most of the noise signals similar to those in the noise training sets for the speech signal to be processed.

[0004] However, terminal devices are used in a wide variety of scenarios, and the types of environmental noise involved are also numerous. Since the noise training set can only cover some types of environmental noise, these methods cannot effectively remove noise components not included in the noise training set for the speech signal to be processed. They have poor generalization performance, poor noise reduction effect, and affect the quality and intelligibility of the noise-reduced speech signal, thereby reducing the user experience. Summary of the Invention

[0005] This application provides an audio signal processing method and related equipment that can improve noise reduction effect, thereby enhancing user experience.

[0006] In a first aspect, an audio signal processing method is provided for use in electronic devices. The method includes: acquiring noise information in an audio signal to be processed; inputting the audio signal and noise information into a denoising model to obtain a denoised first speech signal, wherein the denoising model is a conditional diffusion model trained on the condition of a sample audio signal and sample noise information in the sample audio signal.

[0007] The audio signal processing method provided in this application has better generalization performance and significantly better noise reduction effect. Correspondingly, in terms of listening experience, the audio signal signal after noise reduction by the audio signal processing method provided in this application has less noise, resulting in higher quality and intelligibility of the audio signal and a better user experience.

[0008] Understandably, typical diffusion models derive their final result from Gaussian noise. The diffusion model in this application, however, derives its final result from a given set of information (i.e., noise information and the audio signal), rather than from Gaussian noise. This results in a more accurate inference than directly inferring from Gaussian noise.

[0009] It is also understandable that, in addition to inputting the audio signal to be processed into the diffusion model, noise information is also input into the diffusion model. This allows the noise information to guide the diffusion model's noise reduction process, resulting in better noise reduction performance compared to noise reduction methods that do not input noise information into the diffusion model. Furthermore, it offers better generalization performance compared to current noise reduction methods. Compared to the second related technology, it not only has better generalization performance but also does not require multiple microphones, thus reducing costs.

[0010] In one possible embodiment, obtaining noise information in the audio signal to be processed includes: detecting a second time frame that does not contain a speech signal in a first time frame corresponding to the audio signal; and extracting a first signal segment corresponding to the second time frame from the audio signal as noise information.

[0011] The above scheme extracts the first signal segment corresponding to the second time frame of the audio signal that does not contain the speech signal as noise information. This allows for the extraction of noise information frame by frame, improving the accuracy of noise information extraction and thus enhancing the noise reduction effect.

[0012] In one possible embodiment, inputting the audio signal and noise information into a denoising model to obtain a denoised first speech signal includes: inputting the audio signal and noise information into a denoising model to obtain a first probability distribution information of the audio signal, wherein the first probability distribution information corresponds to the distribution of time and frequency points in the spectrogram of the audio signal; and sampling the first probability distribution to obtain the first speech signal.

[0013] Understanding this, the probability distribution of the audio signal to be processed can be interpreted as the distribution of time-frequency points on the spectrogram and the energy distribution of the audio signal. Time-frequency points with higher energy (corresponding to the speech portion of the audio signal) are more concentrated and have a higher probability of being sampled during the sampling process. Conversely, time-frequency points with lower energy (corresponding to the noise portion of the frequency signal) are more dispersed and have a lower probability of being sampled during the sampling process. This allows for approximation of the overall distribution with a smaller number of sample points, thus achieving noise reduction. Therefore, compared to the spectrogram of the audio signal to be processed, the spectrogram of the denoised speech signal removes the time-frequency points of the noise portion while also restoring the time-frequency distribution of the speech portion.

[0014] In one possible embodiment, the method further includes: acquiring sample noise information based on the sample audio signal; generating a sample input signal based on the sample audio signal, the sample noise information, and a first Gaussian noise, wherein the first Gaussian noise is randomly generated; inputting the sample input signal into a diffusion model to be trained to obtain sample probability distribution information of the sample audio signal; calculating a loss function value based on the difference between the sample probability distribution information and the sample target data, wherein the sample target data is determined based on the first Gaussian noise and a preset diffusion intensity; and updating the parameters to be trained in the diffusion model to be trained based on the loss function value to obtain a noise reduction model.

[0015] The above scheme uses sample audio signals and sample noise information as conditional inputs to the diffusion model to be trained during the training process, and combines them with the loss function value to train the denoising model. This improves the training accuracy of the denoising model and also enables the denoising model to converge as quickly as possible, thereby improving training efficiency. This allows the denoising model to obtain the probability distribution information of the audio signal more accurately on the inference side based on the audio signal and noise information.

[0016] In one possible embodiment, sampling processing of the first probability distribution to obtain a first speech signal includes: obtaining drift coefficients based on the audio signal according to the stochastic differential equation corresponding to the noise reduction model; obtaining inverse drift coefficients according to the first probability distribution information and the drift coefficients; obtaining a sampled signal according to the first probability distribution information, the inverse drift coefficients, and second Gaussian noise, wherein the second Gaussian noise is randomly generated; and obtaining the first speech signal according to the sampled signal.

[0017] As can be understood, the inverse drift coefficient can be used to describe the path of an audio signal containing noisy information towards a clean speech signal during the sampling process. Sampling based on the inverse drift coefficient can achieve noise reduction.

[0018] In one possible embodiment, obtaining a first speech signal based on a sampled signal includes: inputting the sampled signal and noise information into a noise reduction model to obtain a second probability distribution information of the audio signal; correcting the sampled signal based on the second probability distribution information to obtain a corrected signal; and obtaining the first speech signal based on the corrected signal.

[0019] It is understandable that by inputting the audio signal into the noise reduction model and then inputting the sampled signal into the noise reduction model again, the noise reduction model performs one more calculation. The calculated second probability distribution information will be closer to the distribution of the audio signal than the first probability distribution information, thus making the final speech signal more accurate and the noise reduction effect of this application better.

[0020] In one possible embodiment, the signal energy of the first signal segment is greater than a preset energy threshold.

[0021] It is understood that the signal energy of the first signal segment is greater than a preset energy threshold, meaning that the first signal segment contains a first noise segment but does not contain a second noise segment. For details, please refer to the descriptions related to the first and second noise segments below, which will not be repeated here.

[0022] It is understandable that, when the first signal segment does not contain the second noise segment, since the noise characteristics of the first noise segment are more obvious than those of the second noise segment, the computational burden of the denoising model can be reduced without affecting the accuracy of the output results of the denoising model.

[0023] In a second aspect, this application provides an electronic device including one or more processors and one or more memories; wherein the one or more memories are coupled to one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by one or more processors, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0024] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0025] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0026] Fifthly, this application provides a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0027] Understandably, the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0028] Figure 1 is a schematic diagram illustrating an example of a scenario to which the embodiments of this application are applicable;

[0029] Figure 2 is a schematic diagram of an example of the time-domain diagram and spectrogram before and after noise reduction processing provided in the embodiments of this application;

[0030] Figure 3 is a schematic diagram of the audio signal processing method 400 provided in an embodiment of this application;

[0031] Figure 4 is a schematic diagram of the voice activity detection method 600 provided in an embodiment of this application;

[0032] Figure 5 is a schematic diagram of the training method 700 of the noise reduction model provided in the embodiment of this application;

[0033] Figure 6 is a schematic diagram of the hardware structure of an electronic device 1000 provided in an embodiment of this application;

[0034] Figure 7 is a block diagram of the software system of an electronic device 1000 provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] 1. For example, an electronic device records sound through a microphone, obtaining a time-domain audio signal. By performing feature transformation on this time-domain audio signal, a time-frequency domain audio signal can be obtained. The time-frequency domain audio signal can be represented by a spectrogram (or, as referred to in this application, a feature map or time-frequency map).

[0037] In the time-domain graph, the horizontal axis represents time and the vertical axis represents amplitude. The time-domain graph represents the amplitude of the audio signal in the time domain.

[0038] For example, time-domain plot #1, time-domain plot #2, or time-domain plot #3 in Figure 2.

[0039] In a spectrogram, the horizontal axis represents time, and the vertical axis represents frequency. A spectrogram consists of multiple time-frequency points. The brightness or darkness of these points indicates the energy level; for example, a brighter or lighter point indicates greater energy, and vice versa. In other words, a spectrogram can represent the energy level of an audio signal at different times and frequencies.

[0040] For example, if a region in a spectrogram is displayed as black, it means there is no signal in that region; if a region in a spectrogram is close to bright yellow, it means that the signal energy in that region is very high.

[0041] For example, spectrogram #1, spectrogram #2, or spectrogram #3 in Figure 2.

[0042] Understandably, a spectrogram can represent the probability distribution of an audio signal, that is, the distribution of each time-frequency point in the spectrogram.

[0043] 2. Diffusion Models: A type of generative model based on stochastic processes, which can be used to generate various types of data such as images, text, and audio. Unlike traditional generative models, diffusion models do not rely on known labels or target data. Instead, they process random noise to gradually generate high-quality data from the noise. Therefore, diffusion models can be considered a type of unsupervised generative model.

[0044] Among diffusion models, the most common is the unconditional diffusion model, whose main task is to generate random samples of the original data. However, for some specific applications, such as image inpainting, image synthesis, and text-to-image generation, we may need more fine-grained control over the generated samples, which requires the introduction of conditional diffusion models.

[0045] A key characteristic of conditional diffusion models is that the generated samples can be controlled by adding additional conditional information, such as specific parts of an image or text descriptions. For example, in text-to-image generation tasks, we can control which elements the generated image should contain or what style it should present by adding text descriptions.

[0046] In practical implementation, conditional diffusion models typically encode additional conditional information as part of the model during the training phase, and then control the generated results by decoding this conditional information during the generation phase. A key advantage of this approach is that it provides finer control over the generated results while maintaining the generative capabilities of the diffusion model.

[0047] The denoising model involved in this application is a conditional diffusion model. During the training phase, the sample noisy speech signal and the sample noise information of the sample noisy speech signal are used as conditions to train the diffusion model, thereby obtaining the denoising model (for the specific training process of the denoising model, please refer to the relevant description in Figure 5 below).

[0048] 3. Audio signals may contain speech (i.e., the sound of human speech) and / or non-speech (i.e., sounds that are not human speech, such as various noises, or almost or completely no sound). Among them, the part of the audio signal that contains the sound of human speech can be called a speech segment or speech part, and the part that does not contain human speech can be called a noise segment or non-speech segment or noise part or non-speech part.

[0049] It is understood that in the audio signal to be processed in this application, that is, the audio signal before noise reduction, the noise segment does not include speech, and the speech segment may include speech or noise.

[0050] Voice activity detection (VAD) is used to detect speech in an audio signal (including whether speech exists; or, which time periods of the signal contain speech and which time periods do not), thereby distinguishing speech segments from noise segments in the audio signal. For example, in this application, VAD can be implemented by processing statistical signals, by a deep neural network model, or by other methods to achieve similar functionality, without limitation.

[0051] In this application, noise segments in the speech signal to be processed can be obtained through VAD (as an example of the first signal segment mentioned above). These noise segments are used as the aforementioned noise information (one of the conditions in the aforementioned conditional diffusion model).

[0052] Optionally, this application can further distinguish noise segments into a first noise segment (also referred to as an actual noise segment, as another example of the first signal segment mentioned above) and a second noise segment (also referred to as a silent segment). For example, the portion of a noise segment whose signal energy is less than or equal to a preset energy threshold is determined as the second noise segment, and the portion of a noise segment whose signal energy is greater than the preset energy threshold is determined as the first noise segment. The first noise segment is used as the aforementioned noise information (one of the conditions in the aforementioned conditional diffusion model).

[0053] It is understood that the noisy speech signal involved in this application (such as the above-mentioned sample noisy speech signal) can be understood as an audio signal containing noise segments and speech segments. Optionally, the noisy speech signal involved in this application can be understood as an audio signal containing a first noise segment and a speech segment.

[0054] 4. Drift coefficient and reverse drift coefficient.

[0055] Noise is added to a clean signal (e.g., a clean speech signal) through sampling. The drift coefficient and the spread coefficient are used together to characterize the difference between the random signal being denoised and the audio signal to be processed during the noise addition process.

[0056] Wherein, the drift coefficient is the mean of the random signal in the nth sample during N sampling processes; the diffusion coefficient is the variance of the random signal in the nth sample during N sampling processes.

[0057] Denoising is achieved by sampling the audio signal to be processed. The inverse drift coefficient and the inverse diffusion coefficient are used together to characterize the difference between the denoised random signal and the clean signal during the denoising process.

[0058] The inverse drift coefficient is the mean of the random signal in the nth sample during the Nth sampling process; the inverse diffusion coefficient is the variance of the random signal in the nth sample during the Nth sampling process.

[0059] Where 1≤n≤N and n and N are both integers.

[0060] 5. Gaussian noise: A common type of random noise characterized by a Gaussian distribution, meaning its intensity and frequency are uniformly distributed. In signal processing and image processing, Gaussian noise is frequently used to simulate various types of noise. The frequency response of Gaussian noise can be calculated using a Gaussian function. In the frequency domain, the spectral density function of Gaussian noise exhibits a Gaussian curve.

[0061] 6. Random seed is a computer science term referring to a type of random number generator that uses a truly random number (seed) as its initial condition. Generally, computer-generated random numbers are pseudo-random, using a truly random number (seed) as the initial condition and then iteratively generating random numbers using a specific algorithm.

[0062] It is understood that in this application, the random numbers (e.g., Gaussian noise values) generated at different times based on the same random seed are different.

[0063] Terminal devices are commonly used in scenarios that require the acquisition of voice signals, such as recording audio and video, making voice calls (e.g., audio and video calls), or voice recognition (e.g., voice input or voice interaction). The acquired voice signals usually contain a large amount of environmental noise, which seriously affects the quality of the voice signal.

[0064] Currently, numerous speech denoising methods have been proposed to address this problem. These methods train denoising models based on large noise training sets, and the trained denoising models can effectively remove most of the noise signals similar to those in the noise training set for the speech signal to be processed.

[0065] However, terminal devices are used in a wide variety of scenarios, and the types of environmental noise involved are also numerous. Since the noise training set can only cover some types of environmental noise, these methods cannot effectively remove noise components not included in the noise training set for the speech signal to be processed. They have poor generalization performance and poor noise reduction effect, resulting in low intelligibility and quality of the denoised speech signal.

[0066] Therefore, how to improve the noise reduction effect, that is, improve the intelligibility and quality of the noise-reduced speech signal, has become an urgent technical problem to be solved.

[0067] The first related technique involves speech denoising based on discriminative deep neural network models. This type of method applies a mask or mapping function, represented by a discriminative deep neural network model, to the noisy speech signal to obtain a denoised speech signal. Besides poor generalization performance, this method also damages the speech signal.

[0068] The second related technology primarily relies on microphone arrays to acquire speech signals, followed by noise reduction using acoustic models and a comprehensive filter bank. This type of method is model-driven, resulting in limited performance and poor generalization. Errors in the localization module can distort the denoised speech signal. Furthermore, this method requires multiple microphones, leading to higher costs.

[0069] Figure 1 is a schematic diagram illustrating an example of a scenario to which the embodiments of this application are applicable.

[0070] As shown in Figure 1(a), Xiaoming is having a voice call with Xiaohua, who is in a quiet environment (such as a bedroom), in an environment with a variety of noises (such as a large shopping mall).

[0071] For example, in a large shopping mall, a loudspeaker is playing song A, creating musical noise; a machine calls out numbers, creating machine noise; people talk loudly, creating human voice noise. Other examples include footsteps, the sound of objects colliding, and so on.

[0072] For example, in this environment, Xiaoming says to himself via voice call, "I'm eating at shop B, come find me." Xiaohua hears Xiaoming's voice on the phone, but it's severely interfered with by various noises. For instance, Xiaohua hears the conversation as "*%^&$#$shop". Rice*&^( "I (*&*&)." Xiaohua couldn't understand what Xiaoming was saying, resulting in a poor call experience.

[0073] As shown in Figure 1(b), Xiaoming and Xiaohua are having a voice call in the environment shown in Figure 1(a).

[0074] After Xiaoming or Xiaohua's electronic device acquires a voice signal containing various noises, it processes the voice signal with various noises using the audio signal processing method provided in this application, and then plays the processed voice signal back to Xiaohua, which can improve the call experience. For example, if Xiaoming says, "I'm eating at shop B, come find me," Xiaohua will hear the message "I'm eating at shop B, come find me."

[0075] It should be noted that Figure 1 uses a large shopping mall as an example for illustration, but it can be replaced with other environments with a wide variety of noise levels; Figure 1 uses a voice call as an example for illustration, but it can be replaced with other application scenarios that can be processed using the voice processing method provided in this application.

[0076] Figure 2 is a schematic diagram of an example of the time-domain diagram and spectrogram before and after noise reduction processing provided in the embodiments of this application.

[0077] Figure 2(a) shows the time-domain plot #1 and spectrogram #1 of the audio signal #1 with various noises. The spectrogram #1 is obtained by performing a feature transformation on the time-domain plot #1.

[0078] As shown in spectrogram #1, almost all regions include time-frequency points, and most of these time-frequency points have high energy. The time-frequency distribution of the speech signal has been submerged in a large amount of noise.

[0079] In Figure 2(a), regions 101, 103 and 106 have a large number of scattered noise points; in regions 104 and 106, the low-frequency harmonic structure has been completely destroyed by noise; region 102 is located in the mid-high frequency region, in which there are several bright wavy lines formed by video points, and each wavy line corresponds to a different frequency. Since the frequency of the spectrogram is related to the pitch of the audio signal, region 102 is likely to correspond to a piece of music.

[0080] Figure 2(b) shows the time-domain plot #2 and spectrogram #2 of the speech signal #2 obtained by processing the audio signal using the first correlation technique. The spectrogram #2 is obtained by performing feature transformation on the time-domain plot #2.

[0081] As shown in Figure 2(b), after noise reduction, spectrogram #2 has removed a large number of noisy time-frequency points compared to spectrogram #1, and the distribution of time-frequency points of the speech signal can be basically seen; the distinction between the brightness of different time-frequency points is more obvious (for example, the brightness of low-frequency time-frequency points is significantly higher than that of mid- and high-frequency time-frequency points).

[0082] In Figure 2(b), region 203 still contains a large number of time-frequency points that constitute noise, region 201 contains some time-frequency points that constitute noise, and region 206 contains almost no time-frequency points that constitute noise. This indicates that the first correlation technique effectively reduces some noise, but its noise reduction effect on other parts of the noise is not good. Correspondingly, from an auditory perspective, the speech with residual noise will have background noise, which is very uncomfortable for users to listen to and has a poor auditory experience. In other words, the quality of the denoised speech signal is poor. Furthermore, the residual noise will also interfere with the user's understanding of the speech content, meaning that the intelligibility of the denoised speech signal will be reduced.

[0083] As shown in Figure 2(b), the harmonic structures in regions 204 and 205 are still not obvious, or rather, have not been well recovered, indicating poor spectral continuity. Consequently, the sound will sound unnatural, unlike a real person's voice, meaning the quality of the denoised speech signal is poor. Furthermore, the obvious loss of harmonic structures (e.g., region A1 in region 205) indicates damage to the speech signal obtained after denoising using the first correlation technique. Although noise was removed in region 202, the time-frequency points of the speech signal overlapping with the noise were not recovered. Consequently, the sound will sound dry and lack fullness, and the insufficient fullness of some words will increase the difficulty for users to understand the speech content, meaning the intelligibility of the denoised speech signal will be reduced.

[0084] This application provides an audio signal processing method, which uses a sample audio signal and sample noise information of the sample audio signal as conditions to train a diffusion model to obtain a noise reduction model. First, noise information in the audio signal to be processed is obtained, and the noise information and the audio signal are input into the noise reduction model to obtain a noise-reduced speech signal.

[0085] Understandably, typical diffusion models derive their final result from Gaussian noise. The diffusion model in this application, however, derives its final result from a given set of information (i.e., noise information and the audio signal), rather than from Gaussian noise. This results in a more accurate inference than directly inferring from Gaussian noise.

[0086] It is also understandable that, in addition to inputting the audio signal to be processed into the diffusion model, noise information is also input into the diffusion model. This allows the noise information to guide the diffusion model's noise reduction process, resulting in better noise reduction performance compared to noise reduction methods that do not input noise information into the diffusion model. Furthermore, it offers better generalization performance compared to current noise reduction methods. Compared to the second related technology, it not only has better generalization performance but also does not require multiple microphones, thus reducing costs.

[0087] Figure 2(c) shows the time-domain plot #3 and spectrogram #3 of the speech signal #3 obtained by processing the audio signal using the audio signal processing method provided in this application. The spectrogram #3 is obtained by performing feature transformation on the time-domain plot #3.

[0088] As shown in Figure 2(c), after processing by the audio signal processing method provided in this application, the spectrogram #3 has removed a large number of noisy time-frequency points compared to the spectrogram #1, and the distribution of time-frequency points of the speech signal can be clearly seen; the distinction between the brightness of different time-frequency points is more obvious (for example, the brightness of low-frequency time-frequency points is significantly higher than that of mid- and high-frequency time-frequency points).

[0089] Compared to spectrogram #2, spectrogram #3 removes noise at specific time and frequency points more thoroughly, and the intervals between adjacent syllables are more pronounced. For example, almost all noise at specific time and frequency points in regions 303 and 301 are removed, showing significantly better noise removal results than regions 203 and 201, respectively. Furthermore, the noise reduction effect in region 306 is not inferior to that in region 206.

[0090] Therefore, the audio signal processing method provided in this application has better generalization performance and significantly better noise reduction effect; correspondingly, in terms of listening experience, the audio signal signal after noise reduction by the audio signal processing method provided in this application has less noise, resulting in higher quality and intelligibility of the audio signal and a better user experience.

[0091] In spectrogram #3, the harmonic structure is clear and complete. Compared to regions 204 and 205 in spectrogram #2, the harmonic structures in regions 304 and 305 of spectrogram #3 are significantly clearer and more complete (for example, the harmonic structure of region A2 in region 305 is significantly clearer and more complete than that of region A1 in region 205); and, compared to region 202 in spectrogram #2, the time-frequency points of the speech signal overlapping with the noise in region 302 of spectrogram #3 are also well recovered. This indicates that the denoised speech signal has almost no loss and high spectral continuity. Accordingly, from an auditory perspective, the denoised speech signal produced by the audio signal processing method provided in this application is more natural and closer to human voice; the sound is fuller; thus, the quality and intelligibility of the speech signal are higher, and the user experience is better.

[0092] In some embodiments, noise information and the audio signal are input into a noise reduction model to obtain the probability distribution of the audio signal (e.g., represented by the gradient of the audio signal), and the probability distribution of the audio signal is sampled to obtain the probability distribution of the noise-reduced speech signal, thereby obtaining the speech signal.

[0093] Understanding this, the probability distribution of the audio signal to be processed can be interpreted as the distribution of time-frequency points on the spectrogram and the energy distribution of the audio signal. Time-frequency points with higher energy (corresponding to the speech portion of the audio signal) are more concentrated and have a higher probability of being sampled during the sampling process. Conversely, time-frequency points with lower energy (corresponding to the noise portion of the frequency signal) are more dispersed and have a lower probability of being sampled during the sampling process. This allows for approximation of the overall distribution with a smaller number of sample points, thus achieving noise reduction. Therefore, compared to the spectrogram of the audio signal to be processed, the spectrogram of the denoised speech signal removes the time-frequency points of the noise portion while also restoring the time-frequency distribution of the speech portion.

[0094] Figure 3 is a schematic diagram of the audio signal processing method 400 provided in an embodiment of this application.

[0095] S401, acquire noise information in the audio signal to be processed.

[0096] For example, a second time frame that does not include the speech signal is detected in the first time frame corresponding to the audio signal; a first signal segment corresponding to the second time frame is extracted from the audio signal as noise information.

[0097] It is understood that the audio signal here is a time-frequency domain signal. The time domain of the corresponding spectrogram of the audio signal may include multiple first time frames. The multiple first time frames include one or more second time frames and one or more third time frames. The one or more second time frames correspond to the time-frequency points of non-speech signals, and the one or more third time frames correspond to the time-frequency points of speech signals.

[0098] As one implementation, S401 includes steps a1 to a3.

[0099] Step a1: Input the audio signal x to be processed into the feature transformation module 501 to obtain the feature tensor X in the time-frequency domain.

[0100] Here, the audio signal x is a discrete signal in the time domain. For example, to obtain the audio signal x = [x(0), x(1), ..., x(K-1)] T , where K represents the total number of points in the discrete signal, and T represents the transpose of a vector or matrix.

[0101] In this process, the feature transformation module 501 first performs a short-time inverse Fourier transform on the audio signal x, and then performs an inverse amplitude transform to obtain the feature tensor X in the time-frequency domain. X = [X...] Real ,X Image ], where X Real Let X represent the real part of X. Image Let X represent the imaginary part. Real The feature tensor X represented as a time frame dimension Real =[x Real (0),x Real (1),…,x Real (M-1)], or X Image The feature tensor X represented as a time frame dimension Image =[x Image (0),x Image (1),…,x Image [(M-1)], where M represents the total number of time frames corresponding to X, that is, the total number of the first time frames.

[0102] Step a2: Input the feature tensor X into the speech activity detection module 502 to obtain the frame index Index of the time frame excluding the speech signal. noise .

[0103] In other words, it detects the frame indices of all second-time frames within all first-time frames. For example, Index noise = [0, 2, 5, 7, ...] means that frames 0, 2, 5, and 7 do not contain speech. That is, frames 0, 2, 5, and 7 contain only noise, or only noise or no sound.

[0104] For example, the voice activity detection module 502 can be implemented by processing statistical signals, by using a deep neural network model, or by other methods to achieve similar functions, without limitation.

[0105] Step a3, combine the feature tensor X and the frame index Index noise Input the noise segment extraction module 503 to obtain the feature tensor noise of the first signal segment.

[0106] The feature tensor noise of the first signal segment is used as an example of noise information.

[0107] For example, extract the frame index from the feature tensor X. noise The corresponding feature tensor yields the feature tensor noise. noise = [noise] Real noise Image ], where noise Real This represents the real part of noise. Image This represents the imaginary part of noise. Real =X Real [Index noise noise Image =X Image [Index noise ].

[0108] The above scheme extracts the first signal segment corresponding to the second time frame of the audio signal that does not contain the speech signal as noise information. This allows for the extraction of noise information frame by frame, improving the accuracy of noise information extraction and thus enhancing the noise reduction effect.

[0109] Optionally, the first signal segment includes a first noise segment and a second noise segment.

[0110] Optionally, the signal energy of the first signal segment is greater than a preset energy threshold; that is, the first signal segment includes a first noise segment but does not include a second noise segment. For details, please refer to the descriptions related to the first and second noise segments above, which will not be repeated here.

[0111] It is understandable that, when the first signal segment does not contain the second noise segment, since the noise characteristics of the first noise segment are more obvious than those of the second noise segment, the computational burden of the denoising model can be reduced without affecting the accuracy of the output results of the denoising model.

[0112] S402 inputs the audio signal and noise information into the noise reduction model to obtain the first speech signal after noise reduction.

[0113] Among them, the noise reduction model is a conditional diffusion model trained on the sample audio signal and the sample noise information in the sample audio signal.

[0114] As one implementation, S402 includes steps a3 to a4.

[0115] Step a3: Input the feature tensor X and the feature tensor noise into the diffusion model 504 (as an example of a noise reduction model) to obtain the feature tensor Y of the first speech signal in the time-frequency domain after noise reduction.

[0116] The specific implementation of step a3 will be described in detail with reference to Figure 4.

[0117] Step a4: Input the feature tensor Y into the feature inverse transform module 505 to obtain the first speech signal y after noise reduction.

[0118] For example, the feature inverse transform module 505 first performs an inverse amplitude transform on the feature tensor Y, and then performs a short-time Fourier inverse transform. Here, the first speech signal y is a time-domain signal. y = [y(0), y(1), ..., y(D-1)] T , where D is the total number of sampling points.

[0119] For example, in step b0, the noise reduction model is mainly used to obtain the first probability distribution information of the audio signal, wherein the first probability distribution information corresponds to the distribution of time-frequency points in the spectrogram of the audio signal. Subsequently, the first probability distribution is sampled to obtain the first speech signal.

[0120] Understanding this, the probability distribution of the audio signal to be processed can be interpreted as the distribution of time-frequency points on the spectrogram and the energy distribution of the audio signal. Time-frequency points with higher energy (corresponding to the speech portion of the audio signal) are more concentrated and have a higher probability of being sampled during the sampling process. Conversely, time-frequency points with lower energy (corresponding to the noise portion of the frequency signal) are more dispersed and have a lower probability of being sampled during the sampling process. This allows for approximation of the overall distribution with a smaller number of sample points, thus achieving noise reduction. Therefore, compared to the spectrogram of the audio signal to be processed, the spectrogram of the denoised speech signal removes the time-frequency points of the noise portion while also restoring the time-frequency distribution of the speech portion.

[0121] In one possible implementation, the sampling process includes: predicting a sampled signal based on a first probability distribution signal, thereby obtaining a first speech signal from the sampled signal. For example, step b1: obtaining drift coefficients based on the audio signal according to the stochastic differential equation corresponding to the noise reduction model. Step b2: obtaining inverse drift coefficients based on the first probability distribution information and the drift coefficients. Step b3: obtaining the sampled signal based on the first probability distribution information, the inverse drift coefficients, and second Gaussian noise, wherein the second Gaussian noise is randomly generated. Step b4: obtaining the first speech signal from the sampled signal.

[0122] As can be understood, the inverse drift coefficient can be used to describe the path of an audio signal containing noisy information towards a clean speech signal during the sampling process. Sampling based on the inverse drift coefficient can achieve noise reduction.

[0123] Optionally, the sampling process further includes: step b5, inputting the sampled signal and noise information into the noise reduction model to obtain the second probability distribution information of the audio signal; step b6, correcting the sampled signal according to the second probability distribution information to obtain a corrected signal; and step b7, obtaining the first speech signal based on the corrected signal.

[0124] It is understandable that by inputting the audio signal into the noise reduction model and then inputting the sampled signal into the noise reduction model again, the noise reduction model performs one more calculation. The calculated second probability distribution information will be closer to the distribution of the audio signal than the first probability distribution information, thus making the final speech signal more accurate and the noise reduction effect of this application better.

[0125] For details on the sampling process, please refer to the relevant explanation in Figure 4.

[0126] Optionally, method 400 also includes a method for training a denoising model, which includes steps c1 to c5.

[0127] Step c1: Obtain sample noise information based on the sample audio signal.

[0128] Today you can refer to the description related to S401. The difference is that the audio signal to be processed and the noise information are replaced with sample audio signal and sample noise information, respectively.

[0129] Step c2: Generate a sample input signal based on the sample audio signal, sample noise information, and first Gaussian noise, wherein the first Gaussian noise is randomly generated.

[0130] Step c3: Input the sample input signal into the diffusion model to be trained to obtain the sample probability distribution information of the sample audio signal.

[0131] Step c4: Calculate the loss function value based on the difference between the sample probability distribution information and the sample target data. The sample target data is determined based on the first Gaussian noise and the preset diffusion intensity.

[0132] Step c5: Update the parameters that need to be trained in the diffusion model to be trained based on the loss function value to obtain the denoising model.

[0133] For the specific implementation of steps c2 to c5, please refer to the relevant description in Figure 5 below.

[0134] The above scheme uses sample audio signals and sample noise information as conditional inputs to the diffusion model to be trained during the training process, and combines them with the loss function value to train the denoising model. This improves the training accuracy of the denoising model and also enables the denoising model to converge as quickly as possible, thereby improving training efficiency. This allows the denoising model to obtain the probability distribution information of the audio signal more accurately on the inference side based on the audio signal and noise information.

[0135] Figure 4 is a schematic diagram of the speech activity detection method 600 provided in an embodiment of this application. For example, the noise reduction model (e.g., diffusion model 504) includes the initialization module 601 to the judgment module 607 shown in Figure 4.

[0136] Initialize module 601, initialize the sampling count n = 1 (indicating the current sampling is the 1st time), initialize the sampling time s = 0.999, and initialize the sampling signal. The initialized parameters are then output to the drift coefficient calculation module 602.

[0137] Where f represents frequency and t represents time frame.

[0138] For example, when n=1, x(f,t) is the audio signal to be processed as described above. This indicates the first sampled signal.

[0139] For example, if the number of sampling times n satisfies 2 ≤ n ≤ N, when the drift coefficient calculation module 602 to the judgment module 607 all perform the nth processing, The (n-1)th correction signal can be understood as the correction signal obtained in the (n-1)th sampling process, or as the correction signal obtained based on the (n-1)th sampling signal.

[0140] The drift coefficient calculation module 602 takes x(f,t) and s as inputs. This module's function is to calculate the drift coefficient fc based on the stochastic differential equation of the diffusion model, for example... The drift coefficient calculated by this module is output to the reverse drift coefficient calculation module 604 to calculate the reverse drift coefficient.

[0141] For example, the stochastic differential equation here could be a forward stochastic differential equation of a diffusion model.

[0142] The steps performed by the drift coefficient calculation module 602 can be understood as a possible example of step b1.

[0143] Gradient calculation module 603, the input of this module is x(f,t), s, This module's function is to calculate using a trained noise reduction model. The corresponding gradient,

[0144] Understandably, this gradient is used to characterize The probability distribution, that is The distribution of time and frequency points in the spectrogram.

[0145] The steps performed by the gradient calculation module 603 can be understood as a possible example of step b0.

[0146] Specifically, the training process of the noise reduction model will be described in detail below with reference to Figure 5.

[0147] The gradient calculated by gradient calculation module 603 is output to inverse drift coefficient calculation module 604.

[0148] The inverse drift coefficient calculation module 604 takes the drift coefficient fc and gradient grad1 as inputs and calculates the drift coefficient using the formula fcr = -fc + g. 2 The inverse drift coefficient fcr is calculated using g(t)grad1 and output to the prediction module 605. Here, g(t) is a predefined diffusion coefficient. In this application, g(t) can be a constant chosen empirically, such as 1.2 or 2.

[0149] The steps performed by the inverse drift coefficient calculation module 604 can be understood as a possible example of step b2.

[0150] Prediction module 605, also known as sampling prediction module 605, uses the formula To predict the nth sampled signal, z1 is Gaussian noise with zero mean and variance of 1 (i.e., the second Gaussian noise mentioned above). It should be noted that generating Gaussian noise requires setting a random seed. For evidence collection, the random seed can be set when the terminal device is reset.

[0151] Among them, formula It means that it will Updated to The new sampled signal is obtained, which is the sampled signal obtained by the nth sampling, also known as the nth sampled signal.

[0152] For example, sampling methods include, but are not limited to, one or a combination of two of ancestor sampling, Langevin dynamics sampling, and annealed Langevin dynamics sampling. However, regardless of the sampling method, random numbers (i.e., Gaussian noise with a mean of 0 and a variance of 1 in this application) need to be generated based on a random seed, and then sampling is performed based on the random numbers.

[0153] The steps performed by the prediction module 605 can be understood as a possible example of step b3.

[0154] After the prediction module 605 predicts the nth sampled signal, it outputs the prediction result to the gradient calculation module 603.

[0155] Gradient calculation module 603, the input of this module is s, x(f,t). Where, here For the updated nth sampled signal in prediction module 605 The function of this module is to calculate the gradient of the nth sampled signal using a trained neural network model. The gradient grad2 calculated by this module is output to the correction module 606 for correction.

[0156] It is understandable that calculating the gradient again for the nth sampled signal is equivalent to performing an extra calculation by the denoising model. The calculated gradient is better and closer to the distribution of the audio signal, thus making the final speech signal output by the denoising model more accurate. The denoising effect of this application is better.

[0157] The steps performed by the gradient calculation module 603 can be understood as a possible example of step b5.

[0158] The correction module 606 uses the annealing Langevin dynamics sampling method or the Langevin dynamics sampling method to predict the nth sampled signal based on the gradient of the nth sampled signal. Perform correction to obtain the nth correction signal. The annealing Langevin kinetic sampling method can be found in the descriptions in related technologies. Alternatively, the Langevin kinetic sampling method can also be used, which can be found in the descriptions in related technologies. The correction module 606 outputs... Refer to the judgment module 607.

[0159] The steps performed by the correction module 606 can be understood as a possible example of step b6.

[0160] The judgment module 607 is used to determine whether n = N (N is the preset total number of sampling steps, for example, assuming it is 10) is true. If n = N, then... As the second speech signal output, otherwise update the sampling count and sampling time, n←n+1 (i.e., update n with n+1), s=max(0.03,(n-1)×(0.999-0.03) / N)), and then Transmitted to the drift coefficient calculation module 602.

[0161] In other words, when 1 ≤ n < N, the nth correction signal The drift coefficient calculation module 602 performs the (n+1)th processing. When n=N, the nth correction signal This serves as the final output of the noise reduction model (or diffusion model 504).

[0162] The steps performed by the judgment module 607 can be understood as a possible example of step b7.

[0163] The steps performed by the gradient calculation module 603, the correction module 606, and the judgment module 607 can be understood as a possible example of step b4.

[0164] The beneficial effects of method 600 can be found in the beneficial effects of the corresponding scheme in method 400.

[0165] Figure 5 is a schematic diagram of the training method 700 of the noise reduction model provided in the embodiment of this application.

[0166] The addition module 701 is used to linearly add the sample noise from the sample noise set and the sample clean speech signal Ya(k) from the sample clean speech signal set according to a certain signal-to-noise ratio to obtain the sample audio signal Ys(k), which is a time-domain signal.

[0167] Preferably, the quality and intelligibility of the clean speech signals in the sample clean speech signal set should be as high as possible; the types of noise in the sample noise set should be as diverse as possible, so as to facilitate better training results.

[0168] The addition module 701 inputs the sample audio signal Ys(k) into the value feature transformation module 702.

[0169] The feature transformation module 702 performs feature transformation on the sample audio signal Ys(k) to obtain the time-frequency domain sample audio signal Yx(f,t) after feature transformation; it also performs feature transformation on the sample clean speech signal Ya(k) to obtain the time-frequency domain sample clean speech signal YA(f,t) after feature transformation.

[0170] For example, the feature transformation can be a short-time Fourier transform, an amplitude transform, etc.

[0171] The feature transformation module 702 transmits the sample audio signal Yx(f,t) to the speech activity detection module 703, and transmits Yx(f,t) and YA(f,t) to the sample generation module 707.

[0172] The voice activity detection module 703 generates a sample frame index YIndex based on the sample audio signal Yx(f,t). noise And index the sample frame YIndex noise The data is transmitted to the noise segment extraction module 704. For details regarding the speech activity detection module 703, please refer to the description of the speech activity detection module 502 above.

[0173] The noise fragment extraction module 704 extracts noise fragments based on the sample frame index YIndex. noise Generate sample noise fragment YB[YIndex] noise For details regarding the noise segment extraction module 704, please refer to the relevant description of the noise segment extraction module 503 above.

[0174] The sampling time generation module 705 requires no input information. This module is used to randomly generate a sampling time. min to l max The decimal between these two values ​​is used as the sampling time l, where l min To be the minimum value, for example, 0.003, l max This is the maximum value, such as 0.999.

[0175] The sampling time generation module 705 outputs the sampling time l to the sample generation module 707.

[0176] The Gaussian noise generation module 706 is used to generate a Gaussian noise value z2 with a mean of zero and a variance of 1 (i.e., the first Gaussian noise value mentioned above).

[0177] The sample generation module 707 is used to generate a sample sampling signal Yp(f,t) based on Yp(f,t)=μ(f,t,l)+σ(l)z2, where μ(f,t,l) is the mean of Yp(f,t) and σ(l) is the standard deviation of Yp(f,t).

[0178] Among them, μ(f,t,l)=lYx(f,t)+(1-l)YA(f,t),

[0179] The sample generation module 707 is also used to input the sample sampling signal Yp(f,t) into the neural network 708.

[0180] Neural network 708 is used to output the sample gradient Ygrad based on the sample input signal. The sample input signal includes: the sample audio signal Yx(f,t) and the sample noise segment YB[YIndex].noise The sample sampling time l and the sample sampling signal Yp(f,t) are also considered. That is, Ygrad = DNN. s (Yp(f,t),YB[YIndex noise ],l,Yx(f,t)).

[0181] For example, the neural network 708 can be composed of one or more deep neural network models, such as Noise Conditional Scoring Network (NCSN)++, Convolutional Neural Networks (CNN), Convolutional Recurrent Neural Network (CRNN), U-shaped Neural Network Pooled Subgraph Structure (U-Net), and other network models.

[0182] The loss function calculation module 709 is used to calculate the loss function value. The calculation formula is as follows:

[0183] The beneficial effects of training method 700 can be found in the beneficial effects of the corresponding scheme in method 400.

[0184] Figure 6 is a schematic diagram of the hardware structure of an electronic device 1000 provided in an embodiment of this application. Referring to Figure 6, the electronic device 1000 may include a processor 1010, an external memory interface 1020, an internal memory 1021, a universal serial bus (USB) interface 1030, a charging management module 1040, a power management module 1041, a battery 1042, antenna 1, antenna 2, a mobile communication module 1050, a wireless communication module 1060, an audio module 1070, a speaker 1070A, a receiver 1070B, a microphone 1070C, a headphone jack 1070D, a sensor module 1080, buttons 1090, a motor 1091, an indicator 1092, a camera 1093, a display screen 1094, and a subscriber identification module (SIM) card interface 1095, etc. The sensor module 1080 may include a pressure sensor 1080A, a gyroscope sensor 1080B, a barometric pressure sensor 1080C, a magnetic sensor 1080D, an accelerometer sensor 1080E, a distance sensor 1080F, a proximity sensor 1080G, a fingerprint sensor 1080H, a temperature sensor 1080J, a touch sensor 1080K, an ambient light sensor 1080L, a bone conduction sensor 1080M, etc.

[0185] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1000. In other embodiments of this application, the electronic device 1000 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0186] The processor 1010 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU).

[0187] The controller can be the nerve center and command center of the electronic device 1000. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0188] The processor 1010 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1010 is a cache memory. This memory can store instructions or data that the processor 1010 has just used or that are used repeatedly. If the processor 1010 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 1010, and thus improves the efficiency of the system.

[0189] In some embodiments, the processor 1010 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0190] The MIPI interface can be used to connect the processor 1010 to peripheral devices such as the display screen 1094 and the camera 1093. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 1010 and the camera 1093 communicate via the CSI interface to enable the electronic device 1000 to perform its shooting function. The processor 1010 and the display screen 1094 communicate via the DSI interface to enable the electronic device 1000 to perform its display function.

[0191] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 1000. In other embodiments of this application, the electronic device 1000 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0192] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 1000 is selecting a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0193] Video codecs are used to compress or decompress digital video. Electronic device 1000 can support one or more video codecs. Thus, electronic device 1000 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0194] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0195] Electronic device 1000 can implement audio functions, such as music playback and recording, through audio module 1070, speaker 1070A, receiver 1070B, microphone 1070C, headphone jack 1070D, and application processor. Buttons 1090 include a power button and volume buttons. Buttons 1090 can be mechanical buttons or touch buttons. Electronic device 1000 can receive button input and generate key signal inputs related to user settings and function control of electronic device 1000.

[0196] The software system of electronic device 1000 will be described next.

[0197] For example, electronic device 1000 may be a mobile phone. The software system of electronic device 1000 may adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment takes the layered architecture Android system as an example to illustrate the software system of electronic device 1000.

[0198] Figure 7 is a block diagram of a software system of an electronic device 1000 according to an embodiment of this application. Referring to Figure 7, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the hardware abstraction layer (HAL), the driver layer, and the hardware layer.

[0199] The application layer may include a series of application packages, such as dialer applications, gallery applications, etc. (not shown in the figure). In this embodiment, the application package may include applications such as recording and voice call applications, all of which require the use of a microphone to record audio. The terminal device can perform the audio recorded by the microphone according to this application's speech signal processing, including noise reduction processing and high-frequency component restoration processing.

[0200] Alternatively, the application layer may also include other applications that require the audio signals recorded by the microphone to be processed as described in this application; however, this application does not impose any limitations on this.

[0201] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. In this embodiment, the framework layer includes a microphone service interface and a noise reduction service interface. The noise reduction service interface provides APIs and programming frameworks for applications that request noise reduction services. The microphone service provides APIs and programming frameworks for applications that call the microphone.

[0202] The Hardware Abstraction Layer (HAL) is an interface layer located between the operating system kernel and upper-layer software, providing a virtual hardware platform for the operating system. In this embodiment, the HAL may include a microphone hardware abstraction layer and a noise reduction algorithm. The microphone hardware abstraction layer can provide virtual hardware for microphone devices such as microphone 1, microphone 2, or more. The noise reduction algorithm may include runtime code and data that implement the speech signal processing method provided in this embodiment.

[0203] The driver layer is the layer between hardware and software. It includes drivers for various hardware components. Drivers may include microphone device drivers, digital signal processor (DSP) drivers, etc. Microphone device drivers are used to drive microphone sensors to acquire sound signals and to drive audio signal processors to preprocess the sound signals to obtain digital audio signals. DSP drivers are used to drive digital signal processors to process the digital audio signals.

[0204] The hardware layer includes sensors and an audio signal processor. The sensors include microphone 1 and microphone 2. Each microphone within the sensors corresponds one-to-one with a virtual microphone included in the microphone hardware abstraction layer. The audio signal processor converts the sound signals collected by the microphones into digital audio signals. The digital signal processor processes the digital audio signals. It should be noted that the software structure diagram of the electronic device shown in Figure 7 of this application is only an example and does not limit the specific module divisions in different layers of the Android operating system. For details, please refer to the descriptions of the Android operating system software structure in conventional technologies.

[0205] The method in the embodiments of this application will be described in detail below with reference to the above hardware structure and system structure:

[0206] In response to applications that enable recording or voice calls, these applications can call the noise reduction service interface to obtain the application programming interface and programming framework provided by the noise reduction service.

[0207] On one hand, the noise reduction service can call the microphone service in the framework layer to collect sound signals from the environment. Specifically, the microphone service can send a command to the microphone 1 sensor in the hardware layer to collect sound signals by calling microphone 1 in the microphone hardware abstraction layer. The microphone hardware abstraction layer then sends this command to the microphone device driver in the driver layer. Based on the command, the microphone device driver can activate microphone 1, thereby acquiring sound signals from the environment and generating digital audio signals through an audio signal processor.

[0208] On the other hand, the noise reduction service can initialize the noise reduction algorithm. The noise reduction algorithm can obtain the digital audio signal generated by the audio signal processor through the microphone hardware abstraction layer. Then, according to the speech signal processing method stored in the noise reduction algorithm, the noise reduction algorithm can use the digital signal processor to process the acquired digital audio signal to obtain the noise-reduced digital audio signal.

[0209] Specifically, the method for processing digital audio signals to obtain denoised digital audio signals can be found in the flowcharts shown in Figures 3 to 5 above.

[0210] Finally, the noise reduction algorithm can send the noise-reduced digital audio signal back to the noise reduction service, and then back to the application layer.

[0211] This application provides a chip system including one or more processors for calling and executing instructions stored in memory, thereby performing the methods described in this application. The chip system may be composed of chips or may include chips and other discrete devices.

[0212] The chip system may include input circuits or interfaces for transmitting information or data, and output circuits or interfaces for receiving information or data.

[0213] This application also provides a computer program product that, when executed by a processor, implements the methods described in any of the method embodiments of this application.

[0214] The computer program product can be stored in memory and, after processes such as preprocessing, compilation, assembly, and linking, is finally converted into an executable object file that can be executed by a processor.

[0215] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the methods described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0216] The computer-readable storage medium can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0217] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0218] In the several embodiments provided in this application, the systems, apparatuses, and methods disclosed can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. Furthermore, the coupling between units or components can be direct coupling or indirect coupling, including electrical, mechanical, or other forms of connection.

[0219] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0220] It should be understood that "multiple" as used in this application refers to two or more. The term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0221] The terms (or numbers) "first," "second," etc., appearing in the embodiments of this application are for descriptive purposes only, that is, only to distinguish different objects, such as different "coordinates," etc., and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first," "second," etc., may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "at least one (item)" refers to one or more. "Multiple" means two or more. "At least one (item) below" or similar expressions refer to any combination of these items, including any combination of a single (item) or a plurality of (items).

[0222] In summary, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An audio signal processing method, applied to electronic devices, characterized in that, The method includes: detecting a second time frame that does not include a speech signal in a first time frame corresponding to the audio signal; extracting a first signal segment corresponding to the second time frame from the audio signal as noise information, wherein the signal energy of the first signal segment is greater than a preset energy threshold; inputting the audio signal and the noise information into a noise reduction model to obtain a denoised first speech signal, wherein the noise reduction model is a conditional diffusion model trained with the sample audio signal and the sample noise information in the sample audio signal as conditions.

2. The method as described in claim 1, characterized in that, The step of inputting the audio signal and the noise information into a noise reduction model to obtain a noise-reduced first speech signal includes: inputting the audio signal and the noise information into a noise reduction model to obtain a first probability distribution information of the audio signal, wherein the first probability distribution information corresponds to the distribution of time-frequency points in the spectrogram of the audio signal; and sampling the first probability distribution to obtain the first speech signal.

3. The method as described in claim 1 or 2, characterized in that, The method further includes: obtaining sample noise information based on the sample audio signal; generating a sample input signal based on the sample audio signal, the sample noise information, and a first Gaussian noise, wherein the first Gaussian noise is randomly generated; inputting the sample input signal into a diffusion model to be trained to obtain sample probability distribution information of the sample audio signal; calculating a loss function value based on the difference between the sample probability distribution information and the sample target data, wherein the sample target data is determined based on the first Gaussian noise and a preset diffusion intensity; and updating the parameters to be trained in the diffusion model to be trained based on the loss function value to obtain the noise reduction model.

4. The method as described in claim 2, characterized in that, The step of sampling the first probability distribution to obtain the first speech signal includes: obtaining a drift coefficient based on the audio signal according to the stochastic differential equation corresponding to the noise reduction model; obtaining an inverse drift coefficient based on the first probability distribution information and the drift coefficient; obtaining a sampled signal based on the first probability distribution information, the inverse drift coefficient, and second Gaussian noise, wherein the second Gaussian noise is randomly generated; and obtaining the first speech signal based on the sampled signal.

5. The method as described in claim 4, characterized in that, The step of obtaining the first speech signal based on the sampled signal includes: inputting the sampled signal and the noise information into the noise reduction model to obtain the second probability distribution information of the audio signal; correcting the sampled signal based on the second probability distribution information to obtain a corrected signal; and obtaining the first speech signal based on the corrected signal.

6. An electronic device, characterized in that, The electronic device includes: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 5.

7. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 5.

Images