Speech processing method, medium, electronic device, and program product

Abstract

The application relates to the technical field of voice processing, and discloses a voice processing method, a medium, an electronic device and a program product. The method first judges the degree of noise of an environment in which noisy voice data to be processed is located, for example, a signal-to-noise ratio of the noisy voice data to be processed is obtained as a parameter representing the degree of noise, and then different denoising reasoning times are set based on the parameter representing the degree of noise, for example, different denoising reasoning times are set for noisy voice data with different signal-to-noise ratios. In this way, the denoising effect is ensured, and the waste of computing resources in the denoising process is avoided.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech processing method, medium, electronic device and program product. Background Technology

[0002] With the widespread adoption of smart devices such as smartphones and smart speakers, voice interaction scenarios, including voice calls and human-computer interaction, are becoming increasingly common. Typically, in voice interaction scenarios, smart devices use microphones to capture speech, but the speech captured by microphones is often interfered with by noise. This noise severely impacts the call experience and the quality of human-computer interaction. For example, during a voice call, the speech captured by the microphone may contain not only valid speech but also environmental noise such as horns, making it difficult for the user to clearly hear the other person's conversation.

[0003] Currently, such as Figure 1 As shown, traditional methods employ discriminative speech denoising models to denoise the speech captured by the microphone. Specifically, in Figure 1 In the training process of the illustrated speech denoising model, noisy training speech 1 can be input into the neural network for denoising processing, outputting the corresponding denoised training speech 3. Then, by comparing the differences between denoised training speech 3 and the corresponding clean training speech 2, the network weights of the neural network are adjusted to obtain the trained speech denoising model. Furthermore, in Figure 1 During the test shown, the noisy speech 4, which is collected in real time by the microphone, is input into the denoising model, and the denoised speech 5 is output.

[0004] In the training process of the discriminative denoising model, a large amount of labeled data is required for training data such as noisy speech 1 to label the speech data and noise. If this labeled data contains errors or inaccuracies, the model may be unable to correctly distinguish between valid speech and noise, causing valid speech in noisy speech 1 to be misidentified as noise and removed. Similarly, during testing, the model may misidentify valid speech in noisy speech 4 as noise, especially if the model has not learned from noisy speech 4 during training. In such cases, the model may misidentify valid speech as noise, thus damaging valid speech during the denoising process. Summary of the Invention

[0005] This application provides a speech processing method, medium, electronic device, and program product that can automatically match the number of denoising inferences of the diffusion model according to the noise level of the environment, ensuring the denoising effect while avoiding the waste of computational resources in the denoising process.

[0006] In a first aspect, embodiments of this application provide a speech processing method, the method comprising: acquiring first speech data to be denoised (hereinafter referred to as speech data to be processed); calculating a noise level parameter of a first acquisition environment of the first speech data; determining a target inference number based on the noise level parameter, wherein the higher the noise level indicated by the noise level parameter, the greater the target inference number; and using a diffusion model to denoise the first speech data according to the target inference number to obtain denoised second speech data (hereinafter referred to as denoised speech data).

[0007] Thus, this application can automatically match the appropriate number of denoising inference steps based on the parameters representing the noise level of the noisy speech data, ensuring the denoising effect. This avoids performing unnecessary denoising inference in a quiet environment, thereby preventing the waste of computational resources in the denoising process.

[0008] In one possible implementation of the first aspect above, the noise level parameter is the signal-to-noise ratio (SNR), and a higher SNR indicates a lower level of environmental noise.

[0009] In one possible implementation of the first aspect above, the signal-to-noise ratio is calculated as follows: a speech activity sequence of the first speech data is detected, wherein data in the speech activity sequence that takes a first value is speech data, and data that takes a second value is noise data; a first energy and a first data quantity of the data in the speech activity sequence that takes a first value are calculated, and a second energy and a second data quantity of the data in the speech activity sequence that takes a second value are calculated; and the formula is used. Calculate the signal-to-noise ratio (SNR); where snr is the SNR, p s As the first energy, l s p is the first data volume. n For the second energy, l n This is the second data volume. Thus, based on the speech activity sequence, this application can distinguish between speech data (i.e., effective speech data) and noise data in the first speech data, and calculate the energy of the effective speech data and the energy of the noise data in the first speech data. Then, the signal-to-noise ratio of the speech data to be processed is calculated based on the energy of both.

[0010] In one possible implementation of the first aspect above, the speech activity sequence is implemented using at least one of the following algorithms: energy detection algorithm, zero-crossing rate detection, Hidden Markov Model (HMM), or Long Short-Term Memory (LSTM) network.

[0011] In one possible implementation of the first aspect described above, the target inference count is determined as follows: The signal-to-noise ratio (SNR) is determined to fall within a target threshold range, and the inference count corresponding to this range is defined as the target inference count. The target threshold range is one of multiple preset threshold ranges, with the preset threshold range having a larger minimum value corresponding to a smaller inference count. For example, in noisy environments such as factories, train stations, and markets, the SNR of noisy speech data is relatively small, and a larger inference count is set, such as 10. In moderately noisy environments such as restaurants, offices, shopping malls, and parks, the SNR of noisy speech data is relatively large, and a smaller inference count is set, such as 5. In quiet environments such as homes, meeting rooms, small streets, and libraries, the SNR of noisy speech data is even larger, and a smaller inference count is set, such as 2.

[0012] In one possible implementation of the first aspect described above, the second speech data is obtained as follows: the first speech data is converted from the time domain to the frequency domain to obtain first frequency domain data; the first frequency domain data is input into a diffusion model, and denoising is performed on the first frequency domain data according to the target inference number to obtain second frequency domain data, wherein the first frequency domain data and the second frequency domain data have a preset linear relationship; the second frequency domain data is converted from the frequency domain to the time domain to obtain the second speech data. It can be understood that the first frequency domain data and the corresponding second frequency domain data conform to the aforementioned preset linear relationship, and the data mapping trajectory between the first frequency domain data and the corresponding second frequency domain data in the diffusion model is a straight line.

[0013] In this way, it is possible to infer the second frequency domain data (i.e., denoised frequency domain data) from the first frequency domain data (noisy frequency domain data) based on a straight-line data mapping trajectory, and the inference process is relatively fast. Furthermore, the diffusion model provided in this application does not damage the speech data during speech denoising and has strong generalization performance, exhibiting strong suppression capabilities for noise data not encountered during the training phase. Moreover, since this application denoises noisy speech data in the frequency domain, it utilizes the frequency distribution of speech data to ensure the continuity of the speech spectrum in the denoised speech data, thus guaranteeing the clarity of the denoised speech data.

[0014] In one possible implementation of the first aspect above, the second frequency domain data is obtained as follows: based on the numerical value N of the target inference count, N sampling times are obtained, and the N sampling time points are input into the diffusion model; corresponding to i < N, for the i-th sampling time point, the diffusion model obtains the i-th correction information corresponding to the i-th frequency domain data, and performs denoising processing on the i-th frequency domain data using the i-th correction information to obtain the (i+1)-th frequency domain data, where the first frequency domain data is the first frequency domain data, the i-th frequency domain data and the (i+1)-th frequency domain data have a preset linear relationship, and i is a positive integer; corresponding to i = N, the (i+1)-th frequency domain data is used as the second frequency domain data.

[0015] In the inference process of the diffusion model, N correction information corresponding to the noisy speech data to be processed can be obtained at N sampling time points in the frequency domain, thereby performing N denoising processes to obtain the final denoised speech data. It can be understood that when the data mapping trajectory of the diffusion model is a straight line, the value of N can be selected as a small value, so that the inference time of the diffusion model is short and the speed is fast.

[0016] In one possible implementation of the first aspect described above, the (i+1)th frequency domain data is obtained as follows: For the i-th sampling time point, the diffusion model differentiates the i-th frequency domain data with respect to time to obtain the i-th correction information, where the i-th correction information indicates the rate of data change from the i-th frequency domain data to the (i+1)-th frequency domain data; a correction algorithm is then used to denoise the i-th frequency domain data based on the i-th correction information to obtain the (i+1)-th frequency domain data. It can be understood that N represents the number of denoising inference steps of the diffusion model. Therefore, in one denoising inference step of the diffusion model, the rate of data change of the noisy speech data can be used as the correction information corresponding to the current time step.

[0017] In one possible implementation of the first aspect described above, the correction algorithm includes at least one of Euler's algorithm and Chasen extrapolation RK algorithm.

[0018] In one possible implementation of the first aspect described above, the i-th correction information is the ratio of the i-th frequency domain data to N; and the (i+1)-th frequency domain data is obtained by adding the i-th correction information to the i-th frequency domain data. It can be understood that since the data mapping trajectory of the diffusion model can be a straight line, the ratio of the i-th frequency domain data to N, i.e., the i-th correction information, can represent the difference between the noisy speech data and the corresponding denoised speech data at the current time step.

[0019] In one possible implementation of the first aspect above, the diffusion model is trained in the following manner: acquiring first training time-domain data and second training time-domain data of the training speech, wherein the second training time-domain data is obtained by superimposing preset time-domain noise data on the first training time-domain data; converting the first training time-domain data and the second training time-domain data from the time domain to the frequency domain to obtain first training frequency-domain data and second training frequency-domain data respectively; and training the diffusion model based on the first training frequency-domain data and the second training frequency-domain data to obtain the trained diffusion model.

[0020] This application provides a speech processing method. During the training of an ODE-based diffusion model, a pre-defined linear relationship can be constructed between clean training speech data (i.e., first training frequency domain data) and noisy training speech data (second training frequency domain data), ensuring that the data mapping trajectory between the clean and noisy training speech data is a straight line. Specifically, this application trains the diffusion model to transform clean training speech data into noisy training speech data along a straight data mapping trajectory by constructing training samples and training targets in the diffusion model. This enables the pre-trained diffusion model to denoise the noisy speech data to be processed, obtaining denoised speech data.

[0021] In one possible implementation of the first aspect above, the noise indicated by the preset time-domain noise data includes at least one of the following: wind noise, whistle noise, keyboard noise, and human voice noise.

[0022] In one possible implementation of the first aspect above, training the diffusion model based on the first training frequency domain data and the second training frequency domain data includes: acquiring the training sampling time; performing linear interpolation on the first training frequency domain data and the second training frequency domain data according to the training sampling time and based on a preset linear relationship to obtain the third training frequency domain data; superimposing preset frequency domain noise data on the third training frequency domain data to obtain the fourth training frequency domain data, wherein the dimension of the third training frequency domain data is the same as the dimension of the preset frequency domain noise data; inputting the fourth training frequency domain data into the diffusion model, and obtaining the first training correction information by differentiating the fourth training frequency domain data with respect to time through the diffusion model; and adjusting the network parameters of the diffusion model according to the difference between the first training correction information and the preset correction information.

[0023] In one possible implementation of the first aspect described above, the third training frequency domain data x t Obtain it using the following formula: x t = t*x+(1-t)*y, where x represents the first training frequency domain data, y represents the second training frequency domain data, t represents the training sampling time, and x and y in t*x+(1-t)*y have a preset linear relationship.

[0024] In one possible implementation of the first aspect described above, the fourth training frequency domain data x t_noise Obtain x through the following methods: t_noise =μ+δz, where μ represents the mean of the third training frequency domain data, δ represents the standard deviation of the third training frequency domain data, and z represents the preset frequency domain noise data.

[0025] In one possible implementation of the first aspect above, 0 ≤ δ ≤ 1, and the value of δ first increases and then decreases as the training sampling time increases.

[0026] In one possible implementation of the first aspect above, the preset frequency domain noise data is Gaussian noise data with a mean of 0 and a variance of 1.

[0027] In one possible implementation of the first aspect above, the preset correction information is yx.

[0028] In one possible implementation of the first aspect above, the difference between the first training correction information and the preset correction information is determined by the loss function loss = mse(v, yx), where mse() represents the mean squared error.

[0029] In one possible implementation of the first aspect described above, the first frequency domain data, the first training frequency domain data, and the second training frequency domain data are acquired using the Short Time Fourier Transform (STFT) technique.

[0030] Secondly, embodiments of this application provide a readable medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the speech processing method of the first aspect and any possible implementation thereof.

[0031] Thirdly, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, one of the processors of the electronic device, for executing the voice processing method in the first aspect and any possible implementation thereof.

[0032] Fourthly, embodiments of this application provide a computer program product that, when run on an electronic device, enables the electronic device to implement the voice processing method in the first aspect and any possible implementation thereof. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of a speech processing scenario using a discriminative speech denoising model, a traditional method.

[0034] Figure 2 This is a schematic diagram of a speech processing scenario using a diffusion model provided in an embodiment of this application;

[0035] Figure 3 A schematic diagram of the data mapping trajectory for speech processing using a diffusion model provided in this application embodiment;

[0036] Figure 4 A schematic diagram of a data mapping trajectory based on an ODE diffusion model provided in this application embodiment;

[0037] Figure 5 A schematic diagram illustrating the time-domain and frequency-domain data of speech during speech denoising, provided in an embodiment of this application;

[0038] Figure 6 This is a schematic diagram illustrating an application scenario of a speech processing method provided in an embodiment of this application;

[0039] Figure 7 A schematic flowchart of a speech processing method provided in an embodiment of this application;

[0040] Figure 8 A schematic diagram illustrating the signal-to-noise ratio calculation process for voice data to be processed, provided in an embodiment of this application;

[0041] Figure 9 A schematic diagram illustrating the calculation process of a speech activity sequence provided in an embodiment of this application;

[0042] Figure 10 This application provides a schematic diagram of the denoising inference process of a diffusion model for processing speech data, as shown in an embodiment of the present application.

[0043] Figure 11 A flowchart illustrating a training method for a diffusion model provided in an embodiment of this application;

[0044] Figure 12 This is a schematic diagram of the structure of a mobile phone provided for an embodiment of the application. Detailed Implementation

[0045] The illustrative embodiments of this application include, but are not limited to, voice processing methods, media, electronic devices, and program products.

[0046] In some embodiments, in order to address the problem that speech denoising processes in the background art damage effective speech, this application provides a pre-trained diffusion model to denoise noisy speech data to obtain denoised speech data.

[0047] In the field of speech denoising, diffusion models can be used to recover speech data damaged by noise. Specifically, diffusion models learn the data distribution by progressively adding noise to initial clean speech data using a neural network and then attempting to reduce the noise. Ultimately, the neural network can generate denoised speech data that is close to or identical to the initial clean speech data, meaning it restores the initial clean speech data as closely as possible. Therefore, diffusion models denoise noisy speech data by generating new data, rather than by labeling noise and directly removing the labeled noise from the noisy speech data. This avoids damage to the valid speech data in the noisy speech data due to mislabeling or other reasons.

[0048] like Figure 2 As shown, diffusion models typically involve two processes: a diffusion process (i.e., a forward diffusion process) and a reverse diffusion process (i.e., a backward diffusion process). For example, as... Figure 2 As shown, during the forward pass, Gaussian noise is gradually added to the clean speech data x0 over T (e.g., T=4) sampling time points to obtain the noisy data x. T The reverse process is to process the noisy data x. T The process involves gradually denoising and restoring the speech data x0 over T sampling time points. This is understandable. Figure 2 The speech data at each sampling time point can be spectral data, also known as a spectrogram, such as speech data x. T It is a spectrum diagram.

[0049] Currently, speech denoising processes based on diffusion models perform denoising inference on noisy speech data in different environments with the same number of inferences, i.e., denoising inference at the same sampling time point. For example, in noisy and quiet environments, different noisy speech data can be denoised after T inferences (e.g., T=4). Therefore, in the aforementioned techniques, even in quiet environments, noisy speech data undergoes 4 denoising inferences. However, since noisy speech data in quiet environments typically contains less noise, cleaner denoised speech data can be obtained with fewer inferences, such as 2. In this case, the last two denoising processes in the 4 denoising inferences performed in quiet environments are unnecessary, leading to a waste of computational resources.

[0050] To avoid wasting computational resources in the denoising process, this application provides a speech processing method. This method first determines the noise level of the environment in which the noisy speech data to be processed is located. For example, it can obtain parameters characterizing the noise level, such as the signal-to-noise ratio (SNR) of the noisy speech data (denoted as the noise level parameter). Then, based on the noise level parameter, different denoising inference counts are set. For example, different denoising inference counts are set for noisy speech data with different SNRs. It can be understood that a lower SNR indicates a higher noise level, and a larger number of inference counts, such as 10, is set; conversely, a higher SNR indicates a lower noise level, and a smaller number of inference counts, such as 2, is set. Thus, this application can automatically match the appropriate number of denoising inference counts based on parameters characterizing the noise level, such as the SNR of the noisy speech data, ensuring the denoising effect. This avoids performing unnecessary denoising inference in a quiet environment, thereby avoiding wasting computational resources in the denoising process.

[0051] In some embodiments, the parameters characterizing noise levels in this application may further include the following parameters: sound pressure level (SLP), sound energy, sound frequency, sound intensity, and sound energy density. Generally, the higher the values ​​of these parameters, the greater the noise. In this case, a larger parameter characterizing noise levels corresponds to a higher inference time, and vice versa.

[0052] In addition, in some other embodiments, the parameters characterizing the level of noise in this application may also be based on the application or location where the voice data to be processed is located.

[0053] As an example, for the application containing the noisy audio data to be processed, the inference count N is 3 for a recording application, 2 for a call application, and 1 for a voice assistant application. In some embodiments, when the electronic device collects the noisy audio data to be processed through a microphone, the application that calls the microphone, such as an application currently in the foreground, is detected.

[0054] As another example, the value of N is set for the location of the noisy speech data to be processed. For example, a relatively quiet location such as a home, meeting room, small street, or library corresponds to a value of 2; a relatively noisy location such as a restaurant, office, or shopping mall corresponds to a value of 5; and an even noisier location such as a train station, market, or factory corresponds to a value of 10. In some embodiments, when the electronic device collects the noisy speech data to be processed through a microphone, the location can be determined by locating the current position.

[0055] In the following embodiments, the signal-to-noise ratio (SNR) is used as an example to illustrate the level of noise.

[0056] In some embodiments, this application can set one or more signal-to-noise ratio (SNR) thresholds to divide multiple SNR ranges. Furthermore, different inference data are set for noisy speech data in different SNR ranges. It can be understood that noisy speech data with a higher SNR corresponds to smaller inference data. For example, in noisy environments such as factories, stations, and markets, the SNR of noisy speech data is relatively small, and a larger number of inferences is set, such as 10. In moderately noisy environments such as restaurants, offices, shopping malls, and parks, the SNR of noisy speech data is relatively large, and a smaller number of inferences is set, such as 5. In quiet environments such as homes, meeting rooms, small streets, and libraries, the SNR of noisy speech data is even larger, and a smaller number of inferences is set, such as 2.

[0057] In some embodiments, the process of calculating the signal-to-noise ratio (SNR) of noisy speech data in this application includes: distinguishing between valid speech data and noise data in the speech data to be processed, and calculating the energy of the valid speech data and the energy of the noise data in the speech data to be processed. Then, the SNR of the speech data to be processed is calculated based on the energy of both.

[0058] It is understood that the denoising process in this application is implemented through a diffusion model, and the number of denoising inferences refers to the number of times the diffusion model performs denoising processing on a noisy speech data.

[0059] In some embodiments, the diffusion model in this application can be based on stochastic differential equations (SDEs) or ordinary differential equations (ODEs) to construct clean speech data x(0) (i.e., x0) and noisy speech data x(T) (i.e., x0). T The data mapping relationship between ( ). For example, Figure 3 As shown, the data mapping relationship of the diffusion model is used to transform the data distribution, specifically transforming the data distribution p0(x) of clean speech data x(0) to the data distribution p0(x) of noisy speech data x(T). T (x). In practical applications, the diffusion model converts clean speech data x(0) into noisy speech data x(T) through T sampling time points. The data distribution of the speech data can be visualized through a time-frequency domain spectrogram, that is, the data distribution of the speech data can be represented by a spectrogram.

[0060] In some embodiments, the diffusion model in this application can be implemented based on stochastic differential equations (SDEs), ordinary differential equations (ODEs) of arc-shaped trajectories, or ODEs of straight-line trajectories. For example... Figure 3 As shown, the dark curve, light arc, and dark straight line represent the data mapping trajectories corresponding to the SDE, the ODE of the arc trajectory, and the ODE of the straight line trajectory, respectively. Specifically, in the forward process, the clean speech data x is transformed from data x(0) in the time-frequency domain through T sampling time points along the data mapping trajectory to the prior data x(T). Furthermore, in the backward process, the prior data x(T) is transformed into clean speech data x(0) in the time-frequency domain through T sampling time points along the dark curve. It can be understood that the above T sampling time points are equivalent to T denoising inferences.

[0061] like Figure 3As shown, L1 and L1' are the data mapping trajectories of the forward and backward processes of SDE, respectively, while L2 and L2' are the data mapping trajectories of the forward and backward processes of conventional ODE, respectively. The trajectories L1 and L1' of SDE exhibit large fluctuations and are more complex, while the trajectories L2 and L2' of conventional ODE exhibit smaller fluctuations and are simpler.

[0062] Furthermore, in the numerical solution process of ODE, in order to reduce the numerical error of solving ODE, the curved trajectory can be adjusted to a smoother and more direct straight trajectory, which helps to reduce the training time of the diffusion model and improve computational efficiency.

[0063] In some embodiments, this application can straighten the ODE data mapping trajectory of the diffusion model using reflow techniques. (Continue to refer to...) Figure 3 L3 and L3' are the ODE data mapping trajectories straightened using the corrected flow technique, respectively. As an example, the speech processing method in this application can specifically use the straight data mapping trajectory of the diffusion model to denoise the noisy speech data to obtain denoised clean speech data.

[0064] Furthermore, referring to Figure 4 The data mapping trajectory of ODE is explained. Figure 4 In (a) of the diagram, the data mapping trajectory L4, which curves continuously over time between clean speech data x and noisy speech data y, can be achieved through four time steps: L41, L42, L43, and L44 during the sampling process. Furthermore, Figure 4 In (a), the straight data mapping trajectory of clean speech data x and noisy speech data y over continuous time is shown as L5. Correspondingly, the straight trajectory L5 during sampling can be achieved through one time step, L51. Therefore, it can be seen that the training and inference speed of constructing the straight ODE data mapping trajectory in the diffusion model is relatively fast.

[0065] In some embodiments, the speech processing method provided in this application sets the data mapping relationship between clean speech data and noisy speech data in the ODE-based diffusion model as a preset linear relationship, so that the trajectory of this data mapping relationship is a straight line. Thus, based on the straight data mapping trajectory, the training and inference speed of the speech denoising process of the diffusion model is faster.

[0066] In some embodiments, the pre-trained diffusion model in this application can be constructed based on a linear ordinary differential equation, wherein the linear ordinary differential equation is constructed based on a preset linear relationship between noisy speech data and corresponding clean speech data. Specifically, during the denoising inference process of the diffusion model on the noisy speech data, the ordinary differential equation is used to indicate the data transformation speed over time during the denoising process of the noisy speech data, specifically indicating the difference between the noisy speech data and the corresponding clean speech data. For example, at a time step, the correction information of the noisy speech data to be processed can represent the difference between the denoised speech signal (i.e., clean speech data) corresponding to the noisy speech data and the noisy speech data. At this time, adding the correction information to the noisy speech data can obtain the corresponding denoised speech data.

[0067] Therefore, during the denoising inference process, correction information for noisy speech data can be inferred using a diffusion model. Specifically, the correction information for noisy speech data at each time step can be determined, and the corresponding denoised speech data can be identified using this correction information. Subsequently, after one or more time steps, the identified denoised speech data is the corresponding clean speech data.

[0068] Specifically, in the speech denoising inference process of the diffusion model, the number of denoising inference iterations N is first determined based on parameters representing the noise level of the speech data to be processed, such as the signal-to-noise ratio. Then, the frequency domain data to be processed, such as the spectrogram, is obtained. Furthermore, in each denoising inference iteration, correction information corresponding to the frequency domain data to be processed is obtained through the pre-trained diffusion model. This correction information indicates the rate of change of the frequency domain data to be processed over time into the corresponding clean speech data (i.e., effective speech data). Thus, in each denoising inference iteration, the noisy frequency domain data is denoised using this correction information to obtain the corresponding denoised frequency domain data. In this way, the above denoising inference process is executed N times to denoise the noisy frequency domain data, achieving a better denoising effect and avoiding unnecessary denoising inference processes.

[0069] Furthermore, the diffusion model provided in this application does not damage the speech data during speech denoising and exhibits strong generalization performance, demonstrating strong suppression capabilities for noise data not encountered during the training phase. Moreover, this application can first convert the noisy speech data from the time domain to the frequency domain, obtaining noisy frequency domain data, such as a spectrogram. Then, the denoised frequency domain data is converted from the frequency domain to the time domain to obtain denoised speech data. Since this application denoises the noisy speech data in the frequency domain, it utilizes the frequency distribution of the speech data to ensure the continuity of the speech spectrum in the denoised speech data, thereby obtaining clear denoised speech.

[0070] In some embodiments, the training process of the diffusion model in this application includes: establishing a linear relationship between clean training frequency domain data and corresponding noisy training frequency domain data in the frequency domain to construct original frequency domain data. Then, adding Gaussian noise or other noise to the original training frequency domain data in the frequency domain to obtain target training frequency domain data. Next, inputting the target noisy frequency domain data into the neural network and outputting training correction information, which indicates the rate of change of the noisy training frequency domain data into clean training speech data over time, specifically the difference between the denoised speech data corresponding to the noisy training frequency domain data (such as the frequency domain data of the clean training speech data). Finally, adjusting the network parameters of the neural network based on the difference between the training correction information and preset correction information to obtain the trained diffusion model.

[0071] Furthermore, in some embodiments, after determining the number of inferences N using parameters characterizing the noise level, such as the signal-to-noise ratio of the noisy speech data, this application can obtain N correction information corresponding to the frequency domain data to be processed through a diffusion model at N sampling time points, thereby performing N denoising processes to obtain the final denoised frequency domain data. Since the data mapping trajectory of the diffusion model is a straight line, the value of N at the N sampling time points can be selected as a small value, resulting in a shorter inference time and faster speed for the diffusion model. Here, N is a positive integer.

[0072] like Figure 5 The diagram shows the time-domain and frequency-domain data of speech during the speech denoising process. Figure 5 The noisy speech data A1 shown in (a) is noisy speech collected in a supermarket setting, and Figure 5 Figure (a) shows the time-domain waveform A11' and the corresponding spectrogram B1 of a segment of noisy speech A11 from the noisy speech data A1. Figure 5 Figure (b) shows the time-domain waveform A12 and the corresponding spectrogram B2 after denoising the noisy speech data A11 using the technique described in the background. While the time-domain waveform A12 does not contain noisy data, some valid speech data may still be removed. Furthermore, Figure 5 (c) shows the time-domain data waveform A13 and the corresponding spectrogram B3 after denoising the noisy speech A11 based on the diffusion model of this application. At this time, the time-domain data waveform A13 does not contain noise data, but also contains relatively complete effective speech data, and the denoising effect is good.

[0073] The application scenarios of the speech processing method of this application include, but are not limited to: recording (such as classroom recording), human-computer interaction scenarios (such as human-computer interaction based on voice assistant), voice calls, video calls, video conferencing, and other scenarios.

[0074] In some embodiments, after the microphone has captured complete noisy speech data, the electronic device can perform denoising processing on the complete noisy speech data to obtain denoised speech data. For example, in recording or human-computer interaction scenarios, the electronic device can perform denoising processing on the complete noisy speech data. Figure 6 (a) illustrates a human-computer interaction scenario. Specifically, when a user listens to the radio using mobile phone 10 in a park, with the input control 12 for the voice assistant displayed on the desktop interface 11 of mobile phone 10, the user collects noisy voice data containing the voice "play the morning news" as well as human voices and birdsong in the park through the microphone. At this time, mobile phone 10 can perform noise reduction processing on the noisy voice data to obtain denoised voice data containing only the voice "play the morning news", and then use the voice assistant to call the audio software to play the audio of the morning news.

[0075] In other embodiments, the electronic device can perform segmented denoising processing during the microphone's acquisition of noisy speech data. Specifically, it can denoise a segment of noisy speech data, each lasting tens of milliseconds (e.g., 40ms), before denoising another segment of noisy speech data acquired in real-time. For example, in scenarios such as voice calls, video calls, and video conferencing, the real-time acquired speech can be segmented for denoising at 40ms intervals. Figure 6 (b) illustrates a video call scenario. Specifically, when a user uses mobile phone 10 to video call a customer in a factory, mobile phone 10 displays call interface 13 and collects noisy voice data containing the voice message "The production line is working normally" and the sound of the production line equipment operating. At this time, mobile phone 10 can perform segmented denoising processing on the noisy voice data to obtain each segmented denoised voice data, and then transmit the segmented denoised voice data to the friend in the video call in chronological order. In this way, it is ensured that the friend can hear and clearly hear the voice message "Look at the big ship at sea!" in a timely manner.

[0076] The voice processing method provided in this application can be applied to various electronic devices. For example, the electronic device is not limited to mobile phone 10, but can also be a smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, etc. This application does not impose any restrictions on the specific type of electronic device.

[0077] Optionally, the entity executing the speech processing method in this application may be an electronic device, or a device in the electronic device used to execute the speech processing method (referred to as a speech processing device). This application does not specifically limit this.

[0078] The speech processing method provided in the embodiments of this application will be described next.

[0079] In some embodiments, the number of signal-to-noise ratio (SNR) thresholds set in this application is one or more, and the inference number N corresponding to different SNR values ​​is different. SNR is an important standard for measuring signal quality and is usually expressed in decibels (dB).

[0080] As an example, when the signal-to-noise ratio (SNR) threshold is singular, it can be set to 0 dB. Furthermore, for speech data with an SNR greater than 0 dB in a relatively quiet environment such as a library, conference room, or park, the inference iterations N1 of the diffusion model are set to 5. For speech data with an SNR less than or equal to 0 dB in a noisier environment such as a factory or shopping mall, the inference iterations N2 of the diffusion model are set to 10. In this case, N1 = 2 × N2. Combined with... Figure 6 Example, Figure 6 In (a) shown in the diagram, the signal-to-noise ratio of the speech data to be processed in the park is high, for example, greater than 0 dB, allowing the diffusion model to perform inference five times. Furthermore, Figure 6 In (b) shown in the diagram, the signal-to-noise ratio of the speech data to be processed in the factory is small, for example, less than 0 dB, and the number of inferences of the diffusion model can be 10.

[0081] As another example, when there are two signal-to-noise ratio (SNR) thresholds, the SNR threshold can be set to -5 dB, and the larger SNR threshold can be 5 dB. Furthermore, for speech data with an SNR greater than 5 dB in a relatively quiet environment such as a library or conference room, the inference count N3 of the diffusion model is set to 2. For speech data with an SNR less than or equal to 5 dB but greater than -5 dB in a moderately noisy environment such as a restaurant or office, the inference count N4 of the diffusion model is set to 5. For speech data with an SNR less than or equal to -5 dB in a noisy environment such as a train station or factory, the inference count N5 of the diffusion model is set to 10.

[0082] In some embodiments, refer to Figure 7 As shown, taking a signal-to-noise ratio threshold of 1 as an example, the speech processing method will be explained. Specifically, Figure 7 The illustrated process includes the following steps:

[0083] S701: Acquire the voice data to be processed. The voice data to be processed also becomes the first voice data to be denoised.

[0084] The voice data to be processed includes valid voice data and noise data.

[0085] S702: Calculate the signal-to-noise ratio of the speech data to be processed.

[0086] This application can distinguish between valid speech data and noise data in the speech data to be processed through methods such as speech activity detection. Then, a speech activity sequence of the speech data to be processed can be calculated, in which the values ​​of the data points containing valid speech data differ from the values ​​of the data points containing noise data. Furthermore, based on the aforementioned activity sequence, the energy of the valid speech data and the energy of the noise data in the speech data to be processed are calculated. Then, the signal-to-noise ratio (SNR) of the speech data to be processed is calculated based on the energy of both. Specifically, the calculation of the SNR of the speech data to be processed will be described in detail below and will not be repeated here.

[0087] S703: Determine whether the signal-to-noise ratio is greater than the first threshold.

[0088] If the signal-to-noise ratio is greater than the first threshold, proceed to S704; if the signal-to-noise ratio is less than or equal to the first threshold, proceed to S706.

[0089] It is understood that different signal-to-noise ratios of the speech data to be processed in this application correspond to different inference times of the diffusion model.

[0090] S704: Convert the data to be processed from the first time domain data to the first frequency domain data.

[0091] In some embodiments, this application may employ Fourier transform methods such as short-time Fourier transform (STFT) to convert time-domain data into frequency-domain data, specifically time-frequency domain data, through feature transformation. For example, the first frequency-domain data after feature transformation is time-frequency domain data, such as a spectrum.

[0092] S705: Determine the number of inferences as N1, and perform N1 denoising inferences on the first frequency domain data using a pre-trained diffusion model to obtain the second frequency domain data.

[0093] It is understood that the pre-trained diffusion model in this application can perform denoising on the speech data to be processed in the frequency domain and obtain denoised speech data in the frequency domain.

[0094] In some embodiments, the pre-trained diffusion model in this application may be an ODE-based diffusion model, but is not limited thereto. For example, the ODE-based diffusion model in this application, after multiple denoising inference processes, can remove noise from the speech data to be processed along a straight data mapping trajectory and evolve the corresponding denoised speech data (i.e., clean speech data). The training and inference processes of the diffusion model in this application will be described in detail below and will not be repeated here.

[0095] It is understood that the number of inferences performed by the diffusion model in this application to denoise the speech data to be processed depends on the signal-to-noise ratio of the speech data to be processed, i.e., the noise level of the environment in which the speech data to be processed is located.

[0096] S706: Convert the data to be processed from the first time domain data to the first frequency domain data.

[0097] Similarly, the description of S706 can be found in the description of S704 above, and will not be repeated here.

[0098] S707: Determine the number of inferences as N2, and perform N2 denoising inferences on the first frequency domain data using a pre-trained diffusion model to obtain the second frequency domain data.

[0099] Where N1 is less than N2, such as N2 = 2 × N1. For example, if the first threshold is 0 dB, N1 is 5 and N2 is 10.

[0100] It is understandable that S704 and S705, as well as S706 and S707, are two parallel processes, and these two processes are divided based on a pre-set first threshold.

[0101] S708: Converts the second frequency domain data from the frequency domain to the second time domain data, and uses the second time domain data as denoised speech data. The denoised speech data is also called the denoised second speech data.

[0102] The second time-domain data mentioned above refers to denoised speech data in the time domain. In this case, the electronic device can output the denoised speech data through a speaker, etc. For example, in a recording scenario, after the electronic device captures the recording through a microphone, it can output the denoised speech data of the recording after denoising processing using a diffusion model. Similarly, in a voice call scenario, after the electronic device captures the call speech through a microphone, it can transmit the denoised speech data of the call to the other party after denoising processing using a diffusion model.

[0103] Thus, the speech processing method provided in this application can determine the corresponding number of denoising inferences N based on the signal-to-noise ratio of the speech data to be processed, i.e., the noise level of the environment in which the speech data is located. Then, through a pre-trained diffusion model, the speech data to be processed is denoised through N denoising inference processes in the frequency domain to obtain the corresponding denoised frequency domain data. Since the number of inferences is determined based on the actual environment in which the speech data is located, good denoising results are achieved while avoiding unnecessary denoising inference processes, which helps to improve the speed of speech denoising.

[0104] Next, refer to Figure 8 The signal-to-noise ratio calculation process for the speech data to be processed provided in the embodiments of this application is described below. Specifically, as shown... Figure 8 As shown above, Figure 7 S702 in the process may include the following steps:

[0105] S801: Detects the speech activity sequence of the speech data to be processed.

[0106] In this speech activity sequence, the data with the first value is speech data, and the data with the second value is noise data. For example, the first value is 1 and the second value is 0, but it is not limited to this. In this case, the speech activity sequence is a sequence including 0 and 1, where 0 represents noise without speech and 1 represents speech.

[0107] Optionally, the speech activity sequence can be implemented using speech activity detection algorithms such as energy detection, zero-crossing rate detection, hidden Markov models (HMMs), and long short-term memory networks (LSTMs). Speech activity detection algorithms are also known as endpoint detection algorithms. Specifically, the calculation process for the speech activity sequence in this application will be described in detail below, and will not be repeated here.

[0108] S802: The first energy p of the data taking the first value in the statistical speech activity sequence. s and the first data volume l s And calculate the second energy p of the data that takes the second value in the activity sequence. n Second data volume l n .

[0109] It can be understood that the first energy refers to the sum of the energies of the data with a value of 1 in the above speech activity sequence, that is, the energy p of the effective speech data in the speech data to be processed. s Accordingly, the first data quantity is the number of data points with a value of 1 in the aforementioned speech activity sequence, that is, the length l of the data points with a value of 1. s .

[0110] Similarly, the second energy refers to the sum of the energies of the data with a value of 0 in the above speech activity sequence, that is, the energy p of the noise data in the speech data to be processed. n Correspondingly, the second data quantity is the number of data points with a value of 0 in the aforementioned speech activity sequence, that is, the length l of the data points with a value of 0. n .

[0111] S803: According to the first energy p s First data volume l s Second energy p n Second data volume l n Calculate the signal-to-noise ratio of the speech data to be processed.

[0112] Specifically, this application uses formula (1) to calculate the signal-to-noise ratio (SNR) of the speech data to be processed.

[0113]

[0114] Wherein, p in the above formula (1) n / l n p represents the average energy of the data points containing the noise data in the speech data to be processed. s / l s This represents the average energy of the data points containing valid speech data in the speech data to be processed.

[0115] Thus, this application can detect the speech activity sequence of the speech data to be processed, and calculate an accurate signal-to-noise ratio (SNR) based on the energy and data volume distribution of effective speech data and noise data in the speech activity sequence. This improves the adaptability of the denoising inference count obtained from the SNR to the noise level of the current environment, which helps to ensure denoising effectiveness while avoiding waste of computational resources.

[0116] In some embodiments, reference Figure 9 The diagram shown is a schematic representation of the calculation process for a speech activity sequence provided in an embodiment of this application. Figure 9 As shown above, Figure 8 S801 may include the following steps S901 to S905:

[0117] S901: The speech data to be processed is divided into frames to obtain multiple speech frames.

[0118] Specifically, the framing process can divide the continuous speech data represented by the speech data to be processed into a series of overlapping short frames, each of which typically lasts from tens of milliseconds to hundreds of milliseconds.

[0119] S902: Calculate the short-time energy of each speech frame.

[0120] Specifically, short-time energy can be calculated by squaring each sample point in a speech frame and summing the results.

[0121] S903: Calculate the short-time zero-crossing rate for each speech frame.

[0122] For each speech frame, the number of negative waveform crossovers in the speech data can be calculated to obtain the short-time zero-crossing rate. Specifically, a higher zero-crossing rate indicates more frequent waveform changes.

[0123] S904: Obtain the energy threshold and zero-crossing rate threshold.

[0124] The energy threshold and zero-crossing rate threshold mentioned above can be set according to actual needs, and can be adjusted according to the characteristics of the voice data to be processed. This application embodiment does not make specific limitations in this regard.

[0125] S905: Perform endpoint detection on multiple speech frames of the speech data to be processed, and determine the start and end points of each speech segment based on the energy threshold and the zero-crossing rate threshold.

[0126] In the endpoint detection process, this application can scan each frame starting from the first speech frame in the speech data to be processed, and the frame is considered the starting point of the speech segment when the following preset conditions are met. Similarly, in reverse scanning, the frame is considered the ending point of the speech segment when the following preset conditions are met.

[0127] Optionally, the above preset conditions include at least one of the following:

[0128] (1) The energy of the current frame exceeds the energy threshold; (2) The energy of the next frame does not exceed the energy threshold; (3) The zero-crossing rate of the current frame exceeds the zero-crossing rate threshold; (4) The energy of the current frame does not exceed the energy threshold; (5) The zero-crossing rate of the current frame does not exceed the zero-crossing rate threshold.

[0129] S906: Detect each speech segment and each non-speech segment of the speech data to be processed, and obtain the speech activity sequence.

[0130] In this application, based on the start and end points of each detected speech segment, data containing speech segments are marked as 1, and other segments are marked as non-speech segments as 0, thus obtaining a speech activity sequence containing 0 and 1.

[0131] It is understood that in some other embodiments, the speech activity sequence of this application can be detected in other ways, and is not limited to those described above. Figure 9 The methods shown are not specifically limited in the embodiments of this application.

[0132] Next, refer to Figure 10This paper introduces the denoising inference process of the pre-trained diffusion model for processing speech data in this application. For example... Figure 10 As shown above, Figure 7 S705 or S707 may include the following steps:

[0133] S1001: Obtain the N sampling times corresponding to the N inference times.

[0134] Combination Figure 7 For example, N can be N1 or N2, such as 5 or 10.

[0135] Optionally, this application can specify a preset time range t. min ~t max Divide the time into equal intervals to obtain N (e.g., N1 or N2) sampling times, which are N time steps. Where t min and t max The value can be set according to actual needs, for example, t. min The value is 0, t max The value is 1. In practical applications, to achieve normal sampling, the sampling time t is a value greater than 0, for example, t0. min The value can be 0.003.

[0136] As an example, this application can determine four sampling times from 0.003 to 1, that is, divide 0.003 to 1 into four time steps, with the four sampling times being 0.25, 0.5, 0.75, and 1, respectively. The sampling times of 0.25, 0.5, 0.75, and 1 can be denoted as t1, t2, t3, and t4, respectively.

[0137] It can be understood that each of the N sampling times is applied to one inference process of the diffusion model.

[0138] S1002: Take the first frequency domain data of the speech data to be processed as the first frequency domain data, and obtain the i-th correction information corresponding to the i-th frequency domain data through the pre-trained diffusion model.

[0139] Specifically, i takes the value 1 ≤ i ≤ N. In the first inference process of the diffusion model, the first frequency domain data is the first frequency domain data, and i is a positive integer. At this time, in the i-th denoising inference process, the diffusion model obtains the i-th correction information corresponding to the i-th frequency domain data.

[0140] In some embodiments, the pre-trained diffusion model evolves clean speech data x into corresponding noisy speech data y in the time-frequency domain; that is, the evolution process of the diffusion model is x→y. Specifically, the diffusion model can evolve speech data x0 into x over T time steps. T That is, the evolution process is x0→xT Where, at sampling time t = 0, x0 = x, and at sampling time t = T, x... T =y.

[0141] In some embodiments, the evolution process based on the ODE diffusion model in this application is as follows: dx t / dt=f(x t That is, the diffusion model defines the rate of data change in the form of differential equations, indicating how x0→x T The evolution of.

[0142] It is understood that x in this application t It involves pre-processing known clean speech data x (t = 0, x0 = x) to known noisy speech data y (t = T, x0 = x). T The data is obtained by linear interpolation between x and y in the forward process, so that the reverse process can derive the clean speech data x at t=0. Specifically, in this application, x... t The specific construction method will be described in detail below, and will not be repeated here.

[0143] Specifically, the ODE-based diffusion model can calculate the rate of change of speech data at each time step during the process of converting clean speech data into noisy speech data by gradually increasing noise over T time steps. Correspondingly, the value of time step N in the inference process is less than or equal to T. In this case, the N sampling times corresponding to the N time steps can be selected from the T sampling times corresponding to the T time steps.

[0144] It is understandable that in the denoising inference process of the diffusion model, x in the evolution process of the diffusion model... T (i.e., y) represents the first frequency domain data of the speech data to be processed. At this point, during one denoising inference process of the diffusion model, the rate of change of the speech data can be used as the correction information corresponding to the current time step.

[0145] In some embodiments, the i-th correction information in this application can be obtained by differentiating the i-th frequency domain data with respect to the i-th sampling time t using the following formula (2).

[0146]

[0147] Where x in formula (2) ti Let x represent the i-th frequency domain data, specifically the frequency domain data of the speech data to be processed before denoising during the i-th denoising inference process. At this point, x... ti Taking the derivative of the sampling time ti, we get v i v iIt represents the rate of change of the i-th frequency domain data over time, such as the difference between the i-th frequency domain data and the frequency domain data of its denoised speech data.

[0148] S1003: Denoise the i-th frequency domain data using the i-th correction information to obtain the (i+1)-th frequency domain data.

[0149] Specifically, this application can denoise the i-th frequency domain data by using the i-th correction information and N pairs of i-th frequency domain data to obtain the (i+1)-th frequency domain data.

[0150] In some embodiments, this application may employ a correction algorithm to denoise the i-th frequency domain data using the i-th correction information. For example, the correction algorithm may be implemented using Euler's algorithm or Richardson extrapolation (RK) (e.g., the RK-45 algorithm).

[0151] In some embodiments, the i-th correction information of the i-th frequency domain data is calculated using the following formula (3) to obtain the (i+1)-th frequency domain data:

[0152]

[0153] It's understandable, v i This represents the rate of change of the difference between the first frequency domain data and the effective speech data (clean speech data) within the first frequency domain data over time; that is, the rate of change of the first frequency domain data to the second frequency domain data. Where v i This is the i-th correction information, y i This refers to the i-th frequency domain data (i.e., the noisy speech data before the i-th denoising process), y i+1 This refers to the (i+1)th frequency domain data (i.e., the denoised speech data after the i-th denoising process). At this point, y N+1 This refers to the (N+1)th frequency domain data (denoted as the second frequency domain data), which is the frequency domain data after the speech data to be processed has undergone N denoising processes.

[0154] In some embodiments, there is a preset linear relationship between the first frequency domain data of the speech data to be processed and the second frequency domain data after denoising. In this case, the first frequency domain data can be denoised and evolved into the second frequency domain data by a diffusion model along a straight data mapping trajectory in the reverse process.

[0155] S1004: Determine whether the number of inferences i is less than N.

[0156] If i < N, then re-enter S1002 to continue denoising inference; if i = N, then enter S1005 to output the (N+1)th frequency domain data through the diffusion model.

[0157] S1005: For i = N, the (i+1)th frequency domain data is used as the second frequency domain data.

[0158] At this point, the second frequency domain data is the time-frequency domain data of the denoised speech data after the speech data to be processed has been denoised.

[0159] Thus, this application can use a pre-trained diffusion model to denoise the frequency domain data of the speech data to be processed along a straight data mapping trajectory according to a preset linear relationship, thereby obtaining denoised frequency domain data, which is beneficial to improving the speed of the speech denoising process.

[0160] Furthermore, refer to Figure 11 The training process of the diffusion model provided in the embodiments of this application will be described. For example... Figure 11 As shown, the training process of the diffusion model includes the following steps:

[0161] S1101: Obtain the first training time-domain data of the training speech.

[0162] The first training time-domain data may include the time-domain data of the training speech, that is, clean and lossless speech data in the time domain.

[0163] S1102: Obtain preset time-domain noise data, add the preset time-domain noise data to the first training time-domain data, and obtain the second training time-domain data of the training speech.

[0164] The aforementioned preset time-domain noise data can include noise from multiple sound sources, such as one or more noises including wind noise, whistle noise, and voice noise. Correspondingly, the second training time-domain data is a noisy speech data constructed in the time domain, i.e., low-quality speech data.

[0165] S1103: Convert the first training time-domain data and the second training time-domain data from the time domain to the frequency domain to obtain the first training frequency-domain data and the second training frequency-domain data, respectively.

[0166] In some embodiments, this application may employ Fourier transform methods such as short-time Fourier transform to convert time-domain data into frequency-domain data through feature transformation, specifically time-frequency domain data. The first training frequency-domain data and the second training frequency-domain data can be time-frequency domain data, such as a spectrogram.

[0167] S1104: Obtain the training sampling time.

[0168] In some embodiments, this application starts from a preset time range t min ~t max The training sampling time t is obtained by random sampling. Where t... min and t maxThe value can be set according to actual needs, for example, t. min The value is 0, t max The value is 1. In practical applications, the sampling time t is a value greater than 0. To ensure proper sampling, t can be set to a value greater than 0. min The value is 0.003, t max The value is 1.

[0169] Optionally, this application may address t min ~t max Divide the time into equal intervals to obtain T sampling times, which are T time steps.

[0170] For example, this application can determine four sampling times from 0.003 to 1, that is, divide 0.003 to 1 into four time steps, with the four sampling times being 0.25, 0.5, 0.75, and 1, respectively.

[0171] As an example, during a single training session of the diffusion model, the training sampling time t can be t2 (e.g., 0.5).

[0172] S1105: Based on the training sampling time, perform linear interpolation on the first training frequency domain data and the second training frequency domain data to obtain the third training frequency domain data.

[0173] In some embodiments, this application may perform linear interpolation on the first training frequency domain data and the second training frequency domain data according to the following formula (4) to obtain the third training frequency domain data. In this case, the third training frequency domain data is the training sample of the diffusion model.

[0174] x t = t*x + (1-t)*y (4)

[0175] In the training scenario of the model, x in formula (1) represents the first training frequency domain data of clean speech, y represents the second training frequency domain data of noisy speech, and t represents the training sampling time. t This represents the third training frequency domain data at training sampling time t. That is, x t This represents the data points on the data mapping trajectory between the first training frequency domain data x and the second training frequency domain data y at sampling time t in the diffusion model.

[0176] For example, when the training sampling time t is t2, x t It can be At this point, x and y in formula (4) have a pre-defined linear relationship.

[0177] During the training of the diffusion model, x tThe process involves training clean speech data x (t = 0, x0 = x) to training noisy speech data y (t = T, x). T =y) is obtained by linear interpolation along the forward process, so that the reverse process can be derived to the clean training speech data x at t=0.

[0178] S1106: Superimpose the third training frequency domain data with the preset frequency domain noise data to obtain the fourth training frequency domain data.

[0179] The preset frequency domain noise data can be Gaussian noise, such as Gaussian noise with a mean of 0 and a variance of 1.

[0180] It is understandable that Gaussian noise with a mean of 0 and a variance of 1 follows a standard normal distribution, also known as the Gaussian distribution. The standard normal distribution is a symmetrical bell-shaped curve, and its graph is symmetrical about the mean (such as 0).

[0181] In this case, the dimension of the third training frequency domain data is the same as the dimension of the preset frequency domain noise data. At this time, the fourth training frequency domain data is the noisy training sample for the diffusion model.

[0182] In diffusion models, noise distributed in a standard normal distribution is typically added to the data (such as clean, lossless speech data) to simulate the randomness of the data. During the training phase, the model learns how to diffuse noisy speech data (such as X... T The model then reverts to the original data (i.e., X0). In the generation phase, the model uses the learned knowledge to generate new data instances.

[0183] Combining formula (4), and referring to the following formula (5), we show how to construct the fourth training frequency domain data based on the preset frequency domain noise data.

[0184] x t_noise =μ+δ*z=x t +δ*z (5)

[0185] In the training scenario of the model, x t_noise This is the fourth training frequency domain data corresponding to the training speech. Correspondingly, μ represents the third training frequency domain data x. t The mean of z represents the preset frequency domain noise data, and δ represents the variance of the preset frequency domain noise data, such as Gaussian noise.

[0186] Optionally, δ varies with the training sampling time t. For example, δ can first increase and then decrease as the sampling time t increases. Furthermore, δ takes values ​​from 0 to 1. As an example, when the training sampling time t takes values ​​of 0.003, 0.5, and 1, the values ​​of δ are 0, 0.5, and 0, respectively. Of course, the value of δ in this application can be set according to actual needs and is not limited to the above examples.

[0187] For example, when the training sampling time t is t2, δ = δ2 = 0.5.

[0188] S1107: Input the fourth training frequency domain data and the training sampling time into the diffusion model, and use the diffusion model to differentiate the fourth training frequency domain data with respect to the training sampling time to obtain the first training correction information.

[0189] It is understood that this application specifically uses a diffusion model to learn the rate of change between two distributed data points, as described in the first training correction information above.

[0190] Optionally, the diffusion model in this application can be implemented using neural network models such as gradient-based noise-conditional score networks (NCSN) or U-shaped networks (Unet), but is not limited to these.

[0191] In some embodiments, this application may use the following formula (6) to differentiate the fourth training frequency domain data with respect to the sampling time t, that is, to achieve x t_noise Take the derivative with respect to t.

[0192]

[0193] Here, v is the first training correction information mentioned above. Furthermore, since x... t_noise Gaussian noise was introduced, thus increasing the diffusion model by adding Gaussian noise to x. t_noise Robustness of differentiating with respect to sampling time t. Ideally, the first training correction information v = xy. It can be understood that v in formula (1) above... i It can be determined according to formula (5).

[0194] For example, when the training sampling time t is t2, At this point, the first training correction information represented by v2 is

[0195] It can be understood that the first training correction information v mentioned above is used to represent x. t The rate at which the value of changes over time is used to represent the difference in the frequency domain between training clean speech data and training noisy speech data at training sampling time t, i.e., the rate at which the training clean speech data changes to the training noisy speech data.

[0196] S1108: Adjust the network parameters of the diffusion model based on the difference between the first training correction information and the preset correction information.

[0197] Optionally, the aforementioned preset correction information can be yx, which is the difference between the training noisy speech data and the training clean speech data in the time-frequency domain. In this case, the preset correction information can represent the noise data in the training noisy speech data other than the training clean speech data.

[0198] In some embodiments, this application can calculate a loss function based on the difference between the first training correction information and the preset correction information to perform iterative training of the diffusion model. Training is terminated after a preset number of rounds, or when the loss function value is less than a first threshold. It is understood that as the loss function value gradually decreases, the diffusion model used in the backdiffusion process can acquire the ability to infer x from y through N steps.

[0199] It can be understood that the first training correction information mentioned above is the true value of the diffusion model, and the preset correction information is the predicted value of the training target. Optionally, the loss function mentioned above includes, but is not limited to, regression loss functions, such as mean-square error (MSE) and mean absolute error (MAE). MSE calculates the mean of the squares of the differences between the predicted value and the true value, while MAE is the mean of the absolute values ​​of the differences between the predicted value and the true value.

[0200] For example, the loss function of this application can be calculated by loss = mse(v, yx). Where mse() is the mean squared error, v represents the first training correction information, and yx represents the preset correction information.

[0201] Thus, this application provides a speech processing method that, during the training of an ODE-based diffusion model, can construct a preset linear relationship between clean training speech data and noisy training speech data, ensuring that the data mapping trajectory between the clean training speech data and the noisy training speech data is a straight line. Specifically, this application trains the diffusion model to transform clean training speech data into noisy training speech data along a straight data mapping trajectory by constructing training samples and training targets in the diffusion model. This enables the pre-trained diffusion model to denoise the noisy speech data to be processed, obtaining denoised speech data.

[0202] In some embodiments, the speech processing method provided in this application can be executed by various functional modules in a speech processing device. For example, the speech processing device may include the following modules: an inference count acquisition module, used to acquire the inference count N of the speech data to be processed; a feature transformation module, used to convert the first time-domain data of the data to be processed from the time domain to the frequency domain to obtain the first frequency-domain data; a sampling time generation module, used to acquire N sampling times corresponding to the N inference counts N; a neural network calculation module, used to acquire the i-th correction information corresponding to the i-th frequency-domain data during the i-th inference process; a signal enhancement module, used to denoise the i-th frequency-domain data using the i-th correction information to obtain the (i+1)-th frequency-domain data; an inference count judgment module, used to determine whether the inference count i is less than N; and a feature inverse transformation module, used to convert the (N+1)-th frequency-domain signal into time-domain data as denoised speech data.

[0203] In addition, in some other embodiments, the above-mentioned speech processing device may further include the following modules: a noise simulation module for generating preset time-domain noise data such as wind sounds and whistles; a Gaussian noise generation module for acquiring preset frequency-domain noise data such as Gaussian noise; a training sample generation module for linearly interpolating the first training frequency-domain data and the second training frequency-domain data according to the training sampling time to obtain the third training frequency-domain data; a training sample noise addition module for superimposing the third training frequency-domain data with preset frequency-domain noise data to obtain the fourth training frequency-domain data; and a loss calculation module for adjusting the network parameters of the diffusion model according to the difference between the first training correction information corresponding to the fourth training frequency-domain data and the preset correction information. Furthermore, the feature change module converts the first training time-domain data and the second training time-domain data from the time domain to the frequency domain. The neural network calculation module is also used to calculate the first training correction information corresponding to the fourth training frequency-domain data.

[0204] Of course, the various functional modules of the voice processing device provided in this application embodiment include, but are not limited to, the examples above, and may also include other functional modules.

[0205] Next, the hardware structure of the electronic device to which the voice processing method provided in the embodiments of this application is applicable will be described. Specifically, refer to... Figure 12 Taking a mobile phone as an example, this paper explains the hardware structure of electronic devices.

[0206] like Figure 12 As shown, the mobile phone 10 may include a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a sensor module 190, an audio module 150, a camera 170, an interface module 160, buttons 101, and a display screen 102, etc.

[0207] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the mobile phone 10. In other embodiments of this application, the mobile phone 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0208] Processor 110 may include one or more processing units, such as processing modules or circuits of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), microprocessor (MCU), artificial intelligence (AI) processor, or field programmable gate array (FPGA). Different processing units may be independent devices or integrated into one or more processors. Processor 110 may include storage units for storing instructions and data. In some embodiments, the storage unit in processor 110 is a cache memory 180. For example, the cache memory 180 may be used to store the pre-trained ODE-based diffusion model mentioned above, as well as one or more pre-set signal-to-noise ratio (SNR) thresholds. Furthermore, processor 110 can calculate the SNR of the speech data to be processed and the corresponding denoising inference times N, to obtain denoised speech data through N denoising inferences using the pre-trained diffusion model. Specifically, the processor 110 can perform denoising processing on the noisy speech data to be processed based on the straight-line data mapping trajectory, and quickly obtain the undamaged denoised speech data.

[0209] The power module 140 may include a power supply, a power management component, etc. The power supply may be a battery. The power management component manages the charging of the power supply and the power supply to other modules. In some embodiments, the power management component includes a charging management module and a power management module. The charging management module receives charging input from a charger; the power management module connects to the power supply and the processor 110. The power management module receives input from the power supply and / or the charging management module to supply power to the processor 110, the display 102, the camera 170, and the wireless communication module 120, etc.

[0210] The mobile communication module 130 may include, but is not limited to, an antenna, a power amplifier, a filter, and a low-noise amplifier (LNA). The mobile communication module 130 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the mobile phone 10. The mobile communication module 130 can receive electromagnetic waves via the antenna, filter and amplify the received electromagnetic waves, and then transmit them to a modem processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna. In some embodiments, at least some functional modules of the mobile communication module 130 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 130 and at least some modules of the processor 110 may be housed in the same device. Wireless communication technologies can include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wide Band Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Longer Mevolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), Wireless Local Area Networks (WLAN), Near Field Communication (NFC), Frequency Modulation (FM) and / or Field Communication (NFC), Infrared (IR) technology, etc.The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0211] The wireless communication module 120 may include an antenna, which enables the transmission and reception of electromagnetic waves. The wireless communication module 120 can provide solutions for wireless communication applications on the mobile phone 10, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The mobile phone 10 can communicate with networks and other devices through wireless communication technologies.

[0212] In some embodiments, the mobile communication module 130 and the wireless communication module 120 of the mobile phone 10 may also be located in the same module.

[0213] The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc. The display screen 102 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0214] The sensor module 190 may include proximity sensors, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0215] The audio module 150 is used to convert digital audio information into analog audio signals for output, or to convert analog audio input into digital audio signals. The audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 150 may be located in the processor 110, or some functional modules of the audio module 150 may be located in the processor 110. In some embodiments, the audio module 150 may include a speaker, an earpiece, a microphone, and a headphone jack. For example, the microphone in the audio module 150 can be used to receive noisy speech data, and the speaker can be used to output denoised speech data after denoising processing.

[0216] Camera 170 is used to capture still images or videos. An object passes through the lens to generate an optical image that is projected onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then passed to image signal processing (ISP) to be converted into a digital image signal. Mobile phone 10 can achieve its shooting function through ISP, camera 170, video codec, graphics processing unit (GPU), display 102, and application processor.

[0217] Interface module 160 includes an external memory interface, a universal serial bus (USB) interface, and a subscriber identification module (SIM) card interface. The external memory interface can be used to connect an external memory card, such as a microSD card, to expand the storage capacity of the mobile phone 10. The external memory card communicates with the processor 110 through the external memory interface to perform data storage. The USB interface is used for communication between the mobile phone 10 and other electronic devices. The SIM card interface is used to communicate with the SIM card installed in the mobile phone 10, for example, to read or write phone numbers stored in the SIM card.

[0218] In some embodiments, the mobile phone 10 further includes buttons 101, a motor, and indicators. The buttons 101 may include volume buttons, a power button, etc. The motor is used to generate a vibration effect in the mobile phone 10, for example, vibrating when the user's mobile phone 10 is called to prompt the user to answer the call. The indicators may include laser indicators, radio frequency indicators, LED indicators, etc.

[0219] In some embodiments, this application provides a readable medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the speech processing method described above.

[0220] In some embodiments, this application provides an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, one of the processors of the electronic device, for performing the speech processing method described above.

[0221] In some embodiments, this application provides a computer program product including instructions for implementing the speech processing method described above.

[0222] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0223] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0224] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0225] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0226] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0227] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0228] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0229] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A speech processing method, characterized in that, The method includes: Acquire the first speech data to be denoised; Calculate the noise level parameter of the first acquisition environment for the first voice data; The target inference count is determined based on the noise level parameter, wherein the higher the environmental noise level indicated by the noise level parameter, the greater the target inference count. Using a diffusion model, the first speech data is denoised according to the target inference number to obtain the denoised second speech data.

2. The method according to claim 1, characterized in that, The noise level parameter is the signal-to-noise ratio (SNR), and a higher SNR indicates a lower level of environmental noise.

3. The method according to claim 2, characterized in that, The signal-to-noise ratio is calculated in the following way: Detect the speech activity sequence of the first speech data, wherein the data with a first value in the speech activity sequence is speech data, and the data with a second value is noise data; The first energy and first data volume of the data in the speech activity sequence that take the first value are calculated, and the second energy and second data volume of the data in the speech activity sequence that take the second value are calculated. Using formula Calculate the signal-to-noise ratio; Where snr is the signal-to-noise ratio, p s For the first energy, l s p is the first data volume. n For the second energy, l n This refers to the second data volume.

4. The method according to claim 3, characterized in that, The speech activity sequence is implemented using at least one of the following algorithms: energy detection algorithm, zero-crossing rate detection, Hidden Markov Model (HMM), or Long Short-Term Memory (LSTM) network.

5. The method according to claim 2, characterized in that, The target inference number is determined in the following way: Once the signal-to-noise ratio is determined to be within the target threshold range, the number of inferences corresponding to the target threshold range is determined as the target number of inferences; The target threshold range is one of multiple preset threshold ranges, and the preset threshold range with a larger minimum value corresponds to a smaller number of inferences.

6. The method according to claim 1, characterized in that, The second voice data is obtained in the following way: The first voice data is converted from the time domain to the frequency domain to obtain the first frequency domain data; The first frequency domain data is input into the diffusion model, and the first frequency domain data is denoised according to the target inference number to obtain the second frequency domain data, wherein the first frequency domain data and the second frequency domain data have a preset linear relationship; The second frequency domain data is converted from the frequency domain to the time domain to obtain the second speech data.

7. The method according to claim 6, characterized in that, The second frequency domain data is obtained in the following way: Based on the target inference count N, obtain N sampling times and input the N sampling time points into the diffusion model; For i < N, for the i-th sampling time point, the i-th correction information corresponding to the i-th frequency domain data is obtained by the diffusion model, and the i-th frequency domain data is denoised using the i-th correction information to obtain the (i+1)-th frequency domain data, where the first frequency domain data is the first frequency domain data, and the i-th frequency domain data and the (i+1)-th frequency domain data have the preset linear relationship, and i is a positive integer; For i = N, the (i+1)th frequency domain data is used as the second frequency domain data.

8. The method according to claim 7, characterized in that, The (i+1)th frequency domain data is obtained in the following way: For the i-th sampling time point, the diffusion model is used to differentiate the i-th frequency domain data with respect to time to obtain the i-th correction information, wherein the i-th correction information is used to indicate the data change rate from the i-th frequency domain data to the (i+1)-th frequency domain data; A correction algorithm is used to denoise the i-th frequency domain data based on the i-th correction information to obtain the (i+1)-th frequency domain data.

9. The method according to claim 8, characterized in that, The correction algorithm includes at least one of Euler's algorithm and Chasen extrapolation RK algorithm.

10. The method according to claim 8, characterized in that, The i-th correction information is the ratio of the i-th frequency domain data to N; and the (i+1)-th frequency domain data is obtained by adding the i-th correction information to the i-th frequency domain data.

11. The method according to any one of claims 6 to 10, characterized in that, The diffusion model is trained in the following way: Acquire first training time-domain data and second training time-domain data of the training speech, wherein the second training time-domain data is obtained by superimposing preset time-domain noise data on the first training time-domain data; The first training time-domain data and the second training time-domain data are converted from the time domain to the frequency domain to obtain the first training frequency-domain data and the second training frequency-domain data, respectively. The diffusion model is trained based on the first training frequency domain data and the second training frequency domain data to obtain the trained diffusion model.

12. The method according to claim 11, characterized in that, The noise indicated by the preset time-domain noise data includes at least one of the following: wind noise, whistle noise, keyboard noise, and human voice noise.

13. The method according to claim 11, characterized in that, The step of training the diffusion model based on the first training frequency domain data and the second training frequency domain data includes: Obtain the training sampling time; According to the training sampling time, the first training frequency domain data and the second training frequency domain data are linearly interpolated based on the preset linear relationship to obtain the third training frequency domain data; The third training frequency domain data is superimposed with preset frequency domain noise data to obtain the fourth training frequency domain data, wherein the dimension of the third training frequency domain data is the same as the dimension of the preset frequency domain noise data. The fourth training frequency domain data is input into the diffusion model, and the diffusion model is used to differentiate the fourth training frequency domain data with respect to time to obtain the first training correction information. The network parameters of the diffusion model are adjusted based on the difference between the first training correction information and the preset correction information.

14. The method according to claim 13, characterized in that, The third training frequency domain data xt is obtained through the following formula: x t =t*x+(1-t)*y, Where x represents the first training frequency domain data, y represents the second training frequency domain data, t represents the training sampling time, and x and y in t*x+(1-t)*y have the preset linear relationship.

15. The method according to claim 13, characterized in that, The fourth training frequency domain data x t_noise Obtain it through the following methods: x t_noise =μ+δz, Where μ represents the mean of the third training frequency domain data, δ represents the standard deviation of the third training frequency domain data, and z represents the preset frequency domain noise data.

16. The method according to claim 15, characterized in that, 0≤δ≤1, and the value of δ first increases and then decreases as the training sampling time increases.

17. The method according to claim 13, characterized in that, The preset frequency domain noise data is Gaussian noise data with a mean of 0 and a variance of 1.

18. The method according to claim 14, characterized in that, The preset correction information is yx.

19. The method according to claim 18, characterized in that, The difference between the first training correction information and the preset correction information is determined by the loss function loss = mse(v, yx), where mse() represents the mean squared error.

20. The method according to claim 11, characterized in that, The first frequency domain data, the first training frequency domain data, and the second training frequency domain data were obtained using the Short Time Fourier Transform (STFT) technique.

21. A readable medium, characterized in that, The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1 to 20.

22. An electronic device, characterized in that, include: A memory for storing instructions executed by one or more processors of an electronic device, and a processor, one of the processors of the electronic device, for performing the method of any one of claims 1 to 20.

23. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method of any one of claims 1 to 20.

Images