Model training methods, speech processing methods and devices

Abstract

This disclosure presents a model training method, a speech processing method, and an apparatus. The method involves acquiring sample speech time-frequency signals, transforming these signals to obtain a sample Mel spectrogram, performing a pseudo-inverse transform on the sample Mel spectrogram to obtain a first speech time-frequency signal, inputting the first speech time-frequency signal into a diffusion model based on a Schrödinger bridge, denoising the first speech time-frequency signal to obtain a second speech time-frequency signal, determining a target loss based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, training the diffusion model based on the target loss, acquiring the speech time-frequency signal to be processed, and inputting the speech time-frequency signal to be processed into the trained diffusion model for denoising to obtain the target speech time-frequency signal. This method improves the accuracy of speech processing.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model training method, a speech processing method, and an apparatus. Background Technology

[0002] With the development of artificial intelligence technology, speech processing has become increasingly widely used. Currently, diffusion models are commonly used to perform speech processing tasks. The above approach is usually regarded as a generative problem, that is, during model training, noise is gradually added to the speech signal through a forward noise addition process, and the signal is iteratively refined through a backward noise reduction process until the target waveform is reached. The aforementioned forward noise addition process is essentially a process of data to noise, and the structural information of the signal is often easily lost, which reduces the accuracy of speech processing. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail in this disclosure. This overview is not intended to limit the scope of the claims.

[0004] This disclosure provides a model training method, a speech processing method, and an apparatus that can improve the accuracy of speech processing.

[0005] On one hand, embodiments of this disclosure provide a model training method, including:

[0006] Acquire a sample speech time-frequency signal, transform the sample speech time-frequency signal to obtain a sample Mel spectrogram, and perform a pseudo-inverse transform on the sample Mel spectrogram to obtain a first speech time-frequency signal;

[0007] The first speech time-frequency signal is input into a diffusion model based on a Schrödinger bridge to denoise the first speech time-frequency signal and obtain a second speech time-frequency signal. The distribution of the sample speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the initial time, and the distribution of the first speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the target time.

[0008] The target loss is determined based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, and the diffusion model is trained based on the target loss.

[0009] On the other hand, embodiments of this disclosure also provide a model training apparatus, including:

[0010] The first transformation module is used to acquire the sample speech time-frequency signal, transform the sample speech time-frequency signal to obtain the sample Mel spectrum, and perform a pseudo-inverse transformation on the sample Mel spectrum to obtain the first speech time-frequency signal.

[0011] The first diffusion module is used to input the first speech time-frequency signal into a diffusion model based on a Schrödinger bridge, and to denoise the first speech time-frequency signal to obtain a second speech time-frequency signal. The distribution of the sample speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the initial time, and the distribution of the first speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the target time.

[0012] The training module is used to determine the target loss based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, and to train the diffusion model based on the target loss.

[0013] Furthermore, the training module is also used for:

[0014] The signal difference loss is determined based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal;

[0015] The second speech time-frequency signal is transformed to obtain a predicted Mel spectrogram, and the Mel difference loss is determined based on the difference between the predicted Mel spectrogram and the sample Mel spectrogram.

[0016] The target loss is obtained by weighted summation of the signal difference loss and the Mel difference loss.

[0017] Furthermore, the training module is also used for:

[0018] Perform an inverse short-time Fourier transform on the second speech time-frequency signal to obtain the first time-domain signal;

[0019] The first time-domain signal is identified by multiple sub-discriminators to obtain the first identification result of each sub-discriminator;

[0020] The generator loss of the diffusion model is determined based on the mean of multiple first discrimination results, and the target loss is obtained by weighted summation of the signal difference loss, the Mel difference loss, and the generator loss.

[0021] Furthermore, the training module is also used for:

[0022] The inverse short-time Fourier transform is performed on the sample speech time-frequency signal to obtain the second time-domain signal;

[0023] The second time-domain signal is identified based on multiple sub-discriminators to obtain the second discrimination result of each sub-discriminator;

[0024] The discriminator loss of the discriminator is determined based on the average of multiple first discrimination results and the average of multiple second discrimination results;

[0025] The diffusion model and the discriminator are jointly trained based on the target loss and the discriminator loss.

[0026] Furthermore, the training module is also used for:

[0027] For any of the processing layers in the sub-discriminators, a first intermediate feature output when the first time-domain signal is discriminated and a second intermediate feature output when the second time-domain signal is discriminated are obtained;

[0028] For any one of the sub-discriminators, the feature difference between the first intermediate feature and the second intermediate feature of the same processing layer is determined, and the feature matching sub-loss of each sub-discriminator is determined based on the mean of the multiple feature differences.

[0029] The feature matching loss is determined based on the mean of multiple feature matching sub-losses, and the target loss is obtained by weighted summation of the signal difference loss, the Mel difference loss, the generator loss, and the feature matching loss.

[0030] Furthermore, the training module is also used for:

[0031] The first time-domain signal is reshaped into multiple two-dimensional signals with different periods. Each two-dimensional signal is input to the period discriminator of the corresponding period to discriminate the two-dimensional signal and obtain the first discrimination result of each period discriminator.

[0032] The amplitude spectrum of the first time-domain signal is input to each of the spectrum discriminators. The amplitude spectrum is discriminated based on the discrimination parameters corresponding to the spectrum discriminators to obtain the first discrimination result of each spectrum discriminator.

[0033] Furthermore, the training module is also used for:

[0034] For the first period discriminator, the input two-dimensional signal is discriminated to obtain the first discrimination result of the first period discriminator;

[0035] For each of the remaining period discriminators, the input two-dimensional signal is discriminated based on the discrimination result of the previous period discriminator to obtain the first discrimination result of each of the remaining period discriminators.

[0036] Furthermore, the training module is also used for:

[0037] For the first spectrum discriminator, the input amplitude spectrum is discriminated based on the discrimination parameters corresponding to the spectrum discriminator to obtain the first discrimination result of the first spectrum discriminator;

[0038] For each of the remaining spectrum discriminators, the input amplitude spectrum is discriminated based on the discrimination result of the previous spectrum discriminator to obtain the first discrimination result of each of the remaining spectrum discriminators.

[0039] Furthermore, the first transformation module is also used for:

[0040] The phase portion of the sample speech time-frequency signal is discarded, and the sample speech time-frequency signal is compressed in the frequency dimension to obtain the sample Mel spectrum.

[0041] The sample Mel spectrum is mapped to the linear scale domain before compression based on the pseudo-inverse transform matrix to obtain the inverse transform representation. A zero-phase component is added to the imaginary part of the inverse transform representation to obtain the first speech time-frequency signal.

[0042] On the other hand, embodiments of this disclosure also provide a voice processing method, including:

[0043] Obtain the target Mel spectrogram, and perform a pseudo-inverse transform on the target Mel spectrogram to obtain the speech time-frequency signal to be processed;

[0044] The speech time-frequency signal to be processed is input into a diffusion model based on a Schrödinger bridge to denoise the speech time-frequency signal and obtain the target speech time-frequency signal.

[0045] On the other hand, embodiments of this disclosure also provide a voice processing apparatus, including:

[0046] The second transformation module is used to acquire the target Mel spectrogram and perform a pseudo-inverse transformation on the target Mel spectrogram to obtain the speech time-frequency signal to be processed.

[0047] The second diffusion module is used to input the speech time-frequency signal to be processed into a diffusion model based on a Schrödinger bridge, and to denoise the speech time-frequency signal to be processed to obtain the target speech time-frequency signal.

[0048] On the other hand, embodiments of this disclosure also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described model training method or speech processing method.

[0049] On the other hand, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the above-described model training method or speech processing method.

[0050] On the other hand, this disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the model training method or speech processing method described above.

[0051] The embodiments of this disclosure include at least the following beneficial effects: By acquiring sample speech time-frequency signals, transforming the sample speech time-frequency signals to obtain sample Mel spectrograms, and then performing a pseudo-inverse transform on the sample Mel spectrograms, a first speech time-frequency signal with a structure similar to the sample speech time-frequency signals but with some information loss can be obtained. Based on this, the first speech time-frequency signal is input into a diffusion model based on a Schrödinger bridge to denoise the first speech time-frequency signal, resulting in a second speech time-frequency signal. This is equivalent to using the first speech time-frequency signal to replace the original noise-adding result based on a Gaussian distribution, transforming the forward noise-adding process into a data-to-data process, and the distribution of the sample speech time-frequency signal is used as... The edge distribution of the Schrödinger bridge at the initial time, and the distribution of the first speech time-frequency signal, are used as the edge distribution of the Schrödinger bridge at the target time. This enables the recovery between the first speech time-frequency signal and the sample speech time-frequency signal, thus preserving the signal structure during the diffusion process. When determining the target loss based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, and training the diffusion model based on the target loss, the performance of the diffusion model can be effectively improved. Therefore, when acquiring the speech time-frequency signal to be processed and inputting it into the trained diffusion model for denoising, a clearer target speech time-frequency signal can be obtained, thereby improving the accuracy of speech processing.

[0052] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing this disclosure. Attached Figure Description

[0053] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0054] Figure 1 An optional schematic diagram of the model training method provided in the embodiments of this disclosure;

[0055] Figure 2 A schematic diagram of an optional implementation environment provided for an embodiment of this disclosure;

[0056] Figure 3 An optional flowchart of the model training method provided in this disclosure embodiment;

[0057] Figure 4 An optional schematic diagram of a discriminator provided in an embodiment of this disclosure;

[0058] Figure 5 This is a schematic diagram of an optional discriminator structure in the diffusion model provided in this embodiment of the disclosure;

[0059] Figure 6 This is another optional schematic diagram of the discriminator structure in the diffusion model provided in this disclosure embodiment;

[0060] Figure 7 Another optional schematic diagram of the discriminator results in the diffusion model provided in the embodiments of this disclosure;

[0061] Figure 8 A schematic diagram of an optional overall process for a model training method provided in an embodiment of this disclosure;

[0062] Figure 9 An optional flowchart of the speech processing method provided in this embodiment of the disclosure;

[0063] Figure 10 This is a schematic diagram of the structure of the model training device provided in the embodiments of this disclosure;

[0064] Figure 11 This is a schematic diagram of the structure of the voice processing device provided in the embodiments of this disclosure;

[0065] Figure 12 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure;

[0066] Figure 13 This is a partial structural block diagram of a server provided in an embodiment of this disclosure. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0068] It should be noted that in the various specific embodiments of this disclosure, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. The target object can be a user. In addition, when embodiments of this disclosure require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this disclosure be obtained.

[0069] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0070] To facilitate understanding of the technical solutions provided in the embodiments of this disclosure, some key terms used in the embodiments of this disclosure will be explained below:

[0071] Pseudoinverse Transformation: An important transformation used to handle the generalized inverse problem of matrices. From the perspective of spatial mapping, the pseudoinverse transformation can map vectors in the vector space after linear transformation back to the original vector space in a reasonable way that conforms to specific rules, so as to realize a special projection and correspondence relationship, providing a powerful tool for solving various complex problems involving linear transformations.

[0072] Schrödinger Bridge: Used to describe the optimal transmission process between two probability distributions, aiming to find a way for one probability distribution to evolve into another in an optimal manner.

[0073] The diffusion model generates new data samples by simulating the diffusion process of data distribution. The idea is to model the data generation process as an inverse Markov chain. The diffusion process consists of two stages: noise addition and data generation. The noise addition process simulates the gradual transformation of the data distribution into a Gaussian noise distribution, progressively adding noise to the original data until it becomes pure noise containing no information about the original data. The data generation process then reverses this process to recover the original data from the noisy state. This model is widely used in image generation and image inpainting.

[0074] With the development of artificial intelligence technology, the application of speech processing has become increasingly widespread. Currently, model training methods can be broadly categorized into diffusion-based methods and non-diffusion-based methods, as shown below. Figure 1 , Figure 1 This is an optional schematic diagram of the model training method provided in an embodiment of this disclosure. For example... Figure 1 As shown in (a), non-diffusion-based methods typically estimate the amplitude and phase of the Mel spectrogram by inputting it into a black-box model (a generator based on a deep neural network), and then generate the target waveform based on the estimated amplitude and phase, or directly generate the target waveform based on the Mel spectrogram. Figure 1 As shown in (b), diffusion-based methods typically treat speech processing as a generative problem, by inputting the Mel spectrogram as condition c into the diffusion model, based on prior data x. T ~N(μ) c ,∑ c Denoising is performed by progressively adding noise to the speech signal through a forward noise-adding process and iteratively refining the signal during a backward noise-reducing process until the target waveform is generated. However, non-diffusion-based methods lack flexibility, and the forward noise-adding process in the aforementioned diffusion-based methods is essentially a data-to-noise process, which often leads to the loss of structural information in the signal, reducing the accuracy of speech processing.

[0075] Based on this, embodiments of this disclosure provide a model training method, apparatus, electronic device, and storage medium, such as Figure 1 As shown in (c), the inverse transform representation is obtained by performing a pseudo-inverse transform on the Mel spectrum, and the prior distribution x of the Schrödinger bridge is obtained by adding a zero-phase component to the inverse transform representation. T Based on the prior distribution, the edge distribution of the Schrödinger bridge at the target time is determined. The solution of the Schrödinger bridge is obtained by stepwise sampling based on the backward stochastic differential equation, or by sampling based on the ordinary differential equation. The solution of the Schrödinger bridge is then used for denoising to obtain the target waveform, thereby improving the accuracy of speech processing.

[0076] Reference Figure 2 , Figure 2This is a schematic diagram of an optional implementation environment provided by an embodiment of the present disclosure. The implementation environment includes a terminal 201 and a server 202, wherein the terminal 201 and the server 202 are connected through a communication network.

[0077] For example, a sample speech time-frequency signal is acquired in terminal 201 and sent to server 202. Server 202 transforms the sample speech time-frequency signal to obtain a sample Mel spectrogram, performs a pseudo-inverse transform on the sample Mel spectrogram to obtain a first speech time-frequency signal, and inputs the first speech time-frequency signal into a diffusion model based on a Schrödinger bridge. The first speech time-frequency signal is then denoised to obtain a second speech time-frequency signal. During denoising based on the Schrödinger bridge, the distribution of the sample speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the initial time, and the distribution of the first speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the target time. Next, the target loss is determined based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, and the diffusion model is trained based on the target loss. After the diffusion model is trained, the time-frequency signal of the speech to be processed is obtained from the terminal 201. The time-frequency signal of the speech to be processed is input into the trained diffusion model for denoising to obtain the target speech time-frequency signal. The target speech time-frequency signal is sent back to the terminal 201 so that the terminal 201 can perform speech processing based on the target speech time-frequency signal, such as converting the target speech time-frequency signal to obtain the target speech.

[0078] It is understood that the model training method provided in this disclosure embodiment can also be executed independently in terminal 201.

[0079] Server 202 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Additionally, server 202 can also be a node server in a blockchain network.

[0080] Terminal 201 may be a mobile phone, computer, smart voice interaction device, smart wearable device, smart home appliance, vehicle terminal, etc., but is not limited to these. Terminal 201 and server 202 can be directly or indirectly connected through wired or wireless communication, and this embodiment of the disclosure does not impose any limitations.

[0081] Reference Figure 3 , Figure 3This is an optional flowchart of a model training method provided in an embodiment of the present disclosure. The model training method can be executed by a terminal, by a server, or by a combination of both. The model training method includes, but is not limited to, the following steps S301 to S303.

[0082] Step S301: Obtain the sample speech time-frequency signal, transform the sample speech time-frequency signal to obtain the sample Mel spectrogram, and perform a pseudo-inverse transform on the sample Mel spectrogram to obtain the first speech time-frequency signal.

[0083] Among them, the sample speech time-frequency signal is a representation of the sample speech signal in the time-frequency domain, which can be represented in polar coordinates, for example, S=|S|e jθ Let |S| be the amplitude (modulus) of the sample speech time-frequency signal, θ be the phase, and j be the imaginary part. The sample speech time-frequency signal has not undergone any data processing and possesses complete and authentic signal characteristics. It can be considered a real signal sample for comparison with the predicted signal output by the generative model. The sample Mel spectrogram is obtained based on the signal model, which is used to perform data transformation on the sample speech time-frequency signal. This transformation refers to transforming the sample speech time-frequency signal from the linear scale domain to the Mel spectrogram. The first speech time-frequency signal contains a certain amount of noise, and some information is lost after the transformation, but its time-frequency domain representation and dimensions are similar to those of the sample speech time-frequency signal. The pseudo-inverse transformation refers to mapping the Mel spectrogram back to the original uncompressed linear scale domain.

[0084] In one possible implementation, the process of obtaining the first speech time-frequency signal can specifically involve discarding the phase part of the sample speech time-frequency signal and compressing the sample speech time-frequency signal in the frequency dimension to obtain a sample Mel spectrum. Based on the pseudo-inverse transform matrix, the sample Mel spectrum is mapped to the linear scale domain before compression to obtain the inverse transform representation. A zero-phase component is added to the imaginary part of the inverse transform representation to obtain the first speech time-frequency signal.

[0085] Specifically, the sample speech time-frequency signal The input is fed into the signal model, where T is the number of frames and F is the number of frequency units in the sample speech time-frequency signal. Next, the phase component of the sample speech time-frequency signal is discarded; the sample speech time-frequency signal after discarding the phase component can be represented as S = |S|. Then, linear compression is performed on the sample speech time-frequency signal along the frequency dimension to obtain the sample Mel-frequency spectrogram. Among them, F mel This represents the number of frequency elements in the sample Mel spectrum, and F mel <<F. The sample Mel-spectrum Y is obtained through the signal model. mel The process can be represented by the following formula.

[0086] Y mel =|S|A,

[0087] Where A is the linear-Mel transformation matrix, and the dimension of this matrix can be... Used to convert sample speech time-frequency signals from the linear frequency domain to the Mel frequency domain.

[0088] Next, based on the pseudo-inverse transform matrix, the sample Mel spectrum is mapped to the linear scale domain before compression to obtain the inverse transform representation, ensuring that the number of frames and frequency units in the inverse transform representation matches the number of frames and frequency units in the sample speech time-frequency signal. The process of obtaining the inverse transform representation can be expressed by the following formula:

[0089]

[0090] in, This is an inverse transform representation, which can also be viewed as a linear scaling representation, and its dimension can be... This represents the pseudo-inverse transformation matrix, whose dimension can be any number of times.

[0091] Next, a zero-phase component is added to the imaginary part of the inverse transform representation to obtain the first speech time-frequency signal, making the dimension and representation of the first speech time-frequency signal consistent with the sample speech time-frequency signal. The first speech time-frequency signal In the time-frequency domain, it can be represented as Its dimensions can be By converting the sample speech time-frequency signal into a sample Mel spectrogram, key signal information can be preserved while effectively reducing feature dimensionality and computational load, thereby improving the efficiency of subsequent diffusion model speech processing. Based on this, a pseudo-inverse transform and the addition of a zero-phase component ensure that the structure of the first speech time-frequency signal is consistent with that of the sample speech time-frequency signal. Furthermore, the sample Mel spectrogram remains relatively stable in the face of minor interferences or variations in the audio signal, contributing to improved accuracy and reliability of subsequent diffusion model speech processing. Furthermore, by performing a pseudo-inverse transform on the sample Mel spectrogram to obtain the first speech time-frequency signal, preliminary denoising can be performed while restoring the signal, improving the quality of the first speech time-frequency signal.

[0092] Step S302: Input the first speech time-frequency signal into the diffusion model based on the Schrödinger bridge, and denoise the first speech time-frequency signal to obtain the second speech time-frequency signal.

[0093] In this model, the distribution of the sample speech time-frequency signal serves as the marginal distribution of the Schrödinger bridge at the initial time, and the distribution of the first speech time-frequency signal serves as the marginal distribution of the Schrödinger bridge at the target time. The initial time refers to the moment when the diffusion process begins, denoted as t=0, and the target time refers to the moment when the diffusion process ends, denoted as t=T. The marginal distribution acts as a boundary constraint for the construction of the Schrödinger bridge during diffusion model training, determining the probability distribution of the states at the initial and target times. The second speech time-frequency signal is the time-frequency signal estimated by the diffusion model based on the first speech time-frequency signal within a finite time range. Since the first speech time-frequency signal loses some information during the transformation process, the second speech time-frequency signal obtained based on the first speech time-frequency signal also loses some information to some extent. Compared to the sample speech time-frequency signal as complete data, the second speech time-frequency signal can be considered as corrupted data. The generation process of the second speech time-frequency signal can be represented as follows: B is the second speech time-frequency signal. θ (·) represents the neural network of the diffusion model, used to predict the signal at the current time, where t is the current time and x is the signal at the current moment. t Let x be the noise signal at the current time t. T Let be the noise signal at the target time T.

[0094] Specifically, the sample speech time-frequency signal and the first speech time-frequency signal are input into the diffusion model to determine their distributions. These distributions are then used as the prior distributions of the Schrödinger bridge. Next, the distribution of the sample speech time-frequency signal is used as the marginal distribution of the Schrödinger bridge at the initial time, and the distribution of the first speech time-frequency signal is used as the marginal distribution at the target time. Based on this, the Schrödinger bridge can be defined as minimizing the KL divergence between the predicted path metric and the reference path metric. The Schrödinger bridge problem can be defined by the following formula.

[0095]

[0096] Among them, P [0,T] Let p represent the path metric space over a finite time range [0,T], where p is the predicted path metric. ref Let p0 be the marginal distribution of the Schrödinger bridge at the initial time, and p be the reference path metric for the samples. s For the distribution of the sample speech time-frequency signal, p T The edge distribution of the Schrödinger bridge at the target time. Let represent the distribution of the first speech time-frequency signal. When the sample reference path metric is defined by a forward stochastic differential equation (SDE), the Schrödinger bridge can be equivalent to a pair of forward-backward stochastic differential equations. The forward stochastic differential equation can be expressed as:

[0097]

[0098] The backward stochastic differential equation can be expressed as:

[0099]

[0100] Where t∈[0,T] is a finite time range. Let f(·) represent the state of the diffusion process, f(·) be the vector-valued drift function (i.e., the linear drift function), g be the scale-valued diffusion coefficient, and w be the vector-valued diffusion coefficient. t For the standard Wiener process, For the Wiener process in reverse time, Ψ t It is a positive offset. The score function for the positively offset edge distribution. For reverse offset, This is the score function for the edge distribution with reverse offset. When Ψ t For optimal positive offset, For the optimal reverse offset, the Schrödinger bridge in state x t The marginal distribution at time can be represented as

[0101] Next, the distribution of the sample speech time-frequency signal and the distribution of the first speech time-frequency signal are used as Gaussian smoothed data pairs in the forward-backward stochastic differential equation pair. The distribution of the sample speech time-frequency signal can be... The distribution of the first speech time-frequency signal can be As ∈ 0→0, the forward and reverse offsets converge between the first speech time-frequency signal and the sample speech time-frequency signal until the solution of the Schrödinger bridge is obtained. At this time, the forward offset can be expressed as The reverse offset can be expressed as Where x0 is the sample speech time-frequency signal, x1 is the first speech time-frequency signal, and a t is the scaling factor for the sample speech time-frequency signal. The scaling factor for the first speech time-frequency signal. The standard deviation of the noise added to the sample speech time-frequency signal. The standard deviation of the noise added to the first speech time-frequency signal can be expressed by the following formula, where f(·) and g are determined based on the parameters in the forward-backward stochastic differential equation.

[0102]

[0103] Based on this, when the maximum value T = 1 within a finite time range, the marginal distribution p of the Schrödinger bridge... t The following formula can be used for processing.

[0104]

[0105] Finally, the first speech time-frequency signal is gradually denoised according to the edge distribution of the Schrödinger bridge to obtain the second speech time-frequency signal. It should also be noted that during the training of the diffusion model, before denoising the first speech signal, a small amount of noise needs to be gradually added to the first speech time-frequency signal through the diffusion process of the diffusion model, and then the Schrödinger bridge is used to denoise the first speech time-frequency signal with added noise to supplement the mid-to-high frequency information.

[0106] By denoising the noisy first speech time-frequency signal, the original Gaussian distribution-based noise-adding result in the diffusion model is replaced with the first speech time-frequency signal. This transforms the diffusion process (i.e., the forward noise-adding process) into a data-to-data process, reducing information loss caused by multiple noise additions and denoising operations. It better preserves key information and features in the first speech time-frequency signal, avoiding speech distortion problems that may occur during speech processing and improving the accuracy of speech processing. Furthermore, the data-to-data process does not require adding and removing noise step-by-step, effectively reducing computational steps and computational costs, thereby improving the efficiency of speech processing.

[0107] Step S303: Determine the target loss based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, and train the diffusion model based on the target loss.

[0108] The target loss is used to measure the difference between the second speech time-frequency signal and the sample speech time-frequency signal. The diffusion model is trained and optimized based on this difference. The target loss includes signal difference loss, Mel difference loss, generator loss, discriminator loss, and feature matching loss.

[0109] In one possible implementation, the process of determining the target loss based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal can specifically involve determining a signal difference loss based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, transforming the second speech time-frequency signal to obtain a predicted Mel spectrogram, determining the Mel difference loss based on the difference between the predicted Mel spectrogram and the sample Mel spectrogram, and then weighted summing the signal difference loss and the Mel difference loss to obtain the target loss. Here, the signal difference loss is used to evaluate the overall difference between the second speech time-frequency signal and the sample video speech signal, the sample Mel spectrogram is obtained based on the sample speech time-frequency signal, the predicted Mel spectrogram is obtained based on the second speech time-frequency signal, and the Mel difference loss is used to evaluate the difference in amplitude information between the sample Mel spectrogram and the predicted Mel spectrogram.

[0110] Specifically, the number of frequency units and the total number of time steps in the time-frequency domain of the second speech time-frequency signal and the sample speech time-frequency signal are obtained. The differences between the second speech time-frequency signal and the sample speech time-frequency signal at each time step are calculated using the L2 norm and then summed to obtain a first result. Then, the first result is normalized based on the product of the number of frequency units and the total number of time steps to obtain the signal difference loss. According to the above description, the signal difference loss can be the mean-square error (MSE), and the signal difference loss L... mse It can be expressed by the following formula,

[0111]

[0112] Where F represents the number of frequency units, T represents the total number of time steps, t represents any time step, and f represents the frequency value corresponding to any time step. The frequency value can be one, multiple discontinuous frequency values, or a frequency range composed of multiple continuous frequency values. S represents the second speech time-frequency signal at time step t. f,t This represents the sample speech time-frequency signal at time step t.

[0113] Next, the number of frequency units and the total number of time steps corresponding to the sample Mel spectrogram are obtained. Based on the time-frequency domain, the spectral information of the second speech time-frequency signal is obtained. Based on the spectral information of the second speech time-frequency signal, a predicted Mel spectrogram is obtained. The difference between the predicted Mel spectrogram and the sample Mel spectrogram corresponding to each time step is calculated using the L1 norm and then summed to obtain the second result. Then, the second result is normalized based on the product of the number of frequency units corresponding to the Mel spectrogram and the total number of time steps to obtain the Mel difference loss. According to the above description, the Mel difference loss can be the Mean Absolute Error (MAE), and the Mel difference loss can be expressed by the following formula.

[0114]

[0115] Where represents the number of frequency units corresponding to the Mel spectrogram, represents the predicted Mel spectrogram at time step t, and represents the sample Mel spectrogram at time step t.

[0116] Finally, Mel weights are assigned to the Mel difference loss, and the Mel difference loss is weighted based on these weights. The weighted Mel difference loss is then summed with the signal difference loss to obtain the target loss, L. B It can be expressed by the following formula, where λ mel For Mel weights.

[0117] L B =L mse +λ mel L mel

[0118] By training the diffusion model using signal difference loss and Mel difference loss, the model can learn the signal data representation in both the linear and Mel frequency domains. This allows for a more comprehensive extraction of key information from the input speech, improving the accuracy of speech processing. Furthermore, since real audio data is often subject to noise interference, resulting in noise and uncertainty in the Mel graph, Mel difference loss enables the diffusion model to learn how to extract effective features from noisy Mel graphs, enhancing its generalization ability and robustness to noise.

[0119] In one possible implementation, the process of weighted summing of the signal difference loss and the Mel difference loss to obtain the target loss can specifically involve performing an inverse short-time Fourier transform on the second speech time-frequency signal to obtain a first time-domain signal. Multiple sub-discriminators then discriminate the first time-domain signal to obtain the first discrimination result of each sub-discriminator. The generator loss of the diffusion model is determined based on the mean of the multiple first discrimination results. Finally, the signal difference loss, the Mel difference loss, and the generator loss are weighted and summed to obtain the target loss. Here, the first time-domain signal is obtained by performing an inverse short-time Fourier transform on the second speech time-frequency signal; the first discrimination result is obtained based on the first time-domain signal and is used to indicate the output score of the sub-discriminators for the second speech time-frequency signal; and the generator loss is used to evaluate the distribution difference of the second speech time-frequency signal output by the diffusion model.

[0120] Specifically, an inverse short-time Fourier transform is performed on the second speech time-frequency signal to convert it from the time-frequency domain to the time domain, and a first time-domain signal is obtained based on the time domain. This first time-domain signal is then input into multiple sub-discriminators for discrimination, yielding the first discrimination result for each sub-discriminator. The first discrimination results of each sub-discriminator are then transformed, and the generator loss of the diffusion model is determined based on the mean of the transformed first discrimination results. The generator loss L... g It can be expressed by the following formula,

[0121]

[0122] Where M represents the number of sub-discriminators, D m (·) represents the m-th sub-discriminator. The output is the first identification result. No normalization is performed on the first identification result, so the first identification result can be any positive real number or any negative real number. This represents the first time-domain signal obtained after the inverse short-time Fourier transform. The transform process can be expressed as follows: The second speech time-frequency signal is represented by iSTFT(·), which is the inverse short-time Fourier transform; max(·) is the maximum value function used to evaluate the second speech signal. The item is transformed. For any sub-discriminator, when When the first discrimination result is located in the interval (-∞, 1), the output of the function max(·) is: It can be assumed that the distribution similarity between the obtained second speech time-frequency signal and the sample speech time-frequency signal is low, that is, the second speech time-frequency signal and the sample speech time-frequency signal are quite different; when the first discrimination result is located in the interval (1,+∞), the output of the function max(·) is 0, and it can be assumed that the distribution similarity between the obtained second speech time-frequency signal and the speech time-frequency signal is high, that is, the second speech time-frequency signal and the sample speech time-frequency signal are relatively different.

[0123] Next, generator weights are configured for the generator loss. The generator loss and Mel difference loss are weighted and summed based on the generator weights and Mel weights respectively. This summation is then combined with the signal difference loss to obtain the target loss, L. B It can be expressed by the following formula, where λ g The generator weights are used to generate the poor-quality time-frequency signals. The main reasons for generating poor-quality time-frequency signals are noise interference or signal data corruption. By introducing generator loss to train the diffusion model, the model can focus on generating the poor-quality second speech time-frequency signal. This guides the diffusion model to learn how to process noise or corrupted data, thereby generating a high-quality signal.

[0124] L B =Lmse +λ mel L mel +λ g L g

[0125] Furthermore, when fine-tuning the diffusion model or simplifying the training process to quickly obtain results, the diffusion model can be trained using only signal difference loss and Mel difference loss as the target losses. Since signal difference loss evaluates the difference between the sample speech time-frequency signal and the second speech time-frequency signal as a whole, considering both the time-domain and frequency-domain differences, using only signal difference loss and Mel difference loss as the target losses in this case reduces computational cost and complexity, thus improving the training speed of the diffusion model.

[0126] After training the diffusion model, the time-frequency signal of the speech to be processed can be obtained. The time-frequency signal of the speech to be processed is transformed to obtain the target Mel spectrogram. The target Mel spectrogram is then subjected to a pseudo-inverse transform to obtain the time-frequency signal of the speech to be processed. The time-frequency signal of the speech to be processed is then input into the diffusion model based on the Schrödinger bridge to denoise the time-frequency signal of the speech to be processed, and the target time-frequency signal of the speech is obtained.

[0127] The speech time-frequency signal to be processed is obtained by pseudo-inverse transformation based on the target Mel spectrogram. The target Mel spectrogram can be obtained by transforming the input audio. The input audio can be basic audio collected by devices such as microphones, or it can be synthesized audio by converting text into speech.

[0128] Specifically, when the raw data for speech processing is an audio signal, the phase component of the audio signal is discarded, and the audio signal is linearly compressed in the frequency dimension to obtain a target Mel spectrogram. A pseudo-inverse transform is then performed on the target Mel spectrogram to obtain the target inverse transform representation. A zero-phase component is added to the imaginary part of the target inverse transform representation to obtain the speech time-frequency signal to be processed. When the raw data for speech processing is text, the text is converted into synthesized audio using a large language model. Then, based on the synthesized audio and the target Mel spectrogram obtained through the above process, the speech time-frequency signal to be processed is further obtained.

[0129] In addition, in the context of text-to-speech, the aforementioned target Mel spectrogram can also be generated by inputting text into a generative model. After generating the target Mel spectrogram based on the text, a pseudo-inverse transform is then performed based on the target Mel spectrogram to obtain the time-frequency signal of the speech to be processed.

[0130] Alternatively, the input audio can also be a noisy frequency acquired in a noisy environment.

[0131] Next, the speech time-frequency signal to be processed is input into the trained diffusion model for denoising. During denoising, the initial time-frequency signal of the speech time-frequency signal to be processed at the current time and the predicted time-frequency signal predicted by the diffusion model at the current time are obtained. An inverse stochastic differential equation is represented based on the initial time-frequency signal and the predicted time-frequency signal predicted by the diffusion model at the current time. The solution of the Schrödinger bridge is obtained by first-order discretization of this inverse stochastic differential equation. Alternatively, the solution of the Schrödinger bridge can be obtained based on an ordinary differential equation. Denoising is then performed based on the solution of the Schrödinger bridge to obtain the target speech time-frequency signal.

[0132] In one possible implementation, during the training of the diffusion model based on the target loss, specifically, an inverse short-time Fourier transform is performed on the sample speech time-frequency signal to obtain a second time-domain signal. Multiple sub-discriminators then discriminate the second time-domain signal, obtaining the second discrimination result for each sub-discriminator. The discriminator loss of the discriminator is determined based on the mean of multiple first discrimination results and the mean of the second discrimination results. The diffusion model and the discriminator are then jointly trained based on the target loss and the discriminator loss. Here, the second time-domain signal is obtained by performing an inverse short-time Fourier transform on the sample speech time-frequency signal; the second discrimination result is obtained based on the second time-domain signal and is used to indicate the output score of the sub-discriminators for the sample speech time-frequency signal; the discriminator loss is used to evaluate the diffusion model's ability to discriminate between the sample speech time-frequency signal and the second speech time-frequency signal.

[0133] Specifically, an inverse short-time Fourier transform is performed on the sample speech time-frequency signal to convert it from the time-frequency domain to the time domain, resulting in a second time-domain signal. This second time-domain signal is then input into multiple sub-discriminators for discrimination, yielding the second discrimination result for each sub-discriminator. The second discrimination results of each sub-discriminator are then transformed, and the discriminator loss of the discriminator is determined based on the average of the transformed second discrimination result and the transformed first discrimination result. The discriminator loss L... d It can be expressed by the following formula,

[0134]

[0135] Where s represents the second time-domain signal obtained after inverse short-time Fourier transform, s m The output of (s) is the second discrimination result. m (s) is used to evaluate the discriminator's ability to distinguish between sample speech time-frequency signals. This is used to evaluate the discriminator's ability to distinguish between the second speech time-frequency signal and the actual speech signal. During training, since the sample speech time-frequency signal is real data, a higher score for the second discrimination result is desirable (e.g., D). m (s)≥1), such that max(0,1-D)m The output of (s) is 0, at which point the loss is 0; however, since the second speech time-frequency signal is corrupted data, it is desirable for the score of the first discrimination result to be as low as possible (e.g., ),make The output is 0, and the loss for this item is 0. When the score of the second discrimination result is low (e.g., D), m (s)<1) or a high score for the first identification result (e.g. When max(0,1-D) is reached, m The output of (s) is 1-D m (s), The output is At this point, the loss value is relatively large, indicating that the discriminator's ability to distinguish between real and corrupted data is weak, and further optimization is needed.

[0136] After determining the discriminator loss, the diffusion model and discriminator are jointly trained based on the target loss and the discriminator loss. When the diffusion model generates a second speech time-frequency signal, the discriminator is invoked to discriminate the second speech time-frequency signal. The discriminator loss is obtained based on the discrimination result, and the generation process of the diffusion model is guided by the discrimination loss. By jointly training the diffusion model and discriminator with the target loss and the discriminator loss, the diffusion model can learn the overall signal features of the input signal based on the target loss. With the deepening of iterative training, the diffusion model can continuously optimize the generation mechanism based on the discriminator loss, gradually generating a second speech time-frequency signal that is more similar to the sample speech time-frequency signal, thereby improving the generation quality of the diffusion model. At the same time, when faced with increasingly similar sample speech time-frequency signals and second speech time-frequency signals, the discriminator needs to capture the subtle differences between the two signals. By learning the subtle differences, the discrimination accuracy of the discriminator can be enhanced, thereby improving the accuracy of speech processing.

[0137] In one possible implementation, the sub-discriminator includes multiple processing layers. In the process of weighted summing of signal difference loss, Mel difference loss, and generator loss to obtain the target loss, specifically, for any processing layer in the sub-discriminator, a first intermediate feature output when discriminating the first time-domain signal and a second intermediate feature output when discriminating the second time-domain signal are obtained. For any sub-discriminator, the feature difference between the first intermediate feature and the second intermediate feature of the same processing layer is determined. The feature matching sub-loss of each sub-discriminator is determined based on the average of multiple feature differences, and the feature matching loss is determined based on the average of multiple feature matching sub-losses. The sub-discriminator can be viewed as a neural network module composed of multiple processing layers. Each processing layer can consist of substructures such as convolutional layers, normalization layers, and activation functions. For example, refer to... Figure 4 , Figure 4This is an optional schematic diagram of the processing layer of the sub-discriminator provided in an embodiment of the present disclosure. The processing layer of the sub-discriminator includes a pooling layer, a convolutional layer A, a convolutional layer B, a normalization layer, and a ReLU activation function, and uses a residual structure to fuse the input and output of the processing layer.

[0138] Specifically, the number of sub-discriminators and the number of processing layers in each sub-discriminator are obtained. The second time-domain signal of the sampled speech time-frequency signal and the first time-domain signal of the second speech time-frequency signal are input to multiple sub-discriminators for discrimination. For any processing layer in any sub-discriminator, the first intermediate feature corresponding to the first time-domain signal extracted by that processing layer and the second intermediate feature corresponding to the extracted second time-domain signal are obtained, and the feature difference between the first and second intermediate features extracted by that processing layer is obtained. The feature difference between the first and second intermediate features extracted by all processing layers in each sub-discriminator is calculated. The feature matching sub-loss of each sub-discriminator is determined based on the mean of multiple feature differences, and the feature matching loss is determined based on the mean of multiple feature matching sub-losses. When the number of processing layers in each sub-discriminator is the same, according to the above description, the feature matching loss L fm It can be obtained from the following formula,

[0139]

[0140] Where M represents the number of sub-discriminators, and L represents the number of processing layers in the sub-discriminators. f represents the first intermediate feature output by the l-th processing layer of the m-th sub-discriminator after discriminating the first time-domain signal; l m (s) represents the second intermediate feature output by the l-th processing layer of the m-th sub-discriminator after discriminating the second time-domain signal.

[0141] Next, the signal difference loss, Mel difference loss, generator loss, and feature matching loss are weighted to obtain the target loss. Specifically, feature matching weights are assigned to the feature matching loss. The Mel difference loss, generator loss, and feature matching loss are then weighted and summed based on the Mel weights, generator weights, and feature matching weights, respectively. This summation is then combined with the signal difference loss to obtain the target loss, L. B It can be expressed by the following formula, where λ fm For feature matching weights.

[0142] L B =L mse +λ mel L mel +λ g L g +λ fm L fm

[0143] By introducing feature matching loss, the feature differences extracted from all processing layers in the sub-discriminator are calculated layer by layer. This allows the sub-discriminator to learn key information in the time-domain signal, thereby enabling it to extract signal features in a targeted manner and helping to improve the accuracy of speech processing.

[0144] In one possible implementation, the multiple sub-discriminators include multiple periodic discriminators with different periods and multiple spectrum discriminators with different resolutions. In the process of discriminating the first time-domain signal based on the multiple sub-discriminators to obtain the first discrimination result of each sub-discriminator, specifically, the first time-domain signal can be reconstructed into multiple two-dimensional signals with different periods. Each two-dimensional signal is then input to the corresponding periodic discriminator for discrimination, obtaining the first discrimination result of each periodic discriminator. The amplitude spectrum of the first time-domain signal is then input to each spectrum discriminator, and the amplitude spectrum is discriminated based on the discrimination parameters corresponding to the spectrum discriminator, obtaining the first discrimination result of each spectrum discriminator. Here, the two-dimensional signal is obtained by transforming the first time-domain signal based on a specific periodic scale. For example, if the first time-domain signal is [a1, a2, a3, a4] with a periodic scale of 2, the two-dimensional signal obtained after transformation based on the specific periodic scale can be... The discrimination parameters are used to adjust the amplitude spectrum resolution, including window size, step size, and the number of Fast Fourier Transform (FFT) points. The window size controls the number of samples in the first time-domain signal, the step size controls the sampling density of the window in the time dimension of the first time-domain signal, and the number of FFT points refers to the number of sample points used when performing an FFT on the sampled signal in the window. The number of FFT points can be the same as or greater than the window size. When the number of FFT points is greater than the window size, the sampled signal in the window can be padded (e.g., with zeros). The amplitude spectrum indicates the amplitude of different frequency components in the time-domain signal and is a frequency domain representation of the time-domain signal based on the discrimination parameters.

[0145] Specifically, based on the period corresponding to each period discriminator, the first time-domain signal is converted into a data format based on that period, reshaping it into multiple two-dimensional signals with different periods. This allows the diffusion model to extract and analyze features at different period scales based on multiple period discriminators with different periods. Each two-dimensional signal is then input into the corresponding period discriminator for discrimination, yielding the first discrimination result for each period discriminator. Simultaneously, based on the discrimination parameters corresponding to each spectral discriminator, the first time-domain signal is framed according to the window size and step size, resulting in multiple sampled signals. These sampled signals are then subjected to Fast Fourier Transform (FFT) to obtain the complex representation of the first time-domain signal in the frequency domain. The amplitude spectrum at different resolutions is determined based on the modulus of the complex representation. The amplitude spectra at different resolutions are then input into the corresponding spectral discriminators for discrimination, and the first discrimination result for each spectral discriminator is calculated based on multiple stacked two-dimensional convolutional blocks.

[0146] For example, refer to Figure 5 , Figure 5 This is a schematic diagram of an optional discriminator structure in the diffusion model provided in this embodiment of the disclosure. Figure 5The diagram shows 5 period discriminators and 3 spectral discriminators, arranged in parallel to independently discriminate the input data. The periods of period discriminators A, B, C, D, and E are 2, 3, 5, 7, and 11, respectively. The discrimination parameters of spectral discriminators A, B, C, D, and E are (512, 128, 512), (1024, 256, 2014), and (2048, 512, 2048), respectively. When the first time-domain signal is converted into a two-dimensional signal A with a period of 2, the two-dimensional signal A is input into the corresponding period discriminator A with a period of 2 to obtain the discrimination result A. Similarly, the two-dimensional signal B is input into the corresponding period discriminator B to obtain the discrimination result B, the two-dimensional signal C is input into the corresponding period discriminator C to obtain the discrimination result C, the two-dimensional signal D is input into the corresponding period discriminator D to obtain the discrimination result D, and the two-dimensional signal E is input into the corresponding period discriminator E to obtain the discrimination result E. When the amplitude spectrum A is extracted from the first time-domain signal according to the discrimination parameters (512, 128, 512), the amplitude spectrum A is input into the corresponding spectrum discriminator A to obtain the discrimination result F. Similarly, the amplitude spectrum B is input into the spectrum discriminator B to obtain the discrimination result G, and the amplitude spectrum C is input into the spectrum discriminator C to obtain the discrimination result H. By arranging the periodic discriminator and the spectrogram discriminator in parallel, multiple periodic discriminators and multiple spectrogram discriminators can process the input data simultaneously, accelerating the feature extraction and output of discrimination results for each discriminator, thereby improving the speed of speech processing. Furthermore, since different periodic discriminators are used to process signals of different periods, and different spectrogram discriminators are used for amplitude spectra at different resolutions, parallel processing enables the diffusion model to simultaneously capture features at multiple periodic scales and multiple resolutions. This helps improve the diffusion model's understanding and representation of the input signal, thus enhancing its generalization ability.

[0147] In one possible implementation, multiple period discriminators are cascaded in ascending order of period. In the process of discriminating the two-dimensional signal and obtaining the first discrimination result of each period discriminator, specifically, for the first period discriminator, the input two-dimensional signal is discriminated to obtain the first discrimination result of the first period discriminator, and for each of the remaining period discriminators, the input two-dimensional signal is discriminated based on the discrimination result of the previous period discriminator to obtain the first discrimination result of each of the remaining period discriminators.

[0148] Specifically, for the first period discriminator, the input two-dimensional signal is discriminated to obtain the signal periodic features extracted by the first period discriminator. Based on the signal periodic features of the first period discriminator, the first discrimination result of the first period discriminator is obtained. For each of the remaining period discriminators, the signal periodic features of the previous period discriminator are input to the current period discriminator, so that the current discriminator discriminates the input two-dimensional signal based on the signal periodic features of the previous period discriminator, obtains the signal periodic features extracted by the current period discriminator, and obtains the first discrimination result of the current period discriminator based on the signal periodic features of the current discriminator, until the first discrimination results of all period discriminators are obtained. By cascading multiple period discriminators in ascending order of period, the diffusion model can extract features from two-dimensional signals at different period scales. This allows the signal periodic features at different period scales to be passed down layer by layer based on the cascaded structure, enabling the diffusion model to more deeply understand the variation law of the first time-domain signal in the time or spatial dimensions, thereby more comprehensively capturing the periodic features of the first time-domain signal and effectively improving the accuracy and reliability of the diffusion model's discrimination.

[0149] Reference Figure 6 , Figure 6 This is a schematic diagram of another optional discriminator structure in the diffusion model provided in this disclosure embodiment. Figure 6 The diagram illustrates five period discriminators and three spectral discriminators. The five period discriminators are cascaded in ascending order of period, while the three spectral discriminators are arranged in parallel. For the first period discriminator A, it discriminates the two-dimensional signal A to obtain a signal periodic feature f1, and then obtains the discriminant result A based on this feature. The signal periodic feature f1 is then input into period discriminator B, which discriminates the two-dimensional signal B based on f1 to obtain a signal periodic feature f2, and then obtains the discriminant result B based on f2. Similarly, period discriminator C discriminates the two-dimensional signal C based on f2, obtaining the discriminant result C; period discriminator D discriminates the two-dimensional signal D based on f3, obtaining the discriminant result D; and period discriminator E discriminates the two-dimensional signal E based on f4, obtaining the discriminant result E. Simultaneously, the amplitude spectrum A is input into the corresponding spectrum discriminator A to obtain the discrimination result F. Similarly, the amplitude spectrum B is input into the spectrum discriminator B to obtain the discrimination result G, and the amplitude spectrum C is input into the spectrum discriminator C to obtain the discrimination result H.

[0150] In one possible implementation, multiple spectral discriminators are cascaded sequentially in order of increasing resolution. The process of discriminating the amplitude spectrum based on the discriminant parameters corresponding to each discriminator to obtain the first discriminant result can be as follows: for the first discriminator, the input amplitude spectrum is discriminated based on the discriminant parameters corresponding to the first discriminator, resulting in the first discriminant result. For the remaining discriminators, the input amplitude spectrum is discriminated based on the discriminant result of the previous discriminator, resulting in the first discriminant result for each of the remaining discriminators.

[0151] Specifically, for the first spectrogram discriminator, its discrimination parameters are obtained. Based on these parameters, the amplitude spectrum at the corresponding resolution is input into the first spectrogram discriminator for discrimination, resulting in the amplitude spectrum features extracted by the first spectrogram discriminator. The first discrimination result of the first spectrogram discriminator is then obtained based on these amplitude spectrum features. For each of the remaining spectrogram discriminators, the amplitude spectrum features of the previous spectrogram discriminator are input into the current spectrogram discriminator. The current spectrogram discriminator then discriminates the input amplitude spectrum based on the amplitude spectrum features of the first spectrogram discriminator, obtaining the amplitude spectrum features of the current spectrogram discriminator. The first discrimination result of the current spectrogram discriminator is then obtained based on these amplitude spectrum features, and so on, until the first discrimination results of all spectrogram discriminators are obtained. By cascading the spectrogram discriminators in ascending order of resolution, feature fusion of amplitude spectrum features at different resolutions is achieved. This allows the spectrogram discriminators to learn high-resolution information from shallow amplitude spectrum features and semantic information from deep amplitude spectrum features, thereby outputting a discrimination result containing rich information. Based on this discrimination result, the accuracy and reliability of speech processing are improved.

[0152] Reference Figure 7 , Figure 7 This is another optional schematic diagram of the discriminator results in the diffusion model provided in the embodiments of this disclosure. Figure 7 The diagram illustrates five periodic discriminators and three spectral discriminators. The five periodic discriminators are arranged in parallel, while the three spectral discriminators are cascaded in ascending order of resolution. For spectral discriminator A, the amplitude spectrum A corresponding to the discriminant parameters of spectral discriminator A is input into spectral discriminator A to obtain amplitude spectrum feature p1. Based on amplitude spectrum feature p1, the discriminant result F is obtained. Amplitude spectrum feature p1 is input into spectral discriminator B, causing spectral discriminator B to discriminate amplitude spectrum B based on amplitude spectrum feature p1, obtaining amplitude spectrum feature p2. Based on amplitude spectrum feature p2, the discriminant result G is obtained. Similarly, spectral discriminator C discriminates amplitude spectrum C based on amplitude spectrum feature p2 to obtain amplitude spectrum feature p3. Based on amplitude spectrum feature p3, the discriminant result H is obtained.

[0153] Reference Figure 8 , Figure 8 This is a schematic diagram of an optional overall process for the model training method provided in this disclosure embodiment. The principle of the model training method in this disclosure embodiment is described in its entirety below:

[0154] The model training method provided in this embodiment includes data preprocessing, denoising, and other processes. During data preprocessing, the input signal needs to be converted into a Mel spectrogram before being restored to the signal representation. The denoising process is carried out by using a diffusion model based on a Schrödinger bridge. The diffusion model combines a period discriminator and a spectrogram discriminator, which are used to discriminate the signal generated by the diffusion model during the training process.

[0155] First, the diffusion model needs to be trained. Sample speech time-frequency signals are acquired and input into the diffusion model for prediction, resulting in predicted speech time-frequency signals. A signal difference loss is constructed based on the difference between the sample and predicted speech time-frequency signals. The sample speech time-frequency signals are transformed to obtain sample Mel spectrograms, and the predicted speech time-frequency signals are transformed to obtain predicted Mel spectrograms. A Mel difference loss is constructed based on the difference between the sample and predicted Mel spectrograms. The signal difference loss and Mel difference loss are defined as the reconstruction loss.

[0156] Next, short-time inverse Fourier transforms are performed on the predicted speech time-frequency signal and the sample speech time-frequency signal, respectively, to obtain the first time-domain signal and the second time-domain signal. Since the diffusion model can be regarded as a generator, and it combines a periodic discriminator and a spectrogram discriminator, a generator loss is constructed based on the first time-domain signal, and a discriminator loss is constructed based on the first and second time-domain signals. The periodic discriminator is used to acquire signal features of different signal periods, and the spectrogram discriminator is used to acquire signal features of different resolutions. To better extract high-quality features from the periodic and spectrogram discriminators, a feature matching loss is constructed based on the difference between the output features of the first and second time-domain signals at the same processing layer of the same discriminator. Finally, the signal difference loss, Mel spectrogram, generator loss, and feature matching loss are used as the target loss, and the discriminator loss is combined to train the diffusion model, the periodic discriminator, and the spectrogram discriminator. It should be noted that during training, the reconstruction loss and the discriminator loss are used as training criteria to evaluate the training effect.

[0157] After training, the data to be processed is acquired. When the data to be processed is a target Mel spectrogram, a pseudo-inverse transform is performed on the target Mel spectrogram to map it to the linear scale domain before compression, resulting in the inverse transform representation. A zero-phase component is added to the imaginary part of the inverse transform representation to obtain the time-frequency signal of the speech to be processed. When the data to be processed is text, the text is converted into a target Mel spectrogram, and then the time-frequency signal of the speech to be processed is obtained based on the target Mel spectrogram.

[0158] Next, the speech time-frequency signal to be processed is input into the diffusion model for denoising. During denoising, the initial time-frequency signal of the speech time-frequency signal to be processed at the current time and the predicted time-frequency signal predicted by the diffusion model at the current time are obtained. Based on the initial time-frequency signal of the speech time-frequency signal to be processed at the current time and the predicted time-frequency signal predicted by the diffusion model at the current time, an inverse stochastic differential equation is represented. Based on this inverse stochastic differential equation, a first-order discretization is performed to obtain the solution of the Schrödinger bridge. Denoising is performed based on the solution of the Schrödinger bridge. The above process is iterated several times to obtain the target speech time-frequency signal.

[0159] In one possible implementation, the model training method provided in this embodiment can be applied to communication scenarios. The voice signal of the caller is collected via a mobile phone, preprocessed, and then input into a diffusion model for denoising to obtain the target voice signal. This target voice signal is then transmitted to the receiver, and the audio is reconstructed based on the target voice signal and played back to the receiver.

[0160] Additionally, refer to Figure 9 , Figure 9 This is an optional flowchart of a speech processing method provided in an embodiment of the present disclosure. The speech processing can be performed by a terminal, by a server, or by a combination of both. The model training method includes, but is not limited to, the following steps S901 to S902.

[0161] Step S901: Obtain the target Mel spectrogram, and perform a pseudo-inverse transform on the target Mel spectrogram to obtain the speech time-frequency signal to be processed;

[0162] Step S902: Input the speech time-frequency signal to be processed into the diffusion model based on the Schrödinger bridge to denoise the speech time-frequency signal to obtain the target speech time-frequency signal.

[0163] The aforementioned speech processing method and model training method are based on the same inventive concept. By using the time-frequency signal of the speech to be processed to replace the original noise-adding result based on Gaussian distribution, the forward noise-adding process is transformed into a data-to-data process. This enables the recovery between the time-frequency signal of the speech to be processed and the time-frequency signal of the target speech, thereby preserving the structure of the signal during the diffusion process and improving the accuracy of speech processing.

[0164] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0165] Reference Figure 10 , Figure 10 This is a schematic diagram of the structure of the model training device 1000 provided in an embodiment of the present disclosure. The model training device 1000 includes:

[0166] The first transformation module 1001 is used to acquire the sample speech time-frequency signal, transform the sample speech time-frequency signal to obtain the sample Mel spectrum, and perform a pseudo-inverse transformation on the sample Mel spectrum to obtain the first speech time-frequency signal.

[0167] The first diffusion module 1002 is used to input the first speech time-frequency signal into the diffusion model based on the Schrödinger bridge, and to denoise the first speech time-frequency signal to obtain the second speech time-frequency signal. The distribution of the sample speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the initial time, and the distribution of the first speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the target time.

[0168] Training module 1003 is used to determine the target loss based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, and to train the diffusion model based on the target loss.

[0169] Furthermore, training module 1003 is also used for:

[0170] The signal difference loss is determined based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal;

[0171] The second speech time-frequency signal is transformed to obtain the predicted Mel spectrogram, and the Mel difference loss is determined based on the difference between the predicted Mel spectrogram and the sample Mel spectrogram.

[0172] The target loss is obtained by weighted summation of the signal difference loss and the Mel difference loss.

[0173] Furthermore, training module 1003 is also used for:

[0174] The inverse short-time Fourier transform of the second speech time-frequency signal is performed to obtain the first time-domain signal;

[0175] The first time-domain signal is discriminated based on multiple sub-discriminators, and the first discrimination result of each sub-discriminator is obtained.

[0176] The generator loss of the diffusion model is determined by the mean of multiple first discrimination results. The target loss is obtained by weighted summation of the signal difference loss, Mel difference loss and generator loss.

[0177] Furthermore, training module 1003 is also used for:

[0178] The inverse short-time Fourier transform of the sample speech time-frequency signal is performed to obtain the second time-domain signal;

[0179] The second time-domain signal is discriminated based on multiple sub-discriminators, and the second discrimination result of each sub-discriminator is obtained.

[0180] The discriminator loss of the discriminator is determined based on the mean of multiple first discrimination results and the mean of multiple second discrimination results;

[0181] The diffusion model and discriminator are trained jointly using target loss and discriminator loss.

[0182] Furthermore, training module 1003 is also used for:

[0183] For any processing layer in the sub-discriminator, obtain the first intermediate feature output when discriminating the first time-domain signal and the second intermediate feature output when discriminating the second time-domain signal;

[0184] For any sub-discriminator, determine the feature difference between the first intermediate feature and the second intermediate feature in the same processing layer, and determine the feature matching sub-loss of each sub-discriminator based on the mean of multiple feature differences;

[0185] The feature matching loss is determined by the mean of multiple feature matching sub-losses. The target loss is obtained by weighted summing of the signal difference loss, Mel difference loss, generator loss, and feature matching loss.

[0186] Furthermore, training module 1003 is also used for:

[0187] The first time-domain signal is reshaped into multiple two-dimensional signals with different periods. Each two-dimensional signal is input into a period discriminator of the corresponding period to discriminate the two-dimensional signal and obtain the first discrimination result of each period discriminator.

[0188] The amplitude spectrum of the first time-domain signal is input to each spectrum discriminator. The amplitude spectrum is discriminated based on the discrimination parameters corresponding to the spectrum discriminator, and the first discrimination result of each spectrum discriminator is obtained.

[0189] Furthermore, training module 1003 is also used for:

[0190] For the first period discriminator, the input two-dimensional signal is discriminated to obtain the first discrimination result of the first period discriminator;

[0191] For each of the remaining period discriminators, the input two-dimensional signal is discriminated based on the discrimination result of the previous period discriminator, and the first discrimination result of each of the remaining period discriminators is obtained.

[0192] Furthermore, training module 1003 is also used for:

[0193] For the first spectrum discriminator, the input amplitude spectrum is discriminated based on the discrimination parameters corresponding to the spectrum discriminator to obtain the first discrimination result of the first spectrum discriminator;

[0194] For each of the remaining spectrum discriminators, the input amplitude spectrum is discriminated based on the discrimination result of the previous spectrum discriminator, and the first discrimination result of each of the remaining spectrum discriminators is obtained.

[0195] Furthermore, the first transformation module 1001 is also used for:

[0196] The phase component of the sample speech time-frequency signal is discarded, and the sample speech time-frequency signal is compressed in the frequency dimension to obtain the sample Mel spectrum.

[0197] The sample Mel spectrum is mapped to the linear scale domain before compression based on the pseudo-inverse transform matrix to obtain the inverse transform representation. A zero-phase component is added to the imaginary part of the inverse transform representation to obtain the first speech time-frequency signal.

[0198] In summary, the model training apparatus provided in this embodiment acquires sample speech time-frequency signals, transforms the sample speech time-frequency signals to obtain sample Mel spectrograms, and then performs a pseudo-inverse transform on the sample Mel spectrograms to obtain a first speech time-frequency signal with a structure similar to the sample speech time-frequency signals but with some information loss. Based on this, the first speech time-frequency signal is input into a diffusion model based on a Schrödinger bridge to denoise the first speech time-frequency signal, obtaining a second speech time-frequency signal. This is equivalent to using the first speech time-frequency signal to replace the original Gaussian distribution-based denoising result, transforming the forward denoising process into a data-to-data process, and the distribution of the sample speech time-frequency signal is used for... The distribution of the first speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the initial time. This enables the recovery between the first speech time-frequency signal and the sample speech time-frequency signal, thus preserving the signal structure during the diffusion process. When determining the target loss based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, and training the diffusion model based on the target loss, the performance of the diffusion model can be effectively improved. Therefore, when acquiring the speech time-frequency signal to be processed and inputting it into the trained diffusion model for denoising, a clearer target speech time-frequency signal can be obtained, thereby improving the accuracy of speech processing.

[0199] Reference Figure 11 , Figure 11 This is a schematic diagram of the structure of a voice processing device provided in an embodiment of the present disclosure. The voice processing device 1100 includes:

[0200] The second transformation module 1101 is used to acquire the target Mel spectrum and perform a pseudo-inverse transformation on the target Mel spectrum to obtain the speech time-frequency signal to be processed.

[0201] The second diffusion module 1102 is used to input the speech time-frequency signal to be processed into a diffusion model based on Schrödinger bridge, and to denoise the speech time-frequency signal to be processed to obtain the target speech time-frequency signal.

[0202] The speech processing apparatus provided in this disclosure replaces the original Gaussian-based noise addition result with the speech time-frequency signal to be processed, transforming the forward noise addition process into a data-to-data process. This enables the recovery between the speech time-frequency signal to be processed and the target speech time-frequency signal, thereby preserving the structure of the signal during the diffusion process and improving the accuracy of speech processing.

[0203] The electronic device provided in this disclosure for executing the above-described model training method or speech processing method can be a terminal, as shown in the embodiments below. Figure 12 , Figure 12This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure. The terminal includes: a camera assembly 1210, a first memory 1220, an input unit 1230, a display unit 1240, a sensor 1250, an audio circuit 1260, a wireless fidelity (WiFi) module 1270, a first processor 1280, and a first power supply 1290, among other components. Those skilled in the art will understand that... Figure 12 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0204] The camera assembly 1210 can be used to capture images or videos. Optionally, the camera assembly 1210 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions.

[0205] The first memory 1220 can be used to store software programs and modules. The first processor 1280 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the first memory 1220.

[0206] The input unit 1230 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 1230 may include a touch panel 1231 and other input devices 1232.

[0207] The display unit 1240 can be used to display input or provided information, as well as various menus of the terminal. The display unit 1240 may include a display panel 1241.

[0208] Audio circuitry 1260, speaker 1261, and microphone 1262 provide an audio interface.

[0209] The first power source 1290 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery.

[0210] The number of sensors 1250 can be one or more, and these sensors 1250 include, but are not limited to: accelerometers, gyroscopes, pressure sensors, optical sensors, etc.

[0211] An accelerometer can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by the terminal. For example, an accelerometer can be used to detect the components of gravitational acceleration along the three coordinate axes. The first processor 1280 can control the display unit 1240 to display the user interface in either a horizontal or vertical view based on the gravitational acceleration signal acquired by the accelerometer. The accelerometer can also be used for collecting motion data from games or other applications.

[0212] The gyroscope sensor can detect the terminal's orientation and rotation angle. It can work in conjunction with an accelerometer to collect 3D user movements on the terminal. Based on the data collected by the gyroscope sensor, the first processor 1280 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0213] The pressure sensor can be installed on the side bezel of the terminal and / or on the lower layer of the display unit 1240. When the pressure sensor is installed on the side bezel of the terminal, it can detect the user's grip signal on the terminal, and the first processor 1280 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor. When the pressure sensor is installed on the lower layer of the display unit 1240, the first processor 1280 can control the operable controls on the UI interface based on the user's pressure operation on the display unit 1240. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0214] An optical sensor is used to collect ambient light intensity. In one embodiment, the first processor 1280 can control the display brightness of the display unit 1240 based on the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1240 is increased; when the ambient light intensity is low, the display brightness of the display unit 1240 is decreased. In another embodiment, the first processor 1280 can also dynamically adjust the shooting parameters of the camera assembly 1210 based on the ambient light intensity collected by the optical sensor.

[0215] In this embodiment, the first processor 1280 included in the terminal can execute the model training method or the speech processing method of the previous embodiment.

[0216] The electronic device provided in this disclosure for executing the above-described model training method or speech processing method can also be a server, see below. Figure 13 , Figure 13This is a partial structural block diagram of a server provided in an embodiment of the present disclosure. The server can vary significantly due to different configurations or performance. It may include one or more second processors 1310 and second memories 1330, and one or more storage media 1340 (e.g., one or more mass storage devices) for storing application programs 1343 or data 1342. The second memories 1330 and storage media 1340 may be temporary or persistent storage. The program stored in the storage media 1340 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the second processor 1310 may be configured to communicate with the storage media 1340 and execute the series of instruction operations stored in the storage media 1340 on the server.

[0217] The server may also include one or more secondary power supplies 1320, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1360, and / or one or more operating systems 1341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0218] The second processor 1310 in the server can be used to execute model training methods or speech processing methods.

[0219] This disclosure also provides a computer-readable storage medium for storing a computer program for executing the model training method or speech processing method of the foregoing embodiments.

[0220] This disclosure also provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the model training method or speech processing method described above.

[0221] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate to describe embodiments of this disclosure, for example, those that can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0222] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0223] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0224] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0225] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0226] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0227] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0228] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0229] The above is a detailed description of the preferred embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A model training method, characterized in that, include: Acquire a sample speech time-frequency signal, transform the sample speech time-frequency signal to obtain a sample Mel spectrogram, and perform a pseudo-inverse transform on the sample Mel spectrogram to obtain a first speech time-frequency signal; The first speech time-frequency signal is input into a diffusion model based on a Schrödinger bridge to denoise the first speech time-frequency signal and obtain a second speech time-frequency signal. The distribution of the sample speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the initial time, and the distribution of the first speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the target time. The target loss is determined based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, and the diffusion model is trained based on the target loss.

2. The model training method according to claim 1, characterized in that, The step of determining the target loss based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal includes: The signal difference loss is determined based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal; The second speech time-frequency signal is transformed to obtain a predicted Mel spectrogram, and the Mel difference loss is determined based on the difference between the predicted Mel spectrogram and the sample Mel spectrogram. The target loss is obtained by weighted summation of the signal difference loss and the Mel difference loss.

3. The model training method according to claim 2, characterized in that, The step of weighted summing of the signal difference loss and the Mel difference loss to obtain the target loss includes: Perform an inverse short-time Fourier transform on the second speech time-frequency signal to obtain the first time-domain signal; The first time-domain signal is identified by multiple sub-discriminators to obtain the first identification result of each sub-discriminator; The generator loss of the diffusion model is determined based on the mean of multiple first discrimination results, and the target loss is obtained by weighted summation of the signal difference loss, the Mel difference loss, and the generator loss.

4. The model training method according to claim 3, characterized in that, Training the diffusion model based on the target loss includes: The inverse short-time Fourier transform is performed on the sample speech time-frequency signal to obtain the second time-domain signal; The second time-domain signal is identified based on multiple sub-discriminators to obtain the second discrimination result of each sub-discriminator; The discriminator loss of the discriminator is determined based on the average of multiple first discrimination results and the average of multiple second discrimination results; The diffusion model and the discriminator are jointly trained based on the target loss and the discriminator loss.

5. The model training method according to claim 4, characterized in that, The sub-discriminator includes multiple processing layers, and the weighted summation of the signal difference loss, the Mel difference loss, and the generator loss to obtain the target loss includes: For any of the processing layers in the sub-discriminators, a first intermediate feature output when the first time-domain signal is discriminated and a second intermediate feature output when the second time-domain signal is discriminated are obtained; For any one of the sub-discriminators, the feature difference between the first intermediate feature and the second intermediate feature of the same processing layer is determined, and the feature matching sub-loss of each sub-discriminator is determined based on the mean of the multiple feature differences. The feature matching loss is determined based on the mean of multiple feature matching sub-losses, and the target loss is obtained by weighted summation of the signal difference loss, the Mel difference loss, the generator loss, and the feature matching loss.

6. The model training method according to claim 3, characterized in that, The plurality of sub-discriminators includes a plurality of period discriminators with different periods and a plurality of spectral discriminators with different resolutions. The discrimination of the first time-domain signal based on the plurality of sub-discriminators to obtain a first discrimination result for each of the sub-discriminators includes: The first time-domain signal is reshaped into multiple two-dimensional signals with different periods. Each two-dimensional signal is input to the period discriminator of the corresponding period to discriminate the two-dimensional signal and obtain the first discrimination result of each period discriminator. The amplitude spectrum of the first time-domain signal is input to each of the spectrum discriminators. The amplitude spectrum is discriminated based on the discrimination parameters corresponding to the spectrum discriminators to obtain the first discrimination result of each spectrum discriminator.

7. The model training method according to claim 6, characterized in that, Multiple period discriminators are cascaded sequentially in ascending order of period. The process of discriminating the two-dimensional signal to obtain the first discrimination result of each period discriminator includes: For the first period discriminator, the input two-dimensional signal is discriminated to obtain the first discrimination result of the first period discriminator; For each of the remaining period discriminators, the input two-dimensional signal is discriminated based on the discrimination result of the previous period discriminator to obtain the first discrimination result of each of the remaining period discriminators.

8. The model training method according to claim 6, characterized in that, Multiple spectral discriminators are cascaded sequentially in ascending order of resolution. The step of discriminating the amplitude spectrum based on the discrimination parameters corresponding to each spectral discriminator to obtain a first discrimination result for each spectral discriminator includes: For the first spectrum discriminator, the input amplitude spectrum is discriminated based on the discrimination parameters corresponding to the spectrum discriminator to obtain the first discrimination result of the first spectrum discriminator; For each of the remaining spectrum discriminators, the input amplitude spectrum is discriminated based on the discrimination result of the previous spectrum discriminator to obtain the first discrimination result of each of the remaining spectrum discriminators.

9. The model training method according to claim 1, characterized in that, The process of transforming the sample speech time-frequency signal to obtain a sample Mel spectrogram, and performing a pseudo-inverse transform on the sample Mel spectrogram to obtain a first speech time-frequency signal includes: The phase portion of the sample speech time-frequency signal is discarded, and the sample speech time-frequency signal is compressed in the frequency dimension to obtain the sample Mel spectrum. The sample Mel spectrum is mapped to the linear scale domain before compression based on the pseudo-inverse transform matrix to obtain the inverse transform representation. A zero-phase component is added to the imaginary part of the inverse transform representation to obtain the first speech time-frequency signal.

10. A speech processing method, characterized in that, include: Obtain the target Mel spectrogram, and perform a pseudo-inverse transform on the target Mel spectrogram to obtain the speech time-frequency signal to be processed; The speech time-frequency signal to be processed is input into a diffusion model based on a Schrödinger bridge to denoise the speech time-frequency signal and obtain the target speech time-frequency signal.

11. A model training device, characterized in that, include: The first transformation module is used to acquire the sample speech time-frequency signal, transform the sample speech time-frequency signal to obtain the sample Mel spectrum, and perform a pseudo-inverse transformation on the sample Mel spectrum to obtain the first speech time-frequency signal. The first diffusion module is used to input the first speech time-frequency signal into a diffusion model based on a Schrödinger bridge, and to denoise the first speech time-frequency signal to obtain a second speech time-frequency signal. The distribution of the sample speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the initial time, and the distribution of the first speech time-frequency signal is used as the edge distribution of the Schrödinger bridge at the target time. The training module is used to determine the target loss based on the difference between the second speech time-frequency signal and the sample speech time-frequency signal, and to train the diffusion model based on the target loss.

12. A voice processing device, characterized in that, include: The second transformation module is used to acquire the target Mel spectrogram and perform a pseudo-inverse transformation on the target Mel spectrogram to obtain the speech time-frequency signal to be processed. The second diffusion module is used to input the speech time-frequency signal to be processed into a diffusion model based on a Schrödinger bridge, and to denoise the speech time-frequency signal to be processed to obtain the target speech time-frequency signal.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the model training method according to any one of claims 1 to 9, or the speech processing method according to claim 10.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method according to any one of claims 1 to 9, or the speech processing method according to claim 10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method according to any one of claims 1 to 9, or the speech processing method according to claim 10.

Images