Model training method, speech synthesis method, and related product

Abstract

The application discloses a model training method, a speech synthesis method and related products. The model training method comprises the following steps: performing time-frequency conversion on an audio signal to obtain a first time-frequency representation of the audio signal on a first frequency band, wherein the audio signal comprises a first audio signal corresponding to a first text and a second audio signal obtained by performing speech synthesis on the first text by using a model; determining an offset of the audio signal in a time domain based on a sampling rate of the audio signal and the first frequency band; performing time-domain conversion on the audio signal after left shifting the audio signal based on the offset to obtain a second time-frequency representation of the audio signal on the first frequency band; determining a phase change amount of the audio signal on the first frequency band based on the first time-frequency representation and the second time-frequency representation; and adjusting parameters of the model based on the phase change amount.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a model training method, a speech synthesis method, and related products. Background Technology

[0002] With the continuous maturation of generative artificial intelligence, data generation models based on deep neural network models have become a hot area of ​​research and industrialization in the field of artificial intelligence. They have been widely applied in fields such as speech, and can synthesize text into corresponding audio signals.

[0003] However, current data generation models often produce audio signals with phase distortion, which directly leads to increased distortion and poorer synthesis results. Summary of the Invention

[0004] The purpose of this application is to provide a model training method, a speech synthesis method, and related products for training a model that synthesizes speech that more closely matches the real phase distribution, thereby solving the phase distortion problem and improving the speech synthesis effect.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions: In a first aspect, embodiments of this application provide a method for training a model, comprising: The audio signal is subjected to time-frequency transformation to obtain a first time-frequency representation of the audio signal in a first frequency band. The audio signal includes: a first audio signal corresponding to a first text, and a second audio signal obtained by the model through speech synthesis of the first text. Based on the sampling rate of the audio signal and the first frequency band, the offset of the audio signal in the time domain is determined; After shifting the audio signal to the left based on the offset, a time-domain transformation is performed to obtain a second time-frequency representation of the audio signal in the first frequency band. Based on the first time-frequency representation and the second time-frequency representation, the phase change of the audio signal in the first frequency band is determined; The parameters of the model are adjusted based on the phase change.

[0006] Secondly, embodiments of this application provide a speech synthesis method, including: Get the second text; The second text is processed by a model to synthesize speech, resulting in synthesized speech; wherein the model is trained using the training method provided in the first aspect.

[0007] Thirdly, embodiments of this application provide a model training apparatus, comprising: The transformation module is used to perform time-frequency transformation on the audio signal to obtain a first time-frequency representation of the audio signal in a first frequency band. The audio signal includes: a first audio signal corresponding to a first text, and a second audio signal obtained by the model through speech synthesis of the first text. The determining module is configured to determine the time-domain offset of the audio signal based on the sampling rate of the audio signal and the first frequency band; The transformation module is further configured to perform a time-domain transformation based on the offset left-shifting of the audio signal to obtain a second time-frequency representation of the audio signal in the first frequency band; The determining module is further configured to determine the phase change of the audio signal in the first frequency band based on the first time-frequency representation and the second time-frequency representation; An adjustment module is used to adjust the parameters of the model based on the phase change.

[0008] Fourthly, embodiments of this application provide a speech synthesis device, including: The acquisition module is used to acquire the second text; The speech synthesis module is used to synthesize speech from the second text using a model to obtain synthesized speech; wherein the model is trained based on the training method of the model provided in the first aspect.

[0009] Fifthly, embodiments of this application provide an electronic device, including: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the training method of the model provided in the first aspect or the speech synthesis method provided in the second aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform a training method for the model provided in the first aspect or a speech synthesis method provided in the second aspect.

[0011] In a seventh aspect, embodiments of this application provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps in the training method of the model provided in the first aspect or the speech synthesis method provided in the second aspect.

[0012] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: For both the real first audio signal and the model-synthesized second audio signal, a time-frequency transformation is performed on the audio signal to obtain a first time-frequency representation of the audio signal in a specified frequency band. This first time-frequency representation reflects the phase of the audio signal in the specified frequency band. Then, based on the sampling rate of the audio signal and the first frequency within the first frequency band, the offset of the audio signal in the time domain is determined. Based on this offset, the audio signal is shifted to the left, which is equivalent to pulling the "future audio signal" onto the time axis in the past to align it with the current moment. By performing a time-domain transformation on the left-shifted audio signal, a second time-frequency representation of the audio signal in the specified frequency band is obtained. This second time-frequency representation reflects the phase of the "future audio signal" in the specified frequency band. Therefore, based on the first and second time-frequency representations, the amount of phase change of the audio signal in the specified frequency band can be determined. Finally, based on the amount of phase change of the first and second audio signals in the same frequency band, the parameters of the model are adjusted, which helps the model learn the phase information of the real audio signal, thereby synthesizing speech that is more consistent with the real phase distribution. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic flowchart illustrating a model training method provided in one embodiment of this application; Figure 2 A schematic diagram of a time-frequency transformation result provided for one embodiment of this application; Figure 3 A schematic diagram of an audio signal before and after left shift, provided as an embodiment of this application; Figure 4 A schematic diagram illustrating the overlapping time period between audio signals before and after a left shift, provided as an embodiment of this application; Figure 5 A schematic flowchart of a speech synthesis method provided for one embodiment of this application; Figure 6 A schematic diagram of the structure of a model training device provided in one embodiment of this application; Figure 7 A schematic diagram of the structure of a speech synthesis device provided in one embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. The terms "first," "second," etc., used in this specification and claims are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in this specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0015] Explanation of some concepts: Periodic signals: Periodic signals are signals whose transient values ​​repeat over time, such as sinusoidal signals and pulse signals. Because they resemble oscillations when represented in the time domain, periodic signals are sometimes called wave signals or waveform signals. Some periodic signals are not simple periodic signals, but are formed by coupling different periodic signals.

[0016] Linear spectrum: For a continuous periodic time-domain signal, after a short-time Fourier transform, it is converted into a discrete time-frequency domain signal, which facilitates the analysis of the periodicity and frequency domain characteristics of the signal. This spectrum obtained directly by the short-time Fourier transform is also called the linear spectrum.

[0017] Adversarial Training: Adversarial neural networks, often referred to as GANs, are a training method for generative models, characterized by the introduction of adversarial training. Their main features are: a primary generative network model called the generator, and an auxiliary network for training the generator called the discriminator; both are trained together, creating an adversarial process. The generator's goal is to generate samples sufficient to confuse the discriminator's ability to recognize data, while the discriminator's goal is to correctly distinguish between the generator's samples and real data samples. Adversarial training introduces generator loss and discriminator loss; the former is used to optimize the generator network, and the latter is used to optimize the discriminator network.

[0018] Discriminator Network: In generative adversarial training, the primary goal is generally to train a generative model. The discriminator network participates in training as an auxiliary network, aiming to help the generator converge in the correct direction through red-black adversarial training. In principle, the discriminator network resembles a binary classification model, distinguishing between real and fake data. However, in practical applications, a binary classification model is not necessary; our main goal is to train the generator network. Therefore, the discriminator's loss is primarily used during training. The goal is for the discriminator to converge continuously during training. From the discriminator network structure perspective, different discriminator designs are used depending on the generation objective. Since the direct training objective of the discriminator is to distinguish between real and generated samples, it aims for the result of real samples passing through the discriminator network to be as close to 1 as possible, while the result of generated samples passing through the network should be as close to 0 as possible.

[0019] Period, frequency, energy, and phase: The basic physical quantities of a periodic signal include period, frequency, energy, and phase. Among them, energy is generally represented by the amplitude of vibration; phase is generally the angle between the current signal and the starting point of the period, also called the phase angle.

[0020] Discretization of time-domain signals: Time-domain signals are often continuous signals. For ease of study, they are often converted into discrete signal representations. For example, for audio signals, the short-time invariance of audio signals is utilized to convert them into audio frames arranged in time sequence. Each audio frame has a fixed window length, and the conversion has a fixed step size.

[0021] As mentioned earlier, the audio signals synthesized by current data generation models often suffer from phase distortion, which directly leads to increased distortion of the synthesized audio signals and a deterioration in the synthesis effect.

[0022] The inventors, through research on current data generation models, discovered that: On the one hand, current training methods for data generation models often adjust model parameters based on the energy distribution data of audio signals. This makes the model focus more on the energy distribution of the audio signal and less on other related aspects. This is because time-frequency domain signals often have a wide frequency band distribution. For example, a 16kHz audio signal can represent a theoretical vibration frequency range of 0~8000Hz. The higher the frequency band, the more drastic the phase change, and the lower the frequency band, the slower the phase change. Therefore, offline features extracted on the same time window scale are difficult to balance the phase changes across different frequency bands, making it difficult to directly use the phase of the audio signal for model training.

[0023] On the other hand, audio signals, as a time-frequency domain periodic signal, have short-term invariance, meaning that the characteristics of audio signals can be considered relatively stable within a very short time. This means that the spectral characteristics (such as frequency components and energy distribution) of audio signals will not change significantly during this period. However, as time goes by and the signal sampling changes, the phase at the same frequency may change, and the phase change caused by the small displacement of the audio signal along the time direction is relatively stable.

[0024] Furthermore, when performing speech synthesis using a model, if the synthesized audio signal is to be as close as possible to the real audio signal, then the model training process needs to make the synthesized audio signal as close to the real audio signal as possible. When a small time shift occurs in the real audio signal, if the phase change distribution of the real audio signal across various frequency bands is consistent with the phase change distribution of the synthesized audio signal after the same small time shift, then the phase distributions of the two are considered to be very similar, or the model has learned the phase information of the real audio signal. Based on this, by adjusting the model parameters according to the phase changes of the real and synthesized audio signals in the same frequency band, and making the difference between the phase changes of the two audio signals smaller and smaller, the model can learn the phase information of the real audio signal, thereby synthesizing speech that more closely matches the real phase distribution.

[0025] On the other hand, although audio signals are theoretically invariant in the short term, real audio signals are constantly changing, allowing humans to hear a person's voice as time flows. Based on this, shifting the audio signal to the left is equivalent to pulling the "audio signal that will occur at some point in the future" back along the timeline, aligning it with the current moment. In this way, by observing the difference in phase at the same frequency point before and after the left shift, we can observe the phase change of the audio signal over time.

[0026] Taking into account the above factors, a training method for the model is proposed. For both real and synthesized audio signals, a time-frequency transformation is performed to obtain a first time-frequency representation of the audio signal in a specified frequency band. This first time-frequency representation reflects the phase of the audio signal in the specified frequency band. Then, based on the sampling rate of the audio signal and a first frequency within the first frequency band, the offset of the audio signal in the time domain is determined. Based on this offset, the audio signal is shifted to the left, which is equivalent to pulling the "future audio signal" back on the time axis to align it with the current moment. By performing a time-domain transformation on the left-shifted audio signal, a second time-frequency representation of the audio signal in the specified frequency band is obtained. This second time-frequency representation reflects the phase of the "future audio signal" in the specified frequency band. Therefore, based on the first and second time-frequency representations, the phase change of the audio signal in the specified frequency band can be determined. Finally, based on the phase changes of the real and synthesized audio signals in the same frequency band, the model parameters are adjusted, which helps the model learn the phase information of the real audio signal, thereby synthesizing speech that more closely matches the real phase distribution.

[0027] It should be understood that the training method and speech synthesis method of the model provided in the embodiments of this application can be executed by an electronic device, specifically by the processor of the electronic device. The electronic device referred to herein may include terminals, such as, but not limited to, smartphones, tablets, laptops, desktop computers, smart voice interaction devices, smart home appliances, smartwatches, in-vehicle terminals, aircraft, etc.; or, the electronic device may also include a server, such as a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0028] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0029] Please refer to Figure 1 The above is a flowchart illustrating a model training method provided in one embodiment of this application. The method includes the following steps: S102, perform time-frequency transformation on the audio signal to obtain the first time-frequency representation of the audio signal in the first frequency band.

[0030] The audio signal includes: a first audio signal corresponding to the first text, and a second audio signal obtained by the model through speech synthesis of the first text. The first audio signal is the actual audio signal, and the second audio signal is the audio signal synthesized by the model. The model can be any model with speech synthesis capabilities, such as HiFiGAN, etc., and this application embodiment does not limit it.

[0031] For each audio signal, time-frequency transformation can be performed using various Fourier transforms, and this application does not limit the specific implementation of such transformations.

[0032] In one embodiment, to more accurately analyze the changes of the audio signal at different frequencies over time, a short-time Fourier transform is performed on the audio signal to obtain a first linear spectrum of the audio signal; then, a first time-frequency representation of the audio signal in a first frequency band is extracted from the first linear spectrum. The first time-frequency representation of the audio signal in the first frequency band includes: the first time-frequency representation of each audio frame in the audio signal in the first frequency band.

[0033] The first linear spectrum is a two-dimensional matrix. This matrix corresponds to individual audio frames in the time dimension and to frequency bands in the frequency domain dimension. Each frequency band represents a frequency range. Each element in this two-dimensional matrix is ​​a complex number representing the first time-frequency representation of an audio frame within a specific frequency band, denoted as […]. ,in, Indicates the real part, The imaginary part is represented. Based on this first time-frequency representation, the vibration amplitude of a certain audio frame in a certain frequency band can be obtained. and phase .like Figure 2 As shown, the first linear spectrum includes the first time-frequency representations of N audio frames in M ​​frequency bands, where the first time-frequency representation of the i-th audio frame in the j-th frequency band is denoted as... , , .

[0034] For example, an audio signal has a sampling rate of 16000Hz and a vibration frequency range of 0~8000Hz. If the frame window length is 640 sampling points, the sampling step size is 160 sampling points, and the number of sampling points is 1000, then after a short-time Fourier transform, the audio signal is divided into 1000 / 2 = 500 frequency bands (i.e., M = 500), and the width of each frequency band is 8000 / 500 = 16Hz. In this case, the first frequency band represents a frequency range of 0~16Hz, the second frequency band represents a frequency range of 16~32Hz, and so on, with the Mth frequency band representing the frequency range from the (M-1)th to the Mth 16Hz band.

[0035] It is worth noting that when performing a Fourier transform on the audio signal, the window starts from the first sampling point on the left and extracts samples frame by frame until the number of sampling points does not meet the length of a window. The sampling points that do not meet the length of a window are then discarded. In other words, no window padding is performed on the sampling points during the transform process.

[0036] The above describes one implementation of time-frequency transformation of an audio signal. It should be understood that time-domain transformation of an audio signal can also be achieved in other ways, such as, but not limited to, discrete cosine transform, continuous wavelet transform, and synchronous compression transform; this application does not limit these methods.

[0037] As mentioned earlier, audio signals can be transformed into multiple frequency bands through time-frequency conversion. The first frequency band can include some or all of these multiple frequency bands. For example, considering the complexity and variability of audio signals and their wide frequency range, if the model is to comprehensively learn the phase information of the real audio signal across the entire frequency range, all frequency bands can be designated as the first frequency band. Alternatively, if only the phase distortion of a specific frequency range is considered, allowing the model to focus on learning the phase information of the audio signal within that range, the corresponding frequency band can be selected as the first frequency band. Furthermore, considering that the mid-low frequency band and high frequency band are representative of the frequency distribution of human audio signals, one mid-low frequency band and two high frequency bands can be selected as the first frequency band. This allows the model to learn the phase information of the audio signal in representative frequency bands, ensuring the effectiveness of speech synthesis, while also reducing computational load and saving computation time and resources.

[0038] It is worth noting that, since audio signals are complex time-domain signals containing voiced and unvoiced components, the vocal cords do not vibrate when the unvoiced component is pronounced, and therefore it can be considered a non-periodic signal. Conversely, the voiced component vibrates when pronounced, and can be considered a periodic signal. Therefore, a segment corresponding to the voiced component can be extracted from the audio signal, and a time-frequency transformation can be performed on this segment to obtain its first time-frequency representation in the first frequency band.

[0039] S104 determines the time-domain offset of the audio signal based on the sampling rate and the first frequency band of the audio signal.

[0040] For each of the first and second audio signals, the offset of the audio signal in the time domain is used to indicate how many sampling points the audio signal has shifted in the time domain.

[0041] Since different frequency bands represent different frequency ranges, the phase change amplitude varies with each sampling point. Therefore, the time-domain offset of the audio signal is related to the frequency range of the band and the sampling rate of the audio signal. Thus, when there are multiple first frequency bands, the offset corresponding to each first frequency band can be determined based on that first frequency band and the sampling rate of the audio signal. Generally, the offset corresponding to a first frequency band does not exceed the sampling points in the main cycle of that first frequency band. For example, if the main frequency of the first frequency band is 500Hz, for an audio signal with a sampling rate of 16000Hz, each cycle is 1 / 500 = 0.002 seconds, and each cycle contains 16000 * 0.002 = 32 sampling points. Therefore, the offset corresponding to the first frequency band does not exceed 32 sampling points.

[0042] Based on this, in one embodiment, the above-mentioned S104 includes the following steps: S141, determine the first frequency within the first frequency band based on the relationship between the width of the first frequency band and the width threshold.

[0043] The first frequency can be any frequency within the first frequency band, and this application embodiment does not limit this. In one embodiment, to ensure the accuracy of the calculation of the time-domain offset of all audio signals, the first frequency can be determined in the following way: If the width of the first frequency band is greater than the width threshold, the center frequency of the first frequency band is taken as the first frequency; if the width of the first frequency band is less than or equal to the width threshold, the lower limit frequency or the upper limit frequency of the first frequency band is taken as the first frequency.

[0044] Specifically, if the width of the first frequency band is small, the upper and lower frequency limits within the first frequency band are not significantly different, thus using the upper and lower frequency limits to determine the time-domain offset of the audio signal has the same effect. However, if the width of the first frequency band is large, the upper and lower frequency limits within the first frequency band are not significantly different, thus using the upper and lower frequency limits to determine the time-domain offset of the audio signal has a significantly different effect. Therefore, the center frequency of the first frequency band is used as the first frequency, which ensures the accuracy of the offset calculation and provides reliable data support for the subsequent accurate calculation of the phase change between the real first audio signal and the synthesized second audio signal.

[0045] For example, if the first frequency band represents a frequency range of 784~800Hz, and its width is 16Hz, which is less than the width threshold of 100Hz, then the lower limit frequency of 784Hz or the upper limit frequency of 800Hz can be determined as the first frequency. As another example, if the first frequency band represents a frequency range of 600Hz~720Hz, and its width is 100Hz, which is greater than the width threshold of 100Hz, then the center frequency (600+720) / 2=660Hz can be taken as the first frequency.

[0046] S142, based on the ratio between the sampling rate and the first frequency, determine the number of sampling points corresponding to the first frequency band.

[0047] S143, perform a preset calculation on the number of sampling points and the offset ratio corresponding to the first frequency band, and determine the offset of the audio signal in the time domain based on the preset calculation result.

[0048] The offset ratio corresponding to the first frequency band can be set according to actual needs, and this application embodiment does not limit this. The preset operation may include multiplication operation, and then determine the offset of the audio signal in the time domain based on the product between the number of sampling points and the offset ratio.

[0049] Specifically, if the frequency range represented by the first frequency band is high, the offset ratio corresponding to the first frequency band can be a larger value to reflect the degree of phase change; if the frequency range represented by the first frequency band is low, the offset ratio corresponding to the first frequency band can be a smaller value to control the length of the offset.

[0050] For example, assuming the first frequency band includes the 50th frequency band (representing a frequency range of 784~800Hz), the 100th frequency band (representing a frequency range of 1584~1600Hz), and the 150th frequency band (representing a frequency range of 2384~2400Hz), with the center frequency as the first frequency, then the first frequencies in these three frequency bands are 792Hz, 1592Hz, and 2392Hz, respectively.

[0051] If the sampling rate of the audio signal is 16000Hz, then for the 50th frequency band, at the first frequency of 792Hz, the number of sampling points in each cycle (i.e. the number of sampling points corresponding to this frequency band) is 16000 / 792=20. Assuming that the offset ratio corresponding to this frequency band is 0.5, then the offset corresponding to this frequency band is 20*0.5=10 sampling points.

[0052] Similarly, for the 100th frequency band, at the first frequency of 1592Hz, the number of sampling points in each cycle is 16000 / 1592=10. Assuming the offset ratio corresponding to this frequency band is 0.6, then the offset corresponding to this frequency band is 10*0.6=6 sampling points.

[0053] Similarly, for the 150th frequency band, at the first frequency of 2392Hz, the number of sampling points in each cycle is 16000 / 2392Hz=6.69. Assuming that the offset ratio corresponding to this frequency band is 1.0, the offset corresponding to this frequency band is the result of rounding 6.69, which is 6 sampling points.

[0054] S106, based on the left-shifted audio signal by the offset, performs a time-domain transformation to obtain the second time-frequency representation of the audio signal in the first frequency band.

[0055] Specifically, such as Figure 3 As shown, shifting the audio signal to the left in the time domain by an offset representing the number of sampling points is equivalent to pulling the "future audio signal" forward on the time axis to align it with the current moment. Based on this, performing a time-domain transformation on the left-shifted audio signal yields a second time-frequency representation of the audio signal in the first frequency band, which can reflect the phase of the "future audio signal" in the specified frequency band.

[0056] The specific implementation of performing time-domain transformation on the left-shifted audio signal to obtain the second time-frequency representation is similar to the specific implementation of S102 above, and will not be described again.

[0057] S108, based on the first time-frequency representation and the second time-frequency representation, determine the phase change of the audio signal in the first frequency band.

[0058] Since the first time-frequency representation reflects the phase of the audio signal in the first frequency band, and the second time-frequency representation reflects the phase of the "audio signal at a future moment" in the first frequency band, the amount of phase change of the audio signal in the first frequency band can be determined based on the difference between the two. It is worth noting that when there are multiple first frequency bands, each first frequency band has a corresponding amount of phase change.

[0059] In one embodiment, the audio signal includes a plurality of first signal frames, and the left-shifted audio signal includes a plurality of second signal frames. S108 above includes the following steps: S181, determine the overlapping period between the audio signal and the left-shifted audio signal.

[0060] For example, such as Figure 4 As shown, after shifting the audio signal to the left by an offset representing the number of sampling points, the time period corresponding to the overlapping segment between the audio signal and the left-shifted audio segment is the overlapping time period.

[0061] S182, based on the first time-frequency representation of the first signal frame in the first frequency band during the overlapping period, determine the first phase of the first signal frame in the first frequency band.

[0062] like Figure 4As shown, the number of first signal frames and second signal frames within the overlapping time period is N'. Each first signal frame has a corresponding first time-frequency representation in the first frequency band, and each second signal frame has a corresponding second time-frequency representation in the first frequency band. As mentioned above, the first time-frequency representation is a complex number. For each first signal frame, the arctangent operation is performed on the ratio between the imaginary part y and the real part x of the first time-frequency representation of the first signal frame, i.e., arctan(y / x), to obtain the first phase of the first audio frame in the first frequency band. The value range of the first phase is... .

[0063] S183, based on the second time-frequency representation of the second signal frame in the first frequency band during the overlapping period, determine the second phase of the second signal frame in the first frequency band.

[0064] The specific implementation of S183 is similar to that of S182 described above, and will not be repeated here.

[0065] S184, based on the difference between the second phase and the first phase, obtains the phase change of the audio signal in the first frequency band.

[0066] Specifically, the phase change of the audio signal in the first frequency band includes the phase difference between the corresponding first signal frame and the second signal frame in the first frequency band during the overlapping time period.

[0067] For example, with Figure 4 For example, the phase change of the audio signal in the first frequency band is a vector of length N'. The i-th element of this vector represents the difference between the first phase of the i-th first signal frame in the first frequency band and the second phase of the i-th second signal frame in the first frequency band. The phase change of the audio signal in the first frequency band is denoted as... ,in, This represents the phase change between the i-th first signal frame and the i-th second signal frame in the first frequency band. .

[0068] In the above embodiments, the first phase and second phase of each frame in the first frequency band during the overlapping period between the audio signal and the left-shifted audio signal can accurately reflect the phase information of the audio signal before and after the shift in each small time period. Based on the difference between the two, the amount of phase change of the audio signal before and after the shift in each small time period can be accurately determined. The above shows a partial implementation of S108. Of course, it should be understood that S108 can also be implemented in other ways, and the embodiments of this application do not limit it.

[0069] S110, based on the phase change, adjusts the parameters of the model.

[0070] Based on the phase change of the first and second audio signals in the same frequency band, the degree of phase distortion of the second audio signal is reflected. This phase change adjusts the parameters of the model, which helps the model learn the phase information of the real audio signal, thereby synthesizing speech that is more consistent with the real phase distribution.

[0071] In one embodiment, S110 includes the following steps: S1101, determine the first loss based on the phase change of the first audio signal in the first frequency band and the phase change of the second audio signal in the first frequency band.

[0072] If there are multiple first frequency bands, each first frequency band has a corresponding first loss. The first loss is used to represent the degree of phase distortion of the second audio signal in the corresponding first frequency band.

[0073] As an example, the first loss can be determined based on the L1 loss function or the L2 loss function, and the difference between the phase change of the first audio signal in the first frequency band and the phase change of the second audio signal in the first frequency band.

[0074] As another example, when there are multiple first frequency bands, for each audio signal in the first audio signal and the second audio signal, a first matrix corresponding to the audio signal is generated, and the row vectors of the first matrix represent the phase change of the audio signal in a first frequency band; then, based on the distance between the first matrix corresponding to the first audio signal and the first matrix corresponding to the second audio signal, a first loss is obtained.

[0075] For example, assuming the first frequency band has 3 bands, the first matrix corresponding to each audio signal can be represented as follows:

[0076] in, This represents the amount of phase change of the audio signal in the first frequency band. This indicates the amount of phase change of the audio signal in the second first frequency band. This represents the phase change of the audio signal in the third first frequency band. The phase change of the audio signal in each first frequency band includes the phase change between the first signal frame and the second signal frame in the left-shifted audio signal frame within that first frequency band.

[0077] It is worth noting that if the offsets of the different first frequency bands are large, and the number of frames obtained when extracting the Mel spectrum by fast Fourier transform is not the same, the first matrix can be based on the length of the first frequency band with the longest number of frames, and the number of frames of the other first frequency bands can be padded to this length, with the phase change amount used for padding being 0.

[0078] As another example, there are multiple first frequency bands. The phase change of the audio signal in the first frequency band includes the phase change of multiple first signal frames in the audio signal in the first frequency band, that is, the phase difference between multiple first signal frames and their corresponding second signal frames in the first frequency band. In this case, a first matrix corresponding to the first audio signal and a second matrix corresponding to the second audio signal are generated. The row vectors of the first matrix correspond to a first frequency band, and each element in the row vector represents the phase change of a first signal frame in the first audio signal in the corresponding first frequency band. The row vectors of the second matrix also correspond to a first frequency band, and each element in the row vector represents the phase change of a first signal frame in the second audio signal in the first frequency band. For each first frequency band, based on the frequency range of the first frequency band and the pitch of each first signal frame in the first audio signal, the row vectors corresponding to the first frequency band in the first matrix are masked to obtain a third matrix. Similarly, based on the frequency range of the first frequency band and the pitch of each first signal frame in the second audio signal, the row vectors corresponding to the first frequency band in the second matrix are masked to obtain a fourth matrix. The first loss is obtained based on the distance between the third and fourth matrices.

[0079] More specifically, the above-mentioned masking of the row vectors corresponding to the first frequency band in the first matrix based on the frequency range of the first frequency band and the pitch of each first signal frame in the first audio signal to obtain a third matrix, and masking of the row vectors corresponding to the first frequency band in the second matrix based on the frequency range of the first frequency band and the pitch of each first signal frame in the second audio signal to obtain a fourth matrix, includes: if the upper limit frequency of the first frequency band is less than a first frequency threshold or the lower limit frequency is greater than a second frequency threshold, then masking each element in the row vectors corresponding to the first frequency band in the first matrix to obtain a third matrix, and masking each element in the row vectors corresponding to the first frequency band in the second matrix to obtain a fourth matrix; if the first If the upper limit frequency of the frequency band is greater than or equal to a first frequency threshold, and the lower limit frequency of the first frequency band is less than or equal to a second frequency threshold, then from multiple first signal frames of the first audio signal, a third signal frame with a pitch less than a pitch threshold is determined, and the first element in the row vector corresponding to the first frequency band in the first matrix is ​​masked to obtain a third matrix, where the first element represents the phase change amount of the third signal frame in the first frequency band; and from the first signal frames of the second audio signal, a fourth signal frame with a pitch less than a pitch threshold is determined, and the second element in the row vector corresponding to the first frequency band in the second matrix is ​​masked to obtain a fourth matrix, where the second element represents the phase change amount of the fourth signal frame in the first frequency band.

[0080] For example, the frequency range of signals that the human ear can hear is 20Hz to 20000Hz, while the frequency range of most people's voice vibrations is less than 20Hz to 20000Hz. Therefore, the first frequency threshold can be set to 100Hz, and the second frequency threshold can be set to 7000Hz. Based on this, if the lower limit frequency of a certain first frequency band is less than 100Hz or the upper limit frequency is greater than 7000Hz, it indicates that the first frequency band has exceeded the frequency range that the human ear can hear. There is no need to consider the phase distortion in the first frequency band. Therefore, all elements in the row vector corresponding to the first frequency band in the first matrix can be masked to block out the phase change of the audio signal in the first frequency band when calculating the first loss.

[0081] If the lower limit frequency of a certain first frequency band is greater than or equal to 100Hz and the upper limit frequency is less than or equal to 7000Hz, it indicates that the first frequency band is within the frequency range that the human ear can hear. However, for audio signals, signal frames with lower pitches usually belong to the soft part. The phase distortion of such signal frames in the first frequency band is also meaningless for speech synthesis. Therefore, the elements representing the phase change of such signal frames in the row vector corresponding to the first frequency band in the first matrix can be masked to mask the phase change of such signal frames in the first frequency band when calculating the first loss.

[0082] In practical applications, the row vector can be multiplied by the mask vector to mask the elements of the row vector corresponding to the first frequency band. The mask values ​​in the mask vector correspond one-to-one with the elements in the row vector. If an element in the row vector is retained, the mask value corresponding to that element in the mask vector is 1; if an element in the row vector is masked, the mask value corresponding to that element in the mask vector is 0. In other words, the mask vector can be represented as: ,in, This represents the mask value corresponding to the i-th element in the row vector. .

[0083] S1102, Based on the first loss, the parameters of the model are adjusted.

[0084] As an example, the backpropagation algorithm is used to adjust the model's parameters with the goal of reducing the first loss.

[0085] As another example, in the case of a generator-discriminator structure, the loss in the model can also include a second loss caused by the generator and a third loss caused by the discriminator. Here, the first loss is determined based on the difference between the first and second audio signals, and the second loss is... ,in, This represents the expectation for the first audio signal. From the distribution of real data Mid-sampling, This indicates the expectation for the generated data. Sampling from the noise distribution Let represent the generator function, which maps noise to the data space. In practice, we typically use mini-batch data to approximate these expectations, so the loss function can be written as: , Indicates the size of the small batch. It is the first audio signal sampled from a small batch of real data. It consists of samples taken from small batches of noise, and then fake data is generated by generator G. .

[0086] In this case, the first loss, the second loss, and the third loss are weighted and summed to obtain the total loss of the model; then, the backpropagation algorithm is used to adjust the parameters of the model with the goal of reducing the total loss.

[0087] It's worth noting that adding a first loss will increase training time, but it won't affect the model's inference time. Furthermore, the introduced first loss is added to the model's generation loss and has no impact on the original discriminator. Overall training steps, learning rate, and other hyperparameters do not need adjustment; the default settings are sufficient.

[0088] The foregoing illustrates a partial implementation of S110. It should be understood that S110 can also be implemented in other ways, and this application does not limit this implementation.

[0089] In practical applications, S102~S110 above constitutes only one round of model training. In practice, the model can be trained in multiple rounds, that is, S102~S110 can be repeated multiple times until the training stopping condition is met. The training stopping condition can be set according to actual needs, such as the convergence of the first loss or reaching a training round threshold, etc., and this embodiment does not limit this.

[0090] The training method for the model provided in one or more embodiments of this application, regardless of whether it is a real first audio signal or a model-synthesized second audio signal, involves performing a time-frequency transformation on the audio signal to obtain a first time-frequency representation of the audio signal in a specified frequency band. This first time-frequency representation reflects the phase of the audio signal in the specified frequency band. Then, based on the sampling rate of the audio signal and a first frequency within the first frequency band, the offset of the audio signal in the time domain is determined. Based on this offset, the audio signal is shifted to the left, which is equivalent to pulling the "audio signal at a future moment" back along the time axis to align it with the current moment. By performing a time-domain transformation on the left-shifted audio signal, a second time-frequency representation of the audio signal in the specified frequency band is obtained. This second time-frequency representation reflects the phase of the "audio signal at a future moment" in the specified frequency band. Therefore, based on the first and second time-frequency representations, the amount of phase change of the audio signal in the specified frequency band can be determined. Finally, based on the amount of phase change of the first and second audio signals in the same frequency band, the parameters of the model are adjusted, which helps the model learn the phase information of the real audio signal, thereby synthesizing speech that is more consistent with the real phase distribution.

[0091] Based on the training method described above, this application also provides a speech synthesis method. Please refer to... Figure 5 The following is a flowchart illustrating a speech synthesis method according to an embodiment of this application. The method includes: S502, retrieve the second text.

[0092] The second text can be any text to be synthesized into speech.

[0093] S504 uses a model to synthesize speech from the second text, resulting in synthesized speech.

[0094] The speech synthesis method provided by one or more embodiments of this application can synthesize speech that is more consistent with the real phase distribution, thereby solving the phase distortion problem and improving the speech synthesis effect.

[0095] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0096] Based on the same inventive concept, embodiments of this application also provide a model training device. Please refer to... Figure 6 This is a schematic diagram of the structure of a model training device 600 provided in an embodiment of this application. The device 600 includes: a transformation module 610, a determination module 620, and an adjustment module 630.

[0097] The transformation module 610 is used to perform time-frequency transformation on the audio signal to obtain a first time-frequency representation of the audio signal in a first frequency band. The audio signal includes: a first audio signal corresponding to a first text, and a second audio signal obtained by the model through speech synthesis of the first text. The determining module 620 is used to determine the offset of the audio signal in the time domain based on the sampling rate of the audio signal and the first frequency band; The transformation module 610 is further configured to perform a time-domain transformation based on the offset left-shifting of the audio signal to obtain a second time-frequency representation of the audio signal in the first frequency band; The determining module 620 is further configured to determine the phase change of the audio signal in the first frequency band based on the first time-frequency representation and the second time-frequency representation; The adjustment module 630 is used to adjust the parameters of the model based on the phase change amount.

[0098] In another embodiment, the determining module is used to: The first frequency within the first frequency band is determined based on the relationship between the width of the first frequency band and the width threshold. The number of sampling points corresponding to the first frequency band is determined based on the ratio between the sampling rate and the first frequency. The number of sampling points is calculated in a preset ratio to the offset ratio corresponding to the first frequency band, and the offset of the audio signal in the time domain is determined based on the calculation result.

[0099] In another embodiment, when the determining module determines the first frequency within the first frequency band based on the relationship between the width of the first frequency band and the width threshold, it performs the following steps: If the width of the first frequency band is greater than the width threshold, then the center frequency of the first frequency band is taken as the first frequency. If the width of the first frequency band is less than or equal to the width threshold, then the lower limit frequency or the upper limit frequency of the first frequency band is taken as the first frequency.

[0100] In another embodiment, the audio signal includes a plurality of first signal frames, and the left-shifted audio signal includes a plurality of second signal frames; The determining module is used for: Determine the overlap period between the audio signal and the left-shifted audio signal; Based on the first time-frequency representation of the first signal frame in the first frequency band during the overlapping period, the first phase of the first signal frame in the first frequency band is determined; Based on the second time-frequency representation of the second signal frame in the first frequency band during the overlapping period, the second phase of the second signal frame in the first frequency band is determined; The phase change of the audio signal in the first frequency band is obtained based on the difference between the second phase and the first phase.

[0101] In another embodiment, the adjustment module is used to: The first loss is determined based on the phase change of the first audio signal in the first frequency band and the phase change of the second audio signal in the first frequency band; Based on the first loss, the parameters of the model are adjusted.

[0102] In another embodiment, the number of first frequency bands is multiple, and the phase change of the audio signal in the first frequency band includes: the phase change of multiple first signal frames in the audio signal in the first frequency band; When the adjustment module determines the first loss based on the phase change of the first audio signal in the first frequency band and the phase change of the second audio signal in the first frequency band, it performs the following steps: Generate a first matrix corresponding to the first audio signal and a second matrix corresponding to the second audio signal; the row vectors of the first matrix correspond to a first frequency band, and each element in the row vectors of the first matrix represents the phase change of a first signal frame in the first audio signal in the first frequency band; the row vectors of the second matrix correspond to a first frequency band, and each element in the row vectors of the second matrix represents the phase change of a first signal frame in the second audio signal in the first frequency band. For each first frequency band, based on the frequency range of the first frequency band and the pitch of each first signal frame in the first audio signal, the row vector corresponding to the first frequency band in the first matrix is ​​masked to obtain a third matrix; and based on the frequency range of the first frequency band and the pitch of each first signal frame in the second audio signal, the row vector corresponding to the first frequency band in the second matrix is ​​masked to obtain a fourth matrix. The first loss is obtained based on the distance between the third matrix and the fourth matrix.

[0103] In another embodiment, when the adjustment module masks the row vectors corresponding to the first frequency band in the first matrix based on the frequency range of the first frequency band and the pitch of each first signal frame in the first audio signal to obtain a second matrix, and masks the row vectors corresponding to the first frequency band in the second matrix based on the frequency range of the first frequency band and the pitch of each first signal frame in the second audio signal to obtain a fourth matrix, the following steps are performed: If the upper limit frequency of the first frequency band is less than the first frequency threshold or the lower limit frequency is greater than the second frequency threshold, then each element in the row vector corresponding to the first frequency band in the first matrix is ​​masked to obtain the third matrix, and each element in the row vector corresponding to the first frequency band in the second matrix is ​​masked to obtain the fourth matrix. If the upper limit frequency of the first frequency band is greater than or equal to the first frequency threshold, and the lower limit frequency of the first frequency band is less than or equal to the second frequency threshold, then a third signal frame with a pitch less than the pitch threshold is determined from the first signal frame of the first audio signal, and the first element in the row vector corresponding to the first frequency band in the first matrix is ​​masked to obtain a third matrix, where the first element represents the phase change of the third signal frame in the first frequency band; and a fourth signal frame with a pitch less than the pitch threshold is determined from the first signal frame of the second audio signal, and the second element in the row vector corresponding to the first frequency band in the second matrix is ​​masked to obtain a fourth matrix, where the second element represents the phase change of the fourth signal frame in the first frequency band.

[0104] Obviously, the training device for the model provided in this application embodiment can be used as... Figure 1 The entity that performs the training method of the model shown, for example Figure 1 In the training method of the model shown, steps S102 and S106 can be derived from... Figure 6 The transformation module 610 in the training device of the model shown executes steps S104 and S108, which can be performed by... Figure 6 The determination module 620 in the training device of the model shown is executed, and step S110 can be performed by... Figure 6The adjustment module 630 in the training device of the model shown is executed.

[0105] According to another embodiment of this application, Figure 6 The modules in the training device of the model shown can be individually or entirely merged into one or more other modules, or some of the modules can be further divided into multiple functionally smaller modules. This achieves the same operation without affecting the technical effect of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one module can be implemented by multiple modules, or the function of multiple modules can be implemented by one module. The training device of the model in the embodiments of this application may also include other modules. In practical applications, these modules can also be implemented with the assistance of other modules, and can be implemented by multiple modules working together.

[0106] According to another embodiment of this application, a general-purpose computing device, such as a computer, including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can run an application capable of performing tasks such as... Figure 1 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 6 The diagram illustrates a training apparatus for the model and a training method for implementing the model according to embodiments of this application. The computer program may be recorded on, for example, a computer-readable storage medium, and may be transferred to and executed in an electronic device via such a medium.

[0107] Based on the same inventive concept, this application also provides a speech synthesis device. Please refer to... Figure 7 The following is a schematic diagram of the structure of a speech synthesis 700 provided in an embodiment of this application. The device 700 includes: an acquisition module 710 and a speech synthesis module 720.

[0108] The acquisition module 710 is used to acquire the second text.

[0109] The speech synthesis module 720 is used to synthesize the second text into a third audio signal using a model; wherein the model is trained based on the training method of the model provided in the first aspect.

[0110] Obviously, the speech synthesis device provided in this application embodiment can serve as... Figure 5 The entity executing the speech synthesis method shown, for example Figure 1 In the speech synthesis method shown, step S502 can be performed by... Figure 7 The acquisition module 710 in the speech synthesis device shown executes step S504, which can be performed by... Figure 7 The speech synthesis module 720 in the speech synthesis device shown is executed.

[0111] According to another embodiment of this application, Figure 7 The various modules in the speech synthesis device shown can be individually or entirely merged into one or more other modules, or some of the modules can be further divided into multiple functionally smaller modules. This achieves the same operation without affecting the technical effect of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one module can also be implemented by multiple modules, or the function of multiple modules can be implemented by one module. The speech synthesis device in the embodiments of this application may also include other modules. In practical applications, these modules can also be implemented with the assistance of other modules, and can be implemented by multiple modules working together.

[0112] According to another embodiment of this application, a general-purpose computing device, such as a computer, including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can run an application capable of performing tasks such as... Figure 5 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 7 The document describes a speech synthesis apparatus and a speech synthesis method for implementing embodiments of this application. The computer program may be recorded on, for example, a computer-readable storage medium, and may be transferred to and executed in an electronic device via such a medium.

[0113] Figure 8 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Please refer to it. Figure 8 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0114] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0115] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0116] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a training device for the model at the logical level. The processor executes the program stored in memory and specifically performs the following operations: The audio signal is subjected to time-frequency transformation to obtain a first time-frequency representation of the audio signal in a first frequency band. The audio signal includes: a first audio signal corresponding to a first text, and a second audio signal obtained by the model through speech synthesis of the first text. Based on the sampling rate of the audio signal and the first frequency band, the offset of the audio signal in the time domain is determined; After shifting the audio signal to the left based on the offset, a time-domain transformation is performed to obtain a second time-frequency representation of the audio signal in the first frequency band. Based on the first time-frequency representation and the second time-frequency representation, the phase change of the audio signal in the first frequency band is determined; The parameters of the model are adjusted based on the phase change.

[0117] Alternatively, the processor reads the corresponding computer program from non-volatile memory into memory and runs it, forming a speech synthesis device at the logical level. The processor executes the program stored in memory and specifically performs the following operations: Get the second text; The second text is processed by a model to synthesize speech, thereby obtaining synthesized speech; wherein the model is trained using the training method provided in the embodiments of this application.

[0118] The above is as stated in this application. Figure 1The method performed by the training apparatus for the model disclosed in the illustrated embodiments, or as described above in this application. Figure 5 The speech synthesis device method disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0119] The electronic device can also perform Figure 1 The method, and implement the model training device in Figure 1 The illustrated embodiment may also perform the functions of the electronic device, or the electronic device may also perform the functions of the embodiment. Figure 5 The method, and realize the speech synthesis device in Figure 5 The functions of the embodiments shown are not described in detail here.

[0120] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0121] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 1 The method of the illustrated embodiment is specifically used to perform the following operations: The audio signal is subjected to time-frequency transformation to obtain a first time-frequency representation of the audio signal in a first frequency band. The audio signal includes: a first audio signal corresponding to a first text, and a second audio signal obtained by the model through speech synthesis of the first text. Based on the sampling rate of the audio signal and the first frequency band, the offset of the audio signal in the time domain is determined; After shifting the audio signal to the left based on the offset, a time-domain transformation is performed to obtain a second time-frequency representation of the audio signal in the first frequency band. Based on the first time-frequency representation and the second time-frequency representation, the phase change of the audio signal in the first frequency band is determined; The parameters of the model are adjusted based on the phase change.

[0122] Alternatively, when executed by a portable electronic device that includes multiple applications, the instruction can enable the portable electronic device to perform... Figure 5 The method of the illustrated embodiment is specifically used to perform the following operations: Get the second text; The second text is processed by a model to synthesize speech, thereby obtaining synthesized speech; wherein the model is trained using the training method provided in the embodiments of this application.

[0123] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0124] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0125] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0127] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A method for training a model, characterized in that, include: The audio signal is subjected to time-frequency transformation to obtain a first time-frequency representation of the audio signal in a first frequency band. The audio signal includes: a first audio signal corresponding to a first text, and a second audio signal obtained by the model through speech synthesis of the first text. Based on the sampling rate of the audio signal and the first frequency band, the offset of the audio signal in the time domain is determined; After shifting the audio signal to the left based on the offset, a time-domain transformation is performed to obtain a second time-frequency representation of the audio signal in the first frequency band. Based on the first time-frequency representation and the second time-frequency representation, the phase change of the audio signal in the first frequency band is determined; The parameters of the model are adjusted based on the phase change.

2. The method according to claim 1, characterized in that, Determining the time-domain offset of the audio signal based on the sampling rate of the audio signal and the first frequency band includes: The first frequency within the first frequency band is determined based on the relationship between the width of the first frequency band and the width threshold. The number of sampling points corresponding to the first frequency band is determined based on the ratio between the sampling rate and the first frequency. The number of sampling points is used to perform a preset calculation with the offset ratio corresponding to the first frequency band, and the offset of the audio signal in the time domain is determined based on the preset calculation result.

3. The method according to claim 2, characterized in that, Determining the first frequency within the first frequency band based on the relationship between the width of the first frequency band and the width threshold includes: If the width of the first frequency band is greater than the width threshold, then the center frequency of the first frequency band is taken as the first frequency. If the width of the first frequency band is less than or equal to the width threshold, then the lower limit frequency or the upper limit frequency of the first frequency band is taken as the first frequency.

4. The method according to claim 1, characterized in that, The audio signal includes multiple first signal frames, and the left-shifted audio signal includes multiple second signal frames; Determining the phase change of the audio signal in the first frequency band based on the first time-frequency representation and the second time-frequency representation includes: Determine the overlap period between the audio signal and the left-shifted audio signal; Based on the first time-frequency representation of the first signal frame in the first frequency band during the overlapping period, the first phase of the first signal frame in the first frequency band is determined; Based on the second time-frequency representation of the second signal frame in the first frequency band during the overlapping period, the second phase of the second signal frame in the first frequency band is determined; The phase change of the audio signal in the first frequency band is obtained based on the difference between the second phase and the first phase.

5. The method according to claim 1, characterized in that, The adjustment of the model parameters based on the phase change includes: The first loss is determined based on the phase change of the first audio signal in the first frequency band and the phase change of the second audio signal in the first frequency band; Based on the first loss, the parameters of the model are adjusted.

6. The method according to claim 5, characterized in that, The number of the first frequency bands is multiple, and the phase change of the audio signal in the first frequency band includes: the phase change of multiple first signal frames in the audio signal in the first frequency band; The determination of the first loss based on the phase change of the first audio signal in the first frequency band and the phase change of the second audio signal in the first frequency band includes: Generate a first matrix corresponding to the first audio signal and a second matrix corresponding to the second audio signal; the row vectors of the first matrix correspond to a first frequency band, and each element in the row vectors of the first matrix represents the phase change of a first signal frame in the first audio signal in the first frequency band; the row vectors of the second matrix correspond to a first frequency band, and each element in the row vectors of the second matrix represents the phase change of a first signal frame in the second audio signal in the first frequency band. For each first frequency band, based on the frequency range of the first frequency band and the pitch of each first signal frame in the first audio signal, the row vector corresponding to the first frequency band in the first matrix is ​​masked to obtain a third matrix; and based on the frequency range of the first frequency band and the pitch of each first signal frame in the second audio signal, the row vector corresponding to the first frequency band in the second matrix is ​​masked to obtain a fourth matrix. The first loss is obtained based on the distance between the third matrix and the fourth matrix.

7. The method according to claim 6, characterized in that, The method involves masking the row vectors corresponding to the first frequency band in the first matrix based on the frequency range of the first frequency band and the pitch of each first signal frame in the first audio signal to obtain a second matrix, and masking the row vectors corresponding to the first frequency band in the second matrix based on the frequency range of the first frequency band and the pitch of each first signal frame in the second audio signal to obtain a fourth matrix, comprising: If the upper limit frequency of the first frequency band is less than the first frequency threshold or the lower limit frequency is greater than the second frequency threshold, then each element in the row vector corresponding to the first frequency band in the first matrix is ​​masked to obtain the third matrix, and each element in the row vector corresponding to the first frequency band in the second matrix is ​​masked to obtain the fourth matrix. If the upper limit frequency of the first frequency band is greater than or equal to the first frequency threshold, and the lower limit frequency of the first frequency band is less than or equal to the second frequency threshold, then a third signal frame with a pitch less than the pitch threshold is determined from the first signal frame of the first audio signal, and the first element in the row vector corresponding to the first frequency band in the first matrix is ​​masked to obtain a third matrix, where the first element represents the phase change of the third signal frame in the first frequency band; and a fourth signal frame with a pitch less than the pitch threshold is determined from the first signal frame of the second audio signal, and the second element in the row vector corresponding to the first frequency band in the second matrix is ​​masked to obtain a fourth matrix, where the second element represents the phase change of the fourth signal frame in the first frequency band.

8. A speech synthesis method, characterized in that, include: Get the second text; The second text is processed by a model to synthesize speech, thereby obtaining synthesized speech; wherein the model is trained using the training method of any one of claims 1 to 7.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the training method of the model as described in any one of claims 1 to 7 or the speech synthesis method as described in claim 8.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method of the model as described in any one of claims 1 to 7 or the speech synthesis method as described in claim 8.

Images