Speech speed adjusting method and system, computer device and storage medium

Abstract

The application discloses a speech speed adjusting method and system, computer equipment and a storage medium, relates to the technical field of information processing, and comprises the following steps: collecting a plurality of to-be-synthesized texts during speech synthesis, converting the to-be-synthesized texts into corresponding initial-final sequences, and recording the artificial marking pronunciation duration of the initial-final sequences obtained by an artificial marking method; vectorizing the initial-final sequences into phoneme vectors, and after the phoneme vectors pass through n layers of FFT Block and one-dimensional convolution, a predicted duration of the initial-final sequences is predicted; a difference between the predicted duration and the artificial marking pronunciation duration is calculated by using an L1 loss function, so as to train an initial-final duration prediction model during speech synthesis; and a target to-be-synthesized text is input into the initial-final duration prediction model, and corresponding target speech after speech speed adjustment is output according to a prediction result. Through the method of the application, speech speed adjustment of different speech during speech synthesis can be realized, and speech speed diversity can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, in particular to a speech speed adjusting method and system, computer equipment and non-volatile computer readable storage medium. BACKGROUND

[0002] Human natural speech is full of diversity, such as gender, timbre and speech speed, etc. At present, speech synthesis models aim to synthesize human-like speech, which has reached the level of human in naturalness and intelligibility, but still lacks diversity in terms of speech speed. For example, traditional models often model the duration of Chinese consonant and vowel pronunciation with a multi-dimensional Gaussian distribution, and the prediction result is often the mean of the multi-dimensional Gaussian distribution, which lacks diversity. Moreover, such methods are difficult to represent the speech speed of different speakers and cannot realize the mutual switching of different speech speeds. At present, multi-layer convolutional neural networks can be used to model the speech speed of different speakers in the training set, but still lack diversity and cannot model the speech speed of speakers outside the training set.

[0003] Therefore, how to provide a speech speed adjusting method and system, computer equipment and non-volatile computer readable storage medium, which can realize the adjustment of the speech speed of different speech during speech synthesis and achieve speech speed diversity, is a problem that needs to be solved by those skilled in the art. SUMMARY

[0004] In view of the above shortcomings of the prior art, the purpose of the present application is to provide a speech speed adjusting method and system, computer equipment and non-volatile computer readable storage medium which can be used in the field of financial technology or other related fields, can realize the adjustment of the speech speed of different speech during speech synthesis, and can realize speech speed diversity.

[0005] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0006] A speech speed adjusting method, comprising:

[0007] Collecting a plurality of texts to be synthesized during speech synthesis, converting the texts to be synthesized into corresponding consonant and vowel sequences, and recording the manually marked pronunciation duration of the consonant and vowel sequences obtained by manual marking;

[0008] Vectorizing the consonant and vowel sequences into phoneme vectors, and predicting the predicted duration of the consonant and vowel sequences after the phoneme vectors pass through n layers of FFT Block and one-dimensional convolution;

[0009] Using an L1 loss function to calculate the difference between the predicted duration and the manually marked pronunciation duration, thereby training a consonant and vowel duration prediction model during speech synthesis;

[0010] Input the target to-be-synthesized text into the phoneme-vowel duration prediction model, and output the corresponding target speech after speed adjustment according to the prediction result.

[0011] In a further technical solution, the speech speed adjustment method, wherein the to-be-synthesized texts during speech synthesis are collected, the to-be-synthesized texts are converted into corresponding phoneme-vowel sequences, and the manually marked pronunciation durations of the phoneme-vowel sequences obtained through manual marking are recorded, includes:

[0012] Collecting to-be-synthesized texts during speech synthesis;

[0013] Converting the to-be-synthesized texts into corresponding phoneme-vowel sequences;

[0014] Recording the phoneme-vowel duration frames calculated according to the pronunciation start positions and pronunciation end positions of the phoneme-vowel sequences marked by manual marking;

[0015] Recording the manually marked pronunciation durations of the phoneme-vowel sequences calculated according to the phoneme-vowel duration frames.

[0016] In a further technical solution, the speech speed adjustment method, wherein the to-be-synthesized texts during speech synthesis are collected, the to-be-synthesized texts are converted into corresponding phoneme-vowel sequences, and the manually marked pronunciation durations of the phoneme-vowel sequences obtained through manual marking are recorded, further includes:

[0017] Calculating the mel spectrum corresponding to the to-be-synthesized texts.

[0018] In a further technical solution, the speech speed adjustment method, wherein the phoneme-vowel sequence is vectorized into a phoneme vector, and the phoneme vector is subjected to n-layer FFT Block and one-dimensional convolution to predict the predicted duration of the phoneme-vowel sequence, includes:

[0019] Vectorizing the phoneme-vowel sequence into a phoneme vector, and obtaining a speech prompt after the mel spectrum is subjected to a preset promptencoder processing;

[0020] The phoneme vector and the speech prompt are subjected to n-layer FFT Block and one-dimensional convolution to predict the predicted duration of the phoneme-vowel sequence.

[0021] In a further technical solution, the speech speed adjustment method, wherein the mel spectrum corresponding to the to-be-synthesized texts is calculated, includes:

[0022] Calculating the amplitude spectrum of the audio corresponding to the to-be-synthesized texts through Fourier transform;

[0023] convert the amplitude spectrum into a corresponding mel spectrum by using a mel filter bank.

[0024] In a further technical solution, the speech rate adjustment method, wherein the amplitude spectrum of the audio corresponding to the text to be synthesized is calculated by Fourier transform, wherein the Fourier transform frame length is 50 ms and the frame shift is 25 ms.

[0025] In a further technical solution, the speech rate adjustment method, wherein the amplitude spectrum is converted into a corresponding mel spectrum by using a mel filter bank, wherein the mel filter bank is an 80-dimensional mel filter bank.

[0026] A speech rate adjustment system, comprising:

[0027] A conversion module is configured to collect a plurality of texts to be synthesized during speech synthesis, convert the texts to be synthesized into corresponding initial-final sequences, and record artificial marking pronunciation durations of the initial-final sequences obtained by artificial marking;

[0028] A vectorization module is configured to vectorize the initial-final sequences into phoneme vectors, and predict a predicted duration of the initial-final sequences after the phoneme vectors pass through n layers of FFT Block and one-dimensional convolution.

[0029] A model generation module is configured to calculate a difference between the predicted duration and the artificial marking pronunciation duration by using an L1 loss function, so as to train and generate an initial-final duration prediction model during speech synthesis.

[0030] A speech rate adjustment module is configured to input a target text to be synthesized into the initial-final duration prediction model, and output a corresponding target speech after speech rate adjustment according to a prediction result.

[0031] A computer device, comprising at least one processor; and

[0032] A memory in communication connection with the at least one processor; wherein

[0033] The memory has stored thereon a computer program executable by the at least one processor, and the computer program, when executed by the at least one processor, can implement the speech rate adjustment method according to any one of the preceding embodiments.

[0034] A non-volatile computer readable storage medium, wherein the non-volatile computer readable storage medium stores a computer program, and the computer program, when executed by at least one processor, can implement the speech rate adjustment method according to any one of the preceding embodiments.

[0035] Compared with the prior art, the present application provides a speech speed adjustment method, system, computer device and nonvolatile computer readable storage medium, wherein the method comprises: collecting a plurality of to-be-synthesized texts during speech synthesis, converting the to-be-synthesized texts into corresponding initial-final sequences, and recording artificial marking pronunciation durations of the initial-final sequences obtained by artificial marking; vectorizing the initial-final sequences into phoneme vectors, and after the phoneme vectors pass through n layers of FFTBlock and one-dimensional convolution, predicting a predicted duration of the initial-final sequences; calculating a difference between the predicted duration and the artificial marking pronunciation duration by using an L1 loss function, so as to train an initial-final duration prediction model during speech synthesis; inputting a target to-be-synthesized text into the initial-final duration prediction model, and outputting a corresponding target speech after speech speed adjustment according to a prediction result. In this way, the method of the present application can realize speech speed adjustment for different speeches during speech synthesis, and can realize speech speed diversity. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0037] Figure 1 A flowchart of a speech speed adjustment method provided by an embodiment of the present application.

[0038] Figure 2 A network structure diagram of an initial-final duration prediction model provided by an embodiment of the present application.

[0039] Figure 3 A network structure diagram of an FFT Block provided by an embodiment of the present application.

[0040] Figure 4 A network structure diagram of a prompt encoder provided by an embodiment of the present application.

[0041] Figure 5 A functional module diagram of a speech speed adjustment system provided by an embodiment of the present application.

[0042] Figure 6 A hardware structure diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and effects of the present application clearer, more explicit and more comprehensible, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and are not used to limit the present application.

[0044] In the description of the present application, "comprising", "including", "having", "containing" and the like are open-ended terms that mean including but not limited to. The description of the terms "one embodiment", "one specific embodiment", "some embodiments", "for example" and the like means that the specific features, structures or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. The order of steps involved in each embodiment is used to illustrate the implementation of the present application, and the order of steps is not limited and can be adjusted as needed.

[0045] The various non-limiting embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0046] Human natural speech is full of diversity, such as gender, timbre and speech rate, etc. At present, speech synthesis models aim to synthesize human-like speech and have reached the level of human in terms of naturalness and intelligibility, but still lack diversity in terms of speech rate. For example, in the modeling of the duration of Chinese consonant and vowel pronunciation, traditional models often use a multi-dimensional Gaussian distribution, and the prediction result is often the mean of the multi-dimensional Gaussian distribution, which lacks diversity. Moreover, such methods are difficult to represent the speech rate of different speakers and cannot realize the mutual switching of different speech rates. At present, multi-layer convolutional neural networks can be used to model the speech rate of different speakers in the training set, but still lack diversity and cannot model the speech rate of speakers outside the training set.

[0047] Therefore, how to provide a speech rate adjustment method, system, computer device and non-volatile computer readable storage medium, which can realize the adjustment of the speech rate of different speech during speech synthesis to realize speech rate diversity, is a problem that needs to be solved by those skilled in the art at present.

[0048] Therefore, in order to solve the above problems, with reference to Figure 1 The embodiment of the present application provides a speech rate adjustment method, wherein the method comprises the following steps:

[0049] S100, a plurality of texts to be synthesized during speech synthesis are collected, the texts to be synthesized are converted into corresponding consonant and vowel sequences, and the manually marked pronunciation duration of the consonant and vowel sequences obtained by manual marking is recorded;

[0050] S200, vectorize the consonant-vowel sequence into a phoneme vector, and after the phoneme vector passes through n layers of FFTBlock and one-dimensional convolution, the predicted duration of the consonant-vowel sequence is predicted;

[0051] S300, calculate the difference between the predicted duration and the artificially marked pronunciation duration by using an L1 loss function, so as to train a consonant-vowel duration prediction model for speech synthesis;

[0052] S400, input the target text to be synthesized into the consonant-vowel duration prediction model, and output the corresponding target speech after speed adjustment according to the prediction result.

[0053] Further, the speech speed adjustment method, wherein the step S100, collecting a plurality of texts to be synthesized during speech synthesis, converting the text to be synthesized into a corresponding consonant-vowel sequence, and recording the artificially marked pronunciation duration of the consonant-vowel sequence obtained by artificial marking, comprises:

[0054] Collecting a plurality of texts to be synthesized during speech synthesis;

[0055] Converting the text to be synthesized into a corresponding consonant-vowel sequence;

[0056] Recording the consonant-vowel duration frame number calculated by marking the pronunciation start position and the pronunciation end position of the consonant-vowel sequence;

[0057] Recording the artificially marked pronunciation duration of the consonant-vowel sequence corresponding to the consonant-vowel duration frame number.

[0058] Further, the speech speed adjustment method, wherein the step S100, collecting a plurality of texts to be synthesized during speech synthesis, converting the text to be synthesized into a corresponding consonant-vowel sequence, and recording the artificially marked pronunciation duration of the consonant-vowel sequence obtained by artificial marking, further comprises:

[0059] Calculating the mel spectrum corresponding to the text to be synthesized by calculation.

[0060] Further, the speech speed adjustment method, wherein the mel spectrum corresponding to the text to be synthesized is calculated by calculation, comprising:

[0061] Calculating the amplitude spectrum of the audio corresponding to the text to be synthesized by Fourier transform;

[0062] Converting the amplitude spectrum into a corresponding mel spectrum by using a mel filter bank.

[0063] Further, the speech speed adjustment method, wherein the amplitude spectrum corresponding to the text to be synthesized is calculated by Fourier transform, wherein the Fourier transform frame length is 50 ms, and the frame shift is 25 ms.

[0064] Further, the speech speed adjustment method, wherein the amplitude spectrum is converted into a corresponding mel spectrum by using a mel filter bank, wherein the mel filter bank is an 80-dimensional mel filter bank.

[0065] In specific implementation, in the embodiment, a large amount of text to be synthesized during speech synthesis is collected, and the text to be synthesized is converted into a corresponding sequence of initials and finals, denoted as C=C1:T text , T text denotes the length of the sequence of initials and finals.

[0066] Then, the mel spectrum corresponding to the text to be synthesized is calculated, and the specific process is as follows: the amplitude spectrum corresponding to the text to be synthesized is calculated by Fourier transform, and the amplitude spectrum is converted into a corresponding mel spectrum by using an 80-dimensional mel filter bank, denoted as X=X1:T text , T text denotes the number of mel spectrum frames; wherein the Fourier transform frame length is 50 ms, and the frame shift is 25 ms.

[0067] Then, the number of frames of the pronunciation of the sequence of initials and finals calculated by marking the start position and the end position of the pronunciation of the sequence of initials and finals by hand is recorded, and the artificial marking pronunciation duration corresponding to the sequence of initials and finals calculated according to the number of frames of the pronunciation of the sequence of initials and finals is recorded; for example, for an initial and final C i , the start position of the pronunciation of the initial and final is marked as the jth frame to the j+T i frame, and T i is the duration of the initial and final C i , so the artificial marking pronunciation duration corresponding to the initial and final C i is T i *25 ms.

[0068] Further, the speech speed adjustment method, wherein the step S200, the sequence of initials and finals is vectorized into a phoneme vector, and the phoneme vector is processed by n layers of FFTBlock and one-dimensional convolution to predict the predicted duration of the sequence of initials and finals, including:

[0069] The sequence of initials and finals is vectorized into a phoneme vector, and the mel spectrum is processed by a preset promptencoder to obtain a speech prompt;

[0070] The phoneme vector and the speech prompt are simultaneously processed by n layers of FFTBlock and one-dimensional convolution to predict the predicted duration of the sequence of initials and finals.

[0071] In practice, as shown in Figure 2 , the consonant-vowel sequence C = C1:T text is vectorized into a phoneme vector H = h1:T text , and the mel-spectrogram X = X1:T text is processed by a preset prompt encoder to obtain a voice prompt Y; wherein the structure of the prompt encoder is as shown in Figure 4 ;

[0072] Then, the phoneme vector and the voice prompt are simultaneously processed by n layers of FFTBlock and one-dimensional convolution to predict the predicted duration of the consonant-vowel sequence; wherein the structure of the FFTBlock is as shown in Figure 3 , which consists of multi-head attention, one-dimensional convolution, and Layer Normalization (LN);

[0073] In this embodiment, in order to model the diversity of natural human speech, a voice prompt mechanism is designed to constrain the speaker information in the voice prompt for the consonant-vowel duration prediction model, that is, the mel-spectrogram X is calculated by a preset prompt encoder to obtain a voice prompt Y. In order to constrain the voice prompt Y for the consonant-vowel duration prediction model, the voice prompt Y is input into the FFTBlock as the Keys and Values in the multi-head attention mechanism Attention. The calculation formula of Attention is generally as follows:

[0074]

[0075] The multi-head attention mechanism in the FFTBlock is calculated as follows:

[0076]

[0077] wherein Q is the input of each layer of FFTBlock, and d is a coefficient related to the feature dimension of Q.

[0078] In this way, when the consonant-vowel duration prediction model is predicting, the same text can be input, and different consonant-vowel pronunciation durations can be predicted according to different voice prompts.

[0079] Further, the speech speed adjustment method, wherein the step S300 calculates the difference between the predicted duration and the artificially labeled pronunciation duration using an L1 loss function to train a consonant-vowel duration prediction model for speech synthesis;

[0080] Further, the speech rate adjustment method, wherein the step S400, the target text to be synthesized is input into the initial consonant-vowel-consonant length prediction model, and a corresponding target speech after speech rate adjustment is output according to a prediction result.

[0081] In a specific implementation, in this embodiment, the difference between the predicted length and the artificially marked pronunciation length is calculated by using the L1 loss function to train the initial consonant-vowel-consonant length prediction model during speech synthesis; then, the target text to be synthesized is input into the initial consonant-vowel-consonant length prediction model, and the initial consonant-vowel-consonant pronunciation length corresponding to the target text to be synthesized is dynamically adjusted according to a prediction result, so as to model and output a corresponding target speech after speech rate adjustment.

[0082] As can be known from the above method embodiments, the speech rate adjustment method provided by the present application at least includes: collecting a plurality of texts to be synthesized during speech synthesis, converting the texts to be synthesized into corresponding initial consonant-vowel-consonant sequences, and recording artificially marked pronunciation lengths of the initial consonant-vowel-consonant sequences obtained by an artificial marking method; vectorizing the initial consonant-vowel-consonant sequences into phoneme vectors, and after the phoneme vectors pass through n layers of FFT Block and one-dimensional convolution, predicting a predicted length of the initial consonant-vowel-consonant sequence; calculating the difference between the predicted length and the artificially marked pronunciation length by using the L1 loss function, so as to train an initial consonant-vowel-consonant length prediction model during speech synthesis; inputting a target text to be synthesized into the initial consonant-vowel-consonant length prediction model, and outputting a corresponding target speech after speech rate adjustment according to a prediction result; and the like. In this way, the method of the present application can realize speech rate adjustment for different speeches during speech synthesis, and can realize speech rate diversity.

[0083] That is, in the speech rate adjustment method provided by the present application, in order to enable speech synthesis to model the diverse speech rates of human natural speech, the present application proposes an initial consonant-vowel-consonant length prediction model suitable for speech synthesis. The model designs a speech prompt mechanism, calculates a speech prompt Y through a prompt encoder module, Y is used as Keys and Values in a multi-head attention mechanism, provides speaker-related information, and constrains the prediction of initial consonant-vowel-consonant length. The initial consonant-vowel-consonant length prediction model can adjust the initial consonant-vowel-consonant prediction length according to different prompt speeches during prediction, better models different speaker speech rates, and achieves the purpose of speech rate diversity, that is, can adjust the speech rate according to actual needs.

[0084] It should be understood that although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps can be included based on conventional or non-inventive labor, and the operation steps are not necessarily executed in the order of the embodiments or flowcharts. The order of steps listed in the embodiments or flowcharts is only one of the many execution orders, and does not represent the only execution order. It should be noted that there is no certain sequence between the above steps, and those skilled in the art can understand from the description of the embodiments of the present application that the above steps can have different execution orders in different embodiments, that is, they can be executed in parallel, or they can be exchanged and executed, etc. Moreover, at least part of the steps in the embodiments or flowcharts can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation, alternation or synchronization with other steps or sub-steps or stages of other steps.

[0085] Based on the above method embodiments, please refer to Figure 5 Another embodiment of the present application also provides a speech rate adjusting system, wherein the system comprises:

[0086] A conversion module 11 is configured to collect a plurality of to-be-synthesized texts in speech synthesis, convert the to-be-synthesized texts into corresponding initial-final sequences, and record artificial marking pronunciation durations of the initial-final sequences obtained by an artificial marking method;

[0087] A vectorization module 12 is configured to vectorize the initial-final sequences into phoneme vectors, and predict a predicted duration of the initial-final sequences after the phoneme vectors pass through n layers of FFT Block and one-dimensional convolution;

[0088] A model generation module 13 is configured to calculate a difference between the predicted duration and the artificial marking pronunciation duration by using an L1 loss function, so as to train and generate an initial-final duration prediction model in speech synthesis;

[0089] A speech rate adjusting module 14 is configured to input a target to-be-synthesized text into the initial-final duration prediction model, and output a corresponding target speech after speech rate adjustment according to a prediction result.

[0090] Further, the speech rate adjusting system, wherein the collection of a plurality of to-be-synthesized texts in speech synthesis, the conversion of the to-be-synthesized texts into corresponding initial-final sequences, and the recording of artificial marking pronunciation durations of the initial-final sequences obtained by an artificial marking method comprise:

[0091] Collecting a plurality of to-be-synthesized texts in speech synthesis;

[0092] convert the text to be synthesized into a corresponding initial consonant-vowel sequence;

[0093] record the initial consonant-vowel duration frame number calculated by marking the initial and end positions of pronunciation of the initial consonant-vowel sequence by hand;

[0094] record the artificial marking pronunciation duration of the initial consonant-vowel sequence calculated according to the initial consonant-vowel duration frame number.

[0095] Further, the speech speed adjustment system, wherein the collection of a large number of texts to be synthesized during speech synthesis, the conversion of the text to be synthesized into a corresponding initial consonant-vowel sequence, and the recording of the artificial marking pronunciation duration of the initial consonant-vowel sequence obtained by artificial marking further comprise:

[0096] calculate the mel spectrum corresponding to the text to be synthesized by calculation.

[0097] Further, the speech speed adjustment system, wherein the calculation of the mel spectrum corresponding to the text to be synthesized by calculation comprises:

[0098] calculate the amplitude spectrum of the audio corresponding to the text to be synthesized by Fourier transform;

[0099] convert the amplitude spectrum into a corresponding mel spectrum by using a mel filter bank.

[0100] Further, the speech speed adjustment system, wherein the calculation of the amplitude spectrum of the audio corresponding to the text to be synthesized by Fourier transform, wherein the Fourier transform frame length is 50 ms and the frame shift is 25 ms.

[0101] Further, the speech speed adjustment system, wherein the conversion of the amplitude spectrum into a corresponding mel spectrum by using a mel filter bank, wherein the mel filter bank is an 80-dimensional mel filter bank.

[0102] In specific implementation, in the embodiment, a large number of texts to be synthesized during speech synthesis are collected, and the text to be synthesized is converted into a corresponding initial consonant-vowel sequence, denoted as C=C1:T text , T text denotes the length of the initial consonant-vowel sequence;

[0103] Then, the mel spectrum corresponding to the text to be synthesized is calculated by calculation, and the specific process is: the amplitude spectrum of the audio corresponding to the text to be synthesized is calculated by Fourier transform, and the amplitude spectrum is converted into a corresponding mel spectrum by using an 80-dimensional mel filter bank, denoted as X=X1:T text , T text denotes the mel spectrum frame number; wherein the Fourier transform frame length is 50 ms and the frame shift is 25 ms;

[0104] Then, the duration of the syllable-rime sequence corresponding to the manually marked pronunciation time length is recorded according to the duration of the syllable-rime sequence calculated according to the duration of the syllable-rime sequence. Specifically, for the syllable-rime C i The manually marked pronunciation time length corresponding to the syllable-rime C i is T i frames, and T i is the duration of the syllable-rime C i , then the manually marked pronunciation time length corresponding to the syllable-rime C i is T text *25 ms.

[0105] Further, the speech rate adjusting system, wherein the syllable-rime sequence is vectorized into a phoneme vector, and the phoneme vector is processed by n layers of FFT Block and one-dimensional convolution to predict the predicted duration of the syllable-rime sequence, comprising:

[0106] The syllable-rime sequence is vectorized into a phoneme vector, and the mel spectrum is processed by a preset prompt encoder to obtain a voice prompt;

[0107] The phoneme vector and the voice prompt are simultaneously processed by n layers of FFT Block and one-dimensional convolution to predict the predicted duration of the syllable-rime sequence.

[0108] In specific implementation, as shown in Figure 2 , the syllable-rime sequence C=C1:T text is vectorized into a phoneme vector H=h1:T text , and the mel spectrum X=X1:T text is processed by a preset prompt encoder to obtain a voice prompt Y; wherein the structure of the prompt encoder is as shown in Figure 4 ;

[0109] Then, the phoneme vector and the voice prompt are simultaneously processed by n layers of FFT Block and one-dimensional convolution to predict the predicted duration of the syllable-rime sequence; wherein the structure of the FFT Block is as shown in Figure 3 , which is composed of multi-head attention, one-dimensional convolution and Layer Normalization (LN, layer normalization);

[0110] Wherein, in the embodiment, in order to model the diversity of human natural speech, a speech prompt mechanism is designed to constrain the speaker information in the speech prompt that the phoneme duration prediction model follows, that is, the mel spectrum X is calculated by the preset prompt encoder to obtain the speech prompt Y, in order to constrain the phoneme duration prediction model by the speech prompt Y, the speech prompt Y is input into the FFTBlock as the Keys and Values in the multi-head attention mechanism Attention; the Attention calculation formula is generally:

[0111]

[0112] The multi-head attention mechanism in the FFTBlock is calculated as:

[0113]

[0114] Wherein, Q is the input of each layer of FFTBlock, and d is a coefficient related to the feature dimension of Q;

[0115] In this way, when the phoneme duration prediction model is predicted, the same text can be input, and different phoneme pronunciation durations can be predicted according to different prompt speeches.

[0116] Further, the speech speed adjustment system, wherein the difference between the predicted duration and the artificially marked pronunciation duration is calculated by using the L1 loss function to train the phoneme duration prediction model during speech synthesis;

[0117] Further, the speech speed adjustment system, wherein the target text to be synthesized is input into the phoneme duration prediction model, and the corresponding target speech after speed adjustment is output according to the prediction result;

[0118] In specific implementation, in the embodiment, the difference between the predicted duration and the artificially marked pronunciation duration is calculated by using the L1 loss function to train the phoneme duration prediction model during speech synthesis; then, the target text to be synthesized is input into the phoneme duration prediction model, and the corresponding target speech after speed adjustment is output according to the prediction result.

[0119] From the above system embodiments, the speech speed adjusting system provided by the application at least includes: collecting a plurality of to-be-synthesized texts during speech synthesis, converting the to-be-synthesized texts into corresponding initial-final sequences, and recording artificial marking pronunciation durations of the initial-final sequences obtained by an artificial marking manner; vectorizing the initial-final sequences into phoneme vectors, and after the phoneme vectors pass through n layers of FFT Block and one-dimensional convolution, a predicted duration of the initial-final sequences is predicted; a difference between the predicted duration and the artificial marking pronunciation duration is calculated by using an L1 loss function, so as to train an initial-final duration prediction model during speech synthesis; a target to-be-synthesized text is input into the initial-final duration prediction model, and a corresponding target speech after speech speed adjustment is output according to a prediction result; and the like. In this way, the system of the application can realize speech speed adjustment of different speeches during speech synthesis, and speech speed diversity can be realized.

[0120] That is, in the speech speed adjusting system provided by the application, in order to enable speech synthesis to model the diverse speech speed of human natural speech, the application proposes an initial-final duration prediction model suitable for speech synthesis, the model designs a speech prompt mechanism, calculates a speech prompt Y through a prompt encoder module, Y is used as Keys and Values in a multi-head attention mechanism, provides speaker-related information, and constrains the prediction of initial-final duration. The initial-final duration prediction model can adjust the initial-final prediction duration according to different prompt speeches during prediction, better model different speaker speeds, and achieve the purpose of speech speed diversity, that is, the speech speed can be adjusted according to actual needs.

[0121] Based on the above method embodiments, please refer to Figure 6 The application further provides a computer device, wherein the computer device 10 includes:

[0122] a memory 120 and one or more processors 110, Figure 6 In the foregoing, the processor 110 and the memory 120 can be connected through a communication bus or other means, Figure 6 In the foregoing, the connection through the communication bus is taken as an example.

[0123] The processor 110 is configured to implement various control logic of the computer device 10, and can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a single-chip computer, an ARM (Acorn RISC Machine), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. In addition, the processor 110 can also be any conventional processor, microprocessor, or state machine. The processor 110 can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, or any other such configuration.

[0124] The memory 120 is a non-volatile computer-readable storage medium configured to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the computer program corresponding to the speech speed adjustment method in the embodiments of the present application. The processor 110 executes various functions and data processing of the computer device 10 by running the non-volatile software programs, instructions, and units stored in the memory 120, i.e., implements the speech speed adjustment method in the above-mentioned method embodiments.

[0125] The memory 120 can include a program storage area and a data storage area, wherein the program storage area can store application programs required by the operation device and at least one function; and the data storage area can store data created according to the use of the computer device 10, etc. In addition, the memory 120 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 120 can optionally include a memory remotely disposed relative to the processor 110, and these remote memories can be connected to the computer device 10 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0126] The computer device 10, wherein one or more units are stored in the memory 120 and when executed by the one or more processors 110, can implement the following steps:

[0127] S100, collecting a plurality of to-be-synthesized texts during speech synthesis, converting the to-be-synthesized texts into corresponding initial-final sequences, and recording artificial marking pronunciation durations of the initial-final sequences obtained by an artificial marking manner;

[0128] S200, vectorizing the initial-final sequence into a phoneme vector, and predicting a predicted duration of the initial-final sequence after the phoneme vector is subjected to n-layer FFTBlock and one-dimensional convolution;

[0129] S300, calculate the difference between the predicted duration and the artificially marked pronunciation duration by using an L1 loss function, so as to train a phoneme duration prediction model in speech synthesis;

[0130] S400, input the target text to be synthesized into the phoneme duration prediction model, and output the corresponding target speech after speed adjustment according to the prediction result.

[0131] Further, the computer device, wherein the step S100, collecting a plurality of texts to be synthesized in speech synthesis, converting the text to be synthesized into a corresponding phoneme sequence, and recording the artificially marked pronunciation duration of the phoneme sequence obtained by artificial marking, comprises:

[0132] Collecting a plurality of texts to be synthesized in speech synthesis;

[0133] Converting the text to be synthesized into a corresponding phoneme sequence;

[0134] Recording the phoneme duration calculated by marking the starting position and ending position of the phoneme sequence;

[0135] Recording the artificially marked pronunciation duration of the phoneme sequence calculated according to the phoneme duration.

[0136] Further, the computer device, wherein the step S100, collecting a plurality of texts to be synthesized in speech synthesis, converting the text to be synthesized into a corresponding phoneme sequence, and recording the artificially marked pronunciation duration of the phoneme sequence obtained by artificial marking, further comprises:

[0137] Calculating the mel spectrum corresponding to the text to be synthesized by calculation.

[0138] Further, the computer device, wherein the mel spectrum corresponding to the text to be synthesized is calculated by calculation, comprising:

[0139] Calculating the amplitude spectrum of the audio corresponding to the text to be synthesized by Fourier transform;

[0140] Converting the amplitude spectrum into a corresponding mel spectrum by using a mel filter bank.

[0141] Further, the computer device, wherein the amplitude spectrum of the audio corresponding to the text to be synthesized is calculated by Fourier transform, wherein the Fourier transform frame length is 50ms and the frame shift is 25ms.

[0142] Further, the computer device, wherein the amplitude spectrum is converted into a corresponding mel spectrum by using a mel filter bank, wherein the mel filter bank is an 80-dimensional mel filter bank.

[0143] In specific implementation, in the embodiment, a large amount of text to be synthesized during speech synthesis is collected, and the text to be synthesized is converted into a corresponding initial-final sequence, denoted as C=C1:T text , T text denotes the length of the initial-final sequence;

[0144] Then, the mel spectrum corresponding to the text to be synthesized is calculated through calculation, and the specific process is as follows: the audio corresponding to the text to be synthesized is calculated through Fourier transform to obtain an amplitude spectrum, and the amplitude spectrum is converted into a corresponding mel spectrum by using an 80-dimensional mel filter bank, denoted as X=X1:T text , T text denotes the number of mel spectrum frames; wherein the Fourier transform frame length is 50 ms, and the frame shift is 25 ms;

[0145] Then, the initial-final duration frames calculated by marking the initial position and the end position of pronunciation of the initial-final sequence by hand are recorded, and the artificial marking pronunciation duration corresponding to the initial-final sequence calculated according to the initial-final duration frames is recorded; for example, for the initial-final C i , the initial position of the artificial marking is the jth frame to the j+T i frame, and T i is the duration of the initial-final C i , the artificial marking pronunciation duration corresponding to the initial-final C i is T i *25 ms.

[0146] Further, the computer device, wherein the step S200, the initial-final sequence is vectorized into a phoneme vector, and after the phoneme vector is subjected to n-layer FFTBlock and one-dimensional convolution, the predicted duration of the initial-final sequence is predicted, comprising:

[0147] The initial-final sequence is vectorized into a phoneme vector, and the mel spectrum is subjected to a preset promptencoder to obtain a speech prompt;

[0148] After the phoneme vector and the speech prompt are subjected to n-layer FFTBlock and one-dimensional convolution at the same time, the predicted duration of the initial-final sequence is predicted.

[0149] In specific implementation, in the embodiment, as shown in Figure 2 , the initial-final sequence C=C1:T text is vectorized into a phoneme vector H=h1:T text , and the mel spectrum X=X1:T textAfter the preset prompt encoder processing, a voice prompt Y is obtained; wherein, the structure of promptencoder is as shown in Figure 4

[0150] Then, the phoneme vector and the voice prompt are simultaneously subjected to n-layer FFTBlock and one-dimensional convolution, and the predicted duration of the initial consonant-vowel sequence is predicted; wherein, the structure of FFTBlock is as shown in Figure 3

[0151] In the embodiment, in order to model the diversity of human natural speech, a voice prompt mechanism is designed to constrain the initial consonant-vowel duration prediction model to follow the speaker information in the voice prompt, that is, the mel spectrum X is subjected to preset promptencoder to obtain the voice prompt Y, in order to constrain the initial consonant-vowel duration prediction model by the voice prompt Y, the voice prompt Y is input into the FFTBlock as the Keys and Values in the multi-head attention mechanism Attention; the calculation formula of Attention is generally as follows:

[0152]

[0153] The multi-head attention mechanism in FFTBlock is calculated as follows:

[0154]

[0155] Wherein, Q is the input of each layer of FFTBlock, and d is a coefficient related to the feature dimension of Q.

[0156] In this way, when the initial consonant-vowel duration prediction model is predicted, the same text can be input, and different initial consonant-vowel pronunciation durations can be predicted according to different voice prompts.

[0157] Further, the computer device, wherein the step S300, the difference between the predicted duration and the artificially marked pronunciation duration is calculated by using the L1 loss function, so as to train the initial consonant-vowel duration prediction model during voice synthesis;

[0158] Further, the computer device, wherein the step S400, the target text to be synthesized is input into the initial consonant-vowel duration prediction model, and the corresponding target voice after speed adjustment is output according to the prediction result;

[0159] ​​In the embodiment, the difference between the predicted duration and the artificially marked pronunciation duration is calculated by using the L1 loss function to train the phoneme duration prediction model during speech synthesis.

[0160] As can be seen from the above computer device embodiment, the computer device provided by the application can at least achieve the following: collecting a plurality of texts to be synthesized during speech synthesis, converting the texts to be synthesized into corresponding phoneme sequences, and recording artificially marked pronunciation durations of the phoneme sequences obtained by artificial marking; vectorizing the phoneme sequences into phoneme vectors, and predicting the predicted duration of the phoneme sequences after the phoneme vectors pass through n layers of FFT Block and one-dimensional convolution; using the L1 loss function to calculate the difference between the predicted duration and the artificially marked pronunciation duration to train the phoneme duration prediction model during speech synthesis; inputting a target text to be synthesized into the phoneme duration prediction model, and outputting the target speech after speed adjustment according to the prediction result; and the like. In this way, the computer device of the application can achieve speed adjustment of different speeches during speech synthesis, and can achieve speech speed diversity.

[0161] That is, in the computer device provided by the application, in order to enable speech synthesis to model the diverse speech speed of human natural speech, the application proposes a phoneme duration prediction model suitable for speech synthesis. The model designs a speech prompt mechanism, calculates a speech prompt Y through a prompt encoder module, Y is used as Keys and Values in the multi-head attention mechanism, provides speaker-related information, and constrains the prediction of phoneme duration. The phoneme duration prediction model can adjust the phoneme prediction duration according to different prompt speeches during prediction, better model different speaker speeds, and achieve the purpose of speech speed diversity, that is, the speed can be adjusted according to actual needs.

[0162] Those skilled in the art can understand that, Figure 6 The hardware structure diagram shown in the above embodiment is only a schematic diagram of part of the structure related to the application scheme, and does not constitute a limitation on the computer device to which the application scheme is applied. The specific computer device can include more components than those shown in the figure, or combine some components, or have a different component arrangement.

[0163] Based on the above method embodiments, another embodiment of the present application further provides a non-volatile computer readable storage medium, wherein the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by at least one processor to implement the speech speed adjusting method in any one of the above method embodiments, for example, to implement the method steps S100 to S400 in the above description. Figure 1

[0164] By way of example, non-volatile storage can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), as external cache memory. By way of illustration, RAM can be available at various speeds such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synch link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memory of the operational environment described herein are intended to encompass one or more of these and / or any other suitable types of memory.

[0165] Based on the above method embodiments, another embodiment of the present application further provides a computer program product, the computer program product comprising a computer program stored on a non-volatile computer readable storage medium, the computer program comprising program instructions which, when executed by a processor, implement the speech speed adjusting method in any one of the above method embodiments, for example, implement the method steps S100 to S400 in the above description. Figure 1

[0166] The above-described embodiments are merely illustrative for the present application, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0167] ​​Those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can be implemented by hardware. Based on such an understanding, the technical solutions described above essentially or in other words the part that contributes to the related art can be embodied in the form of a software product, and the computer software product can exist in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of each embodiment or some parts of the embodiments.

[0168] Conditional language such as, among others, "can," "could," "might" or "may," unless specifically stated otherwise, generally are intended to convey that a certain feature, element or process can or can not be included in some implementations. Thus, such conditional language is not generally intended to imply that a feature, element or process is required in one or more implementations or that a feature, element or process is necessary or indispensable for one or more implementations. The singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0169] What has been described herein is including the examples of a speech rate adjustment method, system, computer device and non-volatile computer readable storage medium that can provide. Of course, every conceivable combination of elements and / or methods that are described for the purpose of describing various features of the present disclosure cannot be described, but it can be recognized that many additional combinations and permutations of the disclosed features are possible. It will be apparent, therefore, that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure, but all such various modifications are intended to fall within the scope of the appended claims of the present application. Further, or in the alternative, other embodiments of the present disclosure can be apparent from consideration of the present specification and drawings, and practice of the present disclosure as presented herein. It is intended that the examples presented in the present specification and drawings be considered as illustrative in all respects, rather than limiting. Although specific terms are employed in the present disclosure, they are used in a generic and descriptive sense only and not for limitation purposes.

Claims

1. A speech rate adjustment method characterized by, The method comprises the following steps: Collecting a plurality of texts to be synthesized during speech synthesis, converting the texts to be synthesized into corresponding initial-final sequences, and recording artificial marking pronunciation durations of the initial-final sequences obtained by artificial marking; Vectorizing the initial-final sequence into a phoneme vector, and after the phoneme vector is subjected to n-layer FFT Block and one-dimensional convolution, predicting a predicted duration of the initial-final sequence; Using an L1 loss function to calculate the difference between the predicted duration and the artificial marking pronunciation duration, thereby training an initial-final duration prediction model for speech synthesis; Inputting a target text to be synthesized into the initial-final duration prediction model, and outputting a corresponding target speech after adjusting the speech speed according to the prediction result; The collecting a plurality of texts to be synthesized during speech synthesis, converting the texts to be synthesized into corresponding initial-final sequences, and recording artificial marking pronunciation durations of the initial-final sequences obtained by artificial marking comprises the following steps: Obtaining a mel spectrum corresponding to the text to be synthesized by calculation; The vectorizing the initial-final sequence into a phoneme vector, and after the phoneme vector is subjected to n-layer FFT Block and one-dimensional convolution, predicting a predicted duration of the initial-final sequence comprises the following steps: Vectorizing the initial-final sequence into a phoneme vector, and obtaining a speech prompt after the mel spectrum is subjected to a preset prompt encoder; After the phoneme vector and the speech prompt are subjected to n-layer FFT Block and one-dimensional convolution at the same time, a predicted duration of the initial-final sequence is predicted.

2. The speech rate adjustment method according to claim 1, characterized by, The collecting a plurality of texts to be synthesized during speech synthesis, converting the texts to be synthesized into corresponding initial-final sequences, and recording artificial marking pronunciation durations of the initial-final sequences obtained by artificial marking further comprises the following steps: Collecting a plurality of texts to be synthesized during speech synthesis; Converting the texts to be synthesized into corresponding initial-final sequences; Recording an initial-final duration calculated by artificially marking starting positions and ending positions of pronunciation of the initial-final sequence; Recording an artificial marking pronunciation duration corresponding to the initial-final sequence calculated according to the initial-final duration.

3. The speech rate adjustment method according to claim 2, characterized by, The obtaining a mel spectrum corresponding to the text to be synthesized by calculation comprises the following steps: Calculating an amplitude spectrum of audio corresponding to the text to be synthesized by Fourier transform; Converting the amplitude spectrum into a corresponding mel spectrum by using a mel filter bank.

4. The speech rate adjustment method according to claim 3, characterized by, The calculating an amplitude spectrum of audio corresponding to the text to be synthesized by Fourier transform, wherein the Fourier transform frame length is 50 ms, and the frame shift is 25 ms.

5. The speech rate adjustment method according to claim 4, characterized by, The converting the amplitude spectrum into a corresponding mel spectrum by using a mel filter bank, wherein the mel filter bank is an 80-dimensional mel filter bank.

6. A speech rate adjustment system characterized by, The method comprises the following steps: A conversion module is configured to collect a plurality of texts to be synthesized during speech synthesis, convert the texts to be synthesized into corresponding initial-final sequences, and record artificial marking pronunciation durations of the initial-final sequences obtained by artificial marking. vectorize the sequence of diphones into a phoneme vector, and after the phoneme vector is subjected to n-layer FFT Block and one-dimensional convolution, predict a predicted duration of the sequence of diphones; a model generation module configured to calculate a difference between the predicted duration and the artificially marked pronunciation duration by using an L1 loss function, so as to train a diphone duration prediction model in speech synthesis; a speech rate adjustment module configured to input target text to be synthesized into the diphone duration prediction model, and output corresponding target speech after speech rate adjustment according to a prediction result; the collection of a plurality of texts to be synthesized in speech synthesis, conversion of the texts to be synthesized into corresponding sequences of diphones, and recording of artificially marked pronunciation durations of the sequences of diphones obtained by artificial marking, comprising: obtaining a mel-spectrogram corresponding to the text to be synthesized by calculation; the vectorization of the sequence of diphones into a phoneme vector, and after the phoneme vector is subjected to n-layer FFT Block and one-dimensional convolution, prediction of a predicted duration of the sequence of diphones, comprising: vectorization of the sequence of diphones into a phoneme vector, and after the mel-spectrogram is subjected to a preset prompt encoder, obtaining of a speech prompt; after the phoneme vector and the speech prompt are subjected to n-layer FFT Block and one-dimensional convolution at the same time, prediction of a predicted duration of the sequence of diphones.

7. A computer device, comprising: the computer device comprises at least one processor; and a memory in communication connection with the at least one processor; wherein the memory has stored thereon a computer program executable by the at least one processor, and the computer program, when executed by the at least one processor, can implement the speech rate adjustment method according to any one of claims 1-5.

8. A non-transitory computer readable storage medium, comprising: the non-volatile computer readable storage medium has stored thereon a computer program, and the computer program, when executed by at least one processor, can implement the speech rate adjustment method according to any one of claims 1-5.

Images