Model training method, pronunciation duration prediction method, speech generation method and related devices

Abstract

The embodiment of the invention provides a model training method, a pronunciation duration prediction method, a voice generation method and related devices, and the method comprises the steps: generating a first duration sequence based on the pronunciation duration of each phoneme in text annotation data; the first duration sequence comprises a frame number corresponding to the pronunciation duration of each phoneme; adjusting the number of frames in the first duration sequence to obtain a second duration sequence; determining the second duration sequence and the text labeling data as sample data; and training the first model through the sample data to obtain a second model. According to the invention, the frame number in the first time length sequence is adjusted through the preset quantile, and the first model is trained based on the adjusted frame number sequence, so that the sound length prediction precision of the second model can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and involve but are not limited to a model training method, a pronunciation duration prediction method, a speech generation method and related devices. Background Art

[0002] With the development of artificial intelligence technology, text-based speech generation has become an important research topic. In related technologies, speech can be generated through non-autoregressive text-to-speech synthesis methods. When generating speech, if you want to generate more natural speech, you usually need to make a good sound length distribution prediction, that is, to accurately predict the pronunciation duration of each phoneme in the text token, so as to better align the predicted pronunciation duration with the speech token, and then combine the speech tokens with the aligned pronunciation duration to generate the final speech.

[0003] However, in the related art, during the training process of the text-to-speech synthesis model, the training data set used is large in size and the speech quality is uneven. The text-to-speech synthesis model trained with such a training data set has a low sound length prediction accuracy, resulting in low accuracy and naturalness of the final synthesized speech. Summary of the Invention

[0004] The embodiments of the present application provide a model training method, a pronunciation duration prediction method, a speech generation method and related devices, which can at least be applied in the field of artificial intelligence. By adjusting the number of frames in a first duration sequence and training the first model based on the adjusted frame sequence, the sound duration prediction accuracy of the trained second model can be improved.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a model training method, including: generating a first duration sequence based on the pronunciation duration of each phoneme in text annotation data; the first duration sequence includes the number of frames corresponding to the pronunciation duration of each phoneme; adjusting the number of frames in the first duration sequence to obtain a second duration sequence; determining the second duration sequence and the text annotation data as sample data; and training the first model using the sample data to obtain a second model.

[0007] An embodiment of the present application provides a model training device, including: a first generation module, used to generate a first time sequence based on the pronunciation duration of each phoneme in text annotation data; the first time sequence includes the number of frames corresponding to the pronunciation duration of each phoneme; an adjustment module, used to adjust the number of frames in the first time sequence to obtain a second time sequence; a determination module, used to determine the second time sequence and the text annotation data as a sample data training module, used to train the first model through the sample data to obtain a second model.

[0008] An embodiment of the present application provides a method for predicting pronunciation duration, comprising: performing a fourth encoding on second label data to obtain a fourth encoding value; performing a fifth encoding on multiple phonemes in a first text to obtain a fifth encoding value; generating an input sequence based on the fourth encoding value and the fifth encoding value; inputting the input sequence into a second model to obtain the pronunciation duration of each phoneme in the first text; wherein the second model is trained using the training method of the above-mentioned model.

[0009] An embodiment of the present application provides a pronunciation duration prediction device, including: a first encoding module, used to perform a fourth encoding on second label data to obtain a fourth encoding value; the first encoding module is also used to perform a fifth encoding on multiple phonemes in a first text to obtain a fifth encoding value; a second generation module is used to generate an input sequence based on the fourth encoding value and the fifth encoding value; a first prediction module is used to input the input sequence into a second model to obtain the pronunciation duration of each phoneme in the first text.

[0010] An embodiment of the present application provides a speech generation method, the method comprising: performing a sixth encoding on third label data to obtain a sixth encoding value; performing a seventh encoding on multiple phonemes in a second text to obtain a seventh encoding value; generating an input sequence based on the sixth encoding value and the seventh encoding value; inputting the input sequence into a second model to obtain a pronunciation duration of each phoneme in the second text; wherein the second model is trained using the training method of the above-mentioned model; and generating speech corresponding to the second text based on the pronunciation duration.

[0011] An embodiment of the present application provides a speech generation device, including: a second encoding module, used to perform a sixth encoding on third label data to obtain a sixth encoding value; the second encoding module is also used to perform a seventh encoding on multiple phonemes in a second text to obtain a seventh encoding value; a third generation module is used to generate an input sequence based on the sixth encoding value and the seventh encoding value; a second prediction module is used to input the input sequence into a second model to obtain the pronunciation duration of each phoneme in the second text; wherein the second model is trained using the training method of the above-mentioned model; and a speech generation module is used to generate speech corresponding to the second text based on the pronunciation duration.

[0012] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; a processor for implementing the above-mentioned model training method; or implementing the above-mentioned pronunciation duration prediction method; or implementing the above-mentioned speech generation method when executing the executable instructions stored in the memory.

[0013] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned model training method; or the above-mentioned pronunciation duration prediction method; or the above-mentioned speech generation method is implemented.

[0014] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned model training method; or implement the above-mentioned pronunciation duration prediction method; or implement the above-mentioned speech generation method.

[0015] The above solution has the following beneficial effects:

[0016] When training the model, a first duration sequence is first generated based on the pronunciation duration of each phoneme in the text annotation data. Then, by adjusting the number of frames in the first duration sequence, the impact of abnormal sound lengths (too long or too short) in the training data set on the model prediction results can be reduced, thereby enhancing the stability and robustness of the first model. In addition, since there are no abnormal sound lengths in the adjusted second duration sequence, the adjusted second duration sequence and the text annotation data are used together to form sample data, which can provide high-quality input for the training of the first model, improve the training effect of the model, and further improve the sound length prediction accuracy of the trained second model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of a non-autoregressive model predicting hypoarticulation in the related art;

[0018] Figure 2 This is an optional architectural diagram of a training system for a model provided in an embodiment of the present application;

[0019] Figure 3 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0020] Figure 4 This is an optional flowchart of the model training method provided in the embodiment of the present application;

[0021] Figure 5 This is another optional flowchart of the model training method provided in the embodiment of the present application;

[0022] Figure 6 is a schematic diagram of a flow chart for determining a second time sequence provided in an embodiment of the present application;

[0023] Figure 7 This is a schematic diagram of the process of determining sample data provided by an embodiment of the present application;

[0024] Figure 8a Schematic diagram of the implementation flow of the pronunciation duration prediction method provided in the embodiment of the present application;

[0025] Figure 8b This is a schematic diagram of the implementation flow of the speech generation method provided in the embodiment of the present application;

[0026] Figure 9 It is a structural diagram of the pronunciation duration prediction model based on the large language model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it will be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which the embodiments of this application pertain. The terms used in the embodiments of this application are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0029] In the following description, the terms "first", "second", and "third" only distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first", "second", and "third" can be interchanged in a specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0030] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of that module or unit.

[0031] Before describing the training method of the pronunciation duration prediction model provided by the embodiments of the present application, first, the professional terms involved in the embodiments of the present application are described:

[0032] 1) Text-to-Speech (TTS) technology: Also known as speech synthesis technology, it refers to converting the text information generated by a computer or input externally into natural and fluent speech output.

[0033] 2) Phoneme: It is the text representation of an acoustic unit of speech. For example, the initials and finals of Chinese pinyin can represent independent phonemes respectively. Each phoneme represents a very short segment of pronunciation, and the pronunciation of each phoneme is different. Generally, a certain segment of speech in a certain language can be represented by a sequence of phonemes in the phoneme set corresponding to that language. For example, for the Mandarin phrase "nǐ hǎo", the phoneme representation can be "n i3 h ao3".

[0034] 3) Montreal Forced Aligner (MFA): A set of open-source software tools. For a segment of speech and a transcribed text in a certain format of the speech, the MFA tool can be used for time alignment, and the alignment result is the start and end time points of the pronunciation of each text unit. For example, the sentence "Who am I?" has a total pronunciation duration of 1 second. After aligning with the MFA tool, the three characters "wǒ", "shì", and "shuí" can be obtained, corresponding to the start and end seconds of the speech respectively. It can even align the question mark to the silent time at the end of the audio. For example, through the MFA tool, the time distribution of each text unit in the 1-second audio of the recording file "Who am I?" can be determined, such as "wǒ" [0, 0.2], "shì" [0.2, 0.5], "shuí" [0.5, 0.9], "?" [0.9, 1.0].

[0035] 4) Non-Autoregressive Text-to-Speech (NAR TTS): NAR TTS is characterized by parallel prediction and inference of individual speech frames while simultaneously generating speech from a text. The prediction of each speech frame does not depend on the previous historical speech frame.

[0036] 5) Pronunciation duration: referred to as sound duration, refers to the pronunciation time length of each speech unit in natural speech data. For example, how many milliseconds does it take to pronounce a Chinese character. Generally, phonemes are used as the representation unit of speech text, and the number of pronunciation frames of the phoneme is used to represent the pronunciation duration. For example, the phoneme "h" has a pronunciation duration of 5 frames. An audio frame is an audio segment of fixed duration, so the pronunciation duration can be converted into a number of frames. For example: if the frame length is fixed at 20ms, then a 100 millisecond duration can be converted into 5 frames to represent it.

[0037] In order to better understand the training method of the model provided in the embodiment of the present application, the pronunciation duration prediction method in the related art is first described below.

[0038] In related technologies, MFA tools are used to extract the text-to-speech alignment relationship to obtain the pronunciation duration of each text element (which may be a word or a phoneme), and then the {text sequence, sound length sequence} tag pair is directly used to build a neural network model for prediction. The neural network model mainly adopts the design ideas of a classification model or an alignment learning model. With the rise of self-supervised learning technology, the sound length prediction method based on the bidirectional encoder representation from transformer (BERT, Bidirectional Encoder Representations from Transformers) text pre-training feature extractor has improved the prediction effect of sound length distribution. In order to adapt to the needs of multi-speaker TTS synthesis, the timbre features are integrated into the text sound length prediction application, which can better make the sound length distribution prediction personalized for the timbre. In addition, the sound length distribution prediction can be combined with the speech pre-training feature extraction and the cross-attention learning of the text pre-training feature extraction.

[0039] However, the pronunciation duration prediction methods in related technologies have at least the following problems: 1) Traditional sound length prediction methods are generally used as a component module of a non-autoregressive TTS model, which is trained along with the training steps of the TTS model and is naturally a part of the TTS model binding and embedding. 2) In non-autoregressive TTS technology products, there are occasional cases where the synthesized pronunciation duration of individual words is not long enough (or the synthesized pronunciation duration is too long), such as a word is only pronounced half a tone, or it sounds like there is no pronunciation. This situation is related to the failure of the prediction of the sound length distribution of phonemes. This is especially true in boundary input situations such as when the sentence to be synthesized is particularly long or particularly short.

[0040] See Figure 1 , Figure 1 which is a schematic diagram of the prediction of incomplete pronunciation by a non-autoregressive model in the related art. Since the initial phoneme "h" corresponding to the character "hao", the predicted phoneme length obtained by the phoneme length prediction model is only 1 frame, which is much less than the normal pronunciation duration (if the normal is 5 frames), then the user feels that this phoneme is not pronounced. So this sentence should have been pronounced as "nǐ hǎo ma", but the synthesized audio makes the user sound like "nǐ ǎo ma".

[0041] Based on the existing audio duration prediction methods in the related art, there is no way to avoid the problem of insufficient or excessive synthesized pronunciation duration. The reasons are as follows: First, since the pronunciation duration prediction model is trained together with the TTS model, in the training data of the TTS model, there may be some samples where the speaker's speech is unclear, or the pronunciation is omitted during word linking, etc., resulting in incorrect pronunciation durations of some individual phonemes in some samples, or the pronunciation durations of some individual phonemes are too short or too long. Second, since the prediction result of the pronunciation duration prediction model has a certain degree of randomness, generally the phoneme length value ranges from 0 frames (or 1 frame, depending on the specific model settings) to N frames, and N is generally an empirical value. Therefore, it is possible to sample a special extremely long or extremely short value for the phoneme length prediction result.

[0042] Based on the problems existing in the related art, the embodiments of the present application provide a method for training a model. First, based on the pronunciation duration of each phoneme in the text annotation data, a first duration sequence is generated, and then the number of frames in the first duration sequence is adjusted, which helps to reduce the influence of abnormal phoneme lengths (too long or too short phoneme lengths) in the training data set on the prediction result and enhance the stability and robustness of the first model; in addition, the adjusted second duration sequence and the text annotation data are jointly used as sample data to provide high-quality input for training and further improve the phoneme length prediction accuracy of the second model obtained by training.

[0043] Specifically, the embodiments of the present application provide a method for training a model. First, based on the pronunciation duration of each phoneme in the text annotation data, a first duration sequence is generated; the first duration sequence includes the number of frames corresponding to the pronunciation duration of each phoneme; then, the number of frames in the first duration sequence is adjusted to obtain a second duration sequence; then, the second duration sequence and the text annotation data are determined as sample data; finally, the first model is trained with the sample data to obtain a second model. The second model can be a pronunciation duration prediction model for predicting the pronunciation duration of each phoneme in the text.

[0044] Before explaining the training method of the model of the embodiment of the present application, the exemplary application of the model training device of the embodiment of the present application is first described, and the model training device is an electronic device for implementing the training method of the model. In one implementation, the model training device (i.e., electronic device) provided by the embodiment of the present application can be implemented as a terminal or a server. In one implementation, the electronic device provided by the embodiment of the present application can be implemented as any terminal with a model training function, such as a laptop computer, a tablet computer, a desktop computer, an intelligent robot, an intelligent home appliance, and an intelligent vehicle-mounted device; in another implementation, the model training device provided by the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs, Content Delivery Networks), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application. Below, the exemplary application of the model training device when it is implemented as a server will be described.

[0045] See also Figure 2 , Figure 2 This is an optional architectural diagram of the model training system provided in the embodiment of the present application. The model training system 10 in the embodiment of the present application includes at least a terminal 100, a network 200, and a server 300. The terminal 100 is deployed with a model training application, and the server 300 can be a background server for the model training application. The server 300 can constitute the model training device in the embodiment of the present application, that is, the model training method in the embodiment of the present application is implemented through the server 300. The terminal 100 is connected to the server 300 via the network 200. The network 200 can be a wide area network or a local area network, or a combination of the two.

[0046] See also Figure 2, the terminal 100 receives the user's model training operation, and the terminal 100 generates a model training request in response to the model training operation; then, the terminal 100 sends the model training request to the server 300 through the network 200. After receiving the model training request, the server 300 responds to the model training request and obtains a training data set, wherein the training data set includes a speech sample and text annotation data corresponding to the sample speech; the text annotation data includes multiple phonemes of the sample speech; then, the server 300 generates a first time sequence based on the pronunciation duration of each phoneme in the text annotation data; the first time sequence includes the number of frames corresponding to the pronunciation duration of each phoneme; then, the server 300 adjusts the number of frames in the first time sequence to obtain a second time sequence; then, the server 300 determines the second time sequence and the text annotation data as sample data; next, the server 300 trains the first model through the sample data to obtain a second model, and generates a reminder message that the second model is successfully trained; finally, the server 300 sends the reminder message that the second model is successfully trained to the terminal 100 through the network 200, and displays the reminder message that the second model is successfully trained on the display interface of the terminal 100.

[0047] In some embodiments, the training method of the above-mentioned second model can also be executed by the terminal, that is, after receiving the user's model training operation, the terminal 100 can obtain a training data set in response to the model training operation, wherein the training data set includes sample speech and text annotation data corresponding to the sample speech; the text annotation data includes multiple phonemes of the sample speech; then, the terminal 100 generates a first time sequence based on the pronunciation duration of each phoneme in the text annotation data; the first time sequence includes the number of frames corresponding to the pronunciation duration of each phoneme; then, the terminal 100 adjusts the number of frames in the first time sequence to obtain a second time sequence; then, the terminal 100 determines the second time sequence and the text annotation data as sample data; next, the terminal 100 trains the first model through the sample data to obtain the second model, and generates a reminder message that the second model training is successful; finally, the terminal 100 displays a reminder message that the second model training is successful on the display interface.

[0048] Figure 3 is a structural diagram of an electronic device provided in an embodiment of the present application, Figure 3The electronic device shown can be a model training device, a pronunciation duration prediction device, or a speech generation device. The electronic device includes: at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components in the electronic device are coupled together through a bus system 340. It can be understood that the bus system 340 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 3 Various buses are labeled as bus system 340 .

[0049] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0050] The user interface 330 includes one or more output devices 331 that enable presentation of media content, and one or more input devices 332 .

[0051] The memory 350 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, and the like. The memory 350 may optionally include one or more storage devices physically located away from the processor 310. The memory 350 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 350 described in the embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.

[0052] The operating system 351 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks; a network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320. Exemplary network interfaces 320 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB); an input processing module 353 is used to detect one or more user inputs or interactions from one of the one or more input devices 332 and translate the detected inputs or interactions.

[0053] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 3 A model training device 354 stored in memory 350 is shown. This model training device 354 can be a model training device in an electronic device. It can be software in the form of a program or plug-in, and includes the following software modules: a first generation module 3541, an adjustment module 3542, a determination module 3543, and a training module 3544. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0054] In some embodiments, the device of the embodiment of the present application further includes a pronunciation duration prediction device ( Figure 3 ), including the following software modules: a first encoding module, a second generation module and a first prediction module.

[0055] In some embodiments, the device of the embodiment of the present application further includes a speech generating device ( Figure 3 (not shown), including the following software modules: a second encoding module, a third generation module and a second prediction module.

[0056] In some embodiments, the apparatus provided in the embodiments of the present application can be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the model provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0057] The model training method provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the model training method provided in each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal.

[0058] Figure 4 This is an optional flow chart of the training method of the model provided in the embodiment of the present application. Figure 4 The steps shown are explained as Figure 4 As shown, the execution subject of the model training method is a server as an example for description, and the method includes the following steps S101 to S104:

[0059] Step S101 : generating a first duration sequence based on the pronunciation duration of each phoneme in the text annotation data; the first duration sequence includes the number of frames corresponding to the pronunciation duration of each phoneme.

[0060] In an embodiment of the present application, a training data set may be acquired in advance; the training data set includes sample speech and text annotation data corresponding to the sample speech; the text annotation data includes multiple phonemes of the sample speech.

[0061] Here, a training dataset refers to a set of data samples used to train machine learning models. The data in a training dataset can be collected manually or obtained from public datasets. Typically, a training dataset should contain more than 100,000 records, or the total duration of sample speech should exceed 100,000 hours. Sample speech refers to the audio data in the training dataset.

[0062] Text annotation data refers to multiple phonemes obtained by performing phoneme conversion on the text information corresponding to the sample speech. Here, a phoneme is the text representation of an acoustic unit of speech. For example, the initials and finals of Chinese pinyin can represent independent phonemes respectively. For instance, for the Mandarin phrase "你好" (nǐ hǎo), the corresponding phonemes can be represented as "n i3 h ao3". It should be noted that there is a corresponding relationship among the label data, sample speech, and text annotation data in the training dataset, that is, each sample speech has corresponding label data and text annotation data.

[0063] In some embodiments, when the training dataset is obtained by manual collection, the text annotation data in step S101 can be obtained by the following method: First, for the manually collected sample speech, the text information corresponding to each sample speech is obtained through Automatic Speech Recognition (ASR) technology; then, through a Grapheme-to-Phoneme (G2P) conversion tool, the text information is directly converted into the corresponding phonemes, and all the phonemes corresponding to each sample speech are determined as the text annotation data corresponding to that sample speech.

[0064] The pronunciation duration of a phoneme refers to the time interval from the start to the end of the phoneme, usually expressed in milliseconds or seconds. The first duration sequence is a list that includes the number of frames corresponding to the pronunciation duration of each phoneme. The number of frames refers to the number of frames obtained after dividing the continuous speech signal into multiple frames of a fixed length within a specific time period.

[0065] In some embodiments, step S101 can be implemented by the following method: First, use the MFA tool to align the length of each phoneme in the text annotation data with the sample speech to obtain the start and end times of each phoneme's pronunciation, such as Td = (p1(Start p1, end p1), p2(Start p2, end p2),..., pN(Start pN, end pN)), where Start represents the start time point of the speech corresponding to the phoneme, end represents the end time point of the speech corresponding to the phoneme, and N represents the number of phonemes; then, determine the pronunciation duration of each phoneme according to the start and end times of each phoneme's pronunciation; finally, divide the pronunciation duration of each phoneme by the specified duration of each frame to obtain the number of frames of each phoneme. It should be noted that the number of frames is an integer. If the calculated number of frames is a decimal, rounding is used to obtain an integer.

[0066] As an example of step S101, assume that the text annotation data is T = "n i3 h ao3 m a5" (how are you), where "n", "h", and "m" are initial consonants, "i", "ao", and "a" are finals, and "3", "3", and "5" are tones; then, the sound duration corresponding to each phoneme is calculated using the MFA tool, and the number of frames of each phoneme is determined according to the sound duration specified for each frame. The final calculation result is T align = (n: 4, i3: 10, h: 5, ao3: 8, m: 6, a5: 16), where "4" represents the number of frames of the phoneme "n"; finally, the number of frames corresponding to the pronunciation duration of each phoneme in the text annotation data is calculated, and the number of frames corresponding to the pronunciation duration of the phoneme are sorted according to the order of each phoneme in the sample speech to obtain a first duration sequence.

[0067] Step S102: Adjust the number of frames in the first time-length sequence to obtain a second time-length sequence.

[0068] Frame adjustment refers to increasing, decreasing, balancing, or redistributing the number of frames in the first time sequence to meet specific requirements. Frame adjustment methods may include interpolation (adding frames), downsampling (reducing frames), and mean smoothing. The specific frame adjustment method should be selected based on actual circumstances and is not limited in this application.

[0069] In an embodiment of the present application, the number of frames in the first time-length sequence can be adjusted based on a preset quantile to obtain a second time-length sequence. Here, quantile is a concept in statistics, which is used to divide data according to a certain proportion, such as quartiles and percentiles. The quantiles include the more special 50th percentile, that is, the median. The median indicates that 50% of the values ​​in the data are less than or equal to the value corresponding to the median. The preset quantile refers to a set of quantiles that are pre-set based on experience or statistical analysis before adjusting the number of frames, which are used to guide the data adjustment strategy.

[0070] The second time series refers to a frame number list constructed based on the adjusted frame number after the frame number in the first time series is adjusted by using quantiles.

[0071] In some embodiments, the preset quantiles include a minimum quantile and a maximum quantile, and step S102 can be implemented by the following method: first, the number of frames corresponding to the pronunciation duration of the same phoneme is extracted in the first duration sequence to obtain a first frame number sequence, for example, all the frame numbers corresponding to the pronunciation duration of the initial phoneme "n" in the first duration sequence are extracted, which are 100 in total, specifically [1, 3, 5, 1, 8, 2, ..., 8, 20], and the frame number sequence is the first frame number sequence of the initial phoneme "n"; then, for the initial phoneme "n", the extracted frames are extracted. The number of frames in the first frame sequence is adjusted to obtain a second frame sequence. Specifically, the extracted frames can be arranged in ascending order to obtain a sorted frame sequence (i.e., a third frame sequence), such as [1, 1, 1, 1, 2, 2, ..., 19, 20]. Next, the first and second frame numbers are determined in the third frame sequence based on the minimum quantile and the maximum quantile. The first frame number is the minimum frame number in the second frame sequence to be obtained, and the second frame number is the maximum frame number in the second frame sequence to be obtained. Assuming that the minimum quantile is 0.1 and the maximum quantile is 0.9, the 10th frame number (i.e., calculated as 100×0.1=10) in the second frame sequence is determined as the first frame number based on the minimum quantile, such as the first frame number is 5. The 90th frame number (i.e., calculated as 100×0.9=90) in the second frame sequence is determined as the second frame number based on the maximum quantile, such as the second frame number is 18. Next, the number of frames in the third frame sequence that are less than 5 frames and greater than 18 frames are adjusted, such as by increasing the number of frames from the 1st to the 9th position in the third frame sequence that are less than 5 frames to 5 frames through interpolation, and reducing the number of frames from the 91st to the 100th position in the third frame sequence that are greater than 18 frames to 18 frames through downsampling, thereby obtaining a second frame sequence. It should be noted that if the number of frames from the 1st to the 9th position in the third frame sequence is equal to the first number of frames, or if the number of frames from the 91st to the 100th position in the third frame sequence is equal to the second number of frames, then there is no need to adjust the number of frames.

[0072] After obtaining the second frame number sequence, the frame number corresponding to the pronunciation duration of each phoneme in the text annotation data can be obtained from the second frame number sequence, and then the obtained frame numbers are sorted according to the order of each phoneme in the sample speech to obtain the second duration sequence.

[0073] In some embodiments, after adjusting the number of frames in the third frame sequence according to the first frame number and the second frame number, each frame in the third frame sequence after the frame number adjustment can also be normalized to obtain a fourth frame sequence; finally, the number of frames corresponding to the pronunciation duration of each phoneme in the text annotation data is obtained in the fourth frame sequence, and the obtained frames are sorted according to the order of each phoneme in the sample speech to obtain a second duration sequence.

[0074] Step S103: Determine the second time sequence and text annotation data as sample data.

[0075] Sample data refers to the data set finally input into the first model after processing the training data set, wherein the phonemes in the text annotation data and the number of frames corresponding to the phonemes in the second time sequence can be used as a piece of sample data.

[0076] In some embodiments, the training data set also includes first label data, which refers to a speaker label, that is, a speaker label corresponding to the sample speech in the training data set. Step S103 can be implemented by the following method: first, the first label data is encoded to obtain first label encoding data. Here, a simple sort value encoding can be used, that is, each first label data is sorted and encoded starting from 1; then, each phoneme in the text annotation data is encoded according to a preset phoneme encoding table to obtain phoneme encoding data. Assuming that the text is in Chinese, the Chinese is converted into pinyin, and the corresponding encoding data is queried in the preset initial and final encoding table based on the initial consonants and finals with tones in the pinyin.

[0077] Next, the frame numbers in the second time-length sequence are encoded. The specific encoding method is as follows: since the frame numbers in the second time-length sequence are normalized data representations, that is, each frame number is within a fixed range, such as between (0, 1), the fixed numerical range is evenly divided into multiple numerical intervals, and each numerical interval corresponds to an interval identifier. For example, the interval (0, 1) is evenly divided into 100 numerical intervals, (0, 0.01], (0.01, 0.02], ..., (0.99, 1], and these 100 intervals are marked as 1, 2, 3, ..., 100 in sequence; next, it is determined to which numerical interval the frame numbers in the second time-length sequence belong, and the interval identifier corresponding to the numerical interval is used as the frame number encoding data of the corresponding frame number. For example, if the frame number 1 in the second time-length sequence is 0.01, the interval identifier 1 of (0, 0.01] is used as the frame number encoding data of the frame number 1.

[0078] Next, obtain the preset identifiers, which include a background identifier, a phoneme end identifier, and a sound length end identifier. The background identifier serves as the terminator of the first label-encoded data, the phoneme end identifier serves as the separator between the phoneme-encoded data and the frame-encoded data, and the sound length end identifier serves as the terminator of the frame-encoded data. Finally, interleave the first label-encoded data, the phoneme-encoded data, the frame-encoded data, and the separator to obtain sample data. The specific interleaving connection method is as follows:

[0079] [ID][BG][S1][SP][s1][PE][S2][SP][s2][PE]…[SP]

[0080] Where ID is the first label encoding data; BG is the background identifier; S1 is the first phoneme encoding data; SP is the phoneme end identifier; s1 is the frame number encoding data of the first phoneme encoding data; PE is the sound length end identifier; S2 is the second phoneme encoding data; and s2 is the frame number encoding data of the second phoneme encoding data. It should be noted that BG, SP, and PE can be set with corresponding encoding values, such as 320 for BG, 319 for PE, and 318 for SP.

[0081] Step S104: training the first model using sample data to obtain a second model.

[0082] Here, the first model refers to an untrained machine learning model or deep learning model. The first model can be implemented as a long short-term memory network (LSTM) or a transformer model (Transformer). The specific implementation method is not limited in this application.

[0083] The second model may be a pronunciation duration prediction model, which refers to the trained first model. The pronunciation duration prediction model can predict the pronunciation duration of each phoneme in the text data based on the input text data.

[0084] In some embodiments, step S104 can be implemented by the following method: first, the sample data is input into the first model, and the sample data is forward propagated through the first model to obtain a first prediction sequence, wherein the first prediction sequence includes a phoneme prediction value and a frame number prediction value; then, a mask operation is performed on the first prediction sequence, and all data in the first prediction sequence except the frame number prediction value is masked to obtain a second prediction sequence, and masking refers to using a mask to shield out part of the data in the first prediction sequence; finally, the frame number prediction value in the second prediction sequence and the frame number encoding data in the second time sequence are used to perform loss calculation to obtain a loss result, and the network parameters of the first model are adjusted according to the loss result until the loss result stabilizes within a fixed range, and training is stopped to obtain a second model.

[0085] The training method of the model of the embodiment of the present application ensures the comprehensiveness and diversity of the training data by obtaining a training data set containing sample speech and text annotation data, thereby providing a sufficient learning basis for the first model; and by adjusting the number of frames in the first time sequence, it can reduce the impact of abnormal sound lengths (too long or too short) in the training data set on the model prediction results, thereby enhancing the stability and robustness of the pronunciation duration prediction model; in addition, since there are no abnormal sound lengths in the adjusted second time sequence, the adjusted second time sequence and the text annotation data are used together to form sample data, which can provide high-quality input for the training of the second model, improve the training effect of the model, and further improve the sound length prediction accuracy of the second model.

[0086] The following examples illustrate the application scenarios of the model training method provided in the embodiments of the present application. The embodiments of the present application can be applied to at least the following exemplary scenarios:

[0087] In the development process of intelligent voice assistant, it is necessary to accurately predict the pronunciation duration of phonemes to ensure that the generated voice is natural and fluent. In this way, the training method of the model provided in the embodiment of the present application can be used to train the second model, which can be a pronunciation duration prediction model. First, after the terminal receives the model training operation triggered by the developer, it generates a model training request and sends the model training request to the server; then, the server responds to the model training request and obtains a training data set, wherein the training data set includes a speech sample and text annotation data corresponding to the sample speech; the text annotation data includes multiple phonemes of the sample speech; then, the server generates a first duration sequence based on the pronunciation duration of each phoneme in the text annotation data; the first duration sequence includes the number of frames corresponding to the pronunciation duration of each phoneme; then, the server adjusts the number of frames in the first duration sequence to obtain a second duration sequence; next, the second duration sequence and the text annotation data are determined as sample data; finally, the first model is trained by the sample data to obtain a pronunciation duration prediction model. After obtaining the pronunciation duration prediction model, the pronunciation duration of the phoneme is predicted by the pronunciation duration prediction model.

[0088] According to the above scenario, taking the training of the pronunciation duration prediction model as an example, the training method of the model of the embodiment of the present application is explained. Figure 5 This is another optional flow chart of the training method of the model provided in the embodiment of the present application, such as Figure 5 As shown, the method includes the following steps S201 to S211:

[0089] Step S201: The terminal receives a model training operation.

[0090] Here, a training application for a pronunciation duration prediction model may be running on the terminal, and the server constitutes a backend server for the training application for the pronunciation duration prediction model. The model training operation may be a selection operation or an input operation inputted by a client of the training application for the pronunciation duration prediction model running on the terminal. For example, the selection operation may select the storage path of a training dataset for model training, or the input operation may be a user inputting the storage path of the training dataset for model training on the client, or the user may directly input the training dataset for model training on the client.

[0091] In some embodiments, the training application of the pronunciation duration prediction model may provide an input interface or input box, allowing the user to select or enter the storage path of the training data set used for model training. The input interface may be in the form of a form, a text box, or a drop-down menu, and the specific form is not limited in this application. The user may select the storage path of the training data set used for model training from predetermined options, or manually enter the storage path of the training data set used for model training.

[0092] Step S202: The terminal generates a model training request in response to the model training operation.

[0093] Here, the terminal may encapsulate the storage path of the training data set selected or input by the user for model training into the model training request.

[0094] In some embodiments, in order to ensure that the storage path of the training data set selected or input by the user for model training is accurately included in the request data, first, it is necessary to check the integrity of the storage path information. In the process of filling the storage path information into the model training request, ensure that the storage path information is complete and accurate. The method of checking the integrity of the storage path includes checking whether the storage path contains the file name, file extension, and necessary characters such as slash or backslash; then, select a suitable data structure to ensure that the storage path information is correctly embedded in the data structure; finally, special characters are processed. For special characters such as slash, backslash, quotation marks, etc., the special characters can be converted into a specific encoding format for correct transmission and parsing.

[0095] Step S203: The terminal sends a model training request to the server.

[0096] In some embodiments, the terminal sends the encapsulated model training request to the server and requests the server to perform the model training operation, usually using protocols such as HyperText Transfer Protocol (HTTP) or Web Socket to send the model training request.

[0097] In step S204, the server obtains a training data set in response to the model training request; the training data set includes sample speech and text annotation data corresponding to the sample speech; the text annotation data includes multiple phonemes of the sample speech.

[0098] Here, after receiving the model training request, the server will parse the model training request to obtain the storage path of the training data set used for model training; then, the training data set is obtained from the preset database according to the storage path of the training data set.

[0099] In some embodiments, the server can parse the model training request by the following method: assuming that the format of the model training request is an HTTP request, the server can parse the request header and request body, parse the request header to obtain relevant information of the request; parse the request body to obtain the main body data of the request, that is, the storage path of the training data set used for model training. Specific fields or parameters in the request body are parsed, and these fields or parameters contain the storage path of the training data set used for model training. A specific data format, such as a lightweight data exchange format (JSON, JavaScript Object Notation) or Extensible Markup Language (XML) is extracted from the request body, and then the data format is parsed to obtain the storage path of the training data set used for model training.

[0100] In step S205 , the server generates a first duration sequence based on the pronunciation duration of each phoneme in the text annotation data; the first duration sequence includes the number of frames corresponding to the pronunciation duration of each phoneme.

[0101] Here, a phoneme is a textual representation of an acoustic unit of speech. Different languages ​​have specific sets of phonemes, such as English vowels [a, e, i, o, u] and consonants [p, t, k], and Chinese pinyin initials and tonal finals, such as "n i3 hao3." Frames convert the duration of a phoneme's pronunciation into a fixed-length time window (frame). Common frame lengths are 10 milliseconds or 25 milliseconds. Frames represent the detailed distribution of phonemes over time.

[0102] In some embodiments, in step S205, a first duration sequence is generated based on the pronunciation duration of each phoneme in the text annotation data, which can be achieved by the following method: first, each phoneme in the text annotation data is aligned with the sample speech to obtain the pronunciation start and end time of each phoneme; then, based on the pronunciation start and end time, the pronunciation duration of each phoneme is determined; then, based on the preset first duration, the number of frames corresponding to the pronunciation duration of each phoneme is determined; finally, according to the order of each phoneme in the sample speech, the number of frames corresponding to the pronunciation duration of the phoneme is sorted to obtain the first duration sequence.

[0103] Here, length alignment refers to the temporal alignment of the phonemes in the text annotation data with the sample speech to determine the start and end times of each phoneme. For example, after length alignment, the start and end times of the phoneme "h" are obtained to be 0.00s to 0.12s. Common length alignment methods include forced alignment, dynamic time adjustment alignment, and deep learning alignment. The specific length alignment method is not limited in this application.

[0104] The preset first duration refers to the preset duration of each frame, for example, each frame duration is 10 milliseconds. The number of frames for each phoneme is calculated as follows: number of frames = pronunciation duration of the phoneme / preset first duration. For example, if the preset first duration is 10 milliseconds and the pronunciation duration of the phoneme "h" is 120 milliseconds, the number of frames corresponding to the phoneme "h" is 120 / 10 = 12, and the frame length corresponding to the pronunciation time of the phoneme "h" is 12 frames.

[0105] As an example of step S205, assuming that the phonemes in the text annotation data are "n i3 h ao3", the phonemes in the text annotation data are forced to be aligned with the sample speech through the MFA alignment tool to determine the pronunciation start and end time of each phoneme, such as the pronunciation start and end time of the phoneme "n" is 0.00s-0.10s, the pronunciation start and end time of "i3" is 0.10s-0.25s, the pronunciation start and end time of "h" is 0.25s-0.40s, and the pronunciation start and end time of "ao3" is 0.40s-0.60s; then, the pronunciation duration of each phoneme is determined according to the pronunciation start time of each phoneme. It can be seen that the pronunciation duration of the phoneme "n" is 0.1s, the pronunciation duration of the phoneme "i3" is 0.15s, the pronunciation duration of the phoneme "h" is 0.15s, and the pronunciation duration of the phoneme "a The pronunciation duration of "o3" is 0.2s. Assuming that the preset first duration is 10 milliseconds, the number of frames corresponding to the pronunciation duration of each phoneme is determined according to the pronunciation duration of each phoneme and the preset first duration. For example, the number of frames corresponding to the pronunciation duration of the phoneme "n" is 100 / 10=10 frames, the number of frames corresponding to the pronunciation duration of the phoneme "i3" is 150 / 10=15 frames, the number of frames corresponding to the pronunciation duration of the phoneme "h" is 150 / 10=15 frames, and the number of frames corresponding to the pronunciation duration of the phoneme "ao3" is 200 / 10=20 frames. Finally, the number of frames corresponding to the pronunciation duration of the phonemes are sorted according to the order of the phonemes in the sample speech, and the first duration sequence is obtained as [10, 15, 15, 20].

[0106] It should be noted that after determining the number of frames corresponding to the pronunciation duration of each phoneme, the minimum number of frames and the maximum number of frames corresponding to the pronunciation duration of each phoneme can be counted by a preset frame number statistics module to generate a phoneme pronunciation duration statistics table, as shown in Table 1:

[0107] Table 1 Statistics of phoneme pronunciation duration

[0108] phoneme Maximum frame rate Minimum frame rate a 20 7 o 33 5 … … … z 14 2 zh 22 8

[0109] Step S206: The server adjusts the number of frames in the first time-length sequence to obtain a second time-length sequence.

[0110] Frame number adjustment refers to increasing, decreasing, or smoothing the number of frames according to specific criteria (such as quantiles) to optimize the duration distribution of the number of frames corresponding to the pronunciation duration of the phoneme. In an embodiment of the present application, the number of frames in the first duration sequence can be adjusted based on a preset quantile to obtain a second duration sequence. Here, quantile refers to a data division method in statistics, which is used to represent the distribution of data under a certain proportion.

[0111] In some embodiments, the preset quantiles include the minimum quantile and the maximum quantile; see Figure 6 , Figure 6 is a flow chart of determining a second time sequence according to an embodiment of the present application. Figure 6 It is shown that in step S206, the server adjusts the number of frames in the first time sequence to obtain a second time sequence, which can be achieved by the following steps S2061 to S2066:

[0112] Step S2061 : extracting the number of frames corresponding to the pronunciation duration of the same phoneme from the first duration sequence to obtain a first frame number sequence.

[0113] As an example of step S2061, assume that all the frames corresponding to the pronunciation duration of the phoneme "n" in the first duration sequence are extracted, a total of 100, specifically [1, 3, 5, 1, 8, 2, ..., 8, 20]. The sequence is taken as the first frame number sequence. At this time, the frames in the first frame number sequence are sorted according to the order of the phonemes in the sample speech.

[0114] Step S2062: sort the frames in the first frame sequence to obtain a third frame sequence.

[0115] As an example of step S2062, following the example of step S2061 above, the frame numbers in the first frame number sequence are sorted in ascending order to obtain a third frame number sequence, and the third frame number sequence is [1, 1, 1, 1, 2, 2, ..., 19, 20].

[0116] Step S2063 : determining a first frame number and a second frame number in the third frame number sequence based on a preset quantile.

[0117] Here, the preset quantiles may include a minimum quantile and a maximum quantile. The first frame number may be a minimum frame number determined according to the minimum quantile, and the second frame number may be a maximum frame number determined according to the maximum quantile.

[0118] Continuing with the example of step S2062 above, assuming that the minimum quantile is 0.1 and the maximum quantile is 0.9, the 100×0.1=10th frame in the second frame sequence is determined as the first frame based on the minimum quantile, such as if the first frame is 5, and the 100×0.9=90th frame in the second frame sequence is determined as the second frame based on the maximum quantile, such as if the second frame is 18.

[0119] Step S2064: Based on the first frame number and the second frame number, the frame number in the third frame number sequence is adjusted to obtain a second frame number sequence.

[0120] In some embodiments, step S2064 can be implemented by modifying at least one frame number in the third frame sequence that is smaller than the first frame number to the first frame number; and modifying at least one frame number in the third frame sequence that is larger than the second frame number to the second frame number. After modifying the frame numbers in the third frame sequence to obtain a modified third frame sequence, each frame number in the modified third frame sequence can be normalized using the minimum frame number and the maximum frame number to obtain a second frame sequence.

[0121] Here, normalization refers to mapping the data to a specific range, such as (0, 1), so that the data has a uniform scale, which is convenient for subsequent processing and analysis. Normalization processing refers to using the first frame number and the second frame number to standardize the adjusted frame number. The calculation formula for normalization processing can be expressed as the following formula (1):

[0122] Normalized frame rate = (current frame rate - minimum frame rate) / (maximum frame rate - minimum frame rate) (1)

[0123] The second frame number sequence is a decimal sequence between (0, 1), that is, the normalized frame number after normalization represents the frame number corresponding to the pronunciation duration of the phoneme.

[0124] Continuing with the example of step S2063 above, the frame numbers from the 1st to the 9th in the third frame number sequence, which have a frame number less than 5 frames, are increased to 5 frames by interpolation, and the frame numbers from the 91st to the 100th in the third frame number sequence, which have a frame number greater than 18 frames, are reduced to 18 frames by downsampling; then, each frame number in the third frame number sequence after the frame number adjustment is normalized according to the first frame number and the second frame number to obtain a second frame number sequence.

[0125] Step S2065 : Obtain the frame number corresponding to the pronunciation duration of each phoneme in the text annotation data from the second frame number sequence.

[0126] Continuing with the example of step S2064 above, in the second frame sequence, the frame number corresponding to the pronunciation duration of the first phoneme "n" is 0.6, the frame number corresponding to the pronunciation duration of the second phoneme "n" is 0.5, and the frame number corresponding to the pronunciation duration of the third phoneme "n" is 0.8.

[0127] Step S2066: Sort the acquired frames according to the order of each phoneme in the sample speech to obtain a second time sequence.

[0128] Continuing with the example of step S2065 above, the second frame sequence is sorted based on the order of the first, second, and third phonemes "n" in the sample speech. For example, in the sample speech "Can you help me get an orange?", the first phoneme "n" corresponds to "can," the second phoneme "n" corresponds to "you," and the third phoneme "n" corresponds to "take." The phonemes "n" are sorted based on the order of their appearance in the sample speech, resulting in a duration sequence of [0.5, 0.6, 0.8] for the phoneme "n." Then, the frames corresponding to each phoneme in the first duration sequence are sorted based on the order of their appearance in the sample speech, resulting in a second duration sequence.

[0129] It should be noted that steps S2061 to S2066 make fine adjustments to the number of frames through preset quantiles, thereby avoiding the occurrence of extreme frame values ​​and reducing interference data (such as shorter or longer frames) during model training, providing high-quality training data for the training of the pronunciation duration prediction model, while ensuring the accuracy and rationality of the sound length prediction.

[0130] In step S207 , the server determines the second duration sequence and the text annotation data as sample data.

[0131] Here, the server encodes the frames in the second time sequence and the phonemes in the text annotation data respectively, and concatenates the encoding results to obtain sample data.

[0132] In some embodiments, the training data set also includes first label data; see Figure 7 , Figure 7 This is a flow chart of determining sample data provided by an embodiment of the present application. Figure 7 It is shown that in step S207, the server determines the second time sequence and the text annotation data as sample data, which can be achieved by the following steps S2071 to S2074:

[0133] Step S2071: perform a first encoding on the first tag data to obtain a first encoding value.

[0134] Here, the first encoding refers to a specific encoding method for the first tag. The first encoding can use a simple sort value encoding, that is, encoding the first tag data starting from 1. The first encoding value refers to the data representation in numerical form obtained after the first encoding of the first tag data.

[0135] As an example of step S2071 , assuming that there are 20,000 first tag data in total, the first tag data is encoded by simple sort value encoding, and the obtained first encoding value is an integer within [1, 20000].

[0136] Step S2072: Perform a second encoding on each phoneme in the text annotation data to obtain a second encoding value.

[0137] Here, the second encoding refers to a specific encoding method for each phoneme in the text annotation data. The second encoding can determine the corresponding code for each phoneme using a preset phoneme encoding table, and the corresponding code is determined as the second encoding value of each phoneme. The second encoding value is the numerical data representation of the phoneme after the second encoding.

[0138] As an example of step S2072, taking Chinese as an example, the phonemes in Chinese are the initial consonants and finals with tones in pinyin. Since the number of initial consonants and finals with tones in Chinese is fixed, assuming there are 217 in total, an initial consonant and final vowel encoding table can be set according to the number of initial consonants and finals with tones, as shown in Table 2:

[0139] Table 2 Initial and final coding table

[0140] phoneme coding a1 1 a2 2 … zh 217

[0141] By querying the initial and final coding table, the coding of each phoneme can be determined. For example, if the phoneme list A is [n, i3, h, ao3, m, a5], by querying the initial and final coding table, the corresponding coding of each phoneme in the phoneme list A is determined to be [53, 104, 34, 12, 52, 5]. Then, in order to avoid coding value conflicts between the coding of the phoneme and the third coding value, the coding value of each phoneme is added by 100, and the coding interval corresponding to the phoneme is shifted from [1, 217] to [101, 317]. The second coding value corresponding to each phoneme in the phoneme list A is [153, 204, 134, 112, 152, 105].

[0142] Step S2073: Perform a third encoding on the number of frames in the second time-length sequence to obtain a third encoded value.

[0143] Here, the third encoding refers to a specific encoding method for the frames in the second time sequence. The third encoding value refers to the data representation obtained after the frames are encoded by the third encoding.

[0144] In some embodiments, step S2073 can be implemented by the following method: first, obtain a plurality of preset numerical intervals; each numerical interval corresponds to an interval identifier; then, determine the numerical interval to which each frame number in the second time-length sequence belongs in turn; finally, determine the third coding value based on the interval identifier corresponding to the numerical interval to which each frame number belongs.

[0145] Here, a numerical interval is a set of continuous or discrete numerical ranges, each of which defines a range of numerical values. An interval identifier is a symbol used to uniquely represent a specific numerical interval, typically a number, letter, or other identifier.

[0146] As an example of step S2073, assume that the frame number range of the second time-length sequence is (0, 1); then, the (0, 1) interval is evenly divided into 100 numerical intervals, namely (0, 0.01], (0.01, 0.02], ..., (0.99, 1], and interval identifiers are assigned to these 100 numerical intervals, such as marking these 100 intervals as 1, 2, 3, ..., 100 in sequence; finally, according to the size of each integer in the second time-length list, the numerical interval to which the frame number belongs is determined, and the interval identifier corresponding to the numerical interval is determined as the third coded value of the frame number. For example, if the frame number 1 in the second time-length sequence is 0.03, the interval identifier 3 of the numerical interval (0.02, 0.03] is used as the third coded value of the frame number 1.

[0147] Step S2074: concatenate the first coded value, the second coded value, and the third coded value to obtain sample data.

[0148] Here, splicing refers to the process of merging multiple independent data, vectors, or features to form a complete data. For example, if the first encoding value is 10, the second encoding value is 5, and the third encoding value is 3, the sample data obtained after splicing is [10, 5, 3].

[0149] In some embodiments, step S2074 can be implemented by the following method: first, add a first identifier after the first coding value; then, add a second identifier after the second coding value of each phoneme; then, add a third identifier after the third coding value of the frame number corresponding to each phoneme; then, for each phoneme, splice the second coding value with the second identifier added and the third coding value with the third identifier added to obtain the first spliced ​​data of each phoneme; finally, splice the first coding value with the first identifier added and the first spliced ​​data of each phoneme to obtain sample data.

[0150] Here, the first identifier, the second identifier and the third identifier are additional marks or symbols, wherein the first identifier serves as the terminator of the first encoded data, the second identifier serves as a separator between the second encoded data and the third encoded data, and the third identifier serves as the terminator of the third encoded data.

[0151] As an example of step S2074, for example, the first code value is 3, the second code value of each phoneme includes 101 and 121, the third code value of the frame number corresponding to each phoneme includes 5 and 12, the value of the first identifier is encoded as 320, the value of the second identifier is encoded as 318, and the value of the third identifier is encoded as 319. Then, the first identifier is added after the first code value to obtain [3, 320], the second identifier is added after the second code value of each phoneme to obtain [101, 318] and [121, 318], and the third identifier is added after the third code value to obtain [5, 319] and [12, 319]. Next, for each phoneme, the second code value with the second identifier added and the third code value with the third identifier added are concatenated to obtain the first concatenated data of each phoneme. That is, [101, 318] and [5, 319] are concatenated to obtain the first concatenated data [101, 318, 5, 319], and [121, 318] and [12, 319] are concatenated to obtain the first concatenated data [121, 318, 12, 319]. Finally, [3, 320] is concatenated with the first concatenated data [101, 318, 5, 319] and the first concatenated data [121, 318, 12, 319] to obtain sample data. The concatenated sample data is [3, 320, 101, 318, 5, 319, 121, 318, 12, 319].

[0152] In step S208 , the server trains the first model using the sample data to obtain a pronunciation duration prediction model.

[0153] Here, training refers to the process of gradually adjusting the network parameters in the first model through input sample data to minimize the prediction error. Common training methods include supervised learning, unsupervised learning, and non-autoregressive methods.

[0154] In some embodiments, step S208 can be implemented by the following method: first, input the sample data into the first model to obtain a first prediction sequence; the first prediction sequence includes phoneme prediction values ​​and frame number prediction values; then, a mask operation is performed on the phoneme prediction values ​​in the first prediction sequence to obtain a second prediction sequence; then, a loss calculation is performed based on the second prediction sequence and the second duration sequence to obtain a loss result; finally, the network parameters of the first model are adjusted based on the loss result to obtain a second model.

[0155] Here, the first prediction sequence refers to the prediction results obtained by forward propagating the sample data using the first model. The phoneme prediction value refers to the prediction result of the first model for the second encoded value, and the frame number prediction value refers to the prediction result of the first model for the third encoded value. Masking is a data processing technique typically used to ignore unnecessary or irrelevant data to reduce interference or improve computational efficiency. The second prediction sequence refers to the prediction sequence obtained by masking the phoneme prediction values ​​in the first prediction sequence. The second prediction sequence only includes the frame number prediction values.

[0156] Loss calculation involves calculating the error between the predicted value and the true value. Common loss functions include mean squared error loss and cross entropy loss, though this application does not specify specific loss functions. The loss result is the error value obtained through the loss calculation. The loss result reflects the current predictive performance of the model. The smaller the loss value, the better the model's predictive performance.

[0157] Network parameters refer to the adjustable variables in the first model, including weights and biases. The size of network parameters affects the final output of the model. Network parameters can be optimized based on the loss results. Specifically, network parameters can be adjusted based on the loss results using optimization algorithms such as gradient descent. After adjusting the network parameters, forward propagation of sample data continues based on the adjusted network parameters to obtain the loss results. After multiple iterations of the loop, when the loss results stabilize within a fixed range, training is stopped and the second model, i.e., the pronunciation duration prediction model, is obtained.

[0158] Step S209: The server generates a reminder message indicating that the pronunciation duration prediction model training is successful.

[0159] In step S210 , the server sends a reminder message to the terminal indicating that the pronunciation duration prediction model training is successful.

[0160] In step S211, the terminal displays a reminder message on the current interface indicating that the pronunciation duration prediction model has been successfully trained.

[0161] After determining that the pronunciation duration prediction model is successfully trained, the trained pronunciation duration prediction model can be used to predict the pronunciation duration of each phoneme in the input text.

[0162] Based on the training method of the above-mentioned model, an embodiment of the present application also provides a pronunciation duration prediction method. The pronunciation duration prediction method provided by the embodiment of the present application can be executed by an electronic device, which can be a server or a terminal, or can also be executed through interaction between a server and a terminal.

[0163] Figure 8a : is a schematic diagram of the implementation flow of the pronunciation duration prediction method provided in the embodiment of the present application, such as Figure 8aAs shown, taking the execution subject of the pronunciation duration prediction method as a server as an example for explanation, the pronunciation duration prediction method can be implemented by the following steps S301 to S304:

[0164] Step S301: perform fourth encoding on the second tag data to obtain a fourth encoded value.

[0165] In an embodiment of the present application, a first text and second label data may be obtained; the first text includes a plurality of phonemes. The first text refers to the data to be predicted for which the pronunciation duration of the phonemes needs to be predicted. The second label data refers to the speaker label corresponding to the first text.

[0166] The specific encoding method for the second tag data is described in step S2071 and will not be described in detail in this application.

[0167] Step S302 : performing fifth encoding on the plurality of phonemes in the first text to obtain fifth encoding values.

[0168] The specific encoding method for multiple phonemes in the first text is described in step S2072, and this application will not go into details here.

[0169] Step S303: Generate an input sequence based on the fourth code value and the fifth code value.

[0170] Here, the input sequence refers to the data sequence input into the second model for pronunciation duration prediction.

[0171] In some embodiments, step S303 can be implemented by the following method: first, adding the first identifier after the fourth code value; then, adding the second identifier after the fifth code value of each phoneme; and finally, concatenating the fourth code value with the first identifier and the fifth code value with the second identifier of each phoneme to obtain the input sequence. Regarding the method for adding the first identifier, the second identifier, and the first and second identifiers, as well as the data concatenation process, refer to step S2074, which will not be further described in this application.

[0172] Step S304: input the input sequence into the second model to obtain the pronunciation duration of each phoneme in the first text.

[0173] Here, the second model in step S304 is trained using the model training method of steps S101 to S104, and the second model may be a pronunciation duration prediction model.

[0174] In some embodiments, step S304 can be implemented by the following method: first, forward propagation is performed on the inference sequence in the input sequence to obtain an inference result sequence; wherein the inference sequence includes the fourth encoding value of the added first identifier, and the fifth encoding value of the currently added second identifier; the inference result sequence includes the fourth encoding value of the added first identifier, the fifth encoding value of the added second identifier, and the phoneme prediction value; then, the third identifier is added after the phoneme prediction value; then, the next fifth encoding value of the added second identifier is added to the inference result sequence, and the updated inference result sequence is used as the inference sequence for the next round of forward propagation, and forward propagation is continued; wherein, the next fifth encoding value of the added second identifier is adjacent to the fifth encoding value of the currently added second identifier, and is located after the fifth encoding value of the currently added second identifier.

[0175] Then, after forward propagation of all fifth coded values ​​with the second identifier added, all phoneme prediction values ​​in the final inference result sequence are determined as a phoneme prediction value sequence; next, according to each phoneme prediction value in the phoneme prediction value sequence, the normalized frame number corresponding to each phoneme prediction value is determined (this process is the inverse process of the above-mentioned step S2073, and this application will not repeat it here); finally, according to the maximum frame number and minimum frame number of each phoneme in the phoneme pronunciation time statistics table, each normalized frame number is reverse normalized to obtain the pronunciation frame number of each phoneme, and the pronunciation time of each phoneme is determined based on the pronunciation frame number.

[0176] As an example of step S304 , assume that the input sequence is [12, 320, 204, 318, 101, 318, 134, 318, 112, 318, 152, 318, 105, 318]. First, [12, 320, 204, 318] is used as the inference sequence for forward propagation to obtain the phoneme prediction value [3]. The inference sequence and the phoneme prediction value are spliced, and the third identifier is added after the phoneme prediction value to obtain the inference result sequence [12, 320, 204, 318, 3, 319]. Here, 12 represents the fourth coding value, 320 represents the first identifier, 204 represents the fifth coding value of the first phoneme in the first text, 318 represents the second identifier, and 319 represents the third identifier. Then, the next fifth coding value [101, 318] with the second identifier added is added to the inference result sequence to obtain the updated inference result sequence [12, 320, 204, 318, 3, 319, 101, 318]. The updated inference result sequence is continued to be forward propagated until the inference of all fifth coding values ​​in the input sequence is completed to obtain the final inference result sequence.

[0177] Next, the phoneme prediction values ​​in the final inference result sequence are extracted to obtain a phoneme prediction value sequence. For example, the phoneme prediction value sequence is [3, 36, 6, 78, 14, 56]. According to the inverse process of step S2073 above, the normalized frame number sequence corresponding to the phoneme prediction value sequence is [0.03, 0.36, 0.06, 0.78, 0.14, 0.56]. Finally, based on the maximum and minimum frame numbers of each phoneme in the phoneme pronunciation duration statistics table, each normalized frame number is reverse normalized to obtain the pronunciation frame number of each phoneme. Assuming that the normalized frame number of the first phoneme "t" is 0.03, by querying the phoneme pronunciation duration statistics table, it can be seen that the maximum frame number of the pronunciation duration of the phoneme "t" is 30 frames, and the minimum frame number is 2 frames. The pronunciation frame number of the normalized frame number 0.03 is: 0.03×(30-2)+2=2.84. Since the number of frames is an integer, the calculation result is rounded, and the pronunciation frame number corresponding to the normalized frame number 0.03 is 3 frames; the pronunciation duration of the phoneme "t" can also be calculated based on the pronunciation frame number. Assuming that the specified duration of each frame is 10 milliseconds, the pronunciation duration of the phoneme "t" is 3×10=30 milliseconds.

[0178] The pronunciation duration prediction method provided in the embodiment of the present application adopts the second model obtained by training the model training method provided in the above embodiment to predict the pronunciation duration of each phoneme in the first text. Since the frame number is finely adjusted by preset quantiles during the training of the second model, the occurrence of extreme frame values ​​is avoided, and interference data (such as shorter or longer frames) in the model training process is reduced; and since there is no abnormal sound length in the adjusted second duration sequence, the adjusted second duration sequence is used together with the text annotation data to constitute sample data, which provides high-quality training data for the training of the second model, improves the training effect of the second model, and ensures the accuracy and rationality of the sound length prediction.

[0179] Based on the training method of the above-mentioned model and the above-mentioned pronunciation duration prediction method, an embodiment of the present application also provides a speech generation method. The speech generation method provided in the embodiment of the present application can be executed by an electronic device, which can be a server or a terminal, or can also be executed through interaction between a server and a terminal.

[0180] Figure 8b This is a schematic diagram of the implementation flow of the speech generation method provided in the embodiment of the present application. Figure 8b As shown, the speech generation method is described by taking the server as an example. The speech generation method can be implemented by the following steps S401 to S405:

[0181] Step S401 , performing sixth encoding on the third tag data to obtain a sixth encoded value.

[0182] Step S402: Perform seventh encoding on multiple phonemes in the second text to obtain seventh encoding values.

[0183] Step S403: Generate an input sequence based on the sixth coded value and the seventh coded value.

[0184] Step S404: input the input sequence into the second model to obtain the pronunciation duration of each phoneme in the second text.

[0185] Here, the second model in step S404 is trained using the model training method of steps S101 to S104, and the second model may be a pronunciation duration prediction model.

[0186] It should be noted that the implementation process of steps S401 to S404 is the same as that of the above-mentioned steps S301 to S304, and will not be repeated in the embodiment of this application.

[0187] Step S405: Generate speech corresponding to the second text based on the pronunciation duration.

[0188] In the embodiment of the present application, the speech generation process can be performed using TTS technology. When generating speech, a pre-trained speech synthesis model can be used. The pre-trained speech synthesis model may include, but is not limited to, CosyVoice2 and Tacotron 2. The speech synthesis model can generate natural speech based on the input second text.

[0189] The speech generation method provided in the embodiment of the present application adopts the second model obtained by training the model training method provided in the above embodiment to predict the pronunciation duration of each phoneme in the second text. Since the frame number is finely adjusted by preset quantiles during the training of the second model, the occurrence of extreme frame values ​​is avoided, and the interference data (such as shorter or longer frames) in the model training process is reduced; and since there is no abnormal sound length in the adjusted second time length sequence, the adjusted second time length sequence is used together with the text annotation data to constitute the sample data, which provides high-quality training data for the training of the second model, improves the training effect of the second model, ensures the accuracy and rationality of the sound length prediction, and then based on the accurate pronunciation duration, can generate more natural and realistic speech.

[0190] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0191] The embodiment of the present application is directed to the needs of implementing the non-autoregressive large model TTS technology, and proposes an independently trained pronunciation duration prediction model (i.e., the above-mentioned second model) and a training method. The pronunciation duration prediction model can provide a sound length distribution prediction for the NAR TTS large model, or provide a text total pronunciation duration prediction function. The training method of the pronunciation duration prediction model provided by the embodiment of the present application fundamentally avoids the problem that individual words (or phonemes) may be incompletely pronounced or the wrong prolongation of sounds in the synthesized speech due to incorrect sound length predictions.

[0192] For the convenience of description, all texts in the embodiments of the present application are expressed as phoneme texts. For Mandarin texts, the corresponding phonemes are expressed as initials and finals pinyin texts, in which each initial consonant, or each final with tone, is an independent phoneme. In existing TTS technology, the original content of the text (especially for alphabetic languages ​​such as English), or root words (such as byte pair encoding of text) can be directly input. For large-model TTS training, the input data in existing TTS technology is different from the phoneme data in the embodiments of the present application in terms of the only difference in representation method, and there is no essential difference.

[0193] The following describes in detail the training method of the pronunciation duration prediction model provided by the embodiment of the present application:

[0194] 1) Prepare a first dataset (the aforementioned training dataset). The first dataset consists of a large number of speech samples and their corresponding text annotations (S, T, P). S represents the audio recording (the aforementioned sample speech), T represents the corresponding text (the aforementioned text annotation data corresponding to the sample speech), and P represents the speaker label (the aforementioned first label data). Here, T = (p1, p2, p3, ..., pN), where p represents a phoneme and N represents the length of the phoneme sequence in the text.

[0195] 2) Determine pronunciation duration alignment data for the text units (the aforementioned phonemes) in the first dataset. For example: T_align = (p1:3, p2:5, p3:9, ..., pN:4), where the pronunciation duration of the text unit is represented by the number of speech frames corresponding to the actual speech recording S, i.e., the first representation of the pronunciation duration of the text unit (the aforementioned first duration sequence).

[0196] The method for determining the first representation of the pronunciation duration of the text unit is described in step S205 above, and this application will not elaborate on it here.

[0197] 3) Taking the text unit as the dimension, count the maximum and minimum values ​​(the above-mentioned first number of frames and the second number of frames) of the pronunciation duration (the number of frames corresponding to the above-mentioned pronunciation duration) of each text unit in the first data set, and use these two maximum values ​​to normalize each pronunciation duration to obtain a second representation of the pronunciation duration of the text unit (the above-mentioned second duration sequence).

[0198] The method for determining the second representation of the pronunciation duration of the text unit is described in step S206 above, and this application will not elaborate on it here.

[0199] 4) The text annotations T in the first data set are encoded according to a preset encoding method to obtain a numerical encoding representation of the T sequence, that is, the first representation of the text annotations (the above-mentioned second encoding value).

[0200] 5) Constructing a model structure of a text pronunciation duration prediction model (the above-mentioned first model), and using the first data set, the text numerical representation (the above-mentioned second encoding value), and the pronunciation duration numerical representation (the above-mentioned third encoding value) to train the pronunciation duration prediction model until the model converges, thereby obtaining a trained pronunciation duration prediction model (the above-mentioned pronunciation duration prediction model).

[0201] See also Figure 9 , Figure 9 It is a structural diagram of the pronunciation duration prediction model based on the large language model provided by the embodiment of the present application. First, the first coding value 904 is determined according to the first label data 901 (for the specific determination process, see the above step S2071), multiple phonemes 905 are extracted from the text annotation data 902, and the second numerical code is determined according to the multiple phonemes 905 (for the specific determination process, see the above step S2072), and the second duration sequence 906 is determined according to the sample speech 903 (for the specific determination process, see the above step S206); then, the sample data 906 is determined according to the first coding data 904, the second numerical code and the second duration sequence 906 (for the specific determination process, see step S2074); then, the sample data 906 is input into the large language model 9 08 (the above-mentioned first model), model training is performed (see the above-mentioned step S208 for the training process), and the sample data 906 are forward propagated through the large language model 908 to obtain a first prediction sequence, and the first prediction sequence is masked through the mask layer 909 to obtain a phoneme prediction value sequence; then, the third encoded data 910 is extracted from the sample data, and the loss function is used to calculate the loss of the third encoded data 910 and the phoneme prediction sequence to obtain a loss result; finally, the network parameters of the large language model 908 are adjusted according to the loss result until the loss result stabilizes within a fixed range, and the training is terminated to obtain a pronunciation duration prediction model.

[0202] After the pronunciation duration prediction model is trained, reasoning can be performed using the pronunciation duration prediction model to predict the pronunciation duration of each phoneme in the input text (the first text mentioned above). Specifically:

[0203] First, the text sentence for which the pronunciation duration distribution needs to be predicted (the first text mentioned above) is preprocessed according to the style of the model training input and input into the model (the pronunciation duration prediction model mentioned above) for prediction to obtain the sound duration distribution corresponding to each text unit in the text (the phoneme prediction value mentioned above); then, the predicted sound duration distribution representation is reverse normalized to obtain the number of pronunciation frames corresponding to each text unit (the number of frames corresponding to the pronunciation duration of the phoneme mentioned above).

[0204] In an embodiment of the present application, when the NAR large model requires the sound length distribution of the text unit, the prediction result of the pronunciation duration prediction model can be directly applied; when the NAR large model requires the total sound length of the synthesized text, the number of frames of each text unit predicted by the pronunciation duration prediction model is added to obtain the total sound length.

[0205] In an embodiment of the present application, first, in the training data preparation stage, it is necessary to perform statistical sorting of the number of frames at the phoneme level and perform maximum and minimum value reduction on the first data set. The purpose of doing this is that in real recording samples, the speech comes from various collection methods and the speech quality varies. The speakers of some samples may be unclear, and some speakers speak faster or slower. For the same phoneme, there are many possible distributions of the length of each phoneme in up to tens of millions of samples. However, the length distribution of each phoneme should have a reasonable range of lengths. Therefore, the embodiment of the present application performs statistical sorting on the length of each phoneme and adjusts the length distribution of the phonemes based on quantiles to correct the very few phoneme samples with abnormal pronunciation durations, so that the length distribution of the phoneme falls within a reasonable range, thereby fundamentally solving the problem of abnormal pronunciation of synthesized speech.

[0206] The adjusted pronunciation frames are then normalized. Normalization is performed because each phoneme in Chinese has a different pronunciation function, resulting in variations in the sound length distribution for each phoneme. For example, short initials are generally pronounced for a shorter duration, such as a normal "p" which may be pronounced between 3 and 9 frames, while tonal finals are generally pronounced for a longer duration, such as "eng2" which may be pronounced between 8 and 26 frames. Without normalization, the sound length distributions of different phonemes will vary significantly, resulting in a higher probability of prediction errors after model convergence. Therefore, normalizing the adjusted pronunciation frames improves prediction accuracy after model convergence.

[0207] In addition, the embodiment of the present application uses a large language model (LLM) to construct a pronunciation duration prediction model, and the reason why the traditional classification model is not used to construct the pronunciation duration prediction model is that the sound length prediction method in the embodiment of the present application is mainly used to support the training and reasoning of the NAR TTS model, and the biggest disadvantage of the NAR TTS model is its poor rhythmic effect, that is, the sampling of the generated speech lacks "diversity". One of the technical advantages of the large language model is the diversity of generated samples. The same text input may obtain two different sound length distributions after two inferences using a large language model, but they are both sound length distribution results under the same emotional rhythm. Therefore, using a large language model to construct a pronunciation duration prediction model helps to improve the synthetic rhythmic effect of the NAR TTS model.

[0208] In addition, the role of introducing speaker labels when constructing training samples is: the speaker labels can be used as timbre conditions to further enhance the diversity of the pronunciation duration prediction model in predicting the sound length distribution, and have a timbre bias, so that the characteristics of the sound length distribution are consistent with the speaking characteristics of a specific speaker.

[0209] It is understandable that in the embodiments of the present application, if data related to user information or corporate information is involved, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to blur this information to eliminate the correspondence between this information and the user; and the relevant data collection and processing should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0210] The following continues to describe the exemplary structure of the model training device 354 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 3 As shown, the model training device 354 includes: a first generation module 3541, used to generate a first time sequence based on the pronunciation duration of each phoneme in the text annotation data; the first time sequence includes the number of frames corresponding to the pronunciation duration of each phoneme; an adjustment module 3542, used to adjust the number of frames in the first time sequence to obtain a second time sequence; a determination module 3543, used to determine the second time sequence and the text annotation data as sample data; a training module 3544, used to train the first model through the sample data to obtain a second model.

[0211] In some embodiments, the first generation module is further used to: align the sound length of each phoneme in the text annotation data with the sample speech to obtain the pronunciation start and end time of each phoneme; determine the pronunciation duration of each phoneme based on the pronunciation start and end time; determine the number of frames corresponding to the pronunciation duration of each phoneme based on a preset first duration; sort the number of frames corresponding to the pronunciation duration of the phoneme according to the order of each phoneme in the sample speech to obtain the first duration sequence.

[0212] In some embodiments, the adjustment module is further used to: extract the number of frames of the same phoneme in the first time sequence to obtain a first frame sequence; adjust the number of frames in the first frame sequence to obtain a second frame sequence; obtain the number of frames corresponding to the pronunciation duration of each phoneme in the text annotation data from the second frame sequence; sort the obtained frames according to the order of each phoneme in the sample speech to obtain the second time sequence.

[0213] In some embodiments, the adjustment module is further used to: sort the frames in the first frame number sequence to obtain a third frame number sequence; determine the first frame number and the second frame number in the third frame number sequence based on a preset quantile; and adjust the frame number in the third frame number sequence based on the first frame number and the second frame number to obtain the second frame number sequence.

[0214] In some embodiments, the adjustment module is further used to: modify the number of frames in the third frame number sequence that is smaller than the first frame number to the first frame number; modify the number of frames in the third frame number sequence that is larger than the second frame number to the second frame number, to obtain the second frame number sequence.

[0215] In some embodiments, the determination module is further used to: perform a first encoding on the preset first label data to obtain a first encoding value; perform a second encoding on each phoneme in the text annotation data to obtain a second encoding value; perform a third encoding on the number of frames in the second time sequence to obtain a third encoding value; and splice the first encoding value, the second encoding value, and the third encoding value to obtain the sample data.

[0216] In some embodiments, the determination module is further used to: determine the numerical interval to which each frame number in the second time-length sequence belongs based on a preset numerical interval, and each numerical interval corresponds to an interval identifier; and determine the third coding value based on the interval identifier corresponding to the numerical interval to which each frame number belongs.

[0217] In some embodiments, the determination module is also used to: add a first identifier after the first coding value; add a second identifier after the second coding value of each phoneme; add a third identifier after the third coding value of the frame number corresponding to each phoneme; for each phoneme, splice the second coding value with the second identifier added and the third coding value with the third identifier added to obtain the first spliced ​​data of each phoneme; splice the first coding value with the first identifier added and the first spliced ​​data of each phoneme to obtain the sample data.

[0218] In some embodiments, the training module is further used to: input the sample data into the first model to obtain a first prediction sequence; the first prediction sequence includes phoneme prediction values ​​and frame number prediction values; perform a mask operation on the phoneme prediction values ​​in the first prediction sequence to obtain a second prediction sequence; perform loss calculation based on the second prediction sequence and the second duration sequence to obtain a loss result; adjust the network parameters of the first model based on the loss result to obtain the second model.

[0219] An embodiment of the present application further provides a pronunciation duration prediction device, including: a first encoding module, used to perform a fourth encoding on the second label data to obtain a fourth encoding value; the first encoding module is also used to perform a fifth encoding on multiple phonemes in the first text to obtain a fifth encoding value; a second generation module, used to generate an input sequence based on the fourth encoding value and the fifth encoding value; a first prediction module, used to input the input sequence into a second model to obtain the pronunciation duration of each phoneme in the first text.

[0220] An embodiment of the present application further provides a speech generation device, including: a second encoding module, used to perform a sixth encoding on the third label data to obtain a sixth encoding value; the second encoding module is also used to perform a seventh encoding on multiple phonemes in the second text to obtain a seventh encoding value; a third generation module is used to generate an input sequence based on the sixth encoding value and the seventh encoding value; a second prediction module is used to input the input sequence into a second model to obtain the pronunciation duration of each phoneme in the second text; wherein the second model is trained using the training method of the above-mentioned model; and a speech generation module is used to generate speech corresponding to the second text based on the pronunciation duration.

[0221] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0222] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; a processor for implementing the above-mentioned model training method, or implementing the above-mentioned pronunciation duration prediction method, or implementing the above-mentioned speech generation method when executing the executable instructions stored in the memory.

[0223] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned model training method, or the above-mentioned pronunciation duration prediction method, or the above-mentioned speech generation method is implemented.

[0224] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the model training method, pronunciation duration prediction method, or speech generation method provided in the embodiment of the present application.

[0225] In some embodiments, the storage medium can be a computer-readable storage medium, such as a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPR OM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.

[0226] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0227] As an example, the executable instructions may, but need not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed in multiple locations and interconnected by a communication network.

[0228] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Generate a first duration sequence based on the pronunciation duration of each phoneme in the text annotation data; the first duration sequence includes the number of frames corresponding to the pronunciation duration of each phoneme; Adjusting the number of frames in the first time-length sequence to obtain a second time-length sequence; Determine the second time series and the text annotation data as sample data; The first model is trained using the sample data to obtain a second model.

2. The method according to claim 1, characterized in that The generating of a first duration sequence based on the pronunciation duration of each phoneme in the text annotation data includes: Aligning the length of each phoneme in the text annotation data with the sample speech to obtain the start and end time of pronunciation of each phoneme; Determining the pronunciation duration of each phoneme based on the pronunciation start and end time; Determining the number of frames corresponding to the pronunciation duration of each phoneme based on a preset first duration; According to the order of each phoneme in the sample speech, the frame numbers corresponding to the pronunciation duration of the phoneme are sorted to obtain the first duration sequence.

3. The method according to claim 1, characterized in that The step of adjusting the number of frames in the first time-length sequence to obtain a second time-length sequence includes: Extracting the number of frames of the same phoneme from the first time sequence to obtain a first frame number sequence; Adjusting the number of frames in the first frame sequence to obtain a second frame sequence; Obtaining, from the second frame number sequence, the frame number corresponding to the pronunciation duration of each phoneme in the text annotation data; The acquired frames are sorted according to the order of each phoneme in the sample speech to obtain the second time sequence.

4. The method according to claim 3, characterized in that The step of adjusting the number of frames in the first frame sequence to obtain a second frame sequence includes: sorting the frames in the first frame sequence to obtain a third frame sequence; Determining a first frame number and a second frame number in the third frame number sequence based on a preset quantile; Based on the first frame number and the second frame number, the number of frames in the third frame number sequence is adjusted to obtain the second frame number sequence.

5. The method according to claim 4, characterized in that The step of adjusting the number of frames in the third frame sequence based on the first frame number and the second frame number to obtain the second frame sequence includes: Modify the frame numbers in the third frame number sequence that are smaller than the first frame number to the first frame number; The frame numbers in the third frame number sequence that are greater than the second frame number are modified to the second frame number to obtain the second frame number sequence.

6. The method according to claim 1, characterized in that Determining the second time series and the text annotation data as sample data includes: Performing a first encoding on the preset first tag data to obtain a first encoding value; Performing a second encoding on each phoneme in the text annotation data to obtain a second encoding value; performing a third encoding on the number of frames in the second time-length sequence to obtain a third encoded value; The first coding value, the second coding value, and the third coding value are concatenated to obtain the sample data.

7. The method according to claim 6, characterized in that The performing a third encoding on the number of frames in the second time-length sequence to obtain a third encoded value includes: Based on a preset numerical interval, determining a numerical interval to which each frame number in the second time-length sequence belongs, each numerical interval corresponding to an interval identifier; The third encoding value is determined based on the interval identifier corresponding to the numerical interval to which each frame number belongs.

8. The method according to claim 6, characterized in that The step of concatenating the first coding value, the second coding value, and the third coding value to obtain the sample data includes: adding a first identifier after the first encoded value; adding a second identifier after the second encoded value of each phoneme; adding a third identifier after the third encoded value of the number of frames corresponding to each phoneme; For each phoneme, concatenate the second code value with the second identifier added thereto and the third code value with the third identifier added thereto to obtain first concatenated data of each phoneme; The first encoding value with the first identifier added and the first concatenated data of each phoneme are concatenated to obtain the sample data.

9. The method according to any one of claims 1 to 8, characterized in that The step of training the first model using the sample data to obtain the second model includes: Inputting the sample data into the first model to obtain a first prediction sequence; the first prediction sequence includes a phoneme prediction value and a frame number prediction value; performing a masking operation on the phoneme prediction values ​​in the first prediction sequence to obtain a second prediction sequence; Calculate the loss based on the second prediction sequence and the second duration sequence to obtain a loss result; The network parameters of the first model are adjusted based on the loss result to obtain the second model.

10. A method for predicting pronunciation duration, characterized in that: The method comprises: Performing a fourth encoding on the second tag data to obtain a fourth encoded value; Performing a fifth encoding on the plurality of phonemes in the first text to obtain a fifth encoding value; generating an input sequence based on the fourth encoded value and the fifth encoded value; The input sequence is input into a second model to obtain the pronunciation duration of each phoneme in the first text; wherein the second model is trained using the training method of the model described in any one of claims 1 to 9.

11. A speech generation method, characterized in that: The method comprises: Performing a sixth encoding on the third tag data to obtain a sixth encoded value; Performing seventh encoding on multiple phonemes in the second text to obtain seventh encoding values; generating an input sequence based on the sixth encoded value and the seventh encoded value; Inputting the input sequence into a second model to obtain the pronunciation duration of each phoneme in the second text; wherein the second model is trained using the training method of the model according to any one of claims 1 to 9; A speech corresponding to the second text is generated based on the pronunciation duration.

12. An electronic device, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the model training method described in any one of claims 1 to 9 when executing the executable instructions stored in the memory; or implement the pronunciation duration prediction method described in claim 10; or implement the speech generation method described in claim 11.

13. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the model training method described in any one of claims 1 to 9 is implemented; or the pronunciation duration prediction method described in claim 10 is implemented; or the speech generation method described in claim 11 is implemented.

Images