Data processing method and device in voice generation and electronic equipment

Abstract

The invention provides a speech synthesis method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: determining a phoneme sequence corresponding to a text of a to-be-generated speech, and marking the position of a to-be-added non-language event in the phoneme sequence; according to the phoneme sequence, respectively carrying out frame number occupation prediction and rhythm prediction on each phoneme and the non-language event to obtain frame-level first phoneme feature data added with rhythm information; determining to-be-added frame-level non-linguistic feature data in the to-be-generated voice; and according to the non-linguistic feature data, processing the first phoneme feature data into a voice signal added with the non-linguistic event. Therefore, according to the data processing method in voice generation provided by the embodiment of the invention, voice containing diversified, real and natural non-language events can be generated.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, electronic device and computer-readable storage medium in speech generation. Background Art

[0002] Text-to-Speech (TTS) technology is a technology that converts written text into spoken expression. It aims to enable machines to imitate human speech in a natural and fluent way, thereby achieving high-quality voice communication in human-computer interaction. Although existing TTS technology has made remarkable achievements in terms of natural sound quality and emotional expression, it performs poorly in processing non-verbal events (such as laughter, crying, coughing, etc.). Although these sound elements are not direct components of language information, they are related to whether the generated speech has rich communication levels and emotional expressions. Take laughter as an example. It is not only a natural expression of human emotions, but also reflects the speaker's psychological state and interaction style.

[0003] At present, the common practice in the industry is to represent non-language events as a specific language unit. For example, laughter is represented as a phoneme. The location of laughter is indicated by marking specific language units in the text, and then the corresponding laughter is inserted when generating speech. However, non-language events are diverse and complex. For example, laughter can be divided into smile, sneer, wry smile, and laughter. In this method, non-language events are represented as independent phonemes, resulting in the same non-language event showing the same pronunciation in texts of different situations. The inherent diversity of non-language events is lacking, making the generated speech lack naturalness and authenticity.

[0004] Therefore, how to accurately control and generate diverse non-language events while generating natural speech has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a method, device, electronic device and computer-readable storage medium for speech synthesis, which can generate speech containing diverse, real and natural non-language events. The specific scheme is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for data processing in speech generation, the method comprising:

[0007] Determine a phoneme sequence corresponding to the text of the speech to be generated, wherein the position of the non-language event to be added is marked in the phoneme sequence;

[0008] According to the phoneme sequence, the number of frames occupied and the rhythm prediction are performed for each phoneme and the non-language event respectively, so as to obtain first phoneme feature data with rhythm information added at the frame level;

[0009] Determining frame-level non-language feature data to be added to the speech to be generated;

[0010] The first phoneme feature data is processed into a speech signal into which the non-language event is added according to the non-language feature data.

[0011] In a second aspect, an embodiment of the present application provides a data processing device for speech generation, the device comprising:

[0012] A first determining unit is used to determine a phoneme sequence corresponding to the text of the speech to be generated, wherein the position of the non-language event to be added is marked in the phoneme sequence;

[0013] A prediction unit, configured to predict the number of frames occupied and the rhythm of each phoneme and the non-language event according to the phoneme sequence, and obtain first phoneme feature data with rhythm information added at the frame level;

[0014] A second determining unit, used to determine frame-level non-language feature data to be added to the speech to be generated;

[0015] A processing unit is used to process the first phoneme feature data into a speech signal with the non-language event added thereto according to the non-language feature data.

[0016] In a third aspect, the present application further provides an electronic device, including:

[0017] Processor; and

[0018] The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method described in the first aspect is executed.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a data processing program, which is executed by a processor to perform the method described in the first aspect.

[0020] Compared with the prior art, this application has the following advantages:

[0021] The data processing method for speech generation provided in the embodiment of the present application includes the following steps: determining a phoneme sequence corresponding to the text of the speech to be generated, wherein the position of the non-language event to be added is marked in the phoneme sequence; predicting the number of frames occupied and the rhythm prediction of each phoneme and the non-language event according to the phoneme sequence, and obtaining first phoneme feature data at the frame level with rhythm information added; determining non-language feature data at the frame level to be added to the speech to be generated; and processing the first phoneme feature data into a speech signal with the non-language event added according to the non-language feature data.

[0022] It can be seen that the data processing method in speech generation provided by the embodiment of the present application, since the phoneme sequence includes each phoneme and the position of the non-language event to be added is marked, based on the phoneme sequence, the number of speech frames occupied by each phoneme and the number of speech frames occupied by the non-language event can be predicted, and the rhythm information of the frame can be predicted for each frame, thereby obtaining the first phoneme feature data at the frame level with the rhythm information added; after that, the non-language feature data at the frame level to be added to the speech to be generated is determined, since the non-language feature data is fine-grained, and represents the fine distribution information of the non-language event, so that according to the non-language feature data, the first phoneme feature data can be processed into a speech signal containing a real and natural non-language event. In addition, the non-language feature data can be predicted from the text, or extracted from the preset audio, and can be based on different texts, or different preset audios to express non-language events with different pronunciations, thereby realizing the diversity of non-language events. Therefore, the data processing method in speech generation provided by the embodiment of the present application can generate speech containing diverse, real and natural non-language events. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is an architecture diagram of a speech generation system provided in an embodiment of the present application;

[0024] Figure 2 is a flow chart of a data processing method in speech generation provided in an embodiment of the present application;

[0025] Figure 3 is a structural diagram of a prosody prediction model applied by a data processing method for speech generation provided in an embodiment of the present application;

[0026] Figure 4 It is a structural block diagram of an example of a data processing device for speech generation provided in an embodiment of the present application;

[0027] Figure 5 It is a structural block diagram of an example of an electronic device for data processing provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0029] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variants are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] It should be understood that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "Including A, B and / or C" means including any one, any two, or any three of A, B, and C.

[0031] It should be understood that in the embodiments of the present application, "B corresponding to A", "B corresponding to A", "A corresponds to B", or "B corresponds to A" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0032] Before describing the implementation methods of the present application in detail, the prior art is first further described.

[0033] Text-to-Speech (TTS) technology aims to synthesize natural, understandable speech from text. With the development of deep learning, modern neural network-based TTS models can generate speech signals with highly similar timbre and certain emotions. However, human speech contains not only text-related information, but also non-verbal events such as laughter, crying, and coughing. These non-verbal events can convey various emotions or intentions in communication.

[0034] The following two methods are commonly used in related technologies to generate non-language events during speech synthesis. Method 1: Simplify these non-language events into specific language units (such as phonemes or special markers). Method 2: Use a pre-training-fine-tuning scheme based on a language model, by pre-training the conditional language model on large-scale data and fine-tuning it on small-scale high-quality annotated data to achieve better modeling of non-language events.

[0035] However, in human speech, non-language events are diverse and complex. For example, laughter can be divided into smile, sneer, wry smile, and wild laughter. The above method 1 ignores the diversity and complexity of non-language events themselves. The same non-language event (such as laughter) is pronounced the same in texts of different situations, resulting in the lack of diversity, authenticity, and expressiveness of the generated non-language events. Although the above method 2 can improve the performance of the model to a certain extent, obtaining accurately labeled high-quality fine-tuning data requires a lot of manpower and material resources, which is costly. In addition, the complexity of the model and the demand for computing resources increase the actual difficulty of the technology implementation.

[0036] Therefore, it becomes extremely important to generate speech containing diverse, real and natural non-linguistic events at a low cost.

[0037] Based on the above reasons, in order to generate speech containing diverse, real and natural non-verbal events at a lower cost, the first embodiment of the present application provides a data processing method in speech generation, which is applied to electronic devices. The electronic devices can be desktop computers, laptop computers, mobile phones, tablet computers, electronic watches, etc., or other electronic devices that can perform data processing in speech synthesis, which is not specifically limited in the embodiments of the present application.

[0038] The speech generation method provided in this application can be used for speech generation and intelligent question answering for intelligent customer service, intelligent robots, telephone customer service, education and training, voice replies to players during game interactions and game audio generation, answers to users' questions by life assistants, etc. It can also be used for speech generation in other scenarios, which is not specifically limited in this application.

[0039] Exemplarily, the solution provided in the present application can be applied to intelligent conversations with artificial intelligence voice assistants. Specifically, the data processing method for speech generation provided in the present application can be applied to add contextually appropriate and diverse laughter, crying or coughing sounds to the speech to be answered by the artificial intelligence voice assistant.

[0040] Before introducing the data processing method for speech generation provided in the first embodiment of the present application, the speech generation system architecture applied by the data processing method for speech generation provided in the embodiment of the present application is first introduced. Figure 1As shown, it is an architecture diagram of the speech generation system provided by the embodiment of the present application, and the speech generation system 10 includes an acoustic feature determination module 11 and a non-language feature determination module 12. Among them, the acoustic feature determination module 11 includes a text encoder 111, a prosody prediction model 112, a decoder 113, a non-language event modeling model 116 and a vocoder 117; the non-language feature determination module 12 includes a non-language feature prediction model 121 and a non-language feature extraction model 122. Specifically, the phoneme embedding sequence has a mark of the position of the non-language event to be added, and the text encoder 111 is used to convert the phoneme embedding sequence into a phoneme hidden sequence; the rhythm prediction model 112 is used to predict the duration and rhythm of each phoneme and the duration and rhythm of the non-language event based on the phoneme hidden sequence, so as to convert the phoneme hidden sequence into a first phoneme feature data at the frame level with rhythm information added; the decoder 113 is used to decode the first phoneme feature data to obtain a first acoustic spectrum; the non-language feature prediction model 121 is used to predict the frame-level non-language feature data from the first phoneme feature data; the non-language feature extraction model 122 is used to extract the frame-level non-language feature data from the recording (in the inference stage, the recording is a preset audio); the non-language event modeling model 116 is used to process the first acoustic spectrum based on the non-language feature data predicted by the non-language feature prediction model 121, or based on the non-language feature data extracted by the non-language feature extraction model 122, to obtain a second acoustic spectrum with non-language events added; the vocoder 117 is used to convert the second acoustic spectrum into a speech signal in the time domain. It should be noted that Figure 1 The various models in will be described in detail in the introduction to the first embodiment of this application.

[0041] It should be noted that in the embodiment of the present application, the execution subject of the data processing method in speech generation can be a terminal device or a server, wherein the terminal device can be a local terminal device. The embodiment of the present application does not limit the type of the execution subject.

[0042] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0043] The following, combined Figure 2 and Figure 3 The present invention introduces a method for processing data in speech generation provided in an embodiment of the present application.

[0044] like Figure 2 As shown, the data processing method for speech generation provided in the present application includes the following steps S101 to S104.

[0045] Step S101: determining a phoneme sequence corresponding to a text of speech to be generated, wherein the phoneme sequence is marked with positions of non-language events to be added.

[0046] The text of the speech to be generated may be a piece of news, a broadcast script, a handout, an advertisement, an announcement, a commentary, or a reply text to user input. The text of the speech to be generated may be in Chinese, English, Japanese, Russian, French or other languages.

[0047] The text has a corresponding phoneme sequence, and the position of the non-language event to be added can be marked in the phoneme sequence. The non-language event refers to an event that does not have a corresponding text in the text to be generated. The non-language event may include a laughter event, a crying event, a coughing event, etc. A corresponding event marker can be added at the corresponding position of the phoneme sequence. For example, if the non-language event is a laughter event, the event marker can be <laugh>。

[0048] It can be understood that a phone is the smallest speech unit divided according to the natural attributes of speech. It is divided based on the articulatory movements in a syllable, and one articulatory movement constitutes one phone. For example, the Chinese syllable "啊 (a1)" has only one phone, "代 (dai)" has two phones d and ai4, etc. "a1" represents the first tone of "a", and "ai4" represents the fourth tone of "ai". Therefore, each text can be split into a corresponding sequence of phones.

[0049] Optionally, step S101 can be implemented as follows:

[0050] Obtain the text corresponding to the speech to be generated;

[0051] Mark the position of the non - linguistic event in the text to obtain the marked text;

[0052] Convert the marked text into the sequence of phones.

[0053] In this embodiment, the text corresponding to the speech to be generated can be input into the electronic device by the user or the background, so that the electronic device obtains the text of the speech to be generated.

[0054] When the method provided in this application is applied to an artificial intelligence voice assistant, the user can input a question, which can be a question in voice form or in text form. After receiving the question input by the user, the corresponding response text can be determined, and this response text is the text of the speech to be generated.

[0055] After obtaining the text corresponding to the speech to be generated, the position of the non - linguistic event in this text can be marked to obtain the marked text.

[0056] In a specific implementation, the text corresponding to the speech to be generated can be input into a pre - trained BERT model to add the mark of the position of the non - linguistic event in the text corresponding to the speech to be generated through the pre - trained BERT model, so as to obtain the marked text.

[0057] In another specific implementation, the position of the non - linguistic event in the text corresponding to the speech to be generated can be marked in the way of manual annotation to obtain the marked text.

[0058] It can be understood that the non - linguistic event can be located at the end, the beginning, or any position in the middle of the text. For example, the non - linguistic event is the event of clearing the throat before a section of text, or the event of laughing heartily after the text ends.

[0059] Afterwards, the annotated text can be converted into a phoneme sequence.

[0060] For example, the text of the speech to be generated is "I am so happy today.", and its corresponding phonemes are shown in Table 1:

[0061] Table 1.

[0062]

[0063] In the above Table 1, the non-language event is located at the end of the text of the speech to be generated.

[0064] Step S102: According to the phoneme sequence, prediction of the number of frames occupied and prosody prediction are performed for each phoneme and the non-language event, to obtain first phoneme feature data at the frame level with prosody information added.

[0065] The above phoneme sequence is a text representation at the phoneme level. This step is used to convert the text representation at the phoneme level into first phoneme feature data at the frame level with prosody information added.

[0066] Human language usually contains rhythmic information, which represents the intonation of speech. If the rhythmic information is not considered during speech generation, the generated speech will lack intonation, appear monotonous, extremely mechanical, and not real and natural.

[0067] In this step, the number of frames occupied and the rhythm prediction can be performed for each phoneme contained in the phoneme sequence, and the number of frames occupied and the rhythm prediction can be performed for the non-language event corresponding to the event marker, that is, the number of speech frames corresponding to each phoneme and the number of speech frames corresponding to the non-language event are predicted, and the rhythm information of each frame is predicted.

[0068] Since the phoneme sequence contains phonemes and event markers corresponding to non-language events, based on the predicted number of speech frames and the prosodic information corresponding to each frame, the first phoneme feature data at the frame level with the prosodic information added can be obtained. The first phoneme feature data is frame-level data that integrates the phoneme information and prosodic information of each frame.

[0069] It should be noted that since speech is a quasi-steady-state signal, that is, short-term stable, and this short duration is generally 10-30ms, when processing speech signals, in order to reduce the impact of the overall non-steady state and time variation of the speech signal, the signal is framed. In the embodiment of the present application, by predicting the number of speech frames corresponding to each phoneme and non-language event, and predicting the prosodic information of each frame, a basis is provided for generating a realistic short-term stable speech signal.

[0070] Step S103: determining the frame-level non-language feature data to be added to the speech to be generated.

[0071] The above non-language feature data can represent the fine distribution information of non-language events. The fine distribution information may include but is not limited to the frequency of non-language events, the subdivision types of non-language events, the duration of non-language events (number of voice frames), and the prosody information of non-language events.

[0072] Among them, the frequency of non-verbal events can reflect the state of the speaker. For example, high-frequency laughter indicates that the speaker is in a relaxed and happy state, and low-frequency laughter indicates that the speaker is in a serious state. The subdivision type represents the emotional type corresponding to the non-verbal event. For example, the subdivision type of laughter can include smile, sneer, wry smile, wild laughter, contemptuous smile, awkward smile and other types. The duration of non-verbal events is used to convey different emotional intensities. For example, long-term continuous laughter usually expresses happiness or excitement more strongly than short laughter. The rhythmic information of non-verbal events represents the emotional state of the speaker. For example, high-pitched laughter indicates that the speaker is in a happy and excited state, and low-pitched laughter indicates that the speaker is in a nervous and cramped state.

[0073] In the embodiments of the present application, non-language feature data may be determined in different ways based on different speech generation tasks.

[0074] In an optional specific implementation, step S103 may be implemented by the following steps:

[0075] If the speech to be generated is speech to be generated based on text prediction of non-language events, frame-level non-language feature data is predicted based on the first phoneme feature data.

[0076] In this embodiment, the speech generation task is a task of predicting non-language events based on text. For example, if the text is "I am so happy today.", non-language feature data representing fine distribution information of non-language events can be predicted based on the context information of the text.

[0077] Specifically, the first phoneme feature data may be input into a pre-trained non-language feature prediction model, so that the non-language feature data at the frame level may be output through the non-language feature prediction model.

[0078] The pre-trained non-language feature prediction model is used to predict matching non-language feature data based on text data. When the non-language event is a laughter event, the pre-trained non-language feature prediction model is a laughter prediction model. After the first phoneme feature data at the frame level is input into the pre-trained laughter prediction model, the laughter prediction model can output frame-level laughter feature data, and the frame-level laughter feature data represents the fine distribution information of the laughter.

[0079] In another optional specific implementation, step S103 may be implemented by the following steps:

[0080] If the speech to be generated is the speech to be generated by migrating non-language events in the preset audio, frame-level non-language feature data is extracted from the preset audio.

[0081] In this embodiment, the speech generation task is a task of migrating non-language events in a preset audio. For example, the preset audio is the audio corresponding to "I feel really good today, hahaha!", and non-language feature data representing the fine distribution information of non-language events can be extracted from the audio.

[0082] Specifically, the preset audio may be input into a pre-trained non-language feature extraction model, so that frame-level non-language feature data may be output through the non-language feature extraction model.

[0083] The pre-trained non-language feature extraction model is used to identify and separate features corresponding to laughter from audio signals. When the non-language event is a laughter event, the pre-trained non-language feature extraction model is a laughter extraction model. After the preset audio containing the laughter event is input into the pre-trained laughter extraction model, the laughter extraction model can extract frame-level laughter feature data therefrom, and the frame-level laughter feature data represents the fine distribution information of laughter in the preset audio.

[0084] Optionally, the above step of "extracting frame-level non-language feature data from the preset audio" may include the following steps:

[0085] Extracting initial non-language feature data from the preset audio at a first frame rate;

[0086] According to a second frame rate corresponding to the first phoneme feature data, linear interpolation is performed on non-language vectors corresponding to each sampling frame in the initial non-language feature data to obtain frame-level non-language feature data matching the second frame rate.

[0087] When extracting non-language feature data from a preset audio, the preset audio is usually downsampled to a first frame rate through a convolutional layer, a pooling layer, etc. corresponding to a pre-trained non-language feature extraction model, and the data of a 32-dimensional hidden layer with a frame rate of the first frame rate is extracted through the last layer of the non-language feature extraction model as non-language feature data. In practical applications, the first frame rate is less than the second frame rate corresponding to the first phoneme feature data predicted from the text by the prosody prediction model. In this embodiment, non-language feature data aligned with the first phoneme feature data is obtained by linearly interpolating the non-language vectors corresponding to each sampling frame.

[0088] Specifically, the interpolation point to be added between every two adjacent sampling frames can be determined according to the first frame rate and the second frame rate, and then the non-language vector corresponding to the interpolation point is determined based on the non-language vector corresponding to every two adjacent sampling frames, so that the non-language vectors of every two adjacent sampling frames have a smooth transition. Then, the non-language vector corresponding to the interpolation point is added to the initial non-language feature data, so as to obtain non-language feature data aligned with the first phoneme feature data.

[0089] Exemplarily, the laughter extraction model extracts initial laughter feature data from the reference audio at a frame rate of 10 frames per second (i.e., 10 Hz). The rhythm prediction model predicts the number of frames at a frame rate of 50 frames per second (i.e., 50 Hz). Assuming that the initial laughter feature data is [f1, f2, f3, f4, f5], in order to achieve a frame rate of 50 frames per second, it is necessary to add 4 interpolation points between every two adjacent sampling frames, and interpolate and calculate the non-language vectors of the interpolation points to obtain the laughter feature data [f1, f 12 , f 14 , f 16 , f 18 ,f2,f 22 , f 24 , f 26 , f 28 ,f3,f 32 , f 34 , f 36 , f 38 ,f4,f 42 , f 44 , f 46 , f 48 , f5].

[0090] Step S104: According to the non-language feature data, the first phoneme feature data is processed into a speech signal into which the non-language event is added.

[0091] It should be noted that the number of speech frames and prosody information corresponding to the non-language events in the first phoneme feature data predicted in step S102 are initial and relatively rough information, that is, the first phoneme feature data includes rough information of non-language events and cannot well reflect the fine distribution information of non-language events.

[0092] It should be noted that since the non-language feature data represents the fine distribution information of non-language events, the non-language feature data can be used as an additional non-language condition to finely adjust the first phoneme feature data including the rough information of the non-language events, so that the number of speech frames, prosody information, frequency and subdivision type corresponding to the non-language events are more in line with the context. Fine reconstruction of non-language events through non-language feature data can model the rich spectral details related to non-language events, and through conversion processing, obtain speech containing real and natural non-language events.

[0093] When the non-verbal event is a laughter event, the laughter feature data predicted by a pre-trained laughter prediction model or the laughter feature data extracted by a pre-trained laughter extraction model can be used as additional laughter conditions to reconstruct the first phoneme feature data including rough information of the laughter event and generate speech containing real and natural laughter.

[0094] Optionally, the above step S104 can be implemented by the following steps:

[0095] Decoding the first phoneme feature data to obtain a first acoustic spectrum corresponding to the target text, wherein the first acoustic spectrum reflects rough information of the non-language event;

[0096] Processing the first acoustic spectrum according to the non-language feature data to obtain a second acoustic spectrum into which the non-language event is added, wherein the second acoustic spectrum reflects the fine distribution information of the non-language event;

[0097] The second acoustic spectrum is converted into a speech signal in the time domain.

[0098] The first acoustic spectrum and the second acoustic spectrum are both Mel spectrums, and the Mel spectrum can reflect the acoustic features of the speech signal to be generated.

[0099] It can be understood that the Mel spectrum (Mel spectrum diagram) shows the distribution of speech signals at different frequencies. The Mel scale is based on the perceptual judgment of pitch by listeners who are equidistant from each other. Since the human ear is more sensitive to the difference between low-frequency signals and less sensitive to the difference between high-frequency signals, the human ear will more easily identify the frequencies in the low-frequency band with equal distances and the frequencies in the high-frequency band at normal frequencies. Based on this, the Mel scale is proposed, so that the frequencies in the low-frequency band and the frequencies in the high-frequency band with equal distances on the new scale are the same for the human ear. Therefore, the Mel spectrum is a spectrum image that conforms to the auditory characteristics of the human ear.

[0100] In this embodiment, decoding the first phoneme feature data to obtain the first acoustic spectrum corresponding to the target text means: predicting the acoustic features of each frame according to the first phoneme feature data at the frame level to obtain the first acoustic spectrum. Since the first phoneme feature data is frame-level data that integrates the phoneme information and prosodic information of each frame, and the first phoneme feature data includes rough information of non-language events, the first acoustic spectrum obtained by decoding is a Mel spectrum that reflects the phoneme information and prosodic information of each frame and rough information of non-language events.

[0101] In a specific implementation, the first phoneme feature data may be input into an attached Figure 1 In the decoder 113 shown, the first phoneme feature data is decoded by the decoder, thereby converting the first phoneme feature data into a first acoustic spectrum.

[0102] Since the non-language feature data represents the fine distribution information of non-language events, the first acoustic spectrum is processed by the non-language feature data, and the obtained second acoustic spectrum is a Mel spectrum that can reflect the phoneme information and prosody information of each frame and the fine distribution information of non-language events.

[0103] In a specific implementation, the above step of "processing the first acoustic spectrum according to the non-language feature data to obtain a second acoustic spectrum" may include the following steps:

[0104] The first acoustic spectrum and the non-language feature data are input into a pre-trained non-language event modeling model, so that a second acoustic spectrum in which the non-language event is added is output through the non-language event modeling model.

[0105] In this implementation, the first acoustic spectrum and non-language feature data are input into the attached Figure 1 The non-language event modeling model 116 shown enables the non-language event modeling model to finely reconstruct the non-language events in the first acoustic spectrum based on the non-language feature data, thereby outputting a second acoustic spectrum that can reflect the phoneme information and prosodic information of each frame and the fine distribution information of the non-language events.

[0106] Afterwards, the second acoustic spectrum can be transformed to convert the data in the frequency domain into a speech signal in the time domain (i.e., speech waveform). The speech signal in the time domain focuses on the change of sound in time, which refers to the graphical representation of the change of audio signal over time, and can present the amplitude change of sound on the time axis, and can intuitively show the intensity, periodicity and other time-related characteristics of the sound.

[0107] In a specific implementation, the Mel spectrum can be converted into a speech signal in the time domain through the HIFI-GAN model (a generative adversarial network model for efficient and high-fidelity speech synthesis). The input of HIFI-GAN is the Mel spectrum, which is upsampled through multiple convolutional layers until a speech waveform is output, which is the speech signal in the time domain.

[0108] The data processing method for speech generation provided in the embodiment of the present application includes the following steps: determining a phoneme sequence corresponding to the text of the speech to be generated, wherein the position of the non-language event to be added is marked in the phoneme sequence; predicting the number of frames occupied and the rhythm prediction of each phoneme and the non-language event according to the phoneme sequence, and obtaining first phoneme feature data at the frame level with rhythm information added; determining non-language feature data at the frame level to be added to the speech to be generated; and processing the first phoneme feature data into a speech signal with the non-language event added according to the non-language feature data.

[0109] It can be seen that the data processing method in speech generation provided by the embodiment of the present application, since the phoneme sequence includes each phoneme and the position of the non-language event to be added is marked, based on the phoneme sequence, the number of speech frames occupied by each phoneme and the number of speech frames occupied by the non-language event can be predicted, and the rhythm information of the frame can be predicted for each frame, thereby obtaining the first phoneme feature data at the frame level with the rhythm information added; after that, the non-language feature data at the frame level to be added to the speech to be generated is determined, since the non-language feature data is fine-grained, and represents the fine distribution information of the non-language event, so that according to the non-language feature data, the first phoneme feature data can be processed into a speech signal containing a real and natural non-language event. In addition, the non-language feature data can be predicted from the text, or extracted from the preset audio, and can be based on different texts, or different preset audios to express non-language events with different pronunciations, thereby realizing the diversity of non-language events. Therefore, the data processing method in speech generation provided by the embodiment of the present application can generate speech containing diverse, real and natural non-language events.

[0110] In an optional implementation, the above step S102 may include the following steps S1021 to S1023:

[0111] Step S1021: converting the phoneme sequence into a phoneme embedding sequence, wherein the phoneme embedding sequence contains first feature vectors of a first dimension corresponding to each of the phonemes and the non-language events;

[0112] Step S1022: Encode the phoneme embedding sequence to obtain a phoneme hidden sequence containing context information, wherein the phoneme hidden sequence contains second feature vectors of the second dimension corresponding to each of the phonemes and the non-language events;

[0113] Step S1023: According to the phoneme hidden sequence, the number of frames occupied and the rhythm prediction are performed for each of the phonemes and the non-language events, so as to obtain first phoneme feature data with rhythm information added at the frame level.

[0114] In this embodiment, converting the phoneme sequence into a phoneme embedding sequence is a process of converting the phoneme sequence into a low-dimensional vector sequence, and the low-dimensional vector sequence is the phoneme embedding sequence. In a specific implementation, the phoneme sequence can be input into a pre-trained embedding layer to output a phoneme embedding sequence through the pre-trained embedding layer, wherein each phoneme corresponds to a first feature vector (low-dimensional vector) of a first dimension, and a non-language event also corresponds to a first feature vector of a first dimension.

[0115] Afterwards, you can attach Figure 1 The text encoder 111 shown in encodes the phoneme embedding sequence and converts the low-dimensional vector sequence into a high-dimensional vector sequence, which is the phoneme hidden sequence. In the phoneme hidden sequence, each phoneme corresponds to a second feature vector (high-dimensional vector) of the second dimension, which captures the context information and features of the previous and next phonemes. Therefore, the phoneme hidden sequence contains rich context information and features.

[0116] After obtaining the phoneme hidden sequence, the number of frames occupied by each phoneme and the distribution of non-language events can be predicted and the rhythm can be predicted based on the context information and features contained in the phoneme hidden sequence, so as to predict the number of speech frames occupied by each phoneme and the number of speech frames occupied by non-language events, and predict the rhythmic information frame by frame, and then process the phoneme hidden sequence at the phoneme level that does not contain rhythmic information into the first phoneme feature data at the frame level with rhythmic information added.

[0117] This setting converts the phoneme sequence into a phoneme embedding sequence, thereby uniformly representing each phoneme and non-language event in the phoneme sequence as a first feature vector of the first dimension; then the phoneme embedding sequence is encoded to obtain a phoneme hidden sequence. Since the phoneme hidden sequence contains rich contextual information and features, when predicting the number of frames and rhythm of each phoneme and non-language event, it is easy to capture the relationship between each phoneme and the relationship between phonemes and non-language events, thereby extracting richer features from the phoneme hidden sequence, which provides a good foundation for generating real and natural speech that contains non-language events that fit the situation.

[0118] Optionally, the above step S1023 may be implemented by the following steps:

[0119] The phoneme hidden sequence is input into a pre-trained rhythm prediction model, so that the rhythm prediction model can predict the number of frames occupied and the rhythm of each phoneme and non-language event in the phoneme hidden sequence, and obtain the first phoneme feature data with rhythm information added at the frame level.

[0120] In a specific implementation, the phoneme hidden sequence at the phoneme level can be input into the attached Figure 1 In the pre-trained prosody prediction model 112 shown, the pre-trained prosody prediction model is used to predict the number of frames occupied and the prosody of the phoneme hidden sequence, respectively, to obtain first phoneme feature data at the frame level with prosody information added.

[0121] Optionally, the above step S1023 may include the following steps S20 to S22:

[0122] Step S20: predicting the number of frames occupied by each of the phonemes and the non-language events according to the phoneme hidden sequence, and obtaining second phoneme feature data at the frame level;

[0123] Step S21: performing prosody prediction on each frame according to the second phoneme feature data to obtain prosody information corresponding to each frame;

[0124] Step S22: Process the second phoneme feature data according to the prosody information to obtain frame-level first phoneme feature data with the prosody information added thereto.

[0125] The above step S20 is used to convert the phoneme representation (phoneme hidden sequence) at the phoneme level into the phoneme representation (second phoneme feature data) at the frame level. Specifically, by predicting the number of frames occupied by each phoneme and non-language event, the number of speech frames occupied by each phoneme and non-language event is obtained, thereby converting the phoneme hidden sequence into the second phoneme feature data at the frame level. The second phoneme feature data is frame-level data that integrates phoneme information and context information.

[0126] In an optional specific implementation, the above step S20 may include the following steps S20a and S20b:

[0127] Step S20a: predicting the number of frames occupied by each of the phonemes and the non-language events according to the phoneme hidden sequence, and obtaining the number of speech frames occupied by each of the phonemes and the non-language events;

[0128] Step S20b: Copy each second feature vector according to the corresponding number of speech frames to obtain second phoneme feature data at the frame level.

[0129] In a specific implementation, the above steps S20a to S20b refer to: inputting the phoneme hidden sequence into a pre-trained duration predictor, so as to output the second phoneme feature data at the frame level through the duration predictor.

[0130] like Figure 3 , which is a structural diagram of a prosody prediction model used in a data processing method for speech generation provided in an embodiment of the present application, wherein the prosody prediction model 112 includes a duration predictor 1121 .

[0131] The duration predictor 1121 is used to predict the number of speech frames occupied by each phoneme and non-language event based on the phoneme hidden sequence, and to copy the second feature vector of the second dimension corresponding to each phoneme and non-language event according to the corresponding number of speech frames, thereby converting the phoneme hidden sequence at the phoneme level into the second phoneme feature data at the frame level.

[0132] In an optional specific implementation, the above step S21 may include the following steps S21a and S21b:

[0133] Step S21a: predicting pitch information and energy information corresponding to each frame according to the second phoneme feature data;

[0134] Step S21b: Integrate the pitch information and the energy information belonging to the same frame to obtain rhythm information corresponding to each frame.

[0135] It should be noted that the rhythm information usually includes pitch information and energy information. The pitch information represents the height of the sound, and the energy information represents the strength of the sound.

[0136] In a specific implementation, the above-mentioned step S21a refers to: inputting the second phoneme feature data into a pre-trained pitch predictor to output the pitch information corresponding to each frame through the pitch predictor; and inputting the second phoneme feature data into a pre-trained energy predictor to output the energy information corresponding to each frame through the energy predictor.

[0137] It should be noted that there is no order of execution for the prediction of pitch information and the prediction of energy information. In the specific implementation method, the pitch information of each frame can be predicted first, and then the energy information of each frame can be predicted. The energy information of each frame can also be predicted first, and then the pitch information of each frame can be predicted. The pitch information and energy information of each frame can also be predicted at the same time.

[0138] like Figure 3 As shown, the prosody prediction model 112 includes a pitch predictor 1122 and an energy predictor 1123 .

[0139] Among them, the pitch predictor 1122 is used to predict the pitch value of each frame according to the second phoneme feature data, and embed the pitch value into a pitch embedding vector, which is the above-mentioned pitch information; the energy predictor 1123 is used to predict the energy value of each frame according to the second phoneme feature data, and embed the energy value into an energy embedding vector, which is the above-mentioned energy information.

[0140] Afterwards, the pitch information and energy information belonging to the same frame can be integrated to obtain the rhythmic information corresponding to each frame, and the rhythmic information can reflect the height and strength of the sound of each frame.

[0141] Since the second phoneme feature data is frame-level data that integrates phoneme information and context information, based on the rhythmic information that can accurately reflect the pitch and strength of each frame of sound, the second phoneme feature data can be processed into frame-level first phoneme feature data that can reflect the pitch and strength of each frame of sound.

[0142] The following is a detailed introduction to the training of the prosody prediction model, the non-language feature prediction model, and the non-language event modeling model in the data processing method for speech generation provided in the embodiment of the present application:

[0143] Optionally, the data processing method for speech generation provided in the embodiment of the present application may include the following steps:

[0144] Acquire at least one sample audio, wherein the sample audio includes at least one sample non-language event;

[0145] Determining, based on the sample audio data, a sample phoneme sequence marked with a location of the sample non-language event;

[0146] Training the prosody prediction model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained prosody prediction model; or

[0147] Training the non-language feature prediction model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained non-language feature prediction model; or

[0148] The non-language event modeling model to be trained is trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained non-language event modeling model.

[0149] In this embodiment, the sample audio is a real recording containing sample non-language events. For example, the sample audio may be the audio corresponding to "I am so happy today! Hahaha..." After acquiring the sample audio, each phoneme in the sample audio may be determined to obtain a corresponding sample phoneme sequence, in which the location of the sample non-language event may be marked. In addition, based on the sample audio, the number of real speech frames occupied by each sample phoneme in the sample audio and the number of real speech frames occupied by the sample non-language event in the sample audio may also be determined.

[0150] As shown in Table 2, this is an example table of sample phonemes corresponding to sample phoneme audio in the data processing method for speech generation provided in an embodiment of the present application.

[0151] Table 2.

[0152]

[0153] In the above Table 2, the sample phonemes included in the sample audio are j, in1, t, ian1, zh, en1, k, ai1, x, and in1. The sample non-language event included is at the end, and the sample event is marked as laugh. Among them, the number of speech frames occupied by the sample phoneme j is 4 frames, the number of speech frames occupied by the sample phoneme in1 is 7 frames, the number of speech frames occupied by the sample phoneme t is 4 frames, the number of speech frames occupied by the sample phoneme ian1 is 13 frames, the number of speech frames occupied by the sample phoneme zh is 6 frames, the number of speech frames occupied by the sample phoneme en1 is 24 frames, the number of speech frames occupied by the sample phoneme k is 8 frames, the number of speech frames occupied by the sample phoneme ai1 is 9 frames, the number of speech frames occupied by the sample phoneme x is 9 frames, the number of speech frames occupied by the sample phoneme in1 is 18 frames, and the number of speech frames occupied by the sample non-language event is 18 frames.

[0154] The sample audio data and the sample phoneme sequence may be used to train a prosody prediction model to be trained, a non-language feature prediction model to be trained, or a non-language event modeling model to be trained.

[0155] In an optional specific implementation, the above step of "training the prosody prediction model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained prosody prediction model" may include the following steps:

[0156] Determine the real variance information corresponding to the sample audio, wherein the real variance information includes the real speech frame number corresponding to each sample phoneme and the sample non-language event, and the real prosody information corresponding to each sample phoneme and the sample non-language event;

[0157] Inputting the sample phoneme sequence into a prosody prediction model to be trained, so as to output prediction variance information corresponding to the sample phoneme sequence through the prosody prediction model, wherein the prediction variance information includes the predicted speech frame number corresponding to each of the sample phonemes and the sample non-language event, and the predicted prosody information corresponding to each of the sample phonemes and the sample non-language event;

[0158] The model parameters of the prosody prediction model are adjusted so that the difference between the predicted variance information and the true variance information is smaller than a first threshold, thereby obtaining a pre-trained prosody prediction model.

[0159] In an optional implementation, the duration predictor, pitch predictor and energy predictor included in the prosody prediction model may be trained separately.

[0160] Specifically, the real speech frame numbers corresponding to each sample phoneme and sample non-language event in the sample audio have been determined in the previous introduction. In this step, the real pitch information and real energy information corresponding to each sample phoneme and sample non-language event in the sample audio can be determined.

[0161] Afterwards, the sample phoneme sequence can be embedded, thereby converting the sample phoneme sequence into a sample phoneme embedding sequence, and encoding the sample phoneme embedding sequence to obtain a sample phoneme hidden sequence. Afterwards, the sample phoneme hidden sequence is input into the prosody prediction model to be trained, so as to predict the predicted speech frame number corresponding to each sample phoneme and sample non-language event in the sample phoneme sequence through the duration predictor in the prosody prediction model, and predict the predicted pitch information corresponding to each sample phoneme and sample non-language event in the sample phoneme sequence through the pitch predictor in the prosody prediction model, and predict the predicted energy information corresponding to each sample phoneme and sample non-language event in the sample phoneme sequence.

[0162] When training the duration predictor to be trained, the model parameters of the duration predictor to be trained can be adjusted based on the training strategy that the difference between the predicted speech frame number and the actual speech frame number is within the first preset range, thereby obtaining a pre-trained duration predictor. In this way, the difference between the predicted speech frame number predicted by the pre-trained duration predictor and the actual speech frame number is small, and the speech frame number can be accurately predicted based on the text.

[0163] When training the pitch predictor to be trained, the model parameters of the pitch predictor to be trained can be adjusted based on the training strategy that the difference between the predicted pitch information and the real pitch information is within the second preset range, thereby obtaining a pre-trained pitch predictor. In this way, the difference between the predicted pitch information predicted by the pre-trained pitch predictor and the real pitch information is small, and accurate prediction of pitch information based on text can be performed.

[0164] When training the energy predictor to be trained, the model parameters of the energy predictor to be trained can be adjusted based on the training strategy that the difference between the predicted energy information and the real energy information is within the third preset range, so as to obtain a pre-trained energy predictor. In this way, the difference between the predicted energy information predicted by the pre-trained energy predictor and the real energy information is small, and accurate prediction of energy information based on text can be performed.

[0165] In an optional implementation, the prosody prediction model may be trained as a whole.

[0166] Specifically, the real pitch information and the real energy information can be integrated into the real prosody information, and the real prosody information and the real speech frame number can be determined as the real variance information. The predicted pitch information and the predicted energy information can be integrated into the predicted prosody information, and the predicted prosody information and the predicted speech frame number can be determined as the predicted variance information.

[0167] When training the prosody prediction model to be trained, the model parameters of the prosody prediction model to be trained can be adjusted based on the training strategy that the difference between the predicted variance information and the true variance information is less than the first threshold, thereby obtaining a pre-trained prosody prediction model. In this way, the difference between the predicted variance information predicted by the pre-trained prosody prediction model and the true method information is small, and accurate prediction of prosody information based on text can be performed.

[0168] In an optional specific implementation, the above step of "training the non-language feature prediction model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained non-language feature prediction model" may include the following steps:

[0169] Extracting real non-language feature data at a frame level corresponding to the sample audio;

[0170] According to the sample phoneme sequence, the number of frames occupied and the prosody prediction are performed on each of the sample phonemes and the sample non-language events, respectively, to obtain frame-level predicted phoneme feature data with prosody information added;

[0171] Inputting the predicted phoneme feature data into a non-language feature prediction model to be trained, so as to output frame-level predicted non-language feature data through the non-language feature prediction model;

[0172] The model parameters of the non-language feature prediction model are adjusted so that the difference between the predicted non-language feature data and the true non-language feature data is smaller than a second threshold, thereby obtaining a trained non-language feature prediction model.

[0173] In this embodiment, by attaching Figure 1 The pre-trained non-language feature extraction model 122 in the example extracts the real non-language feature data at the frame level corresponding to the sample audio. The sample phoneme hidden sequence is input into the pre-trained prosody prediction model, so that the number of frames occupied and the prosody prediction are respectively performed on each of the sample phonemes and the sample non-language events through the pre-trained prosody prediction model, and the predicted phoneme feature data at the frame level with the prosody information added is output. Afterwards, the predicted phoneme feature data is input into the non-language feature prediction model to be trained, so that the predicted non-language feature data at the frame level is output through the non-language feature prediction model to be trained.

[0174] Based on the training strategy that the difference between the predicted non-language feature data and the actual non-language feature data is less than the second threshold, the model parameters of the non-language feature prediction model to be trained can be adjusted to obtain a pre-trained non-language feature prediction model. In this way, the difference between the predicted non-language feature data predicted by the pre-trained non-language feature prediction model and the actual non-language feature data is small, and accurate prediction of the non-language feature data can be performed based on the text.

[0175] In a specific implementation, the training of the non-language feature prediction model can be optimized by using the MSE function. In the embodiment of the present application, the optimization function used by the non-language feature prediction model is shown in the following formula (1):

[0176]

[0177] In the above formula (1), L laughte To predict non-linguistic feature data, L ref is the real non-linguistic feature data, L laughter Characterizes the difference between the predicted non-linguistic feature data and the true non-linguistic feature data in the MSE loss function.

[0178] In an optional specific implementation, the above step of "training the non-language event modeling model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained non-language event modeling model" may include the following steps:

[0179] Extracting real non-language feature data at a frame level from the sample audio;

[0180] Determining a real acoustic spectrum corresponding to the sample audio;

[0181] Determine a first predicted acoustic spectrum corresponding to the sample phoneme sequence, wherein the first predicted acoustic spectrum reflects rough information of the sample non-language event;

[0182] Inputting the real non-language feature data and the predicted acoustic spectrum into a non-language event modeling model to be trained, so as to output a second predicted acoustic spectrum through the non-language event modeling model, wherein the first predicted acoustic spectrum reflects the fine distribution information of the sample non-language event;

[0183] The model parameters of the non-language event modeling model are adjusted so that the difference between the second predicted acoustic spectrum and the true acoustic spectrum is less than a third threshold, thereby obtaining a trained non-language event modeling model.

[0184] In the foregoing introduction, Figure 1 The pre-trained non-language feature extraction model 122 in the extracts the real non-language feature data at the frame level corresponding to the sample audio. After that, the real acoustic spectrum can be extracted from the sample audio. For example, a mel-spectrogram can be extracted from the sample audio using a fast Fourier transform at a sampling rate of 16kHz, a frame length of 50ms, and a frame shift of 12.5ms. The mel-spectrogram is the real acoustic spectrum corresponding to the sample audio.

[0185] Afterwards, the first predicted acoustic spectrum corresponding to the sample phoneme sequence can be determined. The step "determining the first predicted acoustic spectrum corresponding to the sample phoneme sequence" specifically refers to: according to the sample phoneme sequence, predicting the number of frames occupied and the prosody prediction of each sample phoneme and the sample non-language event, respectively, to obtain frame-level predicted phoneme feature data with prosody information added; decoding the predicted phoneme feature data to obtain the first predicted acoustic spectrum.

[0186] That is to say, through the Figure 1 The prosody prediction model 112 pre-trained in the above embodiment converts the sample phoneme hidden sequence corresponding to the sample phoneme sequence into predicted phoneme feature data through the attached Figure 1 The decoder 113 in the decoder decodes the predicted phoneme feature data to obtain a first predicted acoustic spectrum. Afterwards, the first predicted acoustic spectrum can be refined and reconstructed by the non-language event modeling model to be trained with reference to the real non-language feature data extracted from the sample audio, thereby outputting a second predicted acoustic spectrum.

[0187] Based on the training strategy that the difference between the second predicted acoustic spectrum and the real acoustic spectrum is less than the third threshold, the model parameters of the non-language event reconstruction model to be trained can be adjusted to obtain a pre-trained non-language event reconstruction model. In this way, the difference between the second predicted acoustic spectrum reconstructed by the pre-trained non-language event reconstruction model and the real acoustic spectrum is small, and the acoustic spectrum containing the fine distribution information of the non-language event can be accurately reconstructed based on the non-language feature data.

[0188] In a specific implementation, the optimization goal of the non-language event reconstruction model may be to maximize the likelihood of the training data. The optimization function used by the non-language event reconstruction model is shown in the following formula (2):

[0189]

[0190] In the above formula (2), L glow The value of the optimization function for reconstructing the model for non-language events. The optimization goal is to minimize this value to obtain better model performance. (i) is the real acoustic spectrum corresponding to the i-th sample audio, is the first predicted acoustic spectrum corresponding to the i-th sample audio, is the real non-language feature data corresponding to the i-th sample audio, N is the total number of sample audios, is the log-likelihood function.

[0191] Corresponding to the data processing method in speech generation provided in the first embodiment of the present application, the second embodiment of the present application further provides a data processing device in speech generation, such as Figure 4 As shown, the data processing device 400 in the speech generation includes:

[0192] The first determining unit 401 is used to determine a phoneme sequence corresponding to the text of the speech to be generated, wherein the position of the non-language event to be added is marked in the phoneme sequence;

[0193] The prediction unit 402 is used to predict the number of frames occupied and the prosody of each phoneme and the non-language event according to the phoneme sequence, so as to obtain first phoneme feature data with prosody information added at the frame level;

[0194] The second determining unit 403 is used to determine the frame-level non-language feature data to be added to the speech to be generated;

[0195] The processing unit 404 is configured to process the first phoneme feature data into a speech signal into which the non-language event is added according to the non-language feature data.

[0196] Optionally, the first determining unit 401 is specifically configured to:

[0197] Get the text corresponding to the speech to be generated;

[0198] Marking the location of the non-language event in the text to obtain a marked text;

[0199] The annotated text is converted into the phoneme sequence.

[0200] Optionally, the prediction unit 402 is specifically configured to:

[0201] Converting the phoneme sequence into a phoneme embedding sequence, wherein the phoneme embedding sequence contains first feature vectors of a first dimension corresponding to each of the phonemes and the non-language events;

[0202] Encoding the phoneme embedding sequence to obtain a phoneme hidden sequence containing context information, wherein the phoneme hidden sequence contains second feature vectors of a second dimension corresponding to each of the phonemes and the non-language events;

[0203] According to the phoneme hidden sequence, the number of frames occupied and the rhythm prediction are performed on each of the phonemes and the non-language events, so as to obtain first phoneme feature data with rhythm information added at the frame level.

[0204] Optionally, the prediction unit 402 is specifically configured to:

[0205] According to the phoneme hidden sequence, the number of frames occupied by each of the phonemes and the non-language events is predicted to obtain second phoneme feature data at the frame level;

[0206] According to the second phoneme feature data, prosody prediction is performed on each frame to obtain prosody information corresponding to each frame;

[0207] The second phoneme feature data is processed according to the prosody information to obtain frame-level first phoneme feature data with the prosody information added thereto.

[0208] Optionally, the prediction unit 402 is specifically configured to:

[0209] According to the phoneme hidden sequence, the number of frames occupied by each of the phonemes and the non-language events is predicted to obtain the number of speech frames occupied by each of the phonemes and the non-language events;

[0210] Each second feature vector is copied according to the corresponding number of speech frames to obtain second phoneme feature data at the frame level.

[0211] Optionally, the prediction unit 402 is specifically configured to:

[0212] Predicting pitch information and energy information corresponding to each frame according to the second phoneme feature data;

[0213] The pitch information and the energy information belonging to the same frame are integrated to obtain the rhythm information corresponding to each frame.

[0214] Optionally, the second determining unit 403 is specifically configured to:

[0215] If the speech to be generated is speech to be generated based on text prediction of non-language events, predicting frame-level non-language feature data according to the first phoneme feature data;

[0216] If the speech to be generated is the speech to be generated by migrating non-language events in the preset audio, frame-level non-language feature data is extracted from the preset audio.

[0217] Optionally, the processing unit 404 is specifically configured to:

[0218] Decoding the first phoneme feature data to obtain a first acoustic spectrum corresponding to the target text, wherein the first acoustic spectrum reflects rough information of the non-language event;

[0219] Processing the first acoustic spectrum according to the non-language feature data to obtain a second acoustic spectrum, wherein the second acoustic spectrum reflects the detailed distribution information of the non-language event;

[0220] The second acoustic spectrum is converted into a speech signal in the time domain.

[0221] Optionally, the second determining unit 403 is specifically configured to:

[0222] Extracting initial non-language feature data from the preset audio at a first frame rate;

[0223] According to a second frame rate corresponding to the first phoneme feature data, linear interpolation is performed on non-language vectors corresponding to each sampling frame in the initial non-language feature data to obtain frame-level non-language feature data matching the second frame rate.

[0224] Optionally, the prediction unit 402 is specifically configured to:

[0225] The phoneme hidden sequence is input into a pre-trained rhythm prediction model, so that the rhythm prediction model can predict the number of frames occupied and the rhythm of each phoneme and non-language event in the phoneme hidden sequence, and obtain the first phoneme feature data with rhythm information added at the frame level.

[0226] Optionally, the second determining unit 403 is specifically configured to:

[0227] The first phoneme feature data is input into a pre-trained non-language feature prediction model, so as to output frame-level non-language feature data through the non-language feature prediction model.

[0228] Optionally, the processing unit 404 is specifically configured to:

[0229] The first acoustic spectrum and the non-language feature data are input into a pre-trained non-language event modeling model to output a second acoustic spectrum through the non-language event modeling model.

[0230] Optionally, the second determining unit 403 is specifically configured to:

[0231] The preset audio is input into a pre-trained non-language feature extraction model to output frame-level non-language feature data through the non-language feature extraction model.

[0232] Optionally, the data processing device for speech generation further includes a training unit, wherein the training unit is used to:

[0233] Acquire at least one sample audio, wherein the sample audio includes at least one sample non-language event;

[0234] Determining, based on the sample audio data, a sample phoneme sequence marked with a location of the sample non-language event;

[0235] Training the prosody prediction model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained prosody prediction model; or

[0236] Training the non-language feature prediction model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained non-language feature prediction model; or

[0237] The non-language event modeling model to be trained is trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained non-language event modeling model.

[0238] Optionally, the training unit is specifically used for:

[0239] Determine the real variance information corresponding to the sample audio, wherein the real variance information includes the real speech frame number corresponding to each sample phoneme and the sample non-language event, and the real prosody information corresponding to each sample phoneme and the sample non-language event;

[0240] Inputting the sample phoneme sequence into a prosody prediction model to be trained, so as to output prediction variance information corresponding to the sample phoneme sequence through the prosody prediction model, wherein the prediction variance information includes the predicted speech frame number corresponding to each of the sample phonemes and the sample non-language event, and the predicted prosody information corresponding to each of the sample phonemes and the sample non-language event;

[0241] The model parameters of the prosody prediction model are adjusted so that the difference between the predicted variance information and the true variance information is smaller than a first threshold, thereby obtaining a pre-trained prosody prediction model.

[0242] Optionally, the training unit is specifically used for:

[0243] Extracting real non-language feature data at a frame level corresponding to the sample audio;

[0244] According to the sample phoneme sequence, the number of frames occupied and the prosody prediction are performed on each of the sample phonemes and the sample non-language events, respectively, to obtain frame-level predicted phoneme feature data with prosody information added;

[0245] Inputting the predicted phoneme feature data into a non-language feature prediction model to be trained, so as to output frame-level predicted non-language feature data through the non-language feature prediction model;

[0246] The model parameters of the non-language feature prediction model are adjusted so that the difference between the predicted non-language feature data and the true non-language feature data is smaller than a second threshold, thereby obtaining a trained non-language feature prediction model.

[0247] Optionally, the training unit is specifically used for:

[0248] Extracting real non-language feature data at a frame level from the sample audio;

[0249] Determining a real acoustic spectrum corresponding to the sample audio;

[0250] Determine a first predicted acoustic spectrum corresponding to the sample phoneme sequence, wherein the first predicted acoustic spectrum reflects rough information of the sample non-language event;

[0251] Inputting the real non-language feature data and the predicted acoustic spectrum into a non-language event modeling model to be trained, so as to output a second predicted acoustic spectrum through the non-language event modeling model, wherein the first predicted acoustic spectrum reflects the fine distribution information of the sample non-language event;

[0252] The model parameters of the non-language event modeling model are adjusted so that the difference between the second predicted acoustic spectrum and the true acoustic spectrum is less than a third threshold, thereby obtaining a trained non-language event modeling model.

[0253] Corresponding to the data processing method in speech generation provided in the first embodiment of the present application, the third embodiment of the present application also provides an electronic device for data processing in speech generation.

[0254] like Figure 5 , which is a structural block diagram of an example of an electronic device for data processing provided in an embodiment of the present application.

[0255] In this embodiment, an optional hardware structure of the electronic device 500 can be as follows: Figure 5 As shown, it includes: at least one processor 501, at least one memory 502 and at least one communication bus 505; the memory 502 contains a program 503 and data 504.

[0256] The bus 505 may be a communication device for transmitting data between components inside the electronic device 500, such as an internal bus (e.g., a CPU-memory bus, where the processor is a central processing unit, CPU for short), an external bus (e.g., a universal serial bus port, a peripheral component interconnect express port), etc.

[0257] In addition, the electronic device also includes: at least one network interface 506, at least one peripheral interface 507. The network interface 506 provides wired or wireless communication related to an external network 508 (for example, the Internet, an intranet, a local area network, a mobile communication network, etc.); in some embodiments, the network interface 506 may include any number of network interface controllers (NICs), radio frequency (RF) modules, repeaters, transceivers, modems, routers, gateways, any combination of wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (NFC) adapters, cellular network chips, etc.

[0258] The peripheral interface 507 is used to connect to the peripheral device, and the peripheral device can be the peripheral device 1 ( Figure 5 509), peripheral 2 ( Figure 5 510) and peripheral 3 ( Figure 5 511 in the figure). Peripherals are peripheral devices, which may include but are not limited to cursor control devices (such as mice, touch pads or touch screens), keyboards, displays (such as cathode ray tube displays, liquid crystal displays), displays or light emitting diode displays, video input devices (such as cameras or communication interfaces coupled to video files), etc.

[0259] The processor 501 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0260] The memory 502 may include a high-speed RAM (full name: Random Access Memory) memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0261] The processor 501 calls the program and data stored in the memory 502 to perform the following steps:

[0262] Determine a phoneme sequence corresponding to the text of the speech to be generated, wherein the position of the non-language event to be added is marked in the phoneme sequence;

[0263] According to the phoneme sequence, the number of frames occupied and the rhythm prediction are performed for each phoneme and the non-language event respectively, so as to obtain first phoneme feature data with rhythm information added at the frame level;

[0264] Determining frame-level non-language feature data to be added to the speech to be generated;

[0265] The first phoneme feature data is processed into a speech signal into which the non-language event is added according to the non-language feature data.

[0266] Corresponding to the data processing method in speech generation provided in the first embodiment of the present application, the fourth embodiment of the present application provides a computer-readable storage medium storing a program of the data processing method in speech generation, the program is executed by a processor to perform the following steps:

[0267] Determine a phoneme sequence corresponding to the text of the speech to be generated, wherein the position of the non-language event to be added is marked in the phoneme sequence;

[0268] According to the phoneme sequence, the number of frames occupied and the rhythm prediction are performed for each phoneme and the non-language event respectively, so as to obtain first phoneme feature data with rhythm information added at the frame level;

[0269] Determining frame-level non-language feature data to be added to the speech to be generated;

[0270] The first phoneme feature data is processed into a speech signal into which the non-language event is added according to the non-language feature data.

[0271] It should be noted that for the detailed description of the devices, electronic devices and computer-readable storage media provided in the second, third and fourth embodiments of the present application, reference can be made to the relevant description of the first embodiment of the present application, and no further details will be given here.

[0272] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0273] In a typical configuration, a node device in a blockchain includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0274] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0275] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), random access memory (RAM) of other attributes, read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage media or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0276] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0277] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.< / laugh>

Claims

1. A data processing method in speech generation, characterized in that: The method comprises: Determine a phoneme sequence corresponding to the text of the speech to be generated, wherein the position of the non-language event to be added is marked in the phoneme sequence; According to the phoneme sequence, the number of frames occupied and the rhythm prediction are performed for each phoneme and the non-language event respectively, so as to obtain first phoneme feature data with rhythm information added at the frame level; Determining frame-level non-language feature data to be added to the speech to be generated; The first phoneme feature data is processed into a speech signal into which the non-language event is added according to the non-language feature data.

2. The method according to claim 1, characterized in that The step of determining a phoneme sequence corresponding to the text of the speech to be generated comprises: Get the text corresponding to the speech to be generated; Marking the location of the non-language event in the text to obtain a marked text; The annotated text is converted into the phoneme sequence.

3. The method according to claim 1, characterized in that The method of predicting the number of frames occupied and the rhythm of each phoneme and the non-language event according to the phoneme sequence to obtain first phoneme feature data at the frame level with rhythm information added includes: Converting the phoneme sequence into a phoneme embedding sequence, wherein the phoneme embedding sequence contains first feature vectors of a first dimension corresponding to each of the phonemes and the non-language events; Encoding the phoneme embedding sequence to obtain a phoneme hidden sequence containing context information, wherein the phoneme hidden sequence contains second feature vectors of a second dimension corresponding to each of the phonemes and the non-language events; According to the phoneme hidden sequence, the number of frames occupied and the rhythm prediction are performed on each of the phonemes and the non-language events, so as to obtain first phoneme feature data with rhythm information added at the frame level.

4. The method according to claim 3, characterized in that The method of predicting the number of frames occupied and the rhythm of each of the phonemes and the non-language events according to the phoneme hidden sequence to obtain first phoneme feature data at the frame level with rhythm information added includes: According to the phoneme hidden sequence, the number of frames occupied by each of the phonemes and the non-language events is predicted to obtain second phoneme feature data at the frame level; According to the second phoneme feature data, prosody prediction is performed on each frame to obtain prosody information corresponding to each frame; The second phoneme feature data is processed according to the prosody information to obtain frame-level first phoneme feature data with the prosody information added thereto.

5. The method according to claim 4, characterized in that The method of predicting the number of frames occupied by each of the phonemes and the non-language events according to the phoneme hidden sequence to obtain second phoneme feature data at the frame level includes: According to the phoneme hidden sequence, the number of frames occupied by each of the phonemes and the non-language events is predicted to obtain the number of speech frames occupied by each of the phonemes and the non-language events; Each second feature vector is copied according to the corresponding number of speech frames to obtain second phoneme feature data at the frame level.

6. The method according to claim 4, characterized in that The step of performing prosody prediction on each frame according to the second phoneme feature data to obtain prosody information corresponding to each frame includes: Predicting pitch information and energy information corresponding to each frame according to the second phoneme feature data; The pitch information and the energy information belonging to the same frame are integrated to obtain the rhythm information corresponding to each frame.

7. The method according to claim 1, characterized in that The step of determining the frame-level non-language feature data to be added to the speech to be generated includes: If the speech to be generated is speech to be generated based on text prediction of non-language events, predicting frame-level non-language feature data according to the first phoneme feature data; If the speech to be generated is the speech to be generated by migrating non-language events in the preset audio, frame-level non-language feature data is extracted from the preset audio.

8. The method according to claim 1, characterized in that The step of processing the first phoneme feature data into a speech signal into which the non-language event is added according to the non-language feature data comprises: Decoding the first phoneme feature data to obtain a first acoustic spectrum corresponding to the target text, wherein the first acoustic spectrum reflects rough information of the non-language event; Processing the first acoustic spectrum according to the non-language feature data to obtain a second acoustic spectrum, wherein the second acoustic spectrum reflects the detailed distribution information of the non-language event; The second acoustic spectrum is converted into a speech signal in the time domain.

9. The method according to claim 7, characterized in that: The step of extracting frame-level non-language feature data from the preset audio includes: Extracting initial non-language feature data from the preset audio at a first frame rate; According to a second frame rate corresponding to the first phoneme feature data, linear interpolation is performed on non-language vectors corresponding to each sampling frame in the initial non-language feature data to obtain frame-level non-language feature data matching the second frame rate.

10. The method according to claim 3, characterized in that: The method of predicting the number of frames occupied and the rhythm of each of the phonemes and the non-language events according to the phoneme hidden sequence to obtain first phoneme feature data at the frame level with rhythm information added includes: The phoneme hidden sequence is input into a pre-trained rhythm prediction model, so that the rhythm prediction model can predict the number of frames occupied and the rhythm of each phoneme and non-language event in the phoneme hidden sequence, and obtain the first phoneme feature data with rhythm information added at the frame level.

11. The method according to claim 7, characterized in that The step of predicting the non-language feature data at the frame level according to the first phoneme feature data includes: The first phoneme feature data is input into a pre-trained non-language feature prediction model, so as to output frame-level non-language feature data through the non-language feature prediction model.

12. The method according to claim 8, characterized in that The step of processing the first acoustic spectrum according to the non-language feature data to obtain a second acoustic spectrum includes: The first acoustic spectrum and the non-language feature data are input into a pre-trained non-language event modeling model to output a second acoustic spectrum through the non-language event modeling model.

13. The method according to claim 7, characterized in that The step of extracting frame-level non-language feature data from the preset audio includes: The preset audio is input into a pre-trained non-language feature extraction model to output frame-level non-language feature data through the non-language feature extraction model.

14. The method according to any one of claims 10 to 12, characterized in that: The method further comprises: Acquire at least one sample audio, wherein the sample audio includes at least one sample non-language event; Determining, based on the sample audio data, a sample phoneme sequence marked with a location of the sample non-language event; Training the prosody prediction model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained prosody prediction model; or Training the non-language feature prediction model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained non-language feature prediction model; or The non-language event modeling model to be trained is trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained non-language event modeling model.

15. The method according to claim 14, characterized in that The step of training the prosody prediction model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained prosody prediction model comprises: Determine the real variance information corresponding to the sample audio, wherein the real variance information includes the real speech frame number corresponding to each sample phoneme and the sample non-language event, and the real prosody information corresponding to each sample phoneme and the sample non-language event; Inputting the sample phoneme sequence into a prosody prediction model to be trained, so as to output prediction variance information corresponding to the sample phoneme sequence through the prosody prediction model, wherein the prediction variance information includes the predicted speech frame number corresponding to each of the sample phonemes and the sample non-language event, and the predicted prosody information corresponding to each of the sample phonemes and the sample non-language event; The model parameters of the prosody prediction model are adjusted so that the difference between the predicted variance information and the true variance information is smaller than a first threshold, thereby obtaining a pre-trained prosody prediction model.

16. The method according to claim 14, characterized in that The step of training the non-language feature prediction model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained non-language feature prediction model comprises: Extracting real non-language feature data at a frame level corresponding to the sample audio; According to the sample phoneme sequence, the number of frames occupied and the prosody prediction are performed on each of the sample phonemes and the sample non-language events, respectively, to obtain frame-level predicted phoneme feature data with prosody information added; Inputting the predicted phoneme feature data into a non-language feature prediction model to be trained, so as to output frame-level predicted non-language feature data through the non-language feature prediction model; The model parameters of the non-language feature prediction model are adjusted so that the difference between the predicted non-language feature data and the true non-language feature data is smaller than a second threshold, thereby obtaining a trained non-language feature prediction model.

17. The method according to claim 14, characterized in that The method of training the non-language event modeling model to be trained according to the sample audio data and the sample phoneme sequence to obtain a pre-trained non-language event modeling model includes: Extracting real non-language feature data at a frame level from the sample audio; Determining a real acoustic spectrum corresponding to the sample audio; Determine a first predicted acoustic spectrum corresponding to the sample phoneme sequence, wherein the first predicted acoustic spectrum reflects rough information of the sample non-language event; Inputting the real non-language feature data and the predicted acoustic spectrum into a non-language event modeling model to be trained, so as to output a second predicted acoustic spectrum through the non-language event modeling model, wherein the first predicted acoustic spectrum reflects the fine distribution information of the sample non-language event; The model parameters of the non-language event modeling model are adjusted so that the difference between the second predicted acoustic spectrum and the true acoustic spectrum is less than a third threshold, thereby obtaining a trained non-language event modeling model.

18. A data processing device for speech generation, characterized in that: The device comprises: A first determining unit is used to determine a phoneme sequence corresponding to the text of the speech to be generated, wherein the position of the non-language event to be added is marked in the phoneme sequence; A prediction unit, configured to predict the number of frames occupied and the rhythm of each phoneme and the non-language event according to the phoneme sequence, and obtain first phoneme feature data with rhythm information added at the frame level; A second determining unit, used to determine frame-level non-language feature data to be added to the speech to be generated; A processing unit is used to process the first phoneme feature data into a speech signal with the non-language event added thereto according to the non-language feature data.

19. An electronic device, characterized in that: include: processor; as well as The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method according to any one of claims 1 to 17 is executed.

20. A computer-readable storage medium, characterized in that: A data processing program is stored, and the program is run by a processor to execute the method according to any one of claims 1 to 17.

Images