Speech processing method, device, equipment and computer readable storage medium

Abstract

The embodiment of the present invention provides a speech processing method, which includes: obtaining original speech, original text matching the original speech, and target text corresponding to the original text; determining the final duration information of the speech frame corresponding to the phoneme in the target text, the final duration information including the duration information of the speech frame corresponding to the changed part of the target text relative to the original text; filling the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain the target spectrum; using the target spectrum as rhythmic reference information, combining the final duration information and the target text to perform speech synthesis processing to obtain the target speech. By applying the technical solution provided by the embodiment of the present invention, the rhythmic information of the original speech can be fully utilized to ensure the continuity of the target speech in the auditory sense, so that the rhythm of the entire speech is more natural. The embodiment of the present invention also provides a speech processing device, equipment and storage medium with corresponding technical effects.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a speech processing method, device, equipment and computer-readable storage medium. Background Art

[0002] In video presentation or TV series production, voice adjustment is often required. For example, in a video presentation, after the video to be presented is prepared in advance, if the corresponding materials of the video are repeatedly modified, the narration voice in the video needs to be adjusted multiple times. For another example, in a TV series production, if a completed TV series clip needs to be modified, the dubbing of the corresponding part also needs to be adjusted.

[0003] Currently, a commonly used speech processing method is to synthesize the speech that needs to be adjusted through speech synthesis technology, and then insert or replace the adjusted speech into the original speech.

[0004] This method only synthesizes the part of speech that needs to be adjusted, and inserts or replaces the adjusted speech into the original speech, which can easily make the entire speech sound discontinuous and the rhythm unnatural. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a speech processing method, device, equipment and computer-readable storage medium to achieve the continuity of the entire speech in terms of auditory perception, so that the processed speech rhythm is more natural. The specific technical solution is as follows:

[0006] A speech processing method, comprising:

[0007] Obtaining an original speech to be processed, an original text matching the original speech, and a target text corresponding to the original text, wherein the target text is a text obtained by modifying the original text;

[0008] Determine final duration information of speech frames corresponding to the phonemes in the target text, wherein the final duration information includes duration information of speech frames corresponding to the changed portion of the target text relative to the original text;

[0009] Filling the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain a target spectrum;

[0010] The target spectrum is used as prosodic reference information, and speech synthesis processing is performed in combination with the final duration information and the target text to obtain the target speech.

[0011] In a specific implementation of the present invention, the determining of the final duration information of the speech frame corresponding to the phoneme in the target text includes:

[0012] Determine the actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech;

[0013] Based on the actual duration information of the speech frames corresponding to the phonemes in the original text, predicting the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text;

[0014] The final duration information of the speech frame corresponding to the phoneme in the target text is determined according to the actual duration information and the duration information of the speech frame corresponding to the changed part.

[0015] In a specific implementation of the present invention, the step of predicting the duration information of the speech frame corresponding to the changed portion of the target text relative to the original text based on the actual duration information of the speech frame corresponding to the phoneme in the original text includes:

[0016] Inputting the phoneme sequence of the target text into a first duration prediction model obtained by pre-training, and obtaining initial duration information of speech frames corresponding to the phonemes in the target text through the first duration prediction model;

[0017] Based on the actual duration information of the speech frames corresponding to the phonemes in the original text and the initial duration information of the speech frames corresponding to the phonemes in the target text, the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text is predicted.

[0018] In a specific implementation of the present invention, the method of predicting the duration information of the speech frame corresponding to the changed part of the target text relative to the original text based on the actual duration information of the speech frame corresponding to the phoneme in the original text and the initial duration information of the speech frame corresponding to the phoneme in the target text includes:

[0019] Comparing the actual duration information of the speech frames corresponding to the phonemes in the original text with the initial duration information of the speech frames corresponding to the phonemes in the target text, and determining the ratio between the initial duration information and the actual duration information of the speech frames corresponding to the same part of the target text relative to the original text;

[0020] According to the proportional relationship, the duration information of the speech frame corresponding to the changed part of the target text relative to the original text is predicted.

[0021] In a specific implementation of the present invention, the step of predicting the duration information of the speech frame corresponding to the changed portion of the target text relative to the original text based on the actual duration information of the speech frame corresponding to the phoneme in the original text includes:

[0022] Inputting the phoneme sequence in the original text into a pre-trained second duration prediction model, and obtaining predicted duration information of speech frames corresponding to the phonemes in the original text through the second duration prediction model;

[0023] Retraining the second duration prediction model according to a comparison result of actual duration information and predicted duration information of speech frames corresponding to the phonemes in the original text;

[0024] The phoneme sequence of the target text is input into the retrained second duration prediction model to predict the duration information of the speech frame corresponding to the changed part of the target text relative to the original text.

[0025] In a specific implementation of the present invention, determining the actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech includes:

[0026] Performing alignment processing on the original speech and the original text;

[0027] According to the alignment relationship, actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech is determined.

[0028] In a specific implementation of the present invention, the filling process is performed on the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain the target spectrum, including:

[0029] If the changed part is newly added content, then according to the duration information of the speech frame corresponding to the changed part, in the original spectrum of the original speech, a spectrum insertion process is performed at the position corresponding to the changed part to obtain a target spectrum;

[0030] If the changed part is a modified content, then according to the duration information of the speech frame corresponding to the changed part, in the original spectrum of the original speech, a spectrum replacement process is performed at the position corresponding to the changed part to obtain a target spectrum.

[0031] In a specific implementation of the present invention, the step of obtaining the target speech includes:

[0032] Based on the voiceprint, the newly synthesized speech obtained after the speech synthesis processing is converted into a target speech with a target timbre.

[0033] In a specific implementation of the present invention, the target spectrum is used as prosody reference information, combined with the final duration information and the target text to perform speech synthesis processing to obtain the target speech, including:

[0034] Using the target spectrum as prosodic reference information and extracting a prosodic representation of the prosodic reference information;

[0035] Mapping each phoneme in the target text to a vector space to obtain a vector representation of the target text;

[0036] Based on the final duration information, performing expansion processing on the vector representation of the target text;

[0037] The prosodic representation and the vector representation of the target text after expansion are concatenated and converted into the target speech after transformation.

[0038] A speech processing device, comprising:

[0039] A data acquisition module, used to obtain an original speech to be processed, an original text matching the original speech, and a target text corresponding to the original text, wherein the target text is a text obtained by modifying the original text;

[0040] A final duration information determination module, used to determine the final duration information of the speech frame corresponding to the phoneme in the target text, wherein the final duration information includes the duration information of the speech frame corresponding to the changed part of the target text relative to the original text;

[0041] A target spectrum acquisition module, used for performing filling processing on the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain the target spectrum;

[0042] The target speech acquisition module is used to use the target spectrum as prosody reference information, combine the final duration information and the target text to perform speech synthesis processing, and obtain the target speech.

[0043] An electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0044] The memory is used to store computer programs;

[0045] The processor is used to implement the steps of the above-mentioned speech processing method when executing the program stored in the memory.

[0046] A computer-readable storage medium stores a computer program, which implements the above-mentioned speech processing method when executed by a processor.

[0047] A computer program product includes computer instructions stored in a computer-readable storage medium and suitable for being read and executed by a processor, so that a computer device having the processor executes the above-mentioned speech processing method.

[0048] By applying the technical solution provided in the embodiment of the present invention, the final duration information of the speech frame corresponding to the phoneme in the target text is first determined, and then the original spectrum of the original speech is filled according to the duration information of the speech frame corresponding to the changed part of the target text relative to the original text to obtain the target spectrum, and finally the target spectrum is used as the prosodic reference information, combined with the final duration information and the target text for speech synthesis processing to obtain the target speech. By using the target spectrum obtained after filling the original spectrum of the original speech as the prosodic reference information, combined with the final duration information of the speech frame corresponding to the phoneme in the determined target text and the target text for speech synthesis processing, the prosodic information of the original speech can be fully utilized, the continuity of the target speech in the auditory sense can be ensured, and the rhythm of the entire speech can be made more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0050] Figure 1 is a flow chart of an implementation of a speech processing method in an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of a specific example of a process for determining actual duration information of a speech frame corresponding to an original text in an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of a specific example of a process for predicting final duration information of a speech frame corresponding to a target text in an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of a specific example of a speech synthesis process in an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of a specific example of a speech conversion process in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the structure of a speech processing device according to an embodiment of the present invention;

[0056] Figure 7 The figure is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0058] The core of the present invention is to provide a speech processing method, which can be applied to various scenarios that require speech adjustment, such as video display scenarios, film and television drama production scenarios, etc. The original text that matches the original speech is modified to obtain a target text. The target text has a changed part relative to the original text. Through the technical solution provided by the embodiment of the present invention, the original spectrum of the original speech can be filled according to the duration information of the speech frame corresponding to the changed part, and the target spectrum obtained after filling is used as rhythmic reference information. Combined with the final duration information of the speech frame corresponding to the phoneme in the target text and the target text, speech synthesis processing can be performed, which can make full use of the rhythmic information of the original speech, ensure the continuity of the target speech in the auditory sense, and make the rhythm of the entire speech more natural.

[0059] See also Figure 1 FIG. 1 is a flowchart of a speech processing method according to an embodiment of the present invention. The method may include the following steps:

[0060] S110: Obtaining the original speech to be processed, the original text matching the original speech, and the target text corresponding to the original text.

[0061] In the embodiment of the present invention, the original speech is the speech to be processed, that is, the speech to be modified, the original text is the text matching the original speech, and the target text is the text after the original text is modified.

[0062] After obtaining the original speech to be processed, the original text matching the original speech, and the target text corresponding to the original text, the subsequent steps can be continued.

[0063] S120: Determine final duration information of speech frames corresponding to phonemes in the target text.

[0064] The final duration information includes duration information of speech frames corresponding to the changed portion of the target text relative to the original text.

[0065] It can be considered that both the original text and the target text are composed of phonemes. Phonemes can be speech units defined according to actual conditions. For example, the phonemes of "I" can include "w" and "o", or "wo".

[0066] After obtaining the target text, the final duration information of the speech frame corresponding to the phoneme in the target text can be further determined. For example, the final duration information of the speech frame corresponding to the phoneme in the target text can be determined using a duration prediction model.

[0067] The target text is the text corresponding to the original text. By comparing the target text with the original text, the changed parts and the same parts of the target text relative to the original text can be identified. The duration information of the speech frames corresponding to the changed parts can be predicted using a duration prediction model to obtain the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text. The duration information of the speech frames corresponding to the same parts of the target text relative to the original text can remain unchanged. In this way, the final duration information of the speech frames corresponding to the phonemes in the target text can also be obtained.

[0068] For example, if the original text is "I play ball" and the target text is "I like to play ball", by comparison, it can be known that the changed part of the target text relative to the original text is "like", and the same part of the target text relative to the original text is "I play ball". The final duration information of the speech frames corresponding to the phonemes in the determined target text includes the duration information of the speech frames corresponding to the changed part and the duration information of the speech frames corresponding to the same part, specifically:

[0069] I: [0, 2), indicating that the phoneme "I" corresponds to the speech frames of the 0th and 1st frames, a total of 2 frames;

[0070] Xi: [2, 4), indicating that the phoneme "Xi" corresponds to the speech frames of the 2nd and 3rd frames, a total of 2 frames;

[0071] Huan: [4, 6), indicating that the phoneme "Huan" corresponds to the speech frames of the 4th and 5th frames, a total of 2 frames;

[0072] Da: [6, 10), indicating that the phoneme "Da" corresponds to the speech frames from the 6th to the 9th frames, a total of 4 frames;

[0073] Qiu: [10, 13), indicating that the phoneme "Qiu" corresponds to the speech frames from the 10th to the 12th frames, a total of 3 frames.

[0074] S130: According to the duration information of the speech frames corresponding to the changed part, perform a filling process on the original spectrum of the original speech to obtain the target spectrum.

[0075] After determining the final duration information of the speech frames corresponding to the phonemes in the target text, the duration information of the speech frames corresponding to the changed part of the target text relative to the original text can be obtained. According to the duration information of the speech frames corresponding to the changed part, a filling process can be performed on the original spectrum of the original speech. For example, spectra with the same duration as the speech frames corresponding to the changed part can be inserted or replaced at the positions corresponding to the changed part to obtain the target spectrum. It can be made that the length of the obtained target spectrum is the same as the total duration of the speech frames of the target speech matching the target text.

[0076] S140: Using the target spectrum as prosodic reference information, combined with the final duration information and the target text, to perform speech synthesis processing to obtain the target speech.

[0077] In an embodiment of the present invention, the original spectrum of the original speech is filled according to the duration information of the speech frame corresponding to the changed part of the target text relative to the original text. After the target spectrum is obtained, the target spectrum can be used as prosodic reference information, and then the prosodic reference information is combined with the final duration information of the speech frame corresponding to the phonemes in the target text and the target text for speech synthesis processing to obtain the target speech. The target speech matches the target text and can replace the original speech.

[0078] By using the target spectrum as prosodic reference information, the prosodic information of the original speech is fully utilized to ensure the continuity of the target speech in terms of auditory perception, making the rhythm of the entire speech more natural.

[0079] By applying the method provided in the embodiment of the present invention, the final duration information of the speech frame corresponding to the phoneme in the target text is first determined, and then the original spectrum of the original speech is filled according to the duration information of the speech frame corresponding to the changed part of the target text relative to the original text to obtain the target spectrum, and finally the target spectrum is used as the prosodic reference information, combined with the final duration information and the target text for speech synthesis processing to obtain the target speech. By using the target spectrum obtained after filling the original spectrum of the original speech as the prosodic reference information, combined with the final duration information of the speech frame corresponding to the phoneme in the determined target text and the target text for speech synthesis processing, the prosodic information of the original speech can be fully utilized, the continuity of the target speech in the auditory sense can be ensured, and the rhythm of the entire speech can be made more natural.

[0080] In one embodiment of the present invention, determining the final duration information of the speech frame corresponding to the phoneme in the target text may include the following steps:

[0081] Step 1: Determine the actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech;

[0082] Step 2: Based on the actual duration information of the speech frames corresponding to the phonemes in the original text, the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text is predicted;

[0083] Step 3: Determine the final duration information of the speech frame corresponding to the phoneme in the target text according to the actual duration information and the duration information of the speech frame corresponding to the changed part.

[0084] For the convenience of description, the above three steps are combined for explanation.

[0085] The original text is the text that matches the original speech. Each speech frame of the original speech has its own duration. The actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech can be determined first. For example, if the original text is "I play ball", the actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech can be:

[0086] I: [0,2), 2 frames in total;

[0087] Play: [2,6), 4 frames in total;

[0088] Ball: [6,9), 3 frames in total.

[0089] After determining the actual duration information of the speech frames corresponding to the original speech corresponding to the phonemes in the original text, the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text can be predicted based on the actual duration information of the speech frames corresponding to the phonemes in the original text. For example, the duration information of the speech frames corresponding to the changed parts can be predicted using a duration prediction model.

[0090] The duration information of the speech frames corresponding to the same part of the target text relative to the original text can remain unchanged, so the duration information of the speech frames corresponding to the same part can be obtained based on the actual duration information of the speech frames corresponding to the phonemes in the original text. The final duration information of the speech frames corresponding to the phonemes in the target text can be determined by combining the duration information of the speech frames corresponding to the changed part with the duration information of the speech frames corresponding to the same part.

[0091] By using the duration information of the context of the changing part, the duration information of the speech frame corresponding to the predicted changing part can be made more accurate, ensuring that the temporal distribution of the speech in the changing part is similar to that of the overall speech, and effectively avoiding the speech in the changing part being too long or too short.

[0092] In one embodiment of the present invention, determining the actual duration information of the speech frame of the original speech corresponding to the phoneme in the original text may include the following steps:

[0093] The first step: align the original speech and the original text;

[0094] The second step: According to the alignment relationship, determine the actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech.

[0095] For the convenience of description, the above two steps are combined for explanation.

[0096] After obtaining the original speech and the original text that matches the original speech, the original speech and the original text can be aligned first. Specifically, a speech alignment model can be used to perform speech alignment on the original speech and the original text to obtain the alignment relationship between the original speech and the original text. The speech alignment is an alignment on time slices. For example, in the original text "I play ball", the phoneme "I" in the original speech starts at 0.0 seconds and ends at 0.2 seconds, lasting for 200 milliseconds, which is aligned with this 200-millisecond time slice. The phoneme "play" in the original speech starts at 0.2 seconds and ends at 0.6 seconds, lasting for 400 milliseconds, which is aligned with this 400-millisecond time slice. The phoneme "ball" in the original speech starts at 0.6 seconds and ends at 0.9 seconds, lasting for 300 milliseconds, which is aligned with this 300-millisecond time slice. The speech alignment model can use existing open-source models, such as Montreal Forced Aligner, or can be trained based on pre-obtained sample pairs including speech and text.

[0097] According to the alignment relationship, the actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech can be determined.

[0098] As Figure 2 shown, by inputting the original speech and the original text "I play ball" into the speech alignment model, the actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech can be obtained, that is:

[0099] I: [0, 2), a total of 2 frames;

[0100] play: [2, 6), a total of 4 frames;

[0101] ball: [6, 9), a total of 3 frames.

[0102] That is, after the phoneme "I" in the original text "I play ball" is aligned with the time slice from 0.0 seconds to 0.2 seconds of the original speech, the phoneme "play" is aligned with the time slice from 0.2 seconds to 0.6 seconds of the original speech, and the phoneme "ball" is aligned with the time slice from 0.6 seconds to 0.9 seconds of the original speech. Assuming that 100 milliseconds is one frame, it can be determined that the phoneme "I" corresponds to the speech frames of the 0th and 1st frames, a total of 2 frames, the phoneme "play" corresponds to the speech frames of the 2nd to 5th frames, a total of 4 frames, and the phoneme "ball" corresponds to the speech frames of the 6th to 8th frames, a total of 3 frames.

[0103] By performing alignment processing on the original speech and the original text, the actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech can be accurately determined, providing a basic guarantee for subsequent operations.

[0104] In one implementation of the present invention, based on the actual duration information of the speech frames corresponding to the phonemes in the original text, predicting the duration information of the speech frames corresponding to the changed part of the target text relative to the original text may include the following steps:

[0105] Step 1: Input the phoneme sequence of the target text into the first duration prediction model obtained by pre-training, and obtain the initial duration information of the speech frame corresponding to the phoneme in the target text through the first duration prediction model;

[0106] Step 2: Based on the actual duration information of the speech frames corresponding to the phonemes in the original text and the initial duration information of the speech frames corresponding to the phonemes in the target text, the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text is predicted.

[0107] For the convenience of description, the above two steps are combined for explanation.

[0108] In an embodiment of the present invention, a first duration prediction model may be pre-constructed. The structure of the first duration prediction model may be: input layer (input phoneme) -> embedding layer (embedding) -> encoding layer (Encoder) -> convolution layer (conv1d) -> deep neural network (dnn) -> output layer (output duration information), wherein the encoding layer may be a multi-layer multi-head attention structure.

[0109] The first duration prediction model may be trained using pre-acquired phoneme sequence samples.

[0110] For example, for the phoneme sequence sample: Tian qih en h ao, the actual duration information of the speech frames corresponding to each phoneme are: 4, 5, 3, 2, 1, 3, 3, 3. Input the phoneme sequence sample into the first duration prediction model, and the entire structure changes as follows:

[0111] 1x8---(embedding)-->1x8x256----(encoder)-->1x8x512---conv1d-->1x8x128--dnn-->1x8, 1 represents batch.

[0112] The output of the first duration prediction model is compared with the actual duration information of the speech frame corresponding to each phoneme of the phoneme sequence sample. According to the comparison result, the parameters of the first duration prediction model can be adjusted. When the output accuracy of the first duration prediction model reaches the set accuracy threshold, the training can be stopped and the trained first duration prediction model can be used for duration prediction.

[0113] After obtaining the target text corresponding to the original text, the phoneme sequence of the target text can be input into a pre-trained first duration prediction model, and the initial duration information of the speech frame corresponding to the phonemes in the target text can be obtained through the first duration prediction model.

[0114] Based on the actual duration information of the speech frames corresponding to the phonemes in the original text and the initial duration information of the speech frames corresponding to the phonemes in the target text, the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text can be predicted.

[0115] Specifically, the actual duration information of the speech frames corresponding to the phonemes in the original text and the initial duration information of the speech frames corresponding to the phonemes in the target text can be compared to determine the proportional relationship between the initial duration information and the actual duration information of the speech frames corresponding to the same part of the target text relative to the original text, and then the duration information of the speech frames corresponding to the changed part of the target text relative to the original text can be predicted based on the proportional relationship.

[0116] After determining the actual duration information of the speech frames corresponding to the phonemes in the original text and obtaining the initial duration information of the speech frames corresponding to the phonemes in the target text through the first duration prediction model, the two pieces of information can be compared to obtain the initial duration information of the speech frames corresponding to the same part of the target text relative to the original text, and the actual duration information of the speech frames corresponding to the same part in the original text, and determine the proportional relationship between the initial duration information and the actual duration information of the speech frames corresponding to the same part.

[0117] For example, Figure 3 As shown in the figure, the actual duration information of the speech frame corresponding to the phonemes in the original text "I play ball" is:

[0118] I: [0,2), 2 frames in total;

[0119] Play: [2,6), 4 frames in total;

[0120] Ball: [6,9), 3 frames in total.

[0121] The initial duration information of the speech frame corresponding to the phoneme in the target text "I like to play ball" obtained by the first duration prediction model is:

[0122] I: [0,4), 4 frames in total;

[0123] Hi: [4,7), 4 frames in total;

[0124] Huan: [7,10), 4 frames in total;

[0125] Hit: [10,17), 8 frames in total;

[0126] Ball: [17,23), 6 frames in total.

[0127] The same part of the target text with respect to the original text is "I play ball", the initial duration of the speech frame corresponding to the same part in the target text is 18 frames, and the actual duration of the speech frame corresponding to the same part in the original text is 9 frames, and the ratio between the two is 18 / 9=2.

[0128] According to the determined proportional relationship between the initial duration information of the speech frames corresponding to the same part in the target text and the actual duration information of the speech frames corresponding to the same part in the original text, the duration information of the speech frames corresponding to the changed part of the target text relative to the original text can be predicted.

[0129] It is understandable that the target text has the same part and the changed part relative to the original text, and the duration information of the speech frames corresponding to the same part should be the same. Therefore, the duration of the speech frames corresponding to the changed part of the target text relative to the original text can be adjusted according to the determined proportional relationship, and the initial duration of the speech frame corresponding to each phoneme in the changed part can be divided by the proportional relationship to predict the duration information of the speech frame corresponding to the changed part.

[0130] Still taking the above example, the determined proportional relationship is 2. According to this proportional relationship, the duration of the speech frames corresponding to the changed parts "like" and "happy" of the target text relative to the original text can be adjusted to 4 / 2=2 frames and 4 / 2=2 frames respectively.

[0131] The initial duration information of the speech frames corresponding to the phonemes in the target text is first obtained through the first duration prediction model, and then based on the actual duration information of the speech frames corresponding to the phonemes in the original text and the initial duration information of the speech frames corresponding to the phonemes in the target text, the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text can be accurately predicted.

[0132] In another implementation of the present invention, based on the actual duration information of the speech frames corresponding to the phonemes in the original text, predicting the duration information of the speech frames corresponding to the changed part of the target text relative to the original text may include the following steps:

[0133] The first step: inputting the phoneme sequence of the original text into the pre-trained second duration prediction model, and obtaining the predicted duration information of the speech frame corresponding to the phoneme in the original text through the second duration prediction model;

[0134] The second step: retraining the second duration prediction model according to the comparison result of the actual duration information and the predicted duration information of the speech frame corresponding to the phoneme in the original text;

[0135] The third step: input the phoneme sequence of the target text into the retrained second duration prediction model to predict the duration information of the speech frame corresponding to the changed part of the target text relative to the original text.

[0136] For the convenience of description, the above three steps are combined for explanation.

[0137] In an embodiment of the present invention, a second duration prediction model can be pre-constructed, and the second duration prediction model can be trained based on the pre-acquired phoneme sequence sample. When the output accuracy of the second duration prediction model reaches a set accuracy threshold, the training can be stopped, and the second duration prediction model can be applied to the actual scene. The structure and training process of the second duration prediction model can refer to the description of the first duration prediction model, and will not be repeated here.

[0138] After obtaining the original speech and the original text, and determining the actual duration information of the speech frames of the original speech corresponding to the phonemes in the original text, the phoneme sequence in the original text can be input into a second duration prediction model obtained by pre-training, and the predicted duration information of the speech frames corresponding to the phonemes in the original text can be obtained through the second duration prediction model.

[0139] The actual duration information and predicted duration information of the speech frames corresponding to the phonemes in the original text may be compared. If the prediction accuracy of the second duration prediction model is high enough, the predicted duration information of the speech frames corresponding to the phonemes in the original text obtained by the second duration prediction model should be the same or relatively similar to the actual duration information of the speech frames corresponding to the phonemes in the original text. If the comparison result indicates that the two are quite different, it can be considered that the prediction accuracy of the current second duration prediction model is insufficient, and the second duration prediction model may be retrained using the phoneme sequence of the original text until the predicted duration information of the speech frames corresponding to the phonemes in the original text output by the retrained second duration prediction model is the same as the actual duration information of the speech frames corresponding to the phonemes in the original text or is within a set deviation range.

[0140] The phoneme sequence of the target text is input into the retrained second duration prediction model to predict the duration information of the speech frame corresponding to the changed part of the target text relative to the original text, and the duration information will be more accurate.

[0141] It should be noted that, in the embodiment of the present invention, the duration information of the speech frame corresponding to the changed part of the target text relative to the original text can be obtained in a variety of ways, and then the duration information obtained in a variety of ways is weighted averaged, so that the duration information of the speech frame corresponding to the changed part finally obtained is more accurate. In addition, in unsupervised speech learning, intermediate or future information can be predicted based on the context, and the information predicted by the unsupervised learning model can be added to the first duration prediction model or the second duration prediction model, so that the prediction result of the duration information is more accurate.

[0142] In one embodiment of the present invention, performing filling processing on the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain the target spectrum may include the following steps:

[0143] If the changed part is new content, then according to the duration information of the speech frame corresponding to the changed part, the spectrum is inserted into the position corresponding to the changed part in the original spectrum of the original speech to obtain the target spectrum;

[0144] If the changed part is a modified content, then according to the duration information of the speech frame corresponding to the changed part, a spectrum replacement process is performed at the position corresponding to the changed part in the original spectrum of the original speech to obtain a target spectrum.

[0145] The changed part of the target text relative to the original text can be the new content based on the original text, or the modified content based on the original text.

[0146] The final duration information of the speech frame corresponding to the phoneme in the determined target text includes the duration information of the speech frame corresponding to the changed part. If the changed part is a newly added content, the spectrum can be inserted into the position corresponding to the changed part in the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain the target spectrum. For example, the target text is obtained by adding new content at A of the original text, and in the original spectrum of the original speech, a spectrum with the same duration as the speech frame corresponding to the newly added content can be inserted at the position corresponding to A to obtain the target spectrum.

[0147] If the changed part is a modified content, the spectrum replacement processing can be performed at the position corresponding to the changed part in the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain the target spectrum. For example, the content is modified at BC of the original text to obtain the target text, and in the original spectrum of the original speech, the spectrum with the same duration as the speech frame corresponding to the modified content can be replaced at the position corresponding to BC to obtain the target spectrum.

[0148] Based on the different situations of the changed part, the original spectrum of the original speech is filled according to the duration information of the speech frame corresponding to the changed part to obtain the target spectrum, providing a basic guarantee for its use as prosodic reference information.

[0149] It should be noted that when the original spectrum of the original speech is filled, the spectrum of the set frequency can be filled, such as a full zero spectrum. Using a full zero spectrum for filling processing helps to retain the rhythm of other parts, so that the rhythm of other parts of the synthesized speech is basically consistent with the original.

[0150] In one embodiment of the present invention, obtaining the target speech may include the following steps:

[0151] Based on the voiceprint, the newly synthesized speech obtained after the speech synthesis processing is converted into a target speech with a target timbre.

[0152] In the embodiment of the present invention, Figure 4 As shown in FIG. 1 , the original spectrum of the original speech is filled according to the duration information of the speech frame corresponding to the changed part of the target text relative to the original text. After obtaining the target spectrum, the target spectrum can be used as prosodic reference information, combined with the final duration information of the speech frame corresponding to the phoneme in the target text and the target text for speech synthesis processing, and a new synthesized speech can be obtained. Furthermore, the new synthesized speech can be converted into a target speech with a target timbre based on the voiceprint, such as Figure 5 The target timbre may be the timbre of the original speech, so that the target speech can maintain a high level of timbre similarity and naturalness.

[0153] In one embodiment of the present invention, using the target spectrum as prosody reference information, combining the final duration information and the target text to perform speech synthesis processing to obtain the target speech may include the following steps:

[0154] Step 1: taking the target spectrum as prosodic reference information and extracting the prosodic representation of the prosodic reference information;

[0155] Step 2: Map each phoneme in the target text to the vector space to obtain the vector representation of the target text;

[0156] Step 3: Based on the final duration information, expand the vector representation of the target text;

[0157] Step 4: Concatenate the prosodic representation and the vector representation of the expanded target text, and convert them into the target speech after transformation.

[0158] For the convenience of description, the above steps are combined for explanation.

[0159] After obtaining the original speech to be processed, the original text matching the original speech, and the target text corresponding to the original text, the final duration information of the speech frame corresponding to the phoneme in the target text is determined, and the final duration information includes the duration information of the speech frame corresponding to the changed part of the target text relative to the original text. According to the duration information of the speech frame corresponding to the changed part, the original spectrum of the original speech is filled to obtain the target spectrum. The target spectrum is used as prosodic reference information, and the prosodic representation of the prosodic reference information can be extracted. Specifically, the prosodic reference information can be convolved and subjected to RNN (Recurrent Neural Network) and other operations to extract the prosodic representation, such as extracting a 32-dimensional prosodic representation.

[0160] Through training and inference, each phoneme in the target text can be mapped to the vector space to obtain the vector representation of the target text. The vector space includes vector representations of several phonemes, and each phoneme can be described by the coefficients in the vector representation. For example, the vector space includes vector representations of phonemes such as "A", "B", "C", "D", and "E". The target text includes three phonemes: A, D, and C. By mapping these three phonemes to the vector space, the vector representations of the phonemes "A", "D", and "C" can be extracted, thereby obtaining the vector representation of the target text, such as a 3x256 matrix.

[0161] The final duration information of the speech frame corresponding to the phoneme in the target text has been determined before. Based on the final duration information, the vector representation of the target text can be expanded. For example, the final duration information of the speech frame corresponding to the phoneme "A" included in the target text is 1 frame, the final duration information of the speech frame corresponding to the phoneme "D" is 4 frames, and the final duration information of the speech frame corresponding to the phoneme "C" is 3 frames. Based on the final duration information, the vector representation of the target text is expanded to obtain the vector representation corresponding to "ADDDDCCC", such as an 8x256 matrix.

[0162] The prosodic representation and the vector representation of the expanded target text are concatenated, and then converted into the target speech through some transformations, such as convolution and attention layer operations. For example, concatenating the 32-dimensional prosodic reference information and the 8x256 vector representation can obtain a matrix of 8x(256+32)=8x288, which can then be converted into the target speech through operations such as convolution.

[0163] Using the target spectrum as prosodic reference information and combining it with the final duration information and the target text for speech synthesis processing can ensure that the target speech obtained is more natural and similar to the emotion of the original speech.

[0164] Corresponding to the above method embodiment, the embodiment of the present invention further provides a speech processing device. The speech processing device described below and the speech processing method described above can be referred to each other.

[0165] See also Figure 6 As shown, the speech processing device may include the following modules:

[0166] The data acquisition module 610 is used to obtain the original speech to be processed, the original text matching the original speech, and the target text corresponding to the original text, where the target text is the text after modifying the original text;

[0167] The final duration information determination module 620 is used to determine the final duration information of the speech frame corresponding to the phoneme in the target text, and the final duration information includes the duration information of the speech frame corresponding to the changed part of the target text relative to the original text;

[0168] The target spectrum acquisition module 630 is used to fill the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain the target spectrum;

[0169] The target speech acquisition module 640 is used to use the target spectrum as prosodic reference information, combine the final duration information and the target text to perform speech synthesis processing, and obtain the target speech.

[0170] By using the device provided by the embodiment of the present invention, the final duration information of the speech frame corresponding to the phoneme in the target text is first determined, and then the original spectrum of the original speech is filled according to the duration information of the speech frame corresponding to the changed part of the target text relative to the original text to obtain the target spectrum, and finally the target spectrum is used as the prosodic reference information, combined with the final duration information and the target text for speech synthesis processing to obtain the target speech. By using the target spectrum obtained after filling the original spectrum of the original speech as the prosodic reference information, combined with the final duration information of the speech frame corresponding to the phoneme in the determined target text and the target text for speech synthesis processing, the prosodic information of the original speech can be fully utilized, the continuity of the target speech in the auditory sense can be ensured, and the rhythm of the entire speech can be made more natural.

[0171] In a specific implementation of the present invention, the final duration information determination module 620 is used to:

[0172] Determine the actual duration information of the speech frame of the original speech corresponding to the phoneme in the original text;

[0173] Based on the actual duration information of the speech frames corresponding to the phonemes in the original text, the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text is predicted;

[0174] The final duration information of the speech frame corresponding to the phoneme in the target text is determined according to the actual duration information and the duration information of the speech frame corresponding to the changed part.

[0175] In a specific implementation of the present invention, the final duration information determination module 620 is used to:

[0176] Inputting the phoneme sequence of the target text into a first duration prediction model obtained by pre-training, and obtaining initial duration information of speech frames corresponding to the phonemes in the target text through the first duration prediction model;

[0177] Based on the actual duration information of the speech frames corresponding to the phonemes in the original text and the initial duration information of the speech frames corresponding to the phonemes in the target text, the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text is predicted.

[0178] In a specific implementation of the present invention, the final duration information determination module 620 is used to:

[0179] Comparing the actual duration information of the speech frames corresponding to the phonemes in the original text with the initial duration information of the speech frames corresponding to the phonemes in the target text, and determining the ratio between the initial duration information and the actual duration information of the speech frames corresponding to the same part of the target text relative to the original text;

[0180] According to the proportional relationship, the duration information of the speech frame corresponding to the changed part of the target text relative to the original text is predicted.

[0181] In a specific implementation of the present invention, the final duration information determination module 620 is used to:

[0182] Inputting the phoneme sequence in the original text into a pre-trained second duration prediction model, and obtaining predicted duration information of speech frames corresponding to the phonemes in the original text through the second duration prediction model;

[0183] Retraining the second duration prediction model according to a comparison result of actual duration information and predicted duration information of speech frames corresponding to phonemes in the original text;

[0184] The phoneme sequence of the target text is input into the retrained second duration prediction model to predict the duration information of the speech frame corresponding to the changed part of the target text relative to the original text.

[0185] In a specific implementation of the present invention, the final duration information determination module 620 is used to:

[0186] Align the original speech and the original text;

[0187] According to the alignment relationship, the actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech is determined.

[0188] In a specific implementation of the present invention, the target spectrum acquisition module 630 is used to:

[0189] If the changed part is new content, then according to the duration information of the speech frame corresponding to the changed part, the spectrum is inserted into the position corresponding to the changed part in the original spectrum of the original speech to obtain the target spectrum;

[0190] If the changed part is a modified content, then according to the duration information of the speech frame corresponding to the changed part, a spectrum replacement process is performed at the position corresponding to the changed part in the original spectrum of the original speech to obtain a target spectrum.

[0191] In a specific implementation of the present invention, the target speech acquisition module 640 is used to:

[0192] Based on the voiceprint, the newly synthesized speech obtained after the speech synthesis processing is converted into a target speech with a target timbre.

[0193] In a specific implementation of the present invention, the target speech acquisition module 640 is used to:

[0194] Using the target spectrum as prosodic reference information and extracting prosodic representation of the prosodic reference information;

[0195] Map each phoneme in the target text to the vector space to obtain the vector representation of the target text;

[0196] Based on the final duration information, the vector representation of the target text is expanded;

[0197] The prosodic representation and the vector representation of the expanded target text are concatenated and transformed into the target speech.

[0198] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0199] Memory 703, used for storing computer programs;

[0200] The processor 701 is used to execute the program stored in the memory 703 to implement the following steps:

[0201] Obtaining an original speech to be processed, an original text matching the original speech, and a target text corresponding to the original text, wherein the target text is a text obtained by modifying the original text;

[0202] Determine final duration information of speech frames corresponding to phonemes in the target text, wherein the final duration information includes duration information of speech frames corresponding to a changed portion of the target text relative to the original text;

[0203] According to the duration information of the speech frame corresponding to the changed part, the original spectrum of the original speech is filled to obtain the target spectrum;

[0204] The target spectrum is used as prosodic reference information, combined with the final duration information and the target text for speech synthesis processing to obtain the target speech.

[0205] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0206] The communication interface is used for communication between the above terminal and other devices.

[0207] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0208] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0209] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes any of the speech processing methods in the above embodiments.

[0210] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any of the speech processing methods in the above embodiments.

[0211] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0212] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0213] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0214] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A speech processing method, characterized in that: include: Obtaining an original speech to be processed, an original text matching the original speech, and a target text corresponding to the original text, wherein the target text is a text obtained by modifying the original text; Determine final duration information of speech frames corresponding to the phonemes in the target text, wherein the final duration information includes duration information of speech frames corresponding to the changed portion of the target text relative to the original text; Filling the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain a target spectrum; The target spectrum is used as prosodic reference information, and speech synthesis processing is performed in combination with the final duration information and the target text to obtain the target speech.

2. The speech processing method according to claim 1, characterized in that: The determining of final duration information of the speech frame corresponding to the phoneme in the target text includes: Determine the actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech; Based on the actual duration information of the speech frames corresponding to the phonemes in the original text, predicting the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text; The final duration information of the speech frame corresponding to the phoneme in the target text is determined according to the actual duration information and the duration information of the speech frame corresponding to the changed part.

3. The speech processing method according to claim 2, characterized in that: The step of predicting the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text based on the actual duration information of the speech frames corresponding to the phonemes in the original text comprises: Inputting the phoneme sequence of the target text into a first duration prediction model obtained by pre-training, and obtaining initial duration information of speech frames corresponding to the phonemes in the target text through the first duration prediction model; Based on the actual duration information of the speech frames corresponding to the phonemes in the original text and the initial duration information of the speech frames corresponding to the phonemes in the target text, the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text is predicted.

4. The speech processing method according to claim 3, characterized in that: The method of predicting the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text based on the actual duration information of the speech frames corresponding to the phonemes in the original text and the initial duration information of the speech frames corresponding to the phonemes in the target text comprises: Comparing the actual duration information of the speech frames corresponding to the phonemes in the original text with the initial duration information of the speech frames corresponding to the phonemes in the target text, and determining the ratio between the initial duration information and the actual duration information of the speech frames corresponding to the same part of the target text relative to the original text; According to the proportional relationship, the duration information of the speech frame corresponding to the changed part of the target text relative to the original text is predicted.

5. The speech processing method according to claim 2, characterized in that: The step of predicting the duration information of the speech frames corresponding to the changed parts of the target text relative to the original text based on the actual duration information of the speech frames corresponding to the phonemes in the original text comprises: Inputting the phoneme sequence in the original text into a pre-trained second duration prediction model, and obtaining predicted duration information of speech frames corresponding to the phonemes in the original text through the second duration prediction model; Retraining the second duration prediction model according to a comparison result of actual duration information and predicted duration information of speech frames corresponding to the phonemes in the original text; The phoneme sequence of the target text is input into the retrained second duration prediction model to predict the duration information of the speech frame corresponding to the changed part of the target text relative to the original text.

6. The speech processing method according to claim 2, characterized in that: The determining the actual duration information of the speech frame of the original speech corresponding to the phoneme in the original text comprises: Performing alignment processing on the original speech and the original text; According to the alignment relationship, actual duration information of the phonemes in the original text corresponding to the speech frames of the original speech is determined.

7. The speech processing method according to claim 1, characterized in that: The step of performing filling processing on the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain the target spectrum includes: If the changed part is newly added content, then according to the duration information of the speech frame corresponding to the changed part, in the original spectrum of the original speech, a spectrum insertion process is performed at the position corresponding to the changed part to obtain a target spectrum; If the changed part is a modified content, then according to the duration information of the speech frame corresponding to the changed part, in the original spectrum of the original speech, a spectrum replacement process is performed at the position corresponding to the changed part to obtain a target spectrum.

8. The speech processing method according to claim 1, characterized in that: The obtaining of the target speech comprises: Based on the voiceprint, the newly synthesized speech obtained after the speech synthesis processing is converted into a target speech with a target timbre.

9. The speech processing method according to any one of claims 1 to 8, characterized in that: The step of using the target spectrum as prosody reference information, combining the final duration information and the target text to perform speech synthesis processing to obtain the target speech includes: Using the target spectrum as prosodic reference information and extracting a prosodic representation of the prosodic reference information; Mapping each phoneme in the target text to a vector space to obtain a vector representation of the target text; Based on the final duration information, performing expansion processing on the vector representation of the target text; The prosodic representation and the vector representation of the target text after expansion are concatenated and converted into the target speech after transformation.

10. A speech processing device, characterized in that: include: A data acquisition module, used to obtain an original speech to be processed, an original text matching the original speech, and a target text corresponding to the original text, wherein the target text is a text obtained by modifying the original text; A final duration information determination module, used to determine the final duration information of the speech frame corresponding to the phoneme in the target text, wherein the final duration information includes the duration information of the speech frame corresponding to the changed part of the target text relative to the original text; A target spectrum acquisition module, used for performing filling processing on the original spectrum of the original speech according to the duration information of the speech frame corresponding to the changed part to obtain the target spectrum; The target speech acquisition module is used to use the target spectrum as prosody reference information, combine the final duration information and the target text to perform speech synthesis processing, and obtain the target speech.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the steps of the speech processing method described in any one of claims 1-9 when executing the program stored in the memory.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the speech processing method as described in any one of claims 1 to 9 is implemented.

Images