Voice editing method and apparatus, and related device

Through alignment tools and speech model prediction technology, the problem of difficulty in achieving flexible operation of speech editing is solved, efficient speech editing and natural boundary overeffects are achieved, and the cost and time of re-recording is reduced.

WO2024174787A9PCT designated stage expired Publication Date: 2024-11-21JD DIGITS HAIYI INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/073480
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-02-23
Filing Date
2024-01-22
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

In the prior art, voice editing is difficult to implement flexible addition, deletion, modification and inspection operations, resulting in high re-recording costs, long time, poor final voice quality, and unnatural editing boundaries.

Method used

Obtain the alignment results of speech and text through the alignment tool, determine the target Mel spectrum, including masked area Mel spectrum and non-edited area Mel spectrum, use the speech model to predict masked area Mel spectrum, synthesize the final speech, and support insertion, deletion and By replacing the user only needs to edit the text to edit the audio file.

Benefits of technology

It realizes automatic text-based voice editing, supports flexible editing operations, improves voice editing quality and boundary overeffects, and reduces the cost and time of re-recording.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024073480_21112024_PF_FP_ABST
    Figure CN2024073480_21112024_PF_FP_ABST
Patent Text Reader

Abstract

A voice editing method and apparatus, and a related device, which relate to the technical field of voice processing. The method comprises: acquiring an alignment result for voice and text by means of an alignment tool; determining a target Mel spectrum according to the alignment result and edited text, wherein the target Mel spectrum comprises a masked-region Mel spectrum and a non-edited region Mel spectrum, a masked region being obtained by means of a voice model predicting change text in the edited text; sending the edited text and the target Mel spectrum to the voice model to predict the masked-region Mel spectrum, so as to obtain a predicted Mel spectrum; and synthesizing the predicted Mel spectrum and the non-edited region Mel spectrum by means of a vocoder, so as to obtain final voice. The present disclosure can improve an editing effect and a boundary transition effect.
Need to check novelty before this filing date? Find Prior Art

Description

Voice editing method, device and related equipment

[0001] This disclosure is based on the Chinese patent application with application number 202310193793.2, application date February 23, 2023, and invention name “Voice Editing Method, Device, Electronic Device and Storage Medium”, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this disclosure as a reference. Technical Field

[0002] The present disclosure relates to the field of speech processing technology, and in particular to a speech editing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0003] In many scenarios, voice cannot be directly edited like text, such as adding, deleting, modifying, or checking, to achieve flexible voice generation or modification. For example, if some words are said too much, omitted, or incorrectly in a recording, the cost of re-recording is high due to changes in the recording environment, the recording scene, the voice of the podcast host, or the podcast host cannot be contacted. In addition, the re-recording style of the re-recorded sentence is inconsistent with the recording style of the context, and the re-recording period is relatively long, which is not conducive to the rapid production of audio content. In related technologies, the audio waveform or spectrum of the speech is manipulated, and the editing boundaries are excessive and unnatural, resulting in suboptimal quality of the final edited speech.

[0004] Summary of the Invention

[0005] The present disclosure provides a voice editing method, device, electronic device, computer-readable storage medium, and computer program product, which at least to some extent overcome the problem of low voice editing quality in related technologies.

[0006] According to one aspect of the present disclosure, a speech editing method is provided, comprising: obtaining an alignment result of speech and text through an alignment tool; determining a target mel-spectrogram based on the alignment result and the edited text, wherein the target mel-spectrogram includes a mel-spectrogram of a masked area and a mel-spectrogram of a non-edited area; sending the edited text and the target mel-spectrogram to a speech model to predict the mel-spectrogram of the masked area to obtain a predicted mel-spectrogram; and synthesizing the predicted mel-spectrogram and the mel-spectrogram of the non-edited area to obtain a final speech.

[0007] In one embodiment of the present disclosure, the masked area mel spectrum includes: an edited mel spectrum corresponding to the changed text in the edited text, a mel spectrum of at least one frame before the edited mel spectrum, and a mel spectrum of at least one frame after the edited mel spectrum.

[0008] In one embodiment of the present disclosure, the sending of the edited text and the target mel-spectrogram to a speech model to predict the masked area mel-spectrogram to obtain the predicted mel-spectrogram includes: predicting the edited prosodic features of the edited text through the speech model, wherein the edited prosodic features are prosodic features corresponding to the changed text in the edited text; and predicting the masked area mel-spectrogram based on the edited text and the target mel-spectrogram by the speech model.

[0009] In one embodiment of the present disclosure, the training method of the speech model includes: obtaining training speech and training text, and obtaining a training alignment result of the training speech and training text through an alignment tool; extracting the training prosodic features of each phoneme sequence in the training text and the mel-spectrogram features in the training speech according to the training alignment results; masking the mel-spectrogram features to obtain a training mel-spectrogram, wherein the training mel-spectrogram includes a training masked area mel-spectrogram; encoding the training mel-spectrogram through a mel-spectrogram encoder to obtain a mel-spectrogram implicit representation; splicing the mel-spectrogram implicit representation and the training prosodic features to a decoder, and predicting the training masked area mel-spectrogram.

[0010] In one embodiment of the present disclosure, it also includes: converting the phoneme sequence in the edited text into phoneme embedding information; inputting the phoneme embedding information and position coding into an encoder to obtain a phoneme implicit representation; inputting the phoneme implicit representation into a variable information adapter to obtain the edited prosodic features of the phoneme sequence, wherein the edited prosodic features include: pitch, energy and pronunciation duration.

[0011] In one embodiment of the present disclosure, the variable information adapter includes a duration predictor, an energy predictor, and a pitch predictor.

[0012] In one embodiment of the present disclosure, the implicit representation of the phoneme is input into a variable information adapter to obtain the edited prosodic features of the phoneme sequence: the voiceprint embedded information and the implicit representation of the phoneme are input into the variable information adapter to obtain the edited prosodic features of the phoneme.

[0013] In one embodiment of the present disclosure, the further embodiment includes: masking the mel-spectrogram corresponding to the word phrase according to the training alignment result to obtain the mel-spectrogram of the training masked area; or determining the mel-spectrogram of the training masked area according to an application scenario.

[0014] In one embodiment of the present disclosure, the speech model is iteratively trained based on training data and training loss, wherein the training data includes: the training speech and the training text; the training loss includes: the loss of the mel spectrum of the training masked area, the loss of the training prosodic features, or the loss of the training prosodic features corresponding to the changed text.

[0015] According to another aspect of the present disclosure, there is also provided a voice editing device, comprising:

[0016] Alignment result generation module, which obtains the alignment results of speech and text through alignment tools;

[0017] a target mel spectrum determination module, which determines a target mel spectrum according to the alignment result and the edit text, wherein the target mel spectrum includes: a mel spectrum of the masked area and a mel spectrum of the non-edited area;

[0018] A predicted mel-spectrogram generation module sends the edited text and the target mel-spectrogram to a speech model to predict the mel-spectrogram of the masked area to obtain a predicted mel-spectrogram;

[0019] The final speech synthesis module synthesizes the predicted Mel spectrum and the non-edited area Mel spectrum to obtain the final speech.

[0020] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned voice editing methods by executing the executable instructions.

[0021] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements any one of the above-mentioned speech editing methods.

[0022] According to another aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements the above-mentioned voice editing method when executed by a processor.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0025] FIG1 shows a flow chart of a voice editing method according to an embodiment of the present disclosure;

[0026] FIG2 shows a flow chart of a speech model prediction method according to an embodiment of the present disclosure;

[0027] FIG3 shows a schematic diagram of a speech model structure according to an embodiment of the present disclosure;

[0028] FIG4 shows a flow chart of a speech model training method according to an embodiment of the present disclosure;

[0029] FIG5 shows a flow chart of another voice editing method according to an embodiment of the present disclosure;

[0030] FIG6 shows a schematic diagram of a deletion operation spectrum according to an embodiment of the present disclosure;

[0031] FIG7 shows a schematic diagram of an insertion operation spectrum according to an embodiment of the present disclosure;

[0032] FIG8 is a schematic diagram showing a spectrum of a replacement operation according to an embodiment of the present disclosure;

[0033] FIG9 shows a schematic diagram of a voice editing device according to an embodiment of the present disclosure;

[0034] FIG10 is a schematic diagram showing an exemplary system architecture that can be applied to the voice editing method of the embodiment of the present disclosure; and

[0035] FIG11 shows a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0037] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0038] This exemplary implementation is described in detail below with reference to the accompanying drawings and examples.

[0039] First, an embodiment of the present disclosure provides a voice editing method, which can be executed by any electronic device with computing and processing capabilities.

[0040] FIG1 shows a flow chart of a voice editing method according to an embodiment of the present disclosure. As shown in FIG1 , the voice editing method provided in the embodiment of the present disclosure includes the following steps:

[0041] S102, obtaining the alignment result of the speech and text by using an alignment tool;

[0042] S104, determining a target mel spectrum according to the alignment result and the edited text, wherein the target mel spectrum includes: a mel spectrum of the masked area and a mel spectrum of the non-edited area;

[0043] Masking refers to the process of controlling the area or process of speech processing by blocking selected speech.

[0044] Based on the alignment results of text and speech, you can select the mel spectrum area to be edited, and realize automatic text-based speech editing operations without manually selecting the mel spectrum area to be edited.

[0045] The text in the step of obtaining the alignment result of speech and text through the alignment tool is the original text, the edited text is the new text, and the changed text is the text part that has been changed by replacement, deletion, insertion, etc. in the edited text. For example, the original text is: ABC, the changed text corresponding to the edited text ADE is DE; the changed text corresponding to the edited text AC is B; the changed text corresponding to the edited text ABCF is F, and the area corresponding to the changed text is the editing area.

[0046] In one embodiment, the masked area mel-spectrogram includes: the edited mel-spectrogram corresponding to the changed text in the edited text, the mel-spectrogram for at least one frame before the edited mel-spectrogram, and the mel-spectrogram for at least one frame after the edited mel-spectrogram. For example, if the original text is ABC, and the edited text ADE corresponds to the changed text DE, the edited mel-spectrogram will be the mel-spectrogram corresponding to DE and the mel-spectrogram for at least one frame before and after the edited mel-spectrogram. This allows for more context to be edited, improving the speech editing effect. The mel-spectrograms in other areas are considered non-edited areas.

[0047] S106, sending the edited text and the target mel spectrum to the speech model to predict the masked area mel spectrum to obtain the predicted mel spectrum.

[0048] In one embodiment, the speech model predicts the editing prosodic features of the edited text, wherein the editing prosodic features are prosodic features corresponding to the changed text in the edited text; and the speech model predicts the masked area mel spectrum based on the edited text and the target mel spectrum.

[0049] For example, the original text is: ABC, and the changed text corresponding to the edited text ADE is DE. Then the edited prosodic feature is the prosodic feature corresponding to DE, which is obtained through speech model prediction; the prosodic feature of A that has not been changed in the edited text is directly obtained based on the original speech.

[0050] In one embodiment, the variance adaptor in the speech model predicts the prosodic features corresponding to the changed text in the edited text: duration features, pitch features, energy and other prosodic features. These prosodic features already contain the contextual information of the editing area, and the encoder can learn the contextual features of the entire sentence. In the variance adaptor, multi-layer one-dimensional convolution can also well learn local contextual features. During inference, the variance adaptor can perceive the contextual information and make prosodic predictions that are more consistent with the context. By using explicit modeling of prosodic features such as duration features, pitch features, and energy, these variables can be controlled during editing to achieve editing effects of different styles.

[0051] In one embodiment, when predicting the prosodic features of the edited area, the prosodic features of the non-edited area are extracted. These prosodic features can also be used as inputs of the variance adaptor, enabling the acquisition of contextual information of the edited area, thereby improving the prosodic prediction effect.

[0052] In one embodiment, the speech editing model inputs a masked mel-spectrogram, so that the editing area is equivalent to the masked part. The mel-spectrogram encoder and decoder can both learn contextual information from the input mel-spectrogram to predict the mel-spectrogram of the editing area with more consistent prosody. The FastSpeech2 decoder has the input mel-spectrogram and learns contextual information and boundary information from it, thus achieving a smoother editing boundary transition effect.

[0053] S108, synthesizing the predicted mel-spectrogram and the mel-spectrogram of the non-edited region to obtain the final speech.

[0054] In one embodiment, the boundary of the changed text in the edited text is found, and then the edited area is used as the masked mel spectrum. At the same time, the mel spectrum of the non-edited area is spliced ​​as the decoder input. The decoder is allowed to predict the masked mel spectrum. The mel spectrum of the masked area predicted by the decoder and the mel spectrum of the non-edited area are then spliced ​​together, and the final speech synthesis is performed through the vocoder.

[0055] In the above embodiment, an alignment algorithm is used to automatically align audio and text. Based on the alignment results of text and audio, the mel-spectrogram area corresponding to the edited text area can be found to realize automatic text-based voice editing operations. There is no need to manually select the mel-spectrogram area to be edited. The masked mel-spectrogram is input to the voice model and the voice model is allowed to predict the masked mel-spectrogram to realize text-based voice editing functions, supporting insertion, deletion and replacement operations. The user only needs to edit the text to realize the corresponding audio file editing operation, and expand the editing area. The prosodic features predicted by the voice model based on context information are used in the editing area, and the prosodic features directly extracted from the audio and normalized are used in the non-editing area. The prosodic features of the editing area and the non-editing area are then spliced ​​together to help predict the acoustic features of the editing area, thereby improving the editing effect and boundary transition effect.

[0056] FIG2 shows a flow chart of a speech model prediction method according to an embodiment of the present disclosure. As shown in FIG2 , the speech model prediction method provided in an embodiment of the present disclosure includes the following steps:

[0057] S202, converting the phoneme sequence in the edited text into phoneme embedding information;

[0058] S204, inputting the phoneme embedding information and position code into the encoder to obtain the phoneme implicit representation;

[0059] S206 , inputting the implicit representation of the phoneme into the variable information adapter to obtain the edited prosodic features of the phoneme sequence, wherein the edited prosodic features include: pitch, energy, and pronunciation duration.

[0060] In one embodiment, FIG3 shows a schematic diagram of a speech model structure in accordance with an embodiment of the present disclosure. The speech model is an end-to-end network architecture based on an encoder 304-decoder 306 network structure. Both the encoder 304 and the decoder 306 are multi-layer FFN (feedforward neural network) networks based on the self-attention mechanism. The encoder 304 receives a phoneme sequence as input 301, which is then converted into phoneme embedding information 302. A position encoding 303 is added to the encoder 304 to learn the implicit representation of the phonemes. The encoder 304 then outputs the output and sends it to a variable information adapter 305.

[0061] In one embodiment, the Variance adaptor 305 includes, but is not limited to, the following feature predictors: a duration predictor, an energy predictor, and a pitch predictor. Variance adaptor 305 predicts phoneme-level prosodic features, such as the pitch, energy, and pronunciation duration of a phoneme. The pitch and energy features of the phoneme are added to the implicit representation of the text learned by the backbone network, then expanded to the frame level based on the phoneme duration information. This frame-level sequence representation is then input to the decoder 306 for final acoustic feature prediction. The input to the Variance adaptor 305 is the output of the encoder 304. The duration predictor first predicts the pronunciation duration of each phoneme, then the pitch predictor predicts the pitch feature of each phoneme, and finally the energy predictor predicts the energy feature of each phoneme. The feature vectors corresponding to these features are then added to the backbone network and finally expanded to the frame level based on the phoneme duration information.

[0062] During the training process of the speech model, the acoustic features corresponding to each timbre, as well as the duration, pitch, and energy characteristics of the phonemes corresponding to the input text of the timbre are predicted. In this way, the model learns the timbre and rhythmic style information of the target timbre.

[0063] The pitch and energy features of the phonemes are Gaussian normalized for each speaker, so the pitch and energy features do not contain any speaker information and only represent the overall prosodic performance.

[0064] In one embodiment, the voiceprint embedded information and the phoneme implicit representation are input into a variable information adapter to obtain the edited prosodic features of the phonemes.

[0065] When training a speech model, each timbre has its own independent style features, namely, pronunciation duration, pitch, energy, and other features. The corresponding voiceprint embedding information speaker embedding 311 is obtained through the voiceprint identifier speaker ID 310. Speaker embedding 311 is added to the output of encoder 304. In this way, the input of variance adaptor 305 includes speaker embedding 311 information, which can realize style prediction of different speakers.

[0066] S208 , the speech model predicts the masked region mel spectrum in the masked mel spectrum 308 .

[0067] In one embodiment, the speech model input also includes masked mel-spectra 308, allowing the speech model to predict these masked mel-spectra 308 to implement speech editing operations. The speech model training objective becomes to recover these masked mel-spectra 308. No loss is calculated for other mel-spectra unrelated to the masked regions. These mel-spectra inputs serve solely to provide contextual information, enabling the speech model to predict prosodic features that are more consistent with the context. Consequently, the overall network structure can perceive both the context of the edited region at the text level and the context of the edit pre-fetch at the mel-spectrogram level. It synthesizes the text in the edited region while maintaining a consistent prosody within the context, achieving a speech synthesis effect that is more consistent with the contextual prosody.

[0068] In one embodiment, the masked mel spectrum 308 is passed through several mel spectrum encoders 309 composed of multiple layers of one-dimensional convolution to obtain a mel spectrum implicit representation. The mel spectrum representation is then concatenated with the text implicit representation obtained by the encoder expanded to the frame level and sent together to the decoder 306 to predict the masked mel spectrum 308 to obtain a predicted mel spectrum 307.

[0069] The mel spectrum encoder 309 includes:

[0070] In the first and second Conv1D layers, the activation function is the ReLU function, which is used to perform convolution on the data.

[0071] The first LayerNorm+Dropout layer and the second LayerNorm+Dropout layer normalize the data and alleviate the overfitting phenomenon of the convolutional neural network;

[0072] Linear layer linearlayer, each neuron is connected to the previous neuron to realize linear transformation of the linear combination of the previous layer.

[0073] In the above embodiment, the masked mel-spectrogram is input into the speech model and the speech model is allowed to predict the masked mel-spectrogram to realize the text-based speech editing function, which supports insertion, deletion and replacement operations. The user only needs to edit the text to realize the editing operation of the corresponding audio file. The rhythmic features predicted by the speech model based on context information are used in the editing area, and the rhythmic features directly extracted from the audio and normalized are used in the non-editing area. The synthesized speech editing boundary transition is smooth and natural, and the rhythm of the editing area is consistent with the overall rhythm.

[0074] FIG4 shows a flow chart of a speech model training method according to an embodiment of the present disclosure. As shown in FIG4 , the speech model training method provided in an embodiment of the present disclosure includes the following steps:

[0075] S402, obtaining training speech and training text, and obtaining a training alignment result of the training speech and training text using an alignment tool;

[0076] S404: extracting the training prosodic features of each phoneme sequence in the training text and the mel-spectrogram features in the training speech according to the training alignment result.

[0077] In one embodiment, the training data includes: training speech and training text. A forced alignment tool is used to obtain a training alignment result of the training speech and training text. Then, based on the training alignment result, the pronunciation duration, pitch, and energy features of each phoneme are extracted, and the mel-spectrogram features in the training speech are extracted.

[0078] S406, masking the mel spectrum features to obtain a training mel spectrum, wherein the training mel spectrum includes a training masked area mel spectrum, and the training masked area mel spectrum is the mel spectrum of the masked area during training;

[0079] S408, encoding the trained mel spectrum through a mel spectrum encoder to obtain a mel spectrum implicit representation;

[0080] S410, concatenating the mel spectrum implicit representation and the trained rhythmic features to the decoder, and predicting the mel spectrum of the training masked area.

[0081] When training a speech model, each time a training mel-spectrogram is input, a portion of the mel-spectrogram is randomly masked, allowing the speech model decoder to predict these masked mel-spectrograms. The randomly masked mel-spectrograms are then passed through several mel-spectrogram encoders consisting of multiple layers of one-dimensional convolution to obtain a mel-spectrogram implicit representation. The mel-spectrogram representation is then concatenated with the implicit text representation, such as the training prosodic features obtained by the encoder expanded to the frame level, and sent to the decoder to predict the masked mel-spectrograms.

[0082] In one embodiment, the mel spectrum of the masked area is determined based on the training alignment result, the mel spectrum corresponding to the masked word phrase, or the application scenario.

[0083] In one embodiment, the masking ratio can be fixed or random according to a specified weight, etc.; the mel spectrogram of the masked character, masked word or masked phrase can be masked according to the training alignment results, and then the speech model can be trained; the masking ratio can also be determined according to the application scenario. For example, if the editing area only occupies 10% of the entire sentence, then using this ratio for masking during training can make the speech model more consistent during training and prediction, thereby improving the editing effect of the speech model. Therefore, the masking method and masking ratio can be determined according to the application scenario, etc.

[0084] In one embodiment, the speech model is iteratively trained based on training data and training loss, where the training data includes: training speech and training text; the training loss includes: loss of mel-spectrogram of training masked area, loss of training prosodic features, or loss of training prosodic features corresponding to the changed text.

[0085] The loss of the mel spectrum of the masked area is the loss corresponding to the mel spectrum of the masked area during the training process. The loss of other mel spectrums unrelated to the masked area is not calculated only to provide context information. Please refer to the previous embodiment and will not be repeated here.

[0086] The loss of the training prosodic features is the prosodic loss of the entire input text, and the loss of the training prosodic features corresponding to the changed text is the prosodic loss corresponding to the changed text in the edited text. The speech model uses three prosodic features: duration features, pitch features, and energy features. You can choose to calculate the loss only for these features in the edited area, or you can choose to calculate the loss on the entire sentence. Calculating the loss on the entire sentence is preferred.

[0087] In the above embodiment, during the process of training the speech model, not only can the mel spectrum be randomly masked for training, the masking ratio can be determined according to the application scenario, but the mel spectrum of the characters, words or phrases can also be masked according to the alignment results, and then the speech editing model can be trained to improve the editing effect of the speech model.

[0088] FIG5 shows a flow chart of another voice editing method according to an embodiment of the present disclosure. As shown in FIG5 , the voice editing method provided in an embodiment of the present disclosure includes the following steps:

[0089] S502, finding the location of the changed text in the edited text in the original speech spectrum, that is, the edited area, based on the alignment result;

[0090] S504, extracting rhythmic features such as mel spectrum, duration features, pitch features, and energy features from the non-editing area;

[0091] S506, replacing the original text with the new edited text, using the new edited text as input to the neural network encoder, and then the variance adaptor of the neural network predicts the rhythmic features of the edited region based on the new edited text.

[0092] It should be noted that only the edited area uses the value predicted by the neural network, while other non-edited areas still use the rhythmic feature values ​​extracted from the audio before editing.

[0093] S508: Expand the encoder output to the frame level based on the duration information. At the same time, insert the masked mel-spectrogram at the corresponding position in the mel-spectrogram of the original speech based on the predicted duration of the edited region. The length of the insertion position is the predicted pronunciation duration of the edited region.

[0094] It should be noted that, when performing a deletion operation, there will be no insertion of a new masked mel spectrum, and the deletion operation can be performed directly in the editing area.

[0095] In one embodiment, more mel-spectrograms are masked forward and backward in the editing area, for example, 5-10 frames forward and backward. Because the alignment result always has some deviation, this operation can also reduce the problem of poor speech editing effect caused by the deviation of the alignment result. Even if the alignment result has no deviation, many speech sounds are continuous and cannot be directly separated. If this operation is not performed, the editing boundary will be unnatural. To make the editing boundary transition more natural and smooth, more mel-spectrograms are masked forward and backward in the editing area. This operation can significantly improve the naturalness of the editing boundary transition.

[0096] S510 , using the expanded frame-level encoder output and the masked overall mel-spectrogram sequence as inputs to the decoder and the mel-spectrogram encoder, and allowing the speech model to predict the mel-spectrogram of the masked area based on the context;

[0097] S512: The predicted mel-spectrogram of the masked area and the mel-spectrogram of the non-edited area are concatenated and the final speech synthesis is performed through a vocoder.

[0098] In the above embodiment, based on the alignment results of text and audio, the mel spectrum area corresponding to the edited text area can be found to realize automatic text-based voice editing operations. The masked mel spectrum is input to the voice model and the voice model is allowed to predict the masked mel spectrum to realize text-based voice editing functions, supporting insertion, deletion and replacement operations. The user only needs to edit the text to realize the corresponding audio file editing operation, and expand the editing area. The rhythmic features predicted by the voice model according to the context information are used in the editing area, and the rhythmic features directly extracted from the audio and normalized are used in the non-editing area to improve the editing effect and boundary transition effect.

[0099] FIG6 shows a schematic diagram of a deletion operation spectrum in an embodiment of the present disclosure, as shown in FIG6 .

[0100] S602: Find the mel spectrum region corresponding to the changed text according to the alignment result of the original text and the speech.

[0101] For example, the original text is: AAAAAAAAAAAAAAAAAABBCC, the edited text is AAAAAAAAAAAAAAAAACC, and the changed text is BB. According to the alignment results of the original text and the speech, the mel spectrum area corresponding to the changed text BB is found.

[0102] S604: extract the mel spectrum and rhythmic features (duration feature, pitch feature, energy feature) of the non-editing area, and delete the mel spectrum of the editing area.

[0103] S606: Mask the mel spectrum of the editing area by one or more frames in front and behind.

[0104] In one embodiment, to make the editing boundary transition after deletion more natural, one or more frames of mel spectrum are masked forward and backward at the deletion point, allowing the speech model to correct the editing boundary area and make the deletion transition of the connection area natural.

[0105] For example, the area corresponding to the changed text BB is the editing area, and the rest are non-editing areas. The mel spectrum of the editing area is deleted, and the editing area corresponding to the changed text BB is masked before and after for at least one frame.

[0106] S608: The edited text and the masked overall mel spectrum are fed into the speech model to predict the mel spectrum of the masked area and correct the deletion boundary.

[0107] The prosodic features used in the phoneme sequence obtained by the deletion operation are all truly extracted prosodic features, and there are no predicted prosodic features.

[0108] S610: concatenate the mel-spectrogram of the non-edited region and the predicted mel-spectrogram of the masked region to synthesize the final speech through a vocoder.

[0109] In the above embodiment, the user only needs to delete the text to implement the editing operation of the corresponding audio file. The rhythmic features predicted by the speech model based on the context information are used in the editing area, and the rhythmic features directly extracted from the audio and normalized are used in the non-editing area. The synthesized speech editing boundary transitions are smooth and natural, and the rhythm of the editing area is consistent with the overall rhythm.

[0110] FIG7 shows a schematic diagram of an insertion operation spectrum in an embodiment of the present disclosure, as shown in FIG7 .

[0111] S702, finding the corresponding position of the changed text, that is, the inserted text, in the spectrum based on the alignment result of the original text and the speech;

[0112] For example, the original text is: AAAAAAAAAAAAAAAAAABBCC, the edited text is AAAAAAAAAADDDDDAAAAAAACC, and the changed text is DDDDDD. According to the alignment results of the original text and the speech, the mel spectrum area corresponding to the changed text DDDDDD is found.

[0113] S704, extracting the mel spectrum and rhythm features of the non-editing area;

[0114] S706, using the edited text as encoder input, and the speech model predicting the prosodic features of the edited region based on the changed text in the edited text, wherein the prosodic features include: pronunciation duration;

[0115] S708 : Inserting the masked mel spectrum into the mel spectrum of the original speech according to the predicted pronunciation duration.

[0116] The speech model predicts the pronunciation duration of the edit area based on the changed text DDDDD, determines the edit area corresponding to the changed text based on the pronunciation duration, and masks the edit area corresponding to the changed text and the mel spectrum of at least one frame forward and backward to obtain the masked area mel spectrum.

[0117] In one embodiment, the non-editing region uses the actual extracted rhythmic features (duration, pitch, energy), while the editing region uses the predicted rhythmic features (duration, pitch, energy).

[0118] S710 , the encoder output that integrates the prosodic features and expands to the frame level and the obtained overall Mel spectrum including the mask are sent to the decoder to predict the masked area;

[0119] S712: Concatenate the predicted Mel-spectrogram of the masked area and the Mel-spectrogram of the non-edited area and perform final speech synthesis through a vocoder.

[0120] Similar to the delete operation, to achieve a more natural transition between edit boundaries, the insert operation also masks the mel-spectrogram for an additional five frames, both forward and backward, after inserting the predicted duration into the masked region. This ensures that the resulting audio from the insert edit has a smooth and natural transition between edit boundaries.

[0121] In the above embodiment, the user only needs to insert the text to implement the editing operation of the corresponding audio file. The rhythmic features predicted by the speech model based on the context information are used in the editing area, and the rhythmic features directly extracted from the audio and normalized are used in the non-editing area. The synthesized speech editing boundary transitions are smooth and natural, and the rhythm of the editing area is consistent with the overall rhythm.

[0122] FIG8 shows a schematic diagram of a replacement operation spectrum in an embodiment of the present disclosure, as shown in FIG8 .

[0123] S802: Find the position of the changed text in the edited text in the original speech spectrum according to the alignment result of the original text and the speech.

[0124] For example, the original text is: AAAAAAAAAAAAAAAAAABBCC, and the edited text is AAAEEEEEAAAAAAAAAAABBCC. According to the alignment results of the original text and the speech, the mel spectrum area corresponding to the changed text EEEEE is found.

[0125] S804: Extract the mel spectrum and rhythm features of the non-editing area.

[0126] S806: Replace the original text with the new edited text, send the edited sentence to the encoder, and predict the prosodic features (duration features, pitch features, energy features) of the edited region.

[0127] In the entire sentence, only the edited area uses the predicted prosodic features, while the non-edited area still uses the audio features extracted from the original audio.

[0128] S808: Add a masked mel spectrum to the mel spectrum of the original speech according to the pronunciation duration predicted for the edited region, and mask the mel spectrum by at least one frame forward and backward to make the edit boundary transition more natural.

[0129] The pronunciation duration of the changed text EEEEE is predicted through the speech model. The editing area corresponding to the changed text is determined based on the pronunciation duration. The editing area corresponding to the changed text and the Mel spectrum of at least one frame forward and backward are masked to obtain the Mel spectrum of the masked area.

[0130] S810: The encoder output that integrates the prosodic features and expands to the frame level and the resulting masked overall mel spectrum are sent to the decoder to predict the mel spectrum of the masked area.

[0131] S812: The predicted mel spectrum and the mel spectrum of the non-edited area are concatenated and the final sound is synthesized through a vocoder.

[0132] In the above embodiment, the user only needs to replace the text to realize the editing operation of the corresponding audio file. The rhythmic features predicted by the speech model based on the context information are used in the editing area, and the rhythmic features directly extracted from the audio and normalized are used in the non-editing area. The synthesized speech editing boundary transition is smooth and natural, and the rhythm of the editing area is consistent with the overall rhythm.

[0133] Based on the same inventive concept, the present disclosure also provides a voice editing device, such as the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0134] FIG9 shows a schematic diagram of a speech editing device according to an embodiment of the present disclosure. As shown in FIG9 , the speech editing device 9 includes: an alignment result generation module 901 , a target mel-spectrogram determination module 902 , a predicted mel-spectrogram generation module 903 , and a final speech synthesis module 904 .

[0135] An alignment result generating module 901 obtains the alignment result of speech and text through an alignment tool;

[0136] The target mel spectrum determination module 902 determines the target mel spectrum according to the alignment result and the edited text, wherein the target mel spectrum includes: the mel spectrum of the masked area and the mel spectrum of the non-edited area;

[0137] The predicted mel-spectrum generation module 903 sends the edited text and the target mel-spectrum to the speech model to predict the masked area mel-spectrum and obtain the predicted mel-spectrum;

[0138] The final speech synthesis module 904 synthesizes the predicted mel spectrum and the non-edited area mel spectrum to obtain the final speech.

[0139] In the above embodiment, the masked mel-spectrogram is input into the speech model and the speech model is allowed to predict the masked mel-spectrogram to implement a text-based speech editing function, which supports insertion, deletion and replacement operations. The user only needs to edit the text to implement the corresponding audio file editing operation. The editing boundaries of the synthesized speech are smooth and natural, and the rhythm of the editing area is consistent with the overall rhythm.

[0140] FIG10 shows a schematic diagram of an exemplary system architecture that can be applied to the voice editing method according to an embodiment of the present disclosure.

[0141] As shown in FIG. 10 , the system architecture 1000 may include terminal devices 1001 , 1002 , 1003 , a network 1004 and a server 1005 .

[0142] The terminal devices 1001, 1002, and 1003 that execute the voice editing method may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.

[0143] Optionally, the client of the application installed in different terminal devices 1001, 1002, and 1003 is the same, or is a client of the same type of application based on different operating systems. Based on different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile phone client, a PC client, etc.

[0144] The network 1004 is used as a medium to provide a communication link between the terminal devices 1001, 1002, 1003 and the server 1005, and can be a wired network or a wireless network.

[0145] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0146] The server 1005 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal devices 1001, 1002, and 1003. The background management server may analyze and process received requests and other data, and feed back the processing results to the terminal device.

[0147] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0148] Those skilled in the art will appreciate that the number of terminal devices, networks, and servers in FIG10 is merely illustrative, and any number of terminal devices, networks, and servers may be provided based on actual needs, which is not limited in the present disclosure.

[0149] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0150] The electronic device 1100 according to this embodiment of the present disclosure is described below with reference to Figure 11. The electronic device 1100 shown in Figure 11 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0151] As shown in Figure 11, electronic device 1100 is implemented as a general-purpose computing device. Components of electronic device 1100 may include, but are not limited to, the aforementioned at least one processing unit 1110, the aforementioned at least one storage unit 1120, and a bus 1130 connecting various system components (including storage unit 1120 and processing unit 1110).

[0152] The storage unit stores program codes, which can be executed by the processing unit 1110, so that the processing unit 1110 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0153] For example, the processing unit 1110 can perform the following steps of the above method embodiment: obtain the alignment results of speech and text through an alignment tool; determine the target mel spectrum based on the alignment results and the edited text, wherein the target mel spectrum includes: the masked area mel spectrum and the non-edited area mel spectrum; send the edited text and the target mel spectrum to the speech model to predict the masked area mel spectrum to obtain the predicted mel spectrum; synthesize the predicted mel spectrum and the non-edited area mel spectrum to obtain the final speech.

[0154] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 11201 and / or a cache memory unit 11202 , and may further include a read-only memory unit (ROM) 11203 .

[0155] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0156] The bus 1130 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0157] Electronic device 1100 may also communicate with one or more external devices 1140 (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1100, and / or any device that enables electronic device 1100 to communicate with one or more other computing devices (e.g., a router, modem, etc.). Such communication may occur via input / output (I / O) interface 1150. Furthermore, electronic device 1100 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 1160. As shown, network adapter 1160 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with electronic device 1100, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0158] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0159] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided. The computer-readable storage medium may be a readable signal medium or a readable storage medium. A program product capable of implementing the above-mentioned method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Methods" section above of this specification.

[0160] For example, when the program product in the embodiment of the present disclosure is executed by a processor, the following steps are implemented: obtaining the alignment results of speech and text through an alignment tool; determining the target mel spectrum based on the alignment results and the edited text, wherein the target mel spectrum includes: the mel spectrum of the masked area and the mel spectrum of the non-edited area; sending the edited text and the target mel spectrum to the speech model to predict the mel spectrum of the masked area to obtain the predicted mel spectrum; synthesizing the predicted mel spectrum and the mel spectrum of the non-edited area to obtain the final speech.

[0161] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] In the present disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0163] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0164] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0165] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0166] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0167] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0168] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A voice editing method, wherein: include: Use the alignment tool to obtain the alignment results of speech and text; Determining a target Mel spectrum according to the alignment result and the edited text, wherein the target Mel spectrum includes: a masked area Mel spectrum and a non-edited area Mel spectrum; Sending the edited text and the target Mel spectrum to a speech model to predict the masked area Mel spectrum to obtain a predicted Mel spectrum; The predicted mel-spectrogram and the non-edited region mel-spectrogram are synthesized to obtain the final speech.

2. The voice editing method according to claim 1, wherein: The masked area mel spectrum includes: an edited mel spectrum corresponding to the changed text in the edited text, a mel spectrum of at least one frame before the edited mel spectrum, and a mel spectrum of at least one frame after the edited mel spectrum.

3. The voice editing method according to claim 1, wherein: The step of sending the edited text and the target Mel spectrum to the speech model to predict the masked area Mel spectrum to obtain the predicted Mel spectrum comprises: Predicting the editing prosodic features of the editing text by using the speech model, wherein the editing prosodic features are prosodic features corresponding to the changed text in the editing text; The speech model predicts the masked area mel spectrum according to the edited text and the target mel spectrum.

4. The voice editing method according to claim 3, wherein: The predicting of the editing prosody features of the editing text by the speech model comprises: Converting a phoneme sequence in the edited text into phoneme embedding information; Inputting the phoneme embedding information and the position code into an encoder to obtain an implicit representation of the phoneme; The implicit representation of the phoneme is input into a variable information adapter to obtain the editing prosodic features of the phoneme sequence, wherein the editing prosodic features include: pitch, energy and pronunciation duration.

5. The voice editing method according to claim 4, wherein: The variable information adapter includes a duration predictor, an energy predictor and a pitch predictor.

6. The voice editing method according to claim 4, wherein: The implicit representation of the phoneme is input into the variable information adapter to obtain the editing rhythmic features of the phoneme sequence: The voiceprint embedding information and the implicit representation of the phoneme are input into the variable information adapter to obtain the editing rhythmic features of the phoneme.

7. The voice editing method according to claim 1, wherein: The training method of the speech model includes: Obtain training speech and training text, and obtain the training alignment result of the training speech and training text through the alignment tool; Extracting the training prosodic features of each phoneme sequence in the training text and the Mel-spectrogram features in the training speech according to the training alignment result; Masking the mel spectrum feature to obtain a training mel spectrum, wherein the training mel spectrum includes a training masked area mel spectrum; Encoding the training mel spectrum through a mel spectrum encoder to obtain a mel spectrum implicit representation; The mel spectrum implicit representation and the training rhythmic features are concatenated to the decoder to predict the mel spectrum of the training masked area.

8. The voice editing method according to claim 7, wherein: Also includes: According to the training alignment result, the mel spectrum corresponding to the masked word phrase is obtained to obtain the mel spectrum of the training masked area; Or the training masked area mel spectrum is determined according to an application scenario.

9. The voice editing method according to claim 7, wherein: It also includes: iteratively training the speech model based on training data and training loss, wherein the training data includes: the training speech and the training text; The training loss includes: the loss of the mel spectrum of the training masked area, the loss of the training prosodic feature, or the loss of the training prosodic feature corresponding to the changed text.

10. A voice editing device, wherein: include: The alignment result generation module obtains the alignment results of speech and text through the alignment tool; A target mel spectrum determination module is configured to determine a target mel spectrum according to the alignment result and the edited text, wherein the target mel spectrum includes: a masked area mel spectrum and a non-edited area mel spectrum; A predicted mel-spectrogram generation module sends the edited text and the target mel-spectrogram to a speech model to predict the mel-spectrogram of the masked area to obtain a predicted mel-spectrogram; The final speech synthesis module synthesizes the predicted Mel spectrum and the non-edited area Mel spectrum to obtain the final speech.

11. An electronic device, wherein: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the speech editing method according to any one of claims 1 to 9 by executing the executable instructions.

12. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the speech editing method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the speech editing method according to any one of claims 1 to 9 is implemented.