Speech editing method and apparatus

Abstract

The present application discloses a speech editing method and apparatus. The method comprises: acquiring a target text, the target text being a text obtained after a modification is made to an initial text, and the initial text being a text corresponding to an initial speech; on the basis of the target text, determining the location of the modification in the initial text, and determining a mask vector on the basis of the location of the modification and the initial speech; on the basis of the mask vector and the target text, determining basic frequency information of a frame granularity, extracting a Mel-spectrum feature of the initial speech from the initial speech, and constructing a Mel coding vector on the basis of the target text and the Mel-spectrum feature of the initial speech; on the basis of the target text, the basic frequency information of the frame granularity and the Mel coding vector, determining target speech corresponding to the target text. The present application solves the technical problem in the prior art of poor speech editing quality.

Description

Voice editing method and device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 26, 2024, with application number 2024102120088 and application name “Voice Editing Method and Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and more specifically, to a voice editing method and device. Background Art

[0003] Traditional telephone customer service relies primarily on human agents and push-button menus. However, with the continuous increase in customers and the increasing complexity of business, labor costs are constantly rising, and call volumes during peak hours are rapidly increasing. With the development of artificial intelligence technology, text-to-speech (TTS) technology converts text into speech to facilitate human-computer interaction. However, when it comes to expressing emotions, such as warmth and sympathy, current TTS-synthesized speech often appears mechanical or unnatural compared to real voices, affecting the user experience. In related technologies, one option is to first use TTS to synthesize the audio corresponding to the edited text, then use voice conversion (VC) technology to unify the style of the synthesized audio, and then splice the synthesized audio with the original recording. However, the spliced ​​audio can easily damage the voice quality.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a voice editing method and apparatus to at least solve the technical problem of poor quality of voice editing in related technologies.

[0007] According to one aspect of an embodiment of the present application, a speech editing method is provided, comprising: obtaining a target text, wherein the target text is a text after modifying an initial text, wherein the initial text is a text corresponding to an initial speech; determining a modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech; determining basic frequency information of a frame granularity based on the mask vector and the target text, extracting Mel spectrum features of the initial speech from the initial speech, and constructing a Mel encoding vector based on the target text and the Mel spectrum features of the initial speech; determining a target speech corresponding to the target text based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector.

[0008] In some embodiments, the modified position in the initial text is determined based on the target text, and a mask vector is determined based on the modified position and the initial speech, including: converting the target text into a text sequence, the text sequence at least including: a phoneme sequence; inserting a target identifier between the phonemes corresponding to each word in the target text to obtain a target text sequence; encoding the target text sequence to obtain a target feature, determining the modified position of the initial text from the target feature, and determining the mask vector based on the modified position and the initial speech.

[0009] In some embodiments, the mask vector is determined based on the modified position and the initial speech, including: obtaining the duration and basic frequency characteristics of the initial speech from the initial speech; determining the masked duration in the duration of the initial speech and the masked part in the basic frequency characteristics according to the modified position; determining the duration mask vector in the mask vector according to the masked duration in the duration of the initial speech, and determining the basic frequency mask vector in the mask vector based on the masked part in the basic frequency characteristics.

[0010] In some embodiments, the basic frequency information of the frame granularity is determined based on the mask vector and the target text, including: determining the duration information of the phoneme granularity based on the target feature and the duration mask vector; determining the target feature of the frame granularity based on the duration information of the phoneme granularity and the target feature; and determining the basic frequency information of the frame granularity according to the target feature of the frame granularity and the basic frequency mask vector.

[0011] In some embodiments, determining the duration information of the phoneme granularity based on the target feature and the duration mask vector includes: obtaining the masked phoneme duration information in the duration mask vector and the masked word duration information in the duration mask vector; determining the duration information of the phoneme granularity based on the phoneme duration information and the word duration information, wherein the word duration information is determined by the adjacent target identifier.

[0012] In some embodiments, constructing a mel encoding vector based on the mel spectrum features of the target text and the initial speech includes: determining a masked position in the mel spectrum features of the initial speech based on the modified position; and constructing the mel encoding vector based on the masked position in the mel spectrum features of the initial speech.

[0013] In some embodiments, the target speech corresponding to the target text is determined based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector, including: determining the target text, the basic frequency information of the frame granularity, and the Mel encoding vector as input data, inputting them into the attention block in the streaming decoder for analysis, and obtaining multiple Mel spectra, wherein the size of the attention block is randomly adjusted within a preset range; synthesizing the multiple Mel spectra into a target Mel spectrum corresponding to the target text, wherein the loss function of the streaming decoder at least includes a loss value between a true value obtained by using a whole sentence prediction method and the target Mel spectrum; and determining the target speech based on the target Mel spectrum.

[0014] In some embodiments, determining the target speech based on the target Mel spectrum includes: dividing the target Mel spectrum into multiple Mel spectrum blocks, retaining the tail of the Mel spectrum in the previous Mel spectrum block, and padding it with the Mel spectrum in the current Mel spectrum block to obtain multiple padded Mel spectrum blocks; and synthesizing the target speech with the multiple padded Mel spectrum blocks.

[0015] According to another aspect of an embodiment of the present application, a speech editing device is also provided, including: an acquisition module for acquiring a target text, wherein the target text is a text after modifying an initial text, wherein the initial text is a text corresponding to an initial speech; a determination module for determining a modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech; a construction module for determining basic frequency information of a frame granularity based on the mask vector and the target text, extracting Mel spectrum features of the initial speech from the initial speech, and constructing a Mel encoding vector based on the target text and the Mel spectrum features of the initial speech; a generation module for determining a target speech corresponding to the target text based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector.

[0016] According to another aspect of the embodiments of the present application, a computer device is also provided, including: a memory for storing program instructions; a processor, connected to the memory, for executing the program instructions of the following functions: obtaining a target text, wherein the target text is a text after modifying an initial text, wherein the initial text is a text corresponding to an initial speech; determining a modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech; determining basic frequency information of a frame granularity based on the mask vector and the target text, extracting Mel spectrum features of the initial speech from the initial speech, and constructing a Mel encoding vector based on the target text and the Mel spectrum features of the initial speech; determining a target speech corresponding to the target text based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector.

[0017] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided. The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned voice editing method by running the computer program.

[0018] In an embodiment of the present application, a target text is obtained, where the target text is a modified text of an initial text, wherein the initial text is a text corresponding to an initial speech; a modified position in the initial text is determined based on the target text, and a mask vector is determined based on the modified position and the initial speech; basic frequency information of a frame granularity is determined based on the mask vector and the target text, the Mel spectrum features of the initial speech are extracted from the initial speech, and a Mel encoding vector is constructed based on the target text and the Mel spectrum features of the initial speech; a target speech corresponding to the target text is determined based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector, by obtaining multiple acoustic features of the initial speech and constructing a corresponding mask vector based on the modified position in the initial text, and finally synthesizing the target speech based on the multiple acoustic features and the mask vector, thereby achieving the purpose of making the modified part of the speech consistent with the sound features of the original speech, thereby achieving the technical effect of improving the quality of speech editing, and thus solving the technical problem of poor quality of speech editing in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] FIG1 is a hardware structure block diagram of a computer terminal for implementing a voice editing method according to an embodiment of the present application;

[0021] FIG2 is a flow chart of a voice editing method according to an embodiment of the present application;

[0022] FIG3 is a schematic diagram of an optional text sequence according to an embodiment of the present application;

[0023] FIG4 is a flow chart of another voice editing method according to an embodiment of the present application;

[0024] FIG5 is a schematic diagram of an optional mask vector according to an embodiment of the present application;

[0025] FIG6 is a schematic diagram of an optional Mel-mask spectrum according to an embodiment of the present application;

[0026] FIG7 is a schematic diagram of an optional streaming decoding process according to an embodiment of the present application;

[0027] FIG8 is a structural diagram of a voice editing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] The speech editing method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 shows a hardware structure block diagram of a computer terminal for implementing the speech editing method. As shown in Figure 1, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used in the figure to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be used as one of the ports of the I / O interface), a network interface, and a BUS bus. It will be understood by those skilled in the art that the structure shown in Figure 1 is only for illustration and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may also include more or fewer components than those shown in Figure 1, or have a configuration different from that shown in Figure 1.

[0031] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the voice editing method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned voice editing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0035] It should be noted that, in some alternative embodiments, the computer device shown in FIG. 1 may include hardware components (including circuits), software components (including computer code stored on a computer-readable medium), or a combination of hardware and software components. It should be noted that FIG. 1 is merely an example of a specific embodiment and is intended to illustrate the types of components that may be present in the computer device.

[0036] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a voice editing method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] FIG2 is a flow chart of a voice editing method according to an embodiment of the present application. As shown in FIG2 , the method includes the following steps:

[0038] Step S202, obtaining a target text, where the target text is a modified version of the initial text, wherein the initial text is the text corresponding to the initial speech;

[0039] In the above step S202, the initial speech is a speech generated in advance based on the initial text, or it can be a pre-recorded recording, and the initial text is generated from the recorded recording.

[0040] Step S204, determining the modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech;

[0041] In step S204, the mask vector includes: a duration mask vector and a fundamental frequency (FO) mask vector.

[0042] Step S206, determining basic frequency information of a frame granularity based on the mask vector and the target text, extracting Mel-spectrogram features of the initial speech from the initial speech, and constructing a Mel-coded vector based on the Mel-spectrogram features of the target text and the initial speech;

[0043] In step S206, the mel spectrum (mel spectrogram) is a nonlinear scale unit defined based on frequency, which represents the equidistant changes in pitch perceived by the human ear. The mel spectrum refers to the spectrum converted from the frequency to the mel scale.

[0044] Step S208: determining the target speech corresponding to the target text based on the target text, basic frequency information of the frame granularity, and the Mel coding vector.

[0045] The speech editing method in steps S202 to S208 described above adopts the following steps: obtaining a target text, the target text being a modified text of an initial text, wherein the initial text is the text corresponding to the initial speech; determining the modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech; determining basic frequency information at a frame granularity based on the mask vector and the target text, extracting the mel spectrum features of the initial speech from the initial speech, and constructing a mel encoding vector based on the mel spectrum features of the target text and the initial speech; determining the target speech corresponding to the target text based on the target text, the basic frequency information at a frame granularity, and the mel encoding vector, obtaining multiple acoustic features of the initial speech, constructing a corresponding mask vector based on the modified position in the initial text, and finally synthesizing the target speech based on the multiple acoustic features and the mask vector, thereby achieving the goal of making the modified portion of the speech consistent with the sound features of the original speech, thereby achieving the technical effect of improving the quality of speech editing and solving the technical problem of poor quality of speech editing in the related art. Detailed description is given below.

[0046] In an optional manner, the steps of determining the modified position in the initial text based on the target text, and determining the mask vector based on the modified position and the initial speech are as follows: converting the target text into a text sequence, wherein the text sequence at least includes: a phoneme sequence; inserting a target identifier between the phonemes corresponding to each word in the target text to obtain a target text sequence; encoding the target text sequence to obtain a target feature, determining the modified position of the initial text from the target feature, and determining the mask vector based on the modified position and the initial speech.

[0047] The target text is the text input by the user. Before encoding the target text, the target text is analyzed to extract phonemes, punctuation marks, prosodic information and target identifiers, such as SIL identifiers, to obtain a text sequence. The complete text sequence is shown in Figure 3. The SIL identifier not only has the function of representing silence, but also provides information about word boundaries, so that the training text and the text to be inferred (target text) are aligned. The first row in the lower half consists of phoneme sequences, SIL identifiers and punctuation marks. The second to fourth rows represent: prosodic words, prosodic phrases and intonation phrases, respectively.

[0048] The target identifier (SIL identifier) ​​is used to provide word boundary information. Since the input text sequence is composed of phonemes, inserting a SIL identifier between each word can represent the correspondence between phonemes and words. On the one hand, this enriches the text information, and on the other hand, it can provide word duration information when predicting phoneme duration, thereby improving the accuracy of phoneme duration information.

[0049] When preparing training data, since both audio and text data are available, text-acoustic alignment (aligner) tools can be used to predict the duration and corresponding position of silence in the audio. However, in the actual inference process, the corresponding audio for the text is missing, and the location of silence cannot be known in advance. SIL markers are inserted between the phonemes corresponding to each word. If silence occurs between words, the duration is assigned to the SIL marker; otherwise, the duration corresponding to the SIL marker is 0. This allows the text sequences of the training and inference data to be aligned.

[0050] As shown in Figure 4, the text sequence with the target identifier inserted is input into the text encoder for encoding to obtain the target features, and the target features are used to predict the duration information and F0 information of the frame granularity. The text encoder is mainly composed of the Transformer layer.

[0051] As shown in Figure 5, taking the initial text "My mother held me tightly in her arms at that time" as an example, the modified position is "Mom", and the feature vector corresponding to "Mom" is masked to obtain a mask vector. Specifically, the mask vector can be determined based on the modified position and the initial speech, and the duration and basic frequency characteristics of the initial speech are obtained from the initial speech; the masked duration in the duration of the initial speech and the masked part in the basic frequency characteristics are determined according to the modified position; the duration mask vector in the mask vector is determined according to the masked duration in the duration of the initial speech, and the basic frequency mask vector in the mask vector is determined based on the masked part in the basic frequency characteristics.

[0052] In actual application scenarios, the mask vector generation process is as follows: The feature mask vectors are constructed using the recording and text editing information: the masked duration embedding and the masked f0 embedding. As shown in Figure 5, the complete duration information and f0 features of the initial speech are first extracted from the initial speech. The duration and f0 features are then masked using the modified position information in the initial text. These are then sent to the embedding layer (a neural network layer used to map discrete features to a continuous vector space) to obtain the corresponding vectors.

[0053] Among them, the specific steps of determining the basic frequency information of the frame granularity based on the mask vector and the target text are as follows: determining the duration information of the phoneme granularity based on the target feature and the duration mask vector; determining the target feature of the frame granularity based on the duration information of the phoneme granularity and the target feature; determining the basic frequency information of the frame granularity according to the target feature of the frame granularity and the basic frequency mask vector; the specific steps of determining the duration information of the phoneme granularity based on the target feature and the duration mask vector are as follows: obtaining the masked phoneme duration information in the duration mask vector and the masked word duration information in the duration mask vector; determining the duration information of the phoneme granularity based on the phoneme duration information and the word duration information, wherein the word duration information is determined by the adjacent target identifiers.

[0054] It should be noted that the duration of the frame granularity and the FO information of the frame granularity can be predicted by a feature predictor. It can be understood that in the embodiment of the present application, the feature predictor is mainly optimized from three aspects: autoregressive prediction, feature mask vector and duration information prediction. The overall process is: first, the duration mask vector and f0 mask vector corresponding to the feature are extracted based on the target text and the initial speech; then, the target feature and the mask vector are input into the duration feature predictor to predict the duration information, and the target feature is upsampled based on the duration information and expanded to the target feature at the frame level; then, the frame-level target feature and the f0 mask vector are input into the f0 feature predictor to predict the f0 information at the frame level.

[0055] It should be noted that the feature predictor is divided into a duration predictor based on phoneme granularity and an f0 predictor based on frame granularity. The feature predictor is mainly composed of RNN (Recurrent Neural Network, a neural network structure), Prenet (a structure for preprocessing input data) and Postnet (a structure for post-processing output data), among which Prenet and Postnet are composed of linear layers.

[0056] To improve the sound quality and expressiveness of speech, the feature predictors in the embodiments of this application all use an autoregressive prediction method. The prediction result at time t must rely on the output at time t-1 as input, fully utilizing this information to improve audio quality and expressiveness. The addition of mask vectors and autoregressive predictions enables the model to fully utilize the prior knowledge provided by the recording for the unedited parts, while making predictions based on this prior knowledge for the edited parts. This allows the final predicted duration and f0 information of the audio signal to restore the rhythm and timbre associated with the recorded speaker while improving the naturalness and fluency between the edited and unedited parts.

[0057] It should also be explained that in order to improve the prediction accuracy of the duration information of phoneme granularity, in an embodiment of the present application, a method for assisting the prediction of the duration information of phoneme granularity based on the duration information of the word is provided. Specifically, the duration information of the word is constructed by simultaneously utilizing the duration information of the phoneme and the word boundary provided by the SIL identifier, and the corresponding loss function also includes losses at both the word and phoneme levels.

[0058] The prediction process of duration information and F0 information is shown in the following formula:

[0059] Where, represents the duration of the prediction at time t, represents the state variable of the model, x t Represents the vector corresponding to the target feature output by the text encoder at time t, mx t Represents the duration mask vector at time t.

[0060] Among them, the predicted duration at time t is and the state variables of the model Depends on time t-1 and And the text encoding output vector x at time t t And the time-length mask vector mx at time t t Similarly, the predicted time t and state variables The corresponding input includes the time t-1 and and x at time t t and mf t .

[0061] In some embodiments of the present application, the specific steps of constructing a mel-coded vector based on the mel-spectrogram features of the target text and the initial speech include: determining a masked position in the mel-spectrogram features of the initial speech based on the modified position; and constructing the mel-coded vector based on the masked position in the mel-spectrogram features of the initial speech.

[0062] In actual application scenarios, as shown in Figure 6, a mel-encoded vector can be constructed using a mel encoder. The input of the mel encoder is a masked mel vector constructed based on the initial speech. Specifically, the mel spectrum is first extracted from the initial speech, and then the mel spectrum is masked using the slot information of the modified position in the initial text.

[0063] First, the corresponding Mel spectrum features are extracted from the audio. Then, the slot information of the modified position in the initial text is used to mask the Mel spectrum to construct a masked Mel vector. The masked Mel vector is then input into the Mel encoder to obtain the corresponding Mel encoding vector.

[0064] In some embodiments of the present application, a specific process of determining the target speech corresponding to the target text based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector is as follows:

[0065] The target text, the basic frequency information of the frame granularity, and the Mel-coded vector are determined as input data, and input into the attention block in the streaming decoder for analysis to obtain multiple Mel-spectra, wherein the size of the attention block is randomly adjusted within a preset range; the multiple Mel-spectra are synthesized into a target Mel-spectra corresponding to the target text, wherein the loss function of the streaming decoder at least includes a loss value between a true value obtained by using a whole-sentence prediction method and the target Mel-spectra; and the target speech is determined based on the target Mel-spectra.

[0066] It should be noted that multiple Mel spectra are Mel spectra with frame granularity. The decoder adopts different streaming decoding schemes. Different streaming decoding is reflected in the attention block mechanism. During training, the corresponding streaming decoding scheme will be selected according to a predetermined ratio.

[0067] In actual application scenarios, the Mel spectrum can be calculated by a streaming decoder. The streaming decoder is mainly composed of a Transformer layer, which mainly includes a self-attention layer and a position-wise feed-forward layer based on conv1d. For the streaming decoder, the convolution layer conv1 is changed from non-causal conv1d to causal conv1d, and the attention layer is changed from full attention to chunk-based attention. However, such changes will reduce the effect of the decoder in synthesizing speech. In order to achieve streaming calculation while ensuring that the effect is not reduced, the embodiment of the present application makes the following three adjustments to the streaming decoder:

[0068] During training, the chunk size is dynamically set within a certain range, which makes the model robust to the chunk size of streaming computation.

[0069] 2. As shown in Figure 7, the loss function incorporates the mean square error (MSE) loss between chunk-based streaming computation (chunk input, which divides input data into segments) and sentence-based non-streaming computation (full input, which uses the complete content or information of the input). The non-streaming results are used to guide the training of the streaming model.

[0070] 3. The loss function adds the mean square error loss between the streaming synthesized Mel spectrum and the accurate Mel spectrum label in each chunk to further control the training of the streaming model.

[0071] It can be understood that for the streaming decoder, the optimized loss function includes not only the mel loss corresponding to the entire sentence audio, but also the loss between streaming and non-streaming, as well as the mel loss of each chunk, as shown in the following formula:

[0072] In the formula, m represents the true mel corresponding to the whole sentence, Represents the predicted mel corresponding to the entire sentence in streaming mode, Represents the whole sentence mel,m calculated in non-streaming mode i Represents the real mel corresponding to the i-th chunk, Represents the predicted mel of the i-th chunk in streaming mode, and C represents the number of chunks.

[0073] It can be understood that mel represents Mel spectrum.

[0074] In some embodiments of the present application, determining the target speech based on the target Mel spectrum includes: dividing the target Mel spectrum into multiple Mel spectrum blocks, retaining the tail of the Mel spectrum in the previous Mel spectrum block, and padding it with the Mel spectrum in the current Mel spectrum block to obtain multiple padded Mel spectrum blocks; and synthesizing the target speech with the multiple padded Mel spectrum blocks.

[0075] Specifically, to achieve better streaming performance, the vocoder is trained offline. The target Mel-spectrogram of an entire sentence is input and the audio of the entire sentence is output. 1. The target Mel-spectrogram is divided into chunks of varying size, and a certain number of frames (e.g., 15 frames) from the tail of the previous chunk's Mel-spectrogram are retained to pad the Mel-spectrogram of the current chunk as input to the vocoder-generator. 2. The vocoder generator synthesizes the corresponding waveform based on the input target Mel-spectrogram.

[0076] Among them, the waveform corresponding to the filling part is added to the waveform corresponding to the number of frames of the previous chunk to synthesize the waveform corresponding to this chunk.

[0077] The speech editing method provided in the embodiments of this application uses recordings to extract speaker-related prosodic features and construct duration mask vectors and f0 mask vectors. This allows the model to make predictions based on prior knowledge for unedited parts and fully utilize known prior knowledge for edited parts, thereby improving the naturalness and fluency between the edited and unedited parts. A mel encoder is added to construct a mel mask vector using the recordings, and known acoustic information is input into the model through a series of nonlinear transformations to further improve the naturalness and fluency of audio synthesis. In the input text sequence, SIL markers are inserted between the phonemes corresponding to each word to serve as word boundaries and enrich the text information. Soft boundaries for duration information are provided to align the input data for training and inference. The feature predictor is changed from non-autoregressive to autoregressive, allowing the model to fully utilize the contextual information. The duration predictor predicts duration at both the phoneme and word levels, leveraging the mapping relationship between words and phonemes to improve the accuracy of phoneme duration prediction. The decoder adopts a streaming architecture and is optimized to ensure the alignment of streaming and non-streaming computation. The vocoder uses OLA streaming inference.

[0078] It's understandable that this voice editing method can restore the original recording for unedited portions and synthesize audio that aligns with the original speaker's style and timbre for edited portions. The overall naturalness and smoothness of the audio are comparable to a real person's voice, effectively enhancing the user experience in customer service scenarios. The entire system utilizes a streaming synthesis model, capable of synthesizing and outputting audio in segments, effectively reducing the system's first packet latency and response time. The system is capable of supporting real-time interactive streaming voice products such as voice navigation, intelligent customer service, and smart assistants.

[0079] FIG8 is a voice editing device according to an embodiment of the present application. As shown in FIG8 , the device includes:

[0080] An acquisition module 80 is configured to acquire a target text, wherein the target text is a text obtained by modifying an initial text, wherein the initial text is a text corresponding to the initial speech;

[0081] a determination module 82 for determining a modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech;

[0082] A construction module 84 is configured to determine basic frequency information of a frame granularity based on the mask vector and the target text, extract the Mel spectrum features of the initial speech from the initial speech, and construct a Mel encoding vector based on the Mel spectrum features of the target text and the initial speech;

[0083] The generating module 86 is configured to determine the target speech corresponding to the target text based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector.

[0084] The determination module 82 of the above-mentioned speech editing device includes: a determination submodule for converting the target text into a text sequence, wherein the text sequence includes at least a phoneme sequence; inserting a target identifier between the phonemes corresponding to each word in the target text to obtain a target text sequence; encoding the target text sequence to obtain a target feature, determining the position where the initial text is modified from the target feature, and determining the mask vector based on the modified position and the initial speech.

[0085] The determination submodule includes: a determination unit, which is used to obtain the duration and basic frequency characteristics of the initial speech from the initial speech; determine the masked duration in the duration of the initial speech and the masked part in the basic frequency characteristics according to the modified position; determine the duration mask vector in the mask vector according to the masked duration in the duration of the initial speech, and determine the basic frequency mask vector in the mask vector based on the masked part in the basic frequency characteristics.

[0086] The determination unit includes: a first determination subunit and a second determination subunit, the first determination subunit is used to determine the duration information of the phoneme granularity based on the target feature and the duration mask vector; determine the target feature of the frame granularity based on the duration information of the phoneme granularity and the target feature; determine the basic frequency information of the frame granularity according to the target feature of the frame granularity and the basic frequency mask vector.

[0087] The second determination subunit is used to obtain the masked phoneme duration information in the duration mask vector and the masked word duration information in the duration mask vector; determine the duration information of the phoneme granularity based on the phoneme duration information and the word duration information, wherein the word duration information is determined by the adjacent target identifier.

[0088] The construction module 84 includes: a construction submodule for determining the masked position in the Mel spectrum feature of the initial speech based on the modified position; and constructing the Mel encoding vector based on the masked position in the Mel spectrum feature of the initial speech.

[0089] The generation module 86 includes: a generation submodule, which is used to determine the target text, the basic frequency information of the frame granularity and the Mel-coded vector as input data, input them into the attention block in the streaming decoder for analysis, and obtain multiple Mel-spectra, wherein the size of the attention block is randomly adjusted within a preset range; synthesize the multiple Mel-spectra into a target Mel-spectra corresponding to the target text, wherein the loss function of the streaming decoder at least includes a loss value between a true value obtained by using a whole sentence prediction method and the target Mel-spectra; and determine the target speech based on the target Mel-spectra.

[0090] The generation submodule includes: a generation unit, configured to divide the target mel spectrum into a plurality of mel spectrum blocks, retain the tail of the mel spectrum in the previous mel spectrum block, and pad it with the mel spectrum in the current mel spectrum block to obtain a plurality of padded mel spectrum blocks; and synthesize the target speech with the plurality of padded mel spectrum blocks.

[0091] It should be noted that the voice editing device shown in FIG8 is used to execute the voice editing method shown in FIG2 , so the relevant explanations in the above voice editing method are also applicable to this voice editing device and will not be repeated here.

[0092] An embodiment of the present application also provides a computer device, comprising: a memory for storing program instructions; a processor, connected to the memory, for executing the program instructions of the following functions: obtaining a target text, wherein the target text is a text after modifying an initial text, wherein the initial text is a text corresponding to an initial speech; determining a modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech; determining basic frequency information of a frame granularity based on the mask vector and the target text, extracting Mel spectrum features of the initial speech from the initial speech, and constructing a Mel encoding vector based on the target text and the Mel spectrum features of the initial speech; determining a target speech corresponding to the target text based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector.

[0093] It should be noted that the above-mentioned computer device is used to execute the voice editing method shown in Figure 2, so the relevant explanations in the above-mentioned voice editing method are also applicable to the computer device and will not be repeated here.

[0094] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following speech editing method by running the computer program: obtaining a target text, wherein the target text is a text after modifying an initial text, wherein the initial text is a text corresponding to an initial speech; determining a modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech; determining basic frequency information of a frame granularity based on the mask vector and the target text, extracting Mel spectrum features of the initial speech from the initial speech, and constructing a Mel encoding vector based on the target text and the Mel spectrum features of the initial speech; determining a target speech corresponding to the target text based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector.

[0095] It should be noted that the above-mentioned non-volatile storage medium is used to execute the voice editing method shown in Figure 2, so the relevant explanations in the above-mentioned voice editing method are also applicable to the non-volatile storage medium and will not be repeated here.

[0096] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0097] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0099] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0100] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0102] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A voice editing method, comprising: Obtaining a target text, wherein the target text is a text obtained by modifying an initial text, wherein the initial text is a text corresponding to the initial speech; Determining a modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech; Determining basic frequency information of a frame granularity based on the mask vector and the target text, extracting a mel spectrum feature of the initial speech from the initial speech, and constructing a mel encoding vector based on the mel spectrum feature of the target text and the initial speech; A target speech corresponding to the target text is determined based on the target text, basic frequency information of the frame granularity, and the Mel encoding vector.

2. The method according to claim 1, wherein Determining a modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech, comprising: Converting the target text into a text sequence, wherein the text sequence at least includes: a phoneme sequence; Inserting a target identifier between the phonemes corresponding to each word in the target text to obtain a target text sequence; The target text sequence is encoded to obtain target features, the modified position of the initial text is determined from the target features, and the mask vector is determined based on the modified position and the initial speech.

3. The method according to claim 2, wherein: Determining the mask vector based on the modified position and the initial speech, comprising: Acquiring duration and basic frequency features of the initial speech from the initial speech; Determining, according to the modified position, the masked duration of the initial speech and the masked portion of the basic frequency feature; A duration mask vector in the mask vector is determined according to the masked duration in the duration of the initial speech, and a basic frequency mask vector in the mask vector is determined based on the masked portion in the basic frequency feature.

4. The method according to claim 3, wherein: Determining basic frequency information of a frame granularity based on the mask vector and the target text includes: Determining duration information of phoneme granularity based on the target feature and the duration mask vector; Determining a target feature of a frame granularity based on the duration information of the phoneme granularity and the target feature; The basic frequency information of the frame granularity is determined according to the target feature of the frame granularity and the basic frequency mask vector.

5. The method according to claim 4, wherein Determining duration information of phoneme granularity based on the target feature and the duration mask vector includes: Obtaining masked phoneme duration information in the duration mask vector and masked word duration information in the duration mask vector; The duration information of the phoneme granularity is determined based on the phoneme duration information and the word duration information, wherein the word duration information is determined by the adjacent target identifiers.

6. The method according to claim 1, wherein Constructing a mel encoding vector based on the mel spectrum features of the target text and the initial speech, including: Determining a masked position in the Mel-spectrogram feature of the initial speech based on the modified position; The mel-coded vector is constructed based on the masked position in the mel-spectrometric feature of the initial speech.

7. The method according to claim 1, wherein Determining a target speech corresponding to the target text based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector includes: Determine the target text, the basic frequency information of the frame granularity, and the Mel-coded vector as input data, input them into the attention block in the streaming decoder for analysis, and obtain multiple Mel-spectra, wherein the size of the attention block is randomly adjusted within a preset range; synthesizing the multiple Mel spectrums into a target Mel spectrum corresponding to the target text, wherein the loss function of the streaming decoder at least includes a loss value between a true value obtained by using a whole sentence prediction method and the target Mel spectrum; The target speech is determined based on the target Mel-spectrogram.

8. The method according to claim 7, wherein: Determining the target speech based on the target Mel spectrum includes: Dividing the target mel spectrum into multiple mel spectrum blocks, retaining the tail of the mel spectrum in the previous mel spectrum block, and padding it with the mel spectrum in the current mel spectrum block to obtain multiple padded mel spectrum blocks; The target speech is synthesized by using the multiple padded mel-spectrogram blocks.

9. The method according to claim 1, wherein Constructing a mel encoding vector based on the mel spectrum features of the target text and the initial speech, including: Extracting Mel spectrum features from the initial speech; Using the slot information of the modified position in the initial text, masking the Mel spectrum feature to construct a masked Mel vector; Input the masked Mel vector into the Mel encoder to obtain the corresponding Mel encoding vector.

10. A voice editing device, comprising: An acquisition module, configured to acquire a target text, wherein the target text is a text obtained by modifying an initial text, wherein the initial text is a text corresponding to an initial speech; a determination module, configured to determine a modified position in the initial text based on the target text, and determine a mask vector based on the modified position and the initial speech; A construction module, configured to determine basic frequency information of a frame granularity based on the mask vector and the target text, extract the Mel spectrum features of the initial speech from the initial speech, and construct a Mel encoding vector based on the Mel spectrum features of the target text and the initial speech; A generation module is used to determine the target speech corresponding to the target text based on the target text, the basic frequency information of the frame granularity and the Mel encoding vector.

11. A computer device comprising: a memory for storing program instructions; A processor, connected to the memory, is used to execute program instructions for the following functions: obtaining a target text, wherein the target text is a text modified from an initial text, wherein the initial text is a text corresponding to an initial speech; determining a modified position in the initial text based on the target text, and determining a mask vector based on the modified position and the initial speech; determining basic frequency information of a frame granularity based on the mask vector and the target text, extracting a Mel spectrum feature of the initial speech from the initial speech, and constructing a Mel encoding vector based on the target text and the Mel spectrum feature of the initial speech; and determining a target speech corresponding to the target text based on the target text, the basic frequency information of the frame granularity, and the Mel encoding vector.

12. A non-volatile storage medium comprising a stored computer program, wherein: The device where the non-volatile storage medium is located executes the method according to any one of claims 1 to 9 by running the computer program.

Images