Voice information generation method and device, storage medium and electronic equipment

Abstract

The invention discloses a voice information generation method and device, a storage medium and electronic equipment, and relates to the technical field of artificial intelligence and voice synthesis. The method comprises the steps of obtaining a to-be-processed text; segmenting the to-be-processed text into a plurality of characters, and splicing the plurality of characters with the voice prompt information to obtain an initial input sequence; segmenting the initial input sequence into a plurality of sub-input sequences according to a target window and a target step length of the sliding window; and inputting the plurality of sub-input sequences and tone information corresponding to the voice prompt information to a voice synthesis model, and generating voice information corresponding to the to-be-processed text from the plurality of sub-input sequences and the tone information by using the voice synthesis model, the voice synthesis model being based on a text sample and a voice prompt information sample, and training the tone information sample corresponding to the voice prompt information sample. The technical problem that the effect of generating the voice information is poor is solved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and speech synthesis technology, and more specifically, to a method, device, storage medium and electronic device for generating speech information. Background Art

[0002] At present, speech synthesis can be divided into streaming speech synthesis and non-streaming speech synthesis, but most speech synthesis technologies are only applicable to non-streaming solutions. However, although streaming speech synthesis technology is relatively difficult, its application in real-time interactive scenarios is necessary. Streaming speech synthesis can realize the output of speech while inputting text, which greatly reduces the response time and improves the user experience.

[0003] However, streaming speech synthesis needs to ensure real-time synthesis while maintaining the naturalness of the speech and the stability of the timbre. Therefore, the speech synthesis model must not only quickly generate the audio stream, but also handle the clarity and fluency of the audio, and fine-tune the model parameters to adapt to different input texts and application scenarios. In addition, the implementation of streaming speech synthesis technology also requires efficient algorithms and hardware acceleration technology to ensure that the requirements of real-time synthesis can be met on low-configuration central processing units. Therefore, the above problems are all challenges faced by streaming speech synthesis technology.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method, device, storage medium and electronic device for generating voice information, so as to at least solve the technical problem of poor effect of generating voice information.

[0006] According to one aspect of an embodiment of the present application, a method for generating speech information is provided. The method may include: obtaining a text to be processed; dividing the text to be processed into multiple characters, and concatenating the multiple characters with speech prompt information to obtain an initial input sequence; dividing the initial input sequence into multiple sub-input sequences according to a target window and a target step size of a sliding window; inputting the multiple sub-input sequences and the timbre information corresponding to the speech prompt information into a speech synthesis model, and using the speech synthesis model to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information, wherein the speech synthesis model is trained based on text samples, speech prompt information samples, and timbre information samples corresponding to the speech prompt information samples.

[0007] Optionally, a speech synthesis model is used to generate speech information corresponding to the text to be processed from multiple sub-input sequences and timbre information, including: using a speech synthesis model to generate initial speech information from multiple sub-input sequences and timbre information; adjusting the initial speech information to obtain adjusted initial speech information; inputting the adjusted initial speech information into a vocoder, and using the vocoder to generate speech information from the adjusted initial speech information.

[0008] Optionally, the initial voice information includes the first initial voice information of the current window and the second initial voice information of the previous window, the previous window being the previous window of the current window, and adjusting the initial voice information to obtain the adjusted initial voice information, including: determining the overlapping sequences in the first initial voice information and the second initial voice information; removing the sequences in the first initial voice information to obtain the adjusted first initial voice information.

[0009] Optionally, the method further includes: acquiring training samples including text samples, voice prompt information samples and timbre information samples; and training an initial speech synthesis model based on the training samples to obtain a speech synthesis model.

[0010] Optionally, the speech synthesis model includes a masked convolution layer and a dynamic block attention layer. Based on the training samples, the initial speech synthesis model is trained to obtain the speech synthesis model, including: randomly masking the training samples, and using the masked training samples and the dynamic block strategy to train the initial speech synthesis model to obtain the speech synthesis model, wherein the dynamic block strategy is used to adjust the block size during the training of the initial speech synthesis model.

[0011] Optionally, based on the training samples, the initial speech synthesis model is trained to obtain the speech synthesis model, including: adjusting the trained initial speech synthesis model without classifier guidance to obtain the speech synthesis model.

[0012] Optionally, the method also includes: when performing non-streaming reasoning on the text to be processed, inputting the text to be processed, voice prompt information and timbre information into a speech synthesis model, and using the speech synthesis model to generate speech information corresponding to the text to be processed, voice prompt information and timbre information.

[0013] According to another aspect of the embodiment of the present application, a device for generating speech information is also provided. The device may include: an acquisition unit for acquiring a text to be processed; a splicing unit for segmenting the text to be processed into multiple characters, and splicing the multiple characters with speech prompt information to obtain an initial input sequence; a segmentation unit for segmenting the initial input sequence into multiple sub-input sequences according to a target window and a target step size of a sliding window; a generation unit for inputting multiple sub-input sequences and timbre information corresponding to the speech prompt information into a speech synthesis model, and using the speech synthesis model to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and timbre information, wherein the speech synthesis model is trained based on text samples, speech prompt information samples, and timbre information samples corresponding to the speech prompt information samples.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, including: the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute any method for generating voice information.

[0015] According to another aspect of an embodiment of the present application, there is also provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute instructions to implement any method for generating voice information.

[0016] According to another aspect of an embodiment of the present invention, a processor is further provided. The processor is used to run a program, wherein the method for generating voice information in an embodiment of the present invention is executed when the program is run.

[0017] According to another aspect of the embodiment of the present application, a computer program product is further provided. The computer program product may include a computer program, wherein the computer program is configured to execute any of the above-mentioned methods for generating voice information when running.

[0018] According to another aspect of the embodiment of the present application, a computer program product is also provided. The computer program product may include a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and the computer program is configured to execute any of the above-mentioned methods for generating voice information when running.

[0019] According to another aspect of the embodiment of the present application, a computer program is further provided, wherein the computer program is configured to execute any of the above-mentioned methods for generating voice information when running.

[0020] In an embodiment of the present application, a text to be processed is obtained; the text to be processed is divided into multiple characters, and the multiple characters are concatenated with voice prompt information to obtain an initial input sequence; the initial input sequence is divided into multiple sub-input sequences according to a target window and a target step size of a sliding window; the multiple sub-input sequences and the timbre information corresponding to the voice prompt information are input into a speech synthesis model, and the speech synthesis model is used to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information, wherein the speech synthesis model is trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples, thereby achieving an improvement in the effect of generating voice information, thereby solving the technical problem of poor effect of generating voice information. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 is a flow chart of a method for generating voice information according to an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of a streaming synthetic sliding window according to an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a dynamic mask convolutional layer structure according to an embodiment of the present application;

[0025] Figure 4 is a schematic diagram of a dynamic block attention layer structure according to an embodiment of the present application;

[0026] Figure 5 is a schematic diagram of a basic block of an encoder according to an embodiment of the present application;

[0027] Figure 6 is a schematic diagram of a flow model structure according to an embodiment of the present application;

[0028] Figure 7 is a structural schematic diagram of a device for generating voice information according to an embodiment of the present application;

[0029] Figure 8 is a schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] According to an embodiment of the present application, a method embodiment of a method for generating voice information is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Figure 1 is a flow chart of a method for generating voice information according to an embodiment of the present application. Figure 1 As shown, the method may include the following steps:

[0034] Step S102, obtaining the text to be processed.

[0035] In the above step S102, the text to be processed may be obtained, wherein the text to be processed may be text information that needs to be converted into speech during the speech synthesis process, and may also be referred to as the text to be synthesized.

[0036] In this embodiment, in the streaming speech synthesis system, the text to be processed can be real-time input, continuous, that is, text data synthesized while being input. The text to be synthesized is the core input of speech synthesis, and the content of the text to be synthesized can directly determine the semantics and content of the synthesized speech.

[0037] Step S104, dividing the text to be processed into a plurality of characters, and concatenating the plurality of characters with the voice prompt information to obtain an initial input sequence.

[0038] In the above step S104, after obtaining the text to be processed, the text to be processed can be divided into multiple characters, and the multiple characters can be spliced ​​with the voice prompt information to obtain an initial input sequence. The voice prompt information can be an audio clip containing the timbre and style information of a specific speaker, which can be used to provide a timbre and style reference for the model during the speech synthesis process to ensure that the synthesized speech can accurately imitate the voice characteristics of the specific speaker, and can also be called prompt audio.

[0039] In this embodiment, after obtaining the text to be processed, the text to be processed can be divided into multiple characters. Further, the multiple characters can be spliced ​​with the prompt audio, and the prompt audio can be spliced ​​at the beginning of the multiple characters to obtain an initial input sequence.

[0040] Optionally, Figure 2 is a schematic diagram of a streaming synthesis sliding window according to an embodiment of the present application. After obtaining the text to be processed, the text to be processed can be divided into multiple characters, such as Figure 2 As shown, multiple characters can be represented by 0 to 7. The prompt audio can be further concatenated to the beginning of multiple characters to obtain an initial input sequence.

[0041] Step S106 , dividing the initial input sequence into a plurality of sub-input sequences according to the target window and the target step size of the sliding window.

[0042] In the above step S106, after the text to be processed is divided into a plurality of characters, and the plurality of characters are spliced ​​with the voice prompt information to obtain the initial input sequence, the initial input sequence can be divided into a plurality of sub-input sequences according to the target window and the target step size of the sliding window. The target window can be the window size of the sliding window, for example, the target window can be 90 characters (tokens), which is only for illustration and does not specifically limit the size of the target window. The target step size can be the step size of the sliding window, for example, the target step size can be 80, which is only for illustration and does not specifically limit the value of the target step size.

[0043] In this embodiment, in order to further reduce the demand for video memory of the speech synthesis model, a sliding window method can be used, the window size is set to 90 tokens, the step size is set to 80, and the prompt audio is spliced ​​at the beginning of each packet of data to improve the zero-shot capability of the model.

[0044] It should be noted that zero-shot is a method in the field of machine learning that enables the model to recognize, classify, or understand categories or concepts that have not been seen during training. This learning method is crucial to solving the common long-tail distribution problem, that is, for some unknown samples, traditional supervised learning methods may be difficult to handle. The core of zero-shot learning is to transfer the knowledge learned from the training data (such as features, patterns, or relationships) to unseen categories.

[0045] Optionally, after obtaining the initial input sequence, the initial input sequence can be divided into multiple sub-input sequences according to the window size and step size of the sliding window, and the beginning of each sub-input sequence (each data packet) is spliced ​​with prompt audio. Figure 2 As shown, the first data packet contains characters 0 to 3, the second data packet contains characters 2 to 5, the third data packet contains characters 4 to 7, and prompt audio is spliced ​​at the beginning of each data packet.

[0046] Step S108, input multiple sub-input sequences and timbre information corresponding to the voice prompt information into a speech synthesis model, and use the speech synthesis model to convert the multiple sub-input sequences and timbre information into speech information corresponding to the text to be processed, wherein the speech synthesis model is trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples.

[0047] In the above step S108, after the initial input sequence is divided into multiple sub-input sequences according to the target window and the target step size of the sliding window, the multiple sub-input sequences and the timbre information corresponding to the voice prompt information can be input into the speech synthesis model. The speech synthesis model can further be used to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information. The speech synthesis model can be trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples.

[0048] In this embodiment, the prompt audio is spliced ​​in front of the audio to be synthesized in each window, and the timbre information and the text to be synthesized are input into the speech synthesis model, and the speech synthesis model can be used to generate speech information.

[0049] In an embodiment of the present application, a text to be processed is obtained; the text to be processed is divided into multiple characters, and the multiple characters are concatenated with voice prompt information to obtain an initial input sequence; the initial input sequence is divided into multiple sub-input sequences according to a target window and a target step size of a sliding window; the multiple sub-input sequences and the timbre information corresponding to the voice prompt information are input into a speech synthesis model, and the speech synthesis model is used to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information, wherein the speech synthesis model is trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples, thereby achieving an improvement in the effect of generating voice information, thereby solving the technical problem of poor effect of generating voice information.

[0050] In some embodiments of the present application, a speech synthesis model is used to generate speech information corresponding to the text to be processed from multiple sub-input sequences and timbre information, including: using a speech synthesis model to generate initial speech information from multiple sub-input sequences and timbre information; adjusting the initial speech information to obtain adjusted initial speech information; inputting the adjusted initial speech information into a vocoder, and using the vocoder to generate speech information from the adjusted initial speech information.

[0051] In this embodiment, the speech synthesis model can be used to input multiple sub-sequences and timbre information to generate initial speech information. After the initial speech information is generated, the initial speech information can be adjusted to obtain adjusted initial speech information. The adjusted initial speech information is further input into a vocoder, and the vocoder is used to generate speech information from the adjusted initial speech information. The initial speech information can be Mel spectrum information output by the speech synthesis model, such as a Mel spectrum graph.

[0052] Optionally, the prompt audio is spliced ​​before the audio to be synthesized in each window, and after the timbre information and the text to be synthesized are input into the speech synthesis model, the speech synthesis model can be used to generate a Mel spectrum. The generated Mel spectrum can be further adjusted, for example, the overlapping parts in the window are removed to obtain an adjusted Mel spectrum. Finally, the adjusted Mel spectrum is input into a vocoder, and the vocoder can be used to generate speech information from the adjusted Mel spectrum.

[0053] In some embodiments of the present application, the initial voice information includes the first initial voice information of the current window and the second initial voice information of the previous window, where the previous window is the previous window of the current window. The initial voice information is adjusted to obtain the adjusted initial voice information, including: determining the overlapping sequences in the first initial voice information and the second initial voice information; removing the sequence in the first initial voice information to obtain the adjusted first initial voice information.

[0054] In this embodiment, after using a speech synthesis model to generate initial speech information from multiple sub-input sequences and timbre information, the sequences in the initial speech information of the current window that are identical to the initial speech information of the previous window of the current window can be removed to obtain adjusted initial speech information.

[0055] Optionally, the prompt audio is spliced ​​before the audio to be synthesized in each window, and the timbre information and the text to be synthesized are input into the speech synthesis model. After obtaining the Mel spectrum, the overlapping parts in the window can be removed to obtain the adjusted initial speech information. Finally, the adjusted Mel spectrum is sent to the vocoder to obtain the audio. Figure 2 As shown, if characters 2 and 3 in the second package of data are repeated with the characters in the first package of data, then characters 2 and 3 in the second package of data can be removed; if characters 4 and 5 in the third package of data are repeated with the characters in the second package of data, then characters 4 and 5 in the third package of data can be removed.

[0056] In some embodiments of the present application, the method further includes: obtaining training samples including text samples, voice prompt information samples and timbre information samples; and training an initial speech synthesis model based on the training samples to obtain a speech synthesis model.

[0057] In this embodiment, a training sample consisting of a text sample, a voice prompt information sample and a timbre information sample can be obtained. Further based on the obtained training sample, an initial speech synthesis model can be trained to obtain a speech synthesis model. The speech synthesis model can also be called a flow model.

[0058] Optionally, the Flow model is composed of 64 layers of data blocks, each of which is mainly composed of a convolution layer and an attention layer. In order to support streaming reasoning, this embodiment converts the convolution layer and the attention layer into a dynamic block, and in order to make the convolution operation transition smoothly at the edge, a masked convolution scheme is adopted.

[0059] It should be noted that the attention mechanism is a resource allocation strategy that simulates the function of human visual attention, enabling the model to focus on different parts of information. It is widely used in fields such as speech recognition and natural language processing. The attention model enhances the model's discrimination ability by assigning different weights to each element in the sequence, and can explain the relationship between input and output as an alignment relationship. Convolution involves a mathematical operation between a small matrix called a convolution kernel or filter and the input data in order to extract features of the input data. The convolution operation can capture local features, and more complex features can be extracted by stacking multiple convolution layers.

[0060] It should be noted that chunks usually refer to dividing data into chunks. For example, a tensor can be split into multiple sub-tensors along a specified dimension, or a batch of data can be split into multiple small batches for training, which is suitable for processing scenarios where data needs to be evenly divided. Masking is to cover or modify certain parts of the input data during model training to achieve specific learning goals or optimize model performance.

[0061] Optionally, during the model training process, in order to meet the latency requirements of different scenarios, a dynamic chunk design is adopted during training, and the chunk size range is [3, 8], so that after one training, the model can support both non-streaming synthesis and streaming synthesis.

[0062] In some embodiments of the present application, a speech synthesis model includes a masked convolutional layer and a dynamic block attention layer. Based on training samples, an initial speech synthesis model is trained to obtain a speech synthesis model, including: randomly masking the training samples, and using the masked training samples, and a dynamic block strategy to train the initial speech synthesis model to obtain a speech synthesis model, wherein the dynamic block strategy is used to adjust the size of the block during the training of the initial speech synthesis model.

[0063] In this embodiment, the speech synthesis model includes a masked convolution layer and a dynamic chunk attention layer. In order to support both streaming and non-streaming in one training, the Convolution layer and the Attention layer are replaced by Masked Convolution and Dynamic Attention layers, respectively. Masked is used in Convolution mainly to solve the truncation problem caused by the convolution operation. Because in the reasoning process, future information does not exist in the final convolution receptive field of the current block, but is available in the initial convolution receptive field of the subsequent block. Changes in adjacent convolution input features will cause the output features between the two blocks to be suddenly discontinuous. Although feature discontinuity has little effect on the model, it may cause subtle noise in the synthesized audio. Therefore, it is not feasible to directly use dynamic block convolution for streaming speech conversion. By randomly masking a portion of the data at the end of each data during training, the model is made more robust to different future information during reasoning, so that output features can be continuously generated.

[0064] In some embodiments of the present application, an initial speech synthesis model is trained based on training samples to obtain a speech synthesis model, including: adjusting the trained initial speech synthesis model without classifier guidance to obtain a speech synthesis model.

[0065] In this embodiment, during the model training process, the generated speech token is used as a condition, and the optimal transport conditional flow matching model (OT-CFM) is used to train the corresponding Mel spectrogram. In the continuous time normalized flow, a probability density path is constructed from the prior distribution p0(x) to the data distribution q(x) of the Mel spectrogram. The probability density path is composed of the time-dependent vector field The flow φ(t) is generated by the following ordinary differential equation:

[0066]

[0067] in, can be used to represent the initial value obtained from Gaussian noise. φ1(x)~p1(x) can be used to represent the Mel spectrogram in the dataset, t∈[0,1]. We can further use p1(x) to approximate the speech distribution q(x) and sample from it to solve the above equation. In order to learn the vector field v t (x), defines the optimal transmission flow and forces the neural network to learn by minimizing the following loss:

[0068]

[0069] in, The speaker's timbre information (denoted as e), speech token (denoted as {μ l} 1:L ) and the corresponding masked Mel-spectrogram (denoted as x ′ 1) Input into the flow model for training to obtain the network parameters θ that can predict the vector field:

[0070]

[0071] Among them, x ′ 1 is the result of masking the Mel-spectrogram x1, which is obtained by setting a random range of consecutive frames to zero and adding the time embedding t.

[0072] In addition, the Classifier Free Guidance (CFG) technique is used to improve the generation quality of the diffusion probability model during training. l} 1:L ,x ′ 1}, with a fixed probability of 0.2. In this way, both conditional and unconditional flows can be learned. The vector field during the generation process is as follows:

[0073]

[0074] Wherein, β is a weight coefficient, which is preferably set to 0.7.

[0075] In some embodiments of the present application, the method also includes: when performing non-streaming reasoning on the text to be processed, inputting the text to be processed, voice prompt information and timbre information into a speech synthesis model, and using the speech synthesis model to generate speech information corresponding to the text to be processed, voice prompt information and timbre information.

[0076] In this embodiment, the reasoning can be divided into non-streaming reasoning and streaming reasoning. The non-streaming reasoning method is relatively simple, and the prompt audio, the text to be synthesized, and the speaker's timbre information are directly fed into the model to obtain the voice information.

[0077] In an embodiment of the present application, a text to be processed is obtained; the text to be processed is divided into multiple characters, and the multiple characters are concatenated with voice prompt information to obtain an initial input sequence; the initial input sequence is divided into multiple sub-input sequences according to a target window and a target step size of a sliding window; the multiple sub-input sequences and the timbre information corresponding to the voice prompt information are input into a speech synthesis model, and the speech synthesis model is used to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information, wherein the speech synthesis model is trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples, thereby achieving an improvement in the effect of generating voice information, thereby solving the technical problem of poor effect of generating voice information.

[0078] In order to help those skilled in the art better understand the technical solution of the present application, an example is now provided in conjunction with a specific embodiment.

[0079] Currently, speech synthesis technology has been rapidly developed. In recent years, speech synthesis technology has made significant progress and reached a natural level comparable to human speech. Zero-shot speech synthesis systems can generate the voice of any speaker with minimal registration recordings or audio prompts. Zero-shot speech synthesis has a wide range of applications, including speech-to-speech translation, audio or video content creation, and personal assistant services. However, most synthesis technologies are mostly only applicable to non-streaming scenarios.

[0080] Although streaming speech synthesis technology is technically difficult, its application necessity in real-time interactive scenarios is obvious. Streaming synthesis can realize voice output while inputting text, which greatly reduces system response time and improves user experience. In scenarios such as smart assistants, online customer service, and virtual character interaction, streaming speech synthesis can provide smooth and natural voice interaction, enhancing user experience and satisfaction. However, the difficulty of streaming speech synthesis lies in how to ensure the naturalness of speech and the stability of timbre while synthesizing in real time. This requires the model to not only quickly generate audio streams, but also handle the clarity and fluency of the audio, and fine-tune the model parameters to adapt to different input texts and application scenarios. In addition, efficient algorithms and hardware acceleration technologies are required to ensure that the requirements of real-time synthesis can be met on low-configuration central processing units (CPUs). These are the challenges faced by streaming speech synthesis technology.

[0081] In order to solve the above problems, this application considers the requirements of real-time audio and speech synthesis and video memory occupancy, and proposes a streaming Flow speech synthesis method based on dynamic Chunks with Masked and sliding windows, and performs prompt transmission in each packet of data transmission to generate a stable timbre, thereby enhancing the fluency of human-computer interaction, improving user experience, and meeting the requirements of zero-shot capabilities in real scenarios. On the basis of meeting the requirements of real-time streaming speech synthesis, this embodiment enables the model to also have zero-shot capabilities, so that the synthesized speech has more styles, meets the needs of different groups, and controls the delay within a reasonable range.

[0082] In this embodiment, the entire Flow model is composed of 64 layers of blocks, and each block is mainly composed of a Convolution layer and an Attention layer. In order to support streaming reasoning, the embodiment converts the Convolution layer and the Attention layer into a dynamic Chunk method, and in order to make the Convolution operation transition smoothly at the edge, a Masked Convolution solution is adopted. In order to meet the latency requirements of different scenarios, the Dynamic Chunk design is adopted during training, and the Chunk size range is [3, 8], so that the model can support both non-streaming synthesis and streaming synthesis after one training. In order to further reduce the model's demand for video memory, the embodiment adopts a sliding window method, with the window size set to 90 tokens and the step size set to 80, and prompt audio is spliced ​​at the beginning of each packet of data to improve the model's Zero-shot capability.

[0083] First, for the construction of the model structure, in order to support both streaming and non-streaming in one training, the Convolution layer and Attention layer are replaced by Masked Convolution and Dynamic Attention layers respectively. Figure 3 is a schematic diagram of a dynamic mask convolution layer structure according to an embodiment of the present application, such as Figure 3As shown in the figure, it describes how to process streaming input data through a dynamic mask mechanism in a one-dimensional convolution operation, ensuring that the model can generate smoother and more continuous output features during real-time synthesis or conversion, thereby improving the quality of synthesized speech. Among them, numbers 1 to 7 can be used to represent different positions of input data on the time axis and the processing order of convolution operations. For example, number 1 may represent the current processing time step, while numbers 2, 3, etc. represent subsequent time steps. Padding zero values ​​or specific values ​​are added at both ends of the input sequence to keep the output size after the convolution operation equal to the input size. During training or inference, masking is applied to some parts of the input data, that is, setting its value to zero or performing other types of modifications. Convolution calculations are performed with the input data through the convolution kernel, and features are extracted by sliding windows. The one-dimensional convolution (Conv 1d) layer is used to process one-dimensional input data. The equivalent two-dimensional convolution (Equivalent Conv2d) operation uses a combination of one-dimensional convolution with depthwise separable convolution, convolution enhancement, etc. to achieve an effect similar to two-dimensional convolution to process multi-channel or multi-feature input data. Dynamic Masked Input describes how the model can learn how to continuously generate output features at different locations and different future information availability through dynamic masking.

[0084] Figure 4 is a schematic diagram of a dynamic block attention layer structure according to an embodiment of the present application, such as Figure 4 As shown in the figure, Chunksize=2 means that the input sequence is divided into blocks of size 2. In streaming speech synthesis, each chunk contains a certain number of time steps or feature elements. By setting the dynamic chunk size, the model can flexibly adjust the length of the processed fragment according to the characteristics of the real-time input data during the inference process, so as to better adapt to the real-time input and output of streaming data. Each chunk is only calculated within its own chunk: Since the traditional self-attention mechanism considers all elements in the sequence, this may lead to a large consumption of computing resources and video memory when processing long sequences. The dynamic chunk attention mechanism limits the attention operation of each chunk to only between its internal elements, which reduces the amount of calculation and video memory usage, allowing the model to process real-time streaming data more efficiently.

[0085] Figure 5 is a schematic diagram of a basic block of an encoder according to an embodiment of the present application, such as Figure 5As shown in the figure, the above two modules constitute the encoder basic block. The normalization layer is a preprocessing step before the input data enters the Block, which is used to normalize the data, reduce internal covariate shift, and accelerate the training process of the model. The Feed Forward Module usually contains one or more fully connected layers, which are used to perform nonlinear transformations on the input data and enhance the expressiveness of the model. The Masked Chunk Convolution Module combines the concepts of Masked Convolution and Dynamic Chunk to process local features in the input sequence. The Dynamic Chunk Multi-Head Self Attention Module is an improvement on the traditional multi-head self-attention mechanism, which aims to optimize the processing of streaming data. The Feed Forward Module may be used multiple times in the Block, and each module may adopt different parameters or configurations to more comprehensively process the features of the input sequence. The symbol "+" can be used to represent a residual connection, that is, the output of each module is added to the input to form a residual block, which helps the model learn more complex representations while avoiding the gradient vanishing problem in deep networks. Multiplication by 1 / 2 can be used to represent the halving of feature dimensions, that is, reducing the feature dimensions of the output through certain mechanisms (such as convolution or pooling), which helps reduce the computational complexity and memory usage of the model, especially in streaming processing scenarios.

[0086] The main purpose of using Masked in Convolution is to solve the truncation problem caused by the convolution operation. During inference, future information does not exist in the final convolution receptive field of the current block, but is available in the initial convolution receptive field of the subsequent block. Changes in adjacent convolution input features can cause sudden discontinuities in the output features between the two blocks. Although this feature discontinuity has little effect on the model, it may cause subtle noise in the synthesized audio. Therefore, it is not feasible to use dynamic block convolution directly for streaming speech conversion. By randomly masking a portion of the data at the end of each data during training, the model is made more robust to different future information during inference, so that output features can be generated continuously.

[0087] Figure 6 is a schematic diagram of a flow model structure according to an embodiment of the present application, such as Figure 6As shown in the figure, the one-dimensional residual network (Resnet 1D) is a network structure suitable for one-dimensional data designed based on the residual network principle. In the field of speech synthesis, Resnet 1D can be used to represent a specific layer or module in the model for processing audio features. It adds the input to the complex transformed output through residual connections to maintain the continuity of the information flow. The Transformer Block is part of the Transformer architecture. In speech synthesis applications, the Transformer Block can process long sequence data and capture dependencies in the sequence to generate high-quality audio features.

[0088] Model training is as described in the above formula and will not be repeated here. Model reasoning can be divided into non-streaming reasoning and streaming reasoning. The non-streaming reasoning method is relatively simple. It only needs to directly send the prompt audio, the text to be synthesized, and the speaker's timbre information into the model. The streaming model reasoning is more complicated. This embodiment uses a sliding window method for reasoning. The prompt audio is spliced ​​in front of the audio to be synthesized in each window, and the timbre information and the text to be synthesized are sent to the model at the same time. In the generated Mel spectrum, we remove the overlapping parts in the window and finally send it to the vocoder to obtain the audio. The reasoning process is as follows: Figure 2 As shown in the figure, this embodiment adopts a dynamic Chunk method in training, so the model supports setting different delays according to requirements in different scenarios.

[0089] In this embodiment, a Flow model composed of a block composed of dynamic Chunk Attention and Masked Convolution is constructed, so that the convolution of the model is more robust to different future information during reasoning. By setting the random Chunk method during training, the model can adapt to the requirements of different scenarios for latency. At the same time, streaming and non-streaming models can be obtained at the same time in one training, which greatly simplifies the training of the model and can effectively improve the robustness of the model to time. Reduce the occupancy of video memory in streaming reasoning to ensure that the model also has Zero-shot synthesis capability in streaming reasoning. Further, a sliding window method is used for speech synthesis, and setting the step size smaller than the window size can make the synthesized speech smoother. At the same time, when synthesizing each packet of speech, the prompt audio and the corresponding timbre will be sent to the model to ensure that the synthesized speech has Zero-shot capability.

[0090] In this embodiment, the model supports streaming speech synthesis and can flexibly adjust the model inference chunk according to the needs of different scenarios to meet the requirements of different delays; the model training is simple, and a model that supports both streaming and non-streaming reasoning can be obtained in one training; the model's zero-shot capability in streaming reasoning can be on par with non-streaming, effectively improving the user experience.

[0091] In an embodiment of the present application, a text to be processed is obtained; the text to be processed is divided into multiple characters, and the multiple characters are concatenated with voice prompt information to obtain an initial input sequence; the initial input sequence is divided into multiple sub-input sequences according to a target window and a target step size of a sliding window; the multiple sub-input sequences and the timbre information corresponding to the voice prompt information are input into a speech synthesis model, and the speech synthesis model is used to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information, wherein the speech synthesis model is trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples, thereby achieving an improvement in the effect of generating voice information, thereby solving the technical problem of poor effect of generating voice information.

[0092] Figure 7 is a structural diagram of a device for generating voice information according to an embodiment of the present application, such as Figure 7 As shown, the speech information generating device 700 may include: an acquiring unit 702 , a concatenating unit 704 , a segmenting unit 706 and a generating unit 708 .

[0093] The acquisition unit 702 is used to acquire the text to be processed.

[0094] The concatenation unit 704 is used to divide the text to be processed into multiple characters, and concatenate the multiple characters with the voice prompt information to obtain an initial input sequence.

[0095] The segmentation unit 706 is used to segment the initial input sequence into a plurality of sub-input sequences according to the target window and the target step size of the sliding window.

[0096] The generation unit 708 is used to input multiple sub-input sequences and timbre information corresponding to the voice prompt information into a speech synthesis model, and use the speech synthesis model to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and timbre information, wherein the speech synthesis model is trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples.

[0097] Optionally, the generation unit 708 includes: a first generation module, which is used to use a speech synthesis model to generate initial speech information from multiple sub-input sequences and timbre information; an adjustment module, which is used to adjust the initial speech information to obtain adjusted initial speech information; and a second generation module, which is used to input the adjusted initial speech information into a vocoder, and use the vocoder to generate speech information from the adjusted initial speech information.

[0098] Optionally, the initial voice information includes the first initial voice information of the current window and the second initial voice information of the previous window, the previous window being the previous window of the current window, and the adjustment module includes: a determination submodule, used to determine the overlapping sequences in the first initial voice information and the second initial voice information; and a removal submodule, used to remove the sequences in the first initial voice information to obtain the adjusted first initial voice information.

[0099] Optionally, the device further includes: a first acquisition unit, used to acquire training samples including text samples, voice prompt information samples and timbre information samples; and a training unit, used to train an initial speech synthesis model based on the training samples to obtain a speech synthesis model.

[0100] Optionally, the speech synthesis model includes a masked convolution layer and a dynamic block attention layer, and the training unit includes: a training module for randomly masking the training samples, and using the masked training samples, and a dynamic block strategy to train the initial speech synthesis model to obtain the speech synthesis model, wherein the dynamic block strategy is used to adjust the block size during the training of the initial speech synthesis model.

[0101] Optionally, the training unit includes: an adjustment unit, used to adjust the trained initial speech synthesis model without classifier guidance to obtain a speech synthesis model.

[0102] Optionally, the device also includes: a first generation unit, which is used to input the text to be processed, voice prompt information and timbre information into a speech synthesis model when performing non-streaming reasoning on the text to be processed, and use the speech synthesis model to generate speech information corresponding to the text to be processed from the text to be processed, voice prompt information and timbre information.

[0103] In the device, the text to be processed is obtained by the acquisition unit 702. The text to be processed is divided into multiple characters by the splicing unit 704, and the multiple characters are spliced ​​with the voice prompt information to obtain an initial input sequence. The initial input sequence is divided into multiple sub-input sequences according to the target window and the target step size of the sliding window by the segmentation unit 706. The multiple sub-input sequences and the timbre information corresponding to the voice prompt information are input into the speech synthesis model by the generation unit 708, and the speech synthesis model is used to generate the speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information, wherein the speech synthesis model is trained based on the text sample, the voice prompt information sample, and the timbre information sample corresponding to the voice prompt information sample, thereby achieving the effect of improving the generated speech information, thereby solving the technical problem of poor effect of generated speech information.

[0104] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, the non-volatile storage medium including a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any method for generating voice information.

[0105] Specifically, the above storage medium is used to store program instructions for the following functions to implement the following functions:

[0106] Acquire a text to be processed; divide the text to be processed into multiple characters, and concatenate the multiple characters with voice prompt information to obtain an initial input sequence; divide the initial input sequence into multiple sub-input sequences according to a target window and a target step size of a sliding window; input the multiple sub-input sequences and timbre information corresponding to the voice prompt information into a speech synthesis model, and use the speech synthesis model to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information, wherein the speech synthesis model is trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples.

[0107] Optionally, in this embodiment, the above-mentioned storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above-mentioned contents. More specific examples of the above-mentioned storage medium may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above-mentioned contents.

[0108] In an exemplary embodiment of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods for generating voice information is implemented.

[0109] Optionally, the computer program may implement the following steps when executed by a processor:

[0110] Acquire a text to be processed; divide the text to be processed into multiple characters, and concatenate the multiple characters with voice prompt information to obtain an initial input sequence; divide the initial input sequence into multiple sub-input sequences according to a target window and a target step size of a sliding window; input the multiple sub-input sequences and timbre information corresponding to the voice prompt information into a speech synthesis model, and use the speech synthesis model to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information, wherein the speech synthesis model is trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples.

[0111] According to an embodiment of the present application, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned methods for generating voice information.

[0112] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0113] Figure 8 is a schematic block diagram of an electronic device according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0114] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0115] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the generation method of voice information. For example, in some embodiments, the generation method of voice information may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the generation method of voice information described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the generation method of voice information by any other appropriate means (e.g., by means of firmware).

[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0119] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0123] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0124] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0127] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0129] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating voice information, characterized in that: include: Get the text to be processed; Segmenting the text to be processed into a plurality of characters, and concatenating the plurality of characters with the voice prompt information to obtain an initial input sequence; According to the target window and the target step size of the sliding window, the initial input sequence is divided into a plurality of sub-input sequences; The multiple sub-input sequences and the timbre information corresponding to the voice prompt information are input into a speech synthesis model, and the speech synthesis model is used to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information, wherein the speech synthesis model is trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples.

2. The method according to claim 1, characterized in that The speech synthesis model is used to generate speech information corresponding to the text to be processed by combining the multiple sub-input sequences and the timbre information, including: Using the speech synthesis model to generate initial speech information from the multiple sub-input sequences and the timbre information; Adjusting the initial voice information to obtain adjusted initial voice information; The adjusted initial voice information is input into a vocoder, and the vocoder is used to generate the voice information from the adjusted initial voice information.

3. The method according to claim 2, characterized in that The initial voice information includes first initial voice information of a current window and second initial voice information of a previous window, wherein the previous window is a window previous to the current window, and the initial voice information is adjusted to obtain adjusted initial voice information, including: Determining overlapping sequences in the first initial voice information and the second initial voice information; The sequence in the first initial voice information is removed to obtain adjusted first initial voice information.

4. The method according to claim 1, characterized in that The method further comprises: Acquire a training sample including the text sample, the voice prompt information sample and the timbre information sample; Based on the training samples, an initial speech synthesis model is trained to obtain the speech synthesis model.

5. The method according to claim 4, characterized in that The speech synthesis model includes a masked convolution layer and a dynamic block attention layer. Based on the training sample, the initial speech synthesis model is trained to obtain the speech synthesis model, including: The training samples are randomly masked, and the initial speech synthesis model is trained using the masked training samples and a dynamic block strategy to obtain the speech synthesis model, wherein the dynamic block strategy is used to adjust the block size during the training of the initial speech synthesis model.

6. The method according to claim 4, characterized in that Based on the training samples, an initial speech synthesis model is trained to obtain the speech synthesis model, including: The trained initial speech synthesis model is adjusted without classifier guidance to obtain the speech synthesis model.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When non-streaming reasoning is performed on the text to be processed, the text to be processed, the voice prompt information and the timbre information are input into the speech synthesis model, and the speech synthesis model is used to transform the text to be processed, the voice prompt information and the timbre information to generate the speech information corresponding to the text to be processed.

8. A device for generating voice information, characterized in that: include: An acquisition unit, used for acquiring the text to be processed; A concatenation unit, used for dividing the text to be processed into a plurality of characters, and concatenating the plurality of characters with the voice prompt information to obtain an initial input sequence; A segmentation unit, used for segmenting the initial input sequence into a plurality of sub-input sequences according to a target window and a target step size of the sliding window; A generation unit is used to input the multiple sub-input sequences and the timbre information corresponding to the voice prompt information into a speech synthesis model, and use the speech synthesis model to generate speech information corresponding to the text to be processed from the multiple sub-input sequences and the timbre information, wherein the speech synthesis model is trained based on text samples, voice prompt information samples, and timbre information samples corresponding to the voice prompt information samples.

9. A non-volatile storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

Images