A voice processing method, apparatus, device, medium and program product

Abstract

Embodiments of the application disclose a speech processing method, device, equipment, medium and program product. The method comprises: obtaining text to be converted and reference speech; performing feature extraction processing on the text to obtain a text embedding vector; performing continuous speech labeling processing on the reference speech to obtain a multi-dimensional speech embedding vector; performing speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence; and performing conversion processing on the predicted speech sequence to generate text reading speech. The embodiments of the application can maximize the retention of acoustic details of the reference speech in the text-to-speech process, thereby improving the conversion effect of text-to-speech.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to the field of artificial intelligence, and specifically relates to a speech processing method, a speech processing device, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Text-to-speech technology is a technology that takes text and reference speech as input and outputs text reading speech that reads the text in the style of the reference speech.

[0003] Currently, text-to-speech technology is based on a discrete speech marker to discretely mark the reference speech to extract the acoustic details of the reference speech. However, the discrete speech marker discretely converts the input reference speech into a single numerical value, which loses a large amount of acoustic details of the reference speech, resulting in a loss of speech feature extraction for the reference speech in the text-to-speech process and reducing the conversion effect of text-to-speech. SUMMARY

[0004] The embodiments of the present application provide a speech processing method, device, equipment, medium and program product, which can maximize the preservation of acoustic details of the reference speech in the text-to-speech process, thereby improving the conversion effect of text-to-speech.

[0005] In one aspect, the embodiments of the present application provide a speech processing method, which comprises:

[0006] obtaining text to be converted and reference speech;

[0007] performing feature extraction processing on the text to obtain a text embedding vector, the text embedding vector being used to represent the semantics of the text;

[0008] performing continuous speech marking processing on the reference speech to obtain a multi-dimensional speech embedding vector; the multi-dimensional speech embedding vector being used to represent the acoustic features of the reference speech;

[0009] performing speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence; the predicted speech sequence including a plurality of predicted speech information to which the text reading speech corresponding to the text refers when the text reading speech is read;

[0010] performing conversion processing on the predicted speech sequence to generate text reading speech, the acoustic features of the text reading speech matching the acoustic features of the reference speech.

[0011] In another aspect, the embodiments of the present application provide a speech processing device, which comprises:

[0012] an obtaining unit configured to obtain text to be converted and reference speech;

[0013] a processing unit, configured to perform feature extraction processing on the text to obtain a text embedding vector, the text embedding vector being used to represent semantics of the text;

[0014] the processing unit is further configured to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector, the multi-dimensional speech embedding vector being used to represent acoustic features of the reference speech;

[0015] the processing unit is further configured to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence, the predicted speech sequence comprising a plurality of predicted speech information to which the text reading speech corresponds when being read;

[0016] the processing unit is further configured to perform conversion processing on the predicted speech sequence to generate the text reading speech, an acoustic feature of the text reading speech being matched with an acoustic feature of the reference speech.

[0017] In an implementation manner, when the processing unit is configured to perform continuous speech tagging processing on the reference speech to obtain the multi-dimensional speech embedding vector, the processing unit is specifically configured to:

[0018] perform segmentation processing on the reference speech to obtain at least one speech segment;

[0019] perform continuous speech tagging processing on each speech segment to obtain a continuous speech tag of each speech segment, the continuous speech tag of the speech segment being expressed as a multi-dimensional vector, the continuous speech tag comprising a plurality of continuous values, the plurality of continuous values being capable of continuously representing acoustic features of the speech segment in a time step order of the speech segment, and the acoustic features comprising at least one of the following: pitch, intensity, timbre and duration;

[0020] combine the continuous speech tags of the at least one speech segment to obtain the multi-dimensional speech embedding vector according to a playing position of the at least one speech segment in the reference speech, and a vector dimension of the multi-dimensional speech embedding vector being the same as a total number of the continuous values included in the continuous speech tags of the at least one speech segment.

[0021] In an implementation manner, when the processing unit is configured to perform continuous speech tagging processing on each speech segment to obtain a continuous speech tag of each speech segment, the processing unit is specifically configured to:

[0022] perform resampling processing on each speech segment to obtain a resampled speech segment;

[0023] perform multi-level feature extraction processing on the resampled speech segment to obtain the continuous speech tag of the resampled speech segment.

[0024] In an implementation manner, the text-to-speech voice is read in a time step sequence, and the time step is represented as t, t is a non-negative integer; the processing unit is configured to perform speech prediction processing on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence, and specifically configured to:

[0025] When the time step t is 0, the multi-dimensional speech embedding vector and the text embedding vector are subjected to speech prediction processing to obtain the predicted speech information of the time step 0;

[0026] When the time step t is greater than 0, the multi-dimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before the time step t are subjected to speech prediction processing to obtain the predicted speech information of the time step t;

[0027] Until the time step t reaches a predicted time length required for playing the text-to-speech voice;

[0028] The predicted speech information of each time step t constitutes the predicted speech sequence.

[0029] In an implementation manner, when the processing unit is configured to perform speech prediction processing on the multi-dimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before the time step t to obtain the predicted speech information of the time step t, the processing unit is specifically configured to:

[0030] The multi-dimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before the time step t are subjected to vector connection to obtain a connected vector;

[0031] The connected vector is subjected to regression processing to generate the predicted speech information of the time step t.

[0032] In an implementation manner, when the processing unit is configured to perform conversion processing on the predicted speech sequence to generate the text-to-speech voice corresponding to the text, the processing unit is specifically configured to:

[0033] Each predicted speech information in the predicted speech sequence is subjected to convolution processing to obtain a key speech feature corresponding to each predicted speech information;

[0034] Based on the key speech feature corresponding to each predicted speech information, a predicted speech corresponding to each predicted speech information is generated;

[0035] Each predicted speech is connected in a time step sequence to generate the text-to-speech voice corresponding to the text.

[0036] In an implementation manner, the speech processing method is implemented through a text-to-speech model, and the text-to-speech model comprises a target text marking module, a target continuous speech marking module, a target speech prediction module and a target speech decoding module.

[0037] The target text labeling module is configured to perform feature extraction on the text to obtain a text embedding vector;

[0038] The target continuous speech labeling module is configured to perform continuous speech labeling on the reference speech to obtain a multi-dimensional speech embedding vector;

[0039] The target speech prediction module is configured to perform speech prediction on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence;

[0040] The target speech decoding module is configured to convert the predicted speech sequence to generate text reading speech corresponding to the text.

[0041] In an implementation manner, parameters of the target text labeling module are preset; and the training process of the text-to-speech model comprises:

[0042] obtaining a first data set, the first data set comprising training data audio and corresponding text labels; the text labels being obtained by converting the corresponding training data audio into text;

[0043] optimizing the initial continuous speech labeling module and the initial speech decoding module using the first data set to obtain the target continuous speech labeling module and the target speech decoding module;

[0044] obtaining a second data set, the second data set comprising training text embedding vectors corresponding to training texts, training multi-dimensional speech embedding vectors corresponding to training speeches, and audio labels corresponding to the training speeches; the training multi-dimensional speech embedding vectors being obtained by performing continuous speech labeling on the training speeches using the target continuous speech labeling module;

[0045] optimizing the initial speech prediction module using the second data set to obtain the target speech prediction module.

[0046] In an implementation manner, when the processing unit is used to optimize the initial continuous speech labeling module and the initial speech decoding module using the first data set to obtain the target continuous speech labeling module and the target speech decoding module, the processing unit is specifically configured to:

[0047] performing continuous speech labeling on the training data audio using the initial continuous speech labeling module to obtain the training multi-dimensional speech embedding vectors;

[0048] performing conversion processing on the training multi-dimensional speech embedding vectors using the initial speech decoding module to generate predicted text reading speech corresponding to the training data audio;

[0049] converting the predicted text reading speech into predicted text information in the form of text;

[0050] The initial continuous speech tagging module and the initial speech decoding module are optimized in a direction of reducing a difference between the predicted text information and the text label corresponding to the training data audio, to obtain a target continuous speech tagging module and a target speech decoding module.

[0051] In an implementation manner, when the processing unit optimizes the initial speech prediction module using the second data set to obtain the target speech prediction module, the processing unit is specifically configured to:

[0052] The training text is subjected to speech prediction processing based on the training text embedding vector and the training multi-dimensional speech embedding vector using the initial speech prediction module, to obtain a training predicted speech sequence.

[0053] The audio label is encoded into a predicted speech sequence label, and the predicted speech sequence label includes real speech information of each time step of the training speech after being encoded.

[0054] The initial speech prediction module is optimized in a direction of reducing a difference between the training predicted speech sequence and the predicted speech sequence label, to obtain a target speech prediction module.

[0055] In an implementation manner, the training predicted speech sequence includes training predicted speech information of each time step, and the time step is represented as t. When t is an integer greater than zero, when the processing unit optimizes the initial speech prediction module using the training text embedding vector and the training multi-dimensional speech embedding vector to obtain the training predicted speech sequence, the processing unit is specifically configured to:

[0056] The training speech embedding vector, the training text embedding vector, and real speech information of each time step before the time step t in the predicted speech sequence label are concatenated into a training concatenated vector.

[0057] The training concatenated vector is subjected to regression processing using the initial speech prediction module, to obtain the training predicted speech information of the time step t.

[0058] In another aspect, an embodiment of the present application provides a computer device, which comprises:

[0059] A processor is configured to load and execute a computer program.

[0060] A computer readable storage medium stores the computer program, and the computer program is executed by the processor to implement the speech processing method.

[0061] In another aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is adapted to be loaded and executed by a processor to implement the speech processing method.

[0062] In another aspect, an embodiment of the present application provides a computer program product, which comprises computer instructions, and the computer instructions, when executed by a processor, implement the voice processing method described above.

[0063] In an embodiment of the present application, after obtaining the text to be converted and the reference voice, the text is subjected to feature extraction processing to obtain a text embedding vector for representing the semantics of the text; at the same time, the reference voice is subjected to continuous speech tagging to obtain a multi-dimensional voice embedding vector; since the continuous speech tagging method is used for the reference voice, the continuous speech tagging method can continuously tag the voice information of the reference voice in the form of a multi-dimensional vector according to the voice playing order, thereby retaining the voice information (or acoustic features) of the reference voice to the maximum extent. In this way, based on the multi-dimensional voice embedding vector of the reference voice obtained by continuous tagging and the text embedding vector of the text, the text is subjected to voice prediction processing, and a predicted voice sequence can be obtained, which includes a plurality of predicted voice information to which the text corresponding text reading voice is read as a reference; considering that a large amount of acoustic features of the reference voice are retained by a plurality of continuous values in the multi-dimensional voice embedding vector, a large amount of acoustic features of the reference voice are also retained in the predicted voice sequence obtained based on the multi-dimensional voice embedding vector. In this way, the predicted voice sequence is subjected to conversion processing, and the text corresponding text reading voice can be generated, and the acoustic features of the text reading voice can be ensured to match the acoustic features of the reference voice. In a popular way, the voice style of the text reading voice matches the voice style of the reference voice, and from the auditory effect, the text reading voice and the reference voice are like voice signals emitted by the same object (such as a person or an animal). Through the above aspects, the embodiment of the present application supports continuous speech representation of the reference voice, specifically representing the continuous acoustic features of the reference voice according to the voice playing order in the voice feature extraction process of the reference voice, and representing the continuous acoustic features in the form of a multi-dimensional voice embedding vector, thereby realizing the maximum retention of the voice information of the reference voice. Compared with the discrete speech tagging which only represents the acoustic features of the reference voice as a single numerical value, the voice information loss of the reference voice in the text-to-voice technology can be reduced. In this way, the text-to-voice based on the multi-dimensional voice embedding vector with rich acoustic features can ensure that the acoustic features of the generated text reading voice match the acoustic features of the reference voice (such as the style and timbre of the text reading voice are close to the reference voice), thereby achieving a better conversion effect in the text-to-voice technology. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0065] Figure 1 is a structural schematic diagram of an existing discrete speech tagger;

[0066] Figure 2 is a structural schematic diagram of a speech processing system provided by an exemplary embodiment of the present application;

[0067] Figure 3 is a flowchart of a speech processing method provided by an exemplary embodiment of the present application;

[0068] Figure 4 is a schematic diagram of an information input interface for uploading text to be converted and reference speech provided by an exemplary embodiment of the present application;

[0069] Figure 5 is a flowchart of a continuous speech tagging process for reference speech using a continuous speech tagger provided by an exemplary embodiment of the present application;

[0070] Figure 6 is a structural schematic diagram of a continuous speech tagger provided by an exemplary embodiment of the present application;

[0071] Figure 7 is a schematic diagram of an existing discrete tag;

[0072] Figure 8 is a flowchart of a speech prediction process using a language model provided by an exemplary embodiment of the present application;

[0073] Figure 9 is a flowchart of another speech processing method provided by an exemplary embodiment of the present application;

[0074] Figure 10 is a model structure schematic diagram of text-to-speech provided by an exemplary embodiment of the present application;

[0075] Figure 11 is a training flowchart of a text-to-speech model provided by an exemplary embodiment of the present application;

[0076] Figure 12 is a connection structure schematic diagram between an initial continuous speech tagging module and an initial speech decoding module provided by an exemplary embodiment of the present application;

[0077] Figure 13 is a structural schematic diagram of a voice processing device provided by an example embodiment of the present application;

[0078] Figure 14 is a structural schematic diagram of a computer device provided by an example embodiment of the present application. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0080] In the embodiments of the present application, a voice processing scheme is provided; specifically, the voice processing scheme is a scheme capable of converting text into text reading voice matching acoustic details of reference voice based on text-to-speech (TTS) technology. Through the voice processing scheme, text can be quickly converted into voice, and the style of the converted text reading voice is similar to that of the reference voice. For users, only the reference voice needs to be specified to obtain text reading voice with similar style, thereby meeting the demand of users for personalized text-to-speech.

[0081] wherein: ①Text or text content refers to a form of expression for representing natural language in written form; according to the length of the text, the text can be divided into a single sentence, a paragraph, or a chapter. Text content of any length is composed of characters or character strings, a character string is composed of at least two characters, and a character can include at least one of the following: Chinese characters (i.e., Hanzi), English characters (i.e., letters), numbers, and punctuation marks (such as commas “,”, periods “.”, and square brackets “【】”). The number of characters included in the text is not limited in the embodiments of the present application, and in some cases, the length of the text can be represented by the number of characters included in the text. The more the number of characters, the longer the length of the text.

[0082] ②Speech or called speech signal is a subset of audio signal. Speech specifically refers to the sound emitted by a person or an animal, such as the sound produced when a human speaks; it is mainly used for language communication, and contains semantic information and emotional information, etc., which can be used to identify the identity of the sounder. That is, speech has acoustic characteristics, and the acoustic characteristics of speech emitted by different sounders are different, so as to effectively distinguish the identity of the sounder according to the acoustic characteristics of the speech. Among them, the acoustic characteristics of speech can include at least one of the following: pitch, intensity, timbre and duration, etc. The pitch refers to the high and low of the sound, which is mainly determined by the frequency of the sound wave, and the higher the frequency of the sound wave, the higher the pitch of the sound. The intensity refers to the strength of the sound, which is mainly determined by the amplitude of the sound wave vibration, and the greater the amplitude of the sound wave vibration, the stronger the intensity of the sound. The timbre refers to the unique quality of the sound, and according to the characteristics of the vocal cords of the sounder, the characteristics of the sound produced by the sounder are different, which can be used to distinguish the source of different sounds; for example, the timbre of the sound of a child is usually more childish and crisp than that of an adult. The duration refers to the length of the sound, which depends on the length of the pronunciation time.

[0083] ③Text-to-speech technology is a technology for converting text content into speech signals; it supports converting text content into speech signals through processes such as speech synthesis, speech recognition and speech conversion, and finally outputs natural and smooth speech signals. In order to achieve the purpose of guiding the final generated speech to meet the acoustic characteristics required by the user in the process of text-to-speech, it supports using a reference speech to guide the conversion of text into speech signals that match the acoustic characteristics of the reference speech. At this time, the text-to-speech technology refers to a technology that takes text and reference speech as input, and outputs text reading speech that can read the text content of the text in the style of the reference speech. For example, the reference speech is a sound signal emitted by a 6-year-old girl, and the text to be converted is "Today's weather is good", then in the process of converting the text into speech, the acoustic characteristics of the reference speech are used to guide the generation of the speech signal corresponding to the text, to ensure that the acoustic characteristics (such as style and timbre) of the text reading speech corresponding to the output text are close to the acoustic characteristics of the 6-year-old girl.

[0084] In practical applications, the reference speech with a guiding role will be phonetically labeled (or referred to as phonetic representation) in the text-to-speech process. The so-called phonetic labeling refers to the process of feature extraction on the reference speech to extract the acoustic features of the reference speech. Currently, a phonetic labeler, which can be referred to as a tokenizer, is used to realize the phonetic labeling of the reference speech in the text-to-speech technology. Specifically, a discrete phonetic labeler is used to realize the phonetic labeling of the reference speech. The discrete phonetic labeler, also referred to as a discrete tokenizer, can convert the reference speech into digital data (the digital data is represented as discrete labels, or referred to as discrete labels); in this way, the acoustic features of the reference speech are represented by a single number. The number of discrete labels is limited, which is referred to as a vocabulary, and the size of the vocabulary is 1024; that is, the value of the discrete label obtained by discretely representing the reference speech is a number in [0, 1023], so as to represent the acoustic features of the reference speech by a single number. It is found through practice that, since the discrete labeler uses a single number to represent the acoustic features of the reference speech, it leads to a large amount of loss of speech information in the calculation process of extracting the acoustic features of the reference speech. Specifically, the higher-dimensional features are discretized into a single number, which greatly loses the speech information of the reference speech. Thus, the conversion effect of the text-to-speech (i.e., the matching degree between the acoustic features of the finally generated text reading speech and the acoustic features of the reference speech is low) is reduced.

[0085] To improve the conversion effect of text-to-speech, the speech processing scheme starting from continuous speech representation provided by the embodiments of the present application specifically provides a text-to-speech technology. The so-called "starting from continuous speech representation" means that the text-to-speech technology is implemented using continuous speech markers of reference speech. As can be seen, unlike discrete speech markers, the embodiments of the present application use continuous speech markers to implement feature extraction for reference speech, greatly reducing the loss of speech information of reference speech in the text-to-speech technology, aiming to achieve a better conversion effect in the text-to-speech technology. Among them, the continuous speech marker or continuous representation speech marker supports encoding the reference speech into continuous markers (or continuous speech markers). The continuous speech markers are multi-dimensional vectors and can be used to represent the acoustic features of the reference speech. That is, the continuous speech marker is different from the discrete speech marker, and the output is continuous representation rather than discrete markers. For example, for the same length of reference speech signal, the number of markers output by the discrete speech marker and the continuous speech marker (i.e. the number of markers used to represent the acoustic features of the reference speech) is the same, the difference is that the continuous marker output by the continuous speech marker is a multi-dimensional vector, while the discrete marker output by the discrete marker is a discrete value in a limited range (i.e. a number in the range of [0, 1023]). In this way, using the continuous speech marker to extract the acoustic features of the reference speech can maximize the preservation of the speech information of the reference speech, so that the continuous speech marker based on rich speech information can achieve better text-to-speech effect.

[0086] The general process of the speech processing scheme starting from continuous speech representation provided by the embodiments of the present application can include: a computer device obtains text to be converted and reference speech, the text is text content that needs to be converted into speech signals, and the reference speech is a reference signal for guiding acoustic features. The computer device performs feature extraction processing on the text to obtain a text embedding vector for representing the semantics of the text. Similarly, the reference speech is processed by a continuous speech marker to obtain a multi-dimensional speech embedding vector (i.e. a multi-dimensional vector), which can be used to represent the acoustic features of the reference speech. Then, the computer device performs speech prediction processing on the text based on the multi-dimensional speech embedding vector of the reference speech and the text embedding vector of the text to obtain a predicted speech sequence. The speech prediction processing aims to use the multi-dimensional speech embedding vector to guide the generation of the text embedding vector to generate a plurality of predicted speech signals that the text reading speech is read with reference to. In this way, the computer device converts the predicted speech sequence, specifically a plurality of predicted speech signals in the predicted speech sequence, to generate the text reading speech corresponding to the text, and considering that most of the speech information of the reference speech is retained to guide the generation of the predicted speech sequence, the acoustic features of the text reading speech and the acoustic features of the reference speech can be ensured to be matched.

[0087] It can be seen that the speech processing scheme starting from continuous speech representation provided by the embodiments of the present application has obvious advantages when converting text to speech. The advantages of the embodiments of the present application are described below by taking the comparison between the present application scheme and the existing mainstream text-to-speech technology as an example, wherein:

[0088] The existing mainstream text-to-speech technology based on a discrete speech marker includes an autoregressive text-to-speech technology based on a discrete speech marker. The autoregressive text-to-speech technology based on a discrete speech marker supports representing text as discrete markers, and generating discrete speech markers from reference speech through a discrete speech marker. Then, the two modalities (i.e., the text of discrete markers and the reference speech of discrete markers) are input into a discrete marker semantic space, so as to generate text reading speech corresponding to the text through an autoregressive large language model. There are mainly three kinds of commonly used discrete speech markers, which are a discrete speech marker based on residual quantization, a discrete speech marker based on grouped residual quantization, and a Hubert discrete speech marker. Among them, the discrete speech marker based on residual quantization can repeatedly perform multiple discrete speech marker operations. Each time the nearest distance vector index is selected, the residual of the speech embedding vector of the reference speech minus the embedding vector of the vocabulary is taken as the input of the next discrete speech marker operation. After repeated multiple times, multiple discrete speech markers are obtained. The discrete speech marker based on grouped residual quantization operates on the speech embedding vector of the reference speech according to the feature dimension on the basis of the discrete speech marker based on residual quantization. After the discrete speech marker operation is completed, each marker is combined as the final output. The Hubert discrete speech marker is a model pre-trained by Hubert. Figure 1 Figure 1 As shown in FIG. 1, the discrete speech marker mainly includes an encoder (Encoder), a quantizer (Quantizer) and a decoder (Decoder) and the like. The reference speech is input into the discrete speech marker in the form of a waveform for resampling, and the output after resampling is input into the encoder. The encoder preliminarily represents the reference speech as a vector form speech embedding vector, and then prepares a vector embedding of a vocabulary. In this way, the speech embedding vector is input into the quantizer. For the speech embedding vector, the quantizer selects the index of the vocabulary vector closest to the speech embedding vector in the vocabulary as a marker, that is, selects a discrete value from 0 to 1023 numbers as the discrete marker of the speech embedding vector.

[0089] ​It is not difficult to find that in the feature extraction process of the reference speech using the discrete speech marker, the loss of speech information of the reference speech mainly occurs in the quantizer part in the discrete speech marking operation (i.e., the feature extraction process); that is, in this step from the speech embedding vector to the discrete marker, the semantic embedding vector with a higher dimension is discretized into a single numerical value (one number in 0 to 1023), thereby losing most of the speech information of the reference speech, and further reducing the conversion effect of text-to-speech conversion. Further, to reduce the speech loss caused by the discrete speech marker, the prior art is usually designed in combination with residual quantization; specifically, the speech embedding vector is represented by multiple discrete markers through multiple discretization (multiple layers of residual quantization generate multiple discrete markers, such as shallow residual quantization mainly retains semantic content, and deep residual quantization mainly retains acoustic details; however, such multiple discretization will bring about computational complexity and redundancy. Therefore, due to the information loss of the reference speech in the discrete speech marking operation, there is a loss in the feature extraction of the text-to-speech, which causes the text-to-speech technology based on the discrete speech marker to have serious performance deficiencies, including but not limited to: poor quality of the text-to-speech generated (such as mismatch between the acoustic features of the text-to-speech and the acoustic features of the reference speech); poor naturalness of the text-to-speech, which is easily identified as machine-converted speech from the auditory effect; poor low-noise quality, i.e., more noise in the text-to-speech; poor signal continuity of the text-to-speech; and small sound loudness of the text-to-speech, and the like.

[0090] However, the present application introduces a continuous speech marker into the text-to-speech technology to extract features of the reference speech, and expects to retain the complete speech information of the reference speech through the continuous speech marker output by the continuous speech marker, so as to improve the quality of the text-to-speech based on the continuous speech marker. Compared with the discrete speech marker which only discretizes the acoustic features of the reference speech into a single numerical value to represent, the loss of speech information of the reference speech in the text-to-speech technology can be reduced. In this way, the text-to-speech based on the multi-dimensional speech embedding vector with rich acoustic features can ensure that the acoustic features of the generated text-to-speech and the acoustic features of the reference speech are matched (such as the style and timbre of the text-to-speech are close to the reference speech), thereby achieving a better conversion effect in the text-to-speech technology.

[0091] The speech processing scheme provided by the present application based on continuous speech representation can automatically convert text into text-to-speech with acoustic features matched with the reference speech as a general text-to-speech technology; this makes the speech processing scheme provided by the present application applicable to various text-to-speech scenarios that need to be converted into text-to-speech. The text-to-speech scenario can include but is not limited to:

[0092] (1) Text-to-speech scenarios are dubbing scenarios. In dubbing scenarios, it is necessary to dub characters in videos (such as movies, TV dramas, advertisements, and games) to attract the attention of listeners or viewers with unique sound effects and convey information. Taking a video dubbing scenario as an example, in a scenario where a character in a video is dubbed, the user can provide a reference voice (a voice belonging to the user who provided the video, or a voice from another authorized user) and the text to be converted; the computer device uses the voice processing method provided in this application to perform feature extraction processing on the text to obtain a text embedding vector, and performs continuous voice tagging processing on the reference voice to obtain a multi-dimensional voice embedding vector; the computer device then performs voice prediction processing on the text based on the text embedding vector and the multi-dimensional voice embedding vector to obtain a predicted voice sequence, and finally performs conversion processing on the predicted voice sequence to generate text reading voice. The acoustic details of the text reading voice are similar to / match the acoustic details of the reference voice provided by the user, thereby realizing the dubbing of the character in the video.

[0093] Therefore, the speech processing solution provided in this application embodiment can be used in various dubbing scenarios to improve the speech quality of dubbing. Furthermore, in some dubbing scenarios, it also helps to improve dubbing efficiency; for example, in dubbing scenarios such as advertising or game dubbing, only one reference voice clip is needed for the same character to perform text-to-speech conversion on multiple texts of the same character, achieving batch text-to-speech conversion.

[0094] (2) Text-to-speech scenarios are speech generation scenarios. Speech generation scenarios may include, but are not limited to: ① Scenarios where speech is generated during video clip generation; for example, the generation of video or movie clips requires ensuring the consistency of multiple modal information (such as ensuring consistency between video images and video audio). Using the text-to-speech technology adopted in this application to generate video or movie clips can simplify the generation process of video clips containing multiple modalities such as continuous images, text, and audio, reduce the generation of audio modalities, and obtain high-quality video clips. ② Speech generation scenarios of text during accessible reading; for example, for visually impaired individuals who cannot see the text content, the speech processing solution provided in this application can be used to generate text reading audio based on reference audio and text content; the text content can be read to visually impaired individuals by playing the text reading audio, greatly improving the convenience for visually impaired individuals to read text content.

[0095] (3) The text-to-speech scenario is an intelligent question-and-answer scenario. Intelligent question-and-answer refers to a human-computer interaction question-and-answer method that receives user questions and answers them in accurate and concise natural language. In intelligent question-and-answer scenarios (such as intelligent question-and-answer applications on in-vehicle systems or intelligent question-and-answer functions deployed in hotel robots), after the computer device receives the user's input question and analyzes and retrieves the text to obtain the text-based answer, it can use the speech processing scheme provided in this application to convert the text-based answer into text-to-speech and play the text-to-speech to answer the user's question in natural language. This improves the simplicity of question-and-answer while ensuring the accuracy of text-to-speech conversion.

[0096] It should be understood that the above description is merely an exemplary product performance and text-to-speech scenario provided by the embodiments of this application, and does not limit the product performance and text-to-speech scenario of the speech processing solution provided by the embodiments of this application. The speech processing solution provided by the embodiments of this application can provide efficient, accurate and convenient text-to-speech services in various text-to-speech scenarios, demonstrating high value and practicality in various text-to-speech scenarios, and helping to improve user experience and satisfaction.

[0097] To facilitate understanding of the speech processing scheme provided in the embodiments of this application, the following is combined with... Figure 2 The illustrated scenario diagram provides a brief introduction to the text-to-speech scenarios involved in the embodiments of this application; for example... Figure 2 As shown, the system includes a terminal 201 and a server 202. This application embodiment does not limit the number and naming of the objects 201, terminals 201 and servers 202.

[0098] The terminal 201 can be a terminal device with a text-to-speech function. A user holds the terminal 201 and can perform user operations on the terminal 201. The terminal 201 can include, but is not limited to, a smart phone (such as a smart phone running an Android system or a smart phone running an Internet Operating System (IOS)), a tablet computer, a portable personal computer, a Mobile Internet Device (MID), a vehicle-mounted device, a head-mounted device, a smart chat robot, a flying device, and the like. The embodiments of the present application do not limit the type of terminal device. The server 202 is a server corresponding to the terminal 201, and is used to interact with the terminal 201 to provide computing and application service support for the terminal 201. The server 202 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, and the like. Basic cloud computing services. The various devices included in the text-to-speech system can be directly or indirectly connected through wired or wireless communication. The embodiments of the present application do not limit the communication mode between devices. For example, the communication mode between devices can include, but is not limited to, HTTP requests, remote procedure calls, sockets, and shared content.

[0099] The voice processing scheme provided by the embodiments of the present application can be executed by a computer device, which is one or more devices in the text-to-speech system. In detail, the computer device is the terminal 201 or the server 202 in the system shown in Figure 2 , or the computer device is the terminal 201 and the server 202 in the system shown in Figure 2 . In other words, the voice processing scheme provided by the embodiments of the present application can be executed by one of the terminal 201 or the server 202, or can be executed by the terminal 201 and the server 202 together. The embodiments of the present application do not limit the type and number of computer devices.

[0100] Taking the computer device as an example, the computer device includes the terminal 201 and the server 202, and combining Figure 2The flow of the voice processing scheme provided by the embodiment of the application is introduced. In the specific implementation, the user can input the text to be converted and the reference voice through the terminal 201. For example, the terminal 201 is an accessibility device, and the user inputs the text to be read by voice and the reference voice through the accessibility device. At this time, the terminal 201 forwards the text to be converted and the reference voice to the server 202. In this way, after receiving the text to be converted and the reference voice, the server 202 performs feature extraction processing on the text to obtain a text embedding vector for representing the semantics of the text. Similarly, the reference voice is processed by continuous speech tagging to obtain a multi-dimensional voice embedding vector (i.e., a multi-dimensional vector), which can be used to represent the acoustic features of the reference voice. Then, the server 202 performs voice prediction processing on the text based on the multi-dimensional voice embedding vector of the reference voice and the text embedding vector of the text to obtain a predicted voice sequence. The server 202 further converts the predicted voice sequence to generate a text reading voice similar to the input reference voice in acoustic details. The server 202 packs the text reading voice and returns it to the terminal 201, so as to play the text reading voice through the loudspeaker of the terminal 201.

[0101] Based on the above-described voice processing scheme and system architecture, the following points need to be explained:

[0102] ①The above-mentioned voice processing scheme provided by the embodiment of the application Figure 2 The system shown is to more clearly illustrate the technical scheme of the embodiment of the application, and does not constitute a limitation on the technical scheme provided by the embodiment of the application. It is known to those skilled in the art that, with the evolution of the system architecture and the appearance of new business scenarios, the technical scheme provided by the embodiment of the application is also applicable to similar technical problems. For example, the above is an example of taking the "computer device" including the terminal and the server as the execution subject of the embodiment of the application, i.e., an example of introducing the voice processing scheme provided by the embodiment of the application by the terminal and the server together. It should be understood that in some cases, if the terminal 201 has strong computing power, the terminal 201 can execute the voice processing scheme alone.

[0103] The voice processing scheme provided in the embodiments of the present application is implemented through a text-to-speech model. Specifically, the trained text-to-speech model can be deployed in a computer device; in this way, when the computer device receives text to be converted and reference voice, the text-to-speech model can be directly called to execute the voice processing scheme, which not only improves the speed of text-to-speech conversion, but also effectively ensures the conversion accuracy of text-to-speech conversion through the model. If the computer device used to execute the voice processing scheme provided in the embodiments of the present application is a terminal, the trained text-to-speech model is deployed in the terminal. If the computer device used to execute the voice processing scheme provided in the embodiments of the present application is a server, the trained text-to-speech model is deployed in the server; in this case, the terminal used by the user transmits the text to be converted and the reference voice to the server for text-to-speech conversion, and the server pushes the converted text-to-speech voice to the terminal.

[0104] Further, when the trained text-to-speech model is deployed in the terminal, it can be deployed in the terminal in the form of a plug-in or an application. For example, the trained text-to-speech model is deployed in the terminal as a system-level plug-in, and any application deployed in the terminal can call the plug-in to implement text-to-speech conversion. For another example, the trained text-to-speech model is deployed in a certain application, and after the terminal starts the certain application, the trained text-to-speech model can be called in the certain application to implement text-to-speech conversion; at this time Figure 2The server in the system shown is a server held by an application store of the application program, and can interact with the application program in the terminal to provide a text-to-speech server. The application program can refer to a computer program for completing one or more specific tasks; according to different dimensions (such as the running mode of the application program, function, etc.), the application program can be classified to obtain the type of the same application program in different dimensions. For example, according to the classification of the running mode of the application program, the application program can include but is not limited to: a client installed in the terminal, a small program (as a subprogram of the client) that can be used without downloading and installing, a web (World Wide Web, global wide area network) application program opened through a browser, and the like. For another example, according to the classification of the function type of the application program, the application program can include but is not limited to: an IM (Instant Messaging, instant messaging) application program, a content interaction application program, and the like. The instant messaging application program refers to an application program based on instant messaging and social interaction on the Internet, and can include but is not limited to: a social application program containing a communication function, a map application program containing a social interaction function, a game application program, and the like. The content interaction application program refers to an application program capable of realizing content interaction, which can be, for example, an online banking application program, a sharing platform application program, a personal space application program, a news application program, and the like. The embodiments of the present application do not limit the specific type of the application program running in the terminal and deployed with the trained text-to-speech model, which is described herein.

[0105] ③In the embodiments of the present application, the relevant data collection and processing should strictly comply with the requirements of relevant laws and regulations. Personal information should be obtained with the knowledge or consent of the individual subject (or with the legal basis for information acquisition), and the subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and individual information subjects. For example, when the embodiments of the present application are applied to specific products or technologies, such as reference speech, the permission or consent of the speaker of the reference speech is required, and the collection, use and processing of relevant data (such as converting the text to be converted into speech) should comply with relevant laws, regulations and standards in the relevant region.

[0106] Based on the above-described voice processing scheme, the embodiments of the present application propose a more detailed voice processing method, which will be described in detail below in combination with the accompanying drawings.

[0107] Figure 3 A flowchart of a voice processing method provided by an example embodiment of the present application is shown; the voice processing method can be executed by the aforementioned computer device, such as a terminal; the voice processing method can include but is not limited to steps S301-S305:

[0108] S301: Obtain the text to be converted and the reference speech.

[0109] The text to be converted is text content to be converted into speech, and the reference speech is reference speech used to guide the acoustic features of the speech to be generated in the text-to-speech process; the text-to-speech technology aims to convert the text into text reading speech whose acoustic features match those of the reference speech.

[0110] According to different text-to-speech scenarios, the sources (i.e., acquisition methods) of the text to be converted and the reference speech will be different. As mentioned above, the text-to-speech scenario is a dubbing scenario - a video dubbing scenario; optionally, the text to be converted and the reference speech can both be provided by the user, specifically, the user uploads the text to be converted and the reference speech through an information input interface provided by the terminal (for example, the information input interface is a service interface 401 provided by an application program with text-to-speech function), so that the computer device can obtain the text to be converted and the reference speech; optionally, the text to be converted is the original dialogue of a movie or TV series, which is provided by a video application program, and the reference speech is provided by the user to help the user convert the original dialogue into text reading speech that meets the user's own language style. For another example, the text-to-speech scenario is a dubbing scenario - a game dubbing scenario; at this time, the text to be converted and the reference speech are both set by the game developer. This way of providing the text to be converted and the reference speech by the game developer can unify the voice content and language style of the same game character in the game scenario, meeting the personalized needs of the game developer for the game character.

[0111] It should be noted that the user can input the text to be converted and the reference speech in the same or different ways in the service interface provided by the application program. For example, the user can directly manually edit and input the text in the information input interface, or import local text content or download text content from the Internet; for another example, the user can import reference speech from local memory or the Internet, or select reference audio from the candidate audio provided by the application program, etc.; the embodiments of the present application do not limit the way in which the user inputs the text to be converted and the reference speech in the information input interface. In addition, Figure 4 The interface content and style of the information input interface shown are examples and do not limit the specific implementation of the user inputting the text to be converted and the reference speech in the embodiments of the present application. It is hereby stated.

[0112] S302: performing feature extraction processing on the text to obtain a text embedding vector.

[0113] After obtaining the text to be converted, the computer device needs to perform feature extraction processing on the text, aiming to learn and extract the semantics of the text, so as to better use the extracted text semantics to perform subsequent tasks (such as text-to-speech). Here, the feature extraction processing of the text is to extract a set of feature vectors from the text to represent the semantics of the text; in the embodiments of the present application, the set of feature vectors extracted from the text is referred to as a text embedding vector, and the text embedding vector is used to represent the semantics of the text.

[0114] In the embodiments of the present application, after supporting the use of a text marker to perform feature extraction processing on the text, a vector-represented text embedding vector is obtained through an embedding layer in a neural network. The text marker is a general term for a class of text processing tools that can effectively help users process and analyze text content; in the embodiments of the present application, the text marker can perform text marking on the input text, aiming to mark important strings (or words, characters, etc.) in the text, omit meaningless characters or strings in the text (such as removing stop words in the text, such as words or symbols that have no effect on semantic understanding, etc.), and perform word segmentation on the text to represent the text content as discrete text markers (such as some strings that frequently appear in the text or have important semantics); then use the embedding layer to convert the text markers output by the text marker into a vector-form text embedding vector. Exemplarily, the formula for using the text marker to perform feature extraction processing on the text can be seen as follows:

[0115] x = Tokenizer Text (T) (1)

[0116]

[0117] Tokenizer Text is a text marker; T is the input text; x is a preliminary vector representation obtained by performing feature extraction on the text; Embeds is an embedding layer; is a text embedding vector of the text, which can be represented as [T0, T1, …], where T i is a sub-text embedding vector obtained by performing feature extraction processing on each text block after blocking the text, i = 0, 1, ….

[0118] The embedding layer mentioned above is a special layer in the neural network, mainly used to map discrete feature representations (i.e., text tokens) to a continuous vector space, and use vector forms to represent the semantics of text tokens, so as to facilitate subsequent learning of the vector form of the text embedding vector; in other words, the embedding layer can map high-dimensional discrete features of text (such as text tokens - words or characters) to vector representations in a low-dimensional continuous space. This conversion enables the vector distance between different vector representations to reflect the semantic relationship between the corresponding high-dimensional discrete features, thereby preserving the semantic relationship between text tokens, such as similarity and relevance between semantics. The neural network mentioned above belongs to the field of machine learning (ML) under artificial intelligence (AI) technology; the basic components of the neural network include an input layer, one or more hidden layers, and an output layer, each layer being composed of neurons and weights between neurons. The input layer receives external input information, which is passed to the hidden layer for processing (such as the embedding layer being a kind of hidden layer), and the output layer finally outputs the result.

[0119] S303: Perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector.

[0120] After the computer device obtains the reference speech, it supports performing continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector of the reference speech; the multi-dimensional speech embedding vector is a multi-dimensional vector representation that can be used to completely and continuously represent the acoustic features of the reference speech. In the embodiments of the present application, by representing the reference speech as a continuous multi-dimensional speech embedding vector, compared with representing the semantic of the reference speech as a number in the related art, the acoustic features of the reference speech can be greatly preserved, such as the speech information of each time step (a time step refers to the time interval between two time points according to the time point distribution of the speech time period, such as a time interval of 0 seconds to 1 second) within the speech time period of the reference speech, so that the text reading speech generated based on more abundant acoustic features can improve the matching between the acoustic features of the text reading speech and the acoustic features of the reference speech, i.e., the output style and timbre of the text reading speech are similar to the reference speech.

[0121] This application provides a continuous speech tagger to perform continuous speech tagging processing on a reference speech. As described above, this continuous speech tagger differs from a discrete speech tagger in that it can output a continuous acoustic representation (i.e., the acoustic features of the reference speech throughout the entire speech time period can be represented and preserved), rather than a discrete tag (i.e., only the initial vector representation of the reference speech mapped to the vector space is quantized as the nearest number within [0, 1023]). Therefore, more acoustic features of the reference speech can be preserved for subsequent text-to-speech processing.

[0122] To facilitate understanding, the following is in conjunction with the appendix. Figure 5 The process of using a continuous speech marker to perform continuous speech tagging on reference speech is described exemplarily. For example... Figure 5 As shown, after acquiring reference speech, the computer device considers that the speech duration of the reference speech is often quite long, such as several hours. If continuous speech tagging is performed on the entire reference speech segment, the excessive speech duration would lead to computational complexity and even reduce tagging accuracy. Therefore, this application embodiment supports segmenting the reference speech before continuous speech tagging, such as using a speech segmenter to divide the long reference speech into at least one shorter speech segment that is easier to analyze. Specifically, the speech segmenter involved in this application embodiment is an algorithm or method for splitting large segments of reference speech into shorter speech segments; its goal is to create manageable speech segments that can be processed individually. Figure 5 The reference speech shown is 30 seconds long. Using a speech segmenter, the reference speech is split into three speech segments: speech segment 1 is 10 seconds long, speech segment 2 is 8 seconds long, and speech segment 3 is 12 seconds long.

[0123] Then, the computer device performs continuous speech tagging processing on each of the at least one segmented speech segment using a continuous speech tagger to obtain a continuous speech tag for each of the at least one segmented speech segment. The continuous speech tag for any speech segment is represented as a multi-dimensional vector (or a vector of multiple dimensions). In contrast to the discrete speech tagger which labels a speech segment as a discrete single number, the continuous speech tag preserves all acoustic features of the speech segment within the segment time period of the speech segment in the form of a multi-dimensional vector. Specifically, the continuous speech tag includes a plurality of continuous values which form a multi-dimensional vector, and the plurality of continuous values can represent the acoustic features of the speech segment in a sequential order of time steps of the segment time period (i.e., the speech duration). For example, if the speech duration of a speech segment 1 is 10 seconds and the time step is 1 second, the continuous speech tag obtained by performing continuous speech tagging processing on the speech segment 1 can be represented as a one-row multi-column vector, and the vector values (i.e., the continuous values mentioned above) in the one-row multi-column vector can represent the acoustic features of each time step. For example, the plurality of continuous values in the continuous speech tag can represent the acoustic features (e.g., the pitch, timbre, and loudness at the second) of the speech segment 1 at the first second, the second second, the third second, …, and the tenth second, respectively.

[0124] An exemplary structure of the continuous speech tagger can be seen in Figure 6 , for example. Figure 6As shown, the continuous speech marker includes a resampling module and a convolution module. In a specific implementation, after receiving the at least one speech segment, the continuous speech marker first uses the resampling module to perform resampling processing on each speech segment to obtain each resampled speech segment. In the resampling processing, on the one hand, by changing the sampling rate, sampling format, and channel number of each speech segment, the speech format (or audio format) of the speech segment is unified for subsequent processing in the subsequent module; on the other hand, the frequency range of each speech segment can be appropriately reduced, thereby reducing the amount of data required for calculation, facilitating data storage and transmission. Then, the convolution module is used to perform multi-level feature extraction processing on each resampled speech segment to obtain the continuous speech marker of each resampled speech segment. In detail, the convolution module sequentially includes one one-dimensional convolution layer, N convolution layers, and one one-dimensional convolution layer, where N is a positive integer. The convolution layer can extract features from the input resampled speech segment, and the features are mainly acoustic features of the semantic segment, such as tone, pitch, and intensity. The embodiment of the present application can extract multi-level or multi-layer acoustic features of the speech segment by setting multiple convolution layers in the continuous speech marker, which can not only extract semantic content but also extract acoustic features, thereby avoiding the loss of semantic feature extraction of the reference speech in the text-to-speech process and improving the performance of the text-to-speech technology.

[0125] Finally, after performing the continuous speech marker processing on each speech segment of the reference speech based on the above description to obtain the continuous speech marker of each speech segment, the continuous speech markers of the at least one speech segment are combined according to the playback position of the at least one speech segment in the reference speech, and the multi-dimensional speech embedding vector of the reference speech can be obtained. Here, the combination refers to sequentially connecting the continuous speech markers of the at least one speech segment according to the playback position of each speech segment in the reference speech; for example Figure 5 As shown, the continuous speech marker of the speech segment 2 is connected to the continuous speech marker of the speech segment 1, and the continuous speech marker of the speech segment 3 is connected to the continuous speech marker of the speech segment 2. The vector dimension of the connected multi-dimensional speech embedding vector and the total number of continuous values included in the continuous speech markers of the at least one speech segment are the same, for example, the number of continuous values of the continuous speech marker of the speech segment 1 is 100, the number of continuous values of the continuous speech marker of the speech segment 2 is 80, and the number of continuous values of the continuous speech marker of the speech segment 3 is 200, so the vector dimension of the multi-dimensional speech embedding vector is 100+80+200=380.

[0126] For example, the formula for using the continuous speech marker to perform continuous speech marker processing on the reference speech can be seen as follows:

[0127]

[0128] wherein ContTokenizer Audio is a continuous speech tokenizer, input A is the reference speech, and output is a multi-dimensional speech embedding vector of the reference speech, which can be represented as [A0, A1,...], wherein A j is a sub multi-dimensional speech embedding vector of each speech segment obtained by performing continuous speech tokenization on the reference speech after segmenting the reference speech, j = 0, 1,.... The vector dimension of the multi-dimensional speech embedding vector is the same as the feature dimension of the embedding layer of the subsequent language model used for speech prediction processing, so as to ensure that the output data format of the continuous speech tokenizer conforms to the input data format requirement of the embedding layer in the speech model.

[0129] It can be seen that, by performing continuous speech tokenization on the speech part in the text-to-speech process, specifically, using a continuous speech tokenizer to perform continuous tokenization on the reference speech to represent the acoustic features of the reference speech, rather than using a discrete tokenizer to perform discrete tokenization on the reference speech to represent the acoustic features, the speech information (such as semantic information and acoustic features) of the reference speech can be retained to the maximum extent, thereby improving the conversion effect of text-to-speech. For example, the existing discrete tokenization process can be referred to in Figure 7 ; as shown in Figure 7 , the discrete tokenizer only performs a preliminary vector representation (such as using only one convolutional layer for feature extraction) on the reference speech (or speech segment), and selects a number closest to the vector representation from a vocabulary (i.e., a limited range [0, 1023]) as the discrete tokenization of the reference speech, i.e., using the single number to represent the acoustic features of the reference speech; it is not difficult to find that the discrete speech tokenizer selects a number from the vocabulary to represent the vector representation, which loses many acoustic details of the reference speech, thereby resulting in poor performance of text-to-speech. In contrast, the continuous speech tokenizer of the present application performs continuous speech tokenization on the reference speech (or speech segment), which not only continuously represents the acoustic features of each time step within the speech time period of the reference speech, but also uses a multi-dimensional speech embedding vector to not only completely represent the acoustic features but also ensure the accuracy of the acoustic feature representation.

[0130] S304: performing speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector, to obtain a predicted speech sequence.

[0131] Based on the foregoing steps, the reference speech is processed using a continuous speech tagger to obtain a multi-dimensional speech embedding vector, and the text is processed using a text tagger to extract features to obtain a text embedding vector, and then a language model is used to process the text speech based on the multi-dimensional speech embedding vector and the text embedding vector (or referred to as autoregressive processing), and the output is a predicted predicted speech sequence; the predicted speech sequence includes multiple predicted speech information used / referenced by the text corresponding text reading speech when read at each time step, and one time step references one predicted speech information. That is, through the language model, the multi-dimensional speech embedding vector based on the reference speech and the text embedding vector of the text can be used to predict the predicted speech information corresponding to the text reading speech at each time step when the text is converted into text reading speech; the predicted speech information corresponding to any time step can be understood as information used to generate the voice signal to be played corresponding to the time step.

[0132] The language model, also known as a large language model, is a model based on machine learning and natural language processing technology. It learns the ability to understand and generate human language by training on a large amount of text data. Specifically, the large language model aims to understand and generate human language; it is trained on a large amount of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. The embodiments of the present application mainly involve autoregressive models (AR) under the language model; in the field of natural language processing, autoregressive models have the ability to predict the next token based on the previous series of tokens, specifically the language model task of predicting the next (or previous) possible following string based on the previous (or next) content. Autoregressive models usually use stacked decoders to implement the prediction of output tokens, and have the characteristic of predicting future values based on their own historical values. The autoregressive model can include but is not limited to a model with autoregressive properties obtained by pre-training and fine-tuning, and the model type of the autoregressive model is not limited in the embodiments of the present application.

[0133] The text-to-speech voice to be generated is read in a time step order, that is, each time step predicted voice signal is read in turn when the text-to-speech voice is read; therefore, embodiments of the present application support using an autoregressive model to predict the predicted voice information of each time step in a time step unit, and then forming a predicted voice sequence according to the predicted voice information of all time steps. Based on this, the following describes an implementation process of using an autoregressive model to perform voice prediction processing on the text based on the multi-dimensional voice embedding vector of the reference voice and the text embedding vector of the text to obtain a predicted voice sequence. In a specific implementation, a time step is denoted as t, t is a non-negative integer, that is, t = 0, 1, 2, …; when the time step t = 0, the multi-dimensional voice embedding vector and the text embedding vector are input into the autoregressive model to perform voice prediction processing / autoregressive processing to obtain the predicted voice information of the time step 0; when the time step t > 0, the multi-dimensional voice embedding vector, the text embedding vector, and the predicted voice information of each time step before the time step t are input into the autoregressive model to perform voice prediction processing to obtain the predicted voice information of the time step t; until the time step t reaches the predicted length of time required for the text-to-speech voice to be played, that is, until the autoregressive model predicts that the time step t has reached the end time step of the text-to-speech voice. In this way, the predicted voice information of each time step t predicted by the autoregressive model forms a predicted voice sequence.

[0134] More specifically, when the time step t = 0, the process of inputting the multi-dimensional voice embedding vector and the text embedding vector into the autoregressive model to perform voice prediction processing / autoregressive processing can include: first, connecting or concatenating the multi-dimensional voice embedding vector and the text embedding vector to obtain a connected vector; and then, using a language model (specifically, an autoregressive model) to perform regression processing on the connected vector to generate the predicted voice information of the time step 0. Similarly, when the time step t > 0, the process of inputting the multi-dimensional voice embedding vector, the text embedding vector, and the predicted voice information of each time step before the time step t into the autoregressive model to perform voice prediction processing can include: connecting the multi-dimensional voice embedding vector, the text embedding vector, and the predicted voice information of each time step before the time step t to obtain a connected vector; and then, using a language model (specifically, an autoregressive model) to perform regression processing on the connected vector to generate the predicted voice information of the time step t. The calculation formula of the predicted voice information of any time step t is as follows:

[0135]

[0136] wherein, LM is a language model; is the predicted voice information of the time step t; is the text embedding vector of the text; is the multi-dimensional voice embedding vector of the reference voice; is the predicted voice information of each time t in all time steps before the time step t. Indicates will Perform vector concatenation, specifically embedding multidimensional speech into vectors. Connect to text embedding vector Then, the predicted speech information for each time step t in all time steps t preceding time step t is... After connecting to the multidimensional speech embedding vector, by concatenating the text-based text tags (i.e., text embedding vectors) first and the continuous speech tags of the reference speech (i.e., multidimensional speech embedding vectors) last as input conditions for the language model, the language model can output predictive speech information with a style and timbre similar to the reference speech according to this input condition format.

[0137] To facilitate understanding of the speech prediction processing performed by the language model described above, the following section will refer to the appendix. Figure 8 Taking time steps t = 0, 1, 2, 3 as an example, the process of speech prediction processing performed by the language model to obtain the predicted speech information at each time step is introduced. Figure 8 As shown, at time step t = 0, the multidimensional speech embedding vector of the reference speech is first concatenated to the text embedding vector of the text to obtain the concatenated vector; such as the multidimensional speech embedding vector. Text embedding vector Then the concatenated vector can be represented as: Then, the concatenated vector is input into a language model (such as an autoregressive model) for regression processing to generate the predicted speech information at time step 0; this predicted speech information at time step 0 can be represented as... Similarly, at time step t = 1, the multidimensional speech embedding vector of the reference speech is concatenated to the text embedding vector of the text, and the predicted speech information at time step 0 is concatenated to the multidimensional speech embedding vector to obtain the concatenated vector, which is then represented as follows: The concatenated vector is then input into the language model for regression processing to generate predicted speech information at time step t=1. This predicted speech information at time step t=1 can be represented as... And so on, to obtain the predicted speech information at time step t=2. Predicted speech information at time step t=3 Therefore, the final predicted speech sequence includes

[0138] It can be seen that, on the one hand, the embodiment of the present application supports predicting the predicted speech information referred to by each time step when the text reading speech is read from the time step dimension, which can ensure that the predicted speech information corresponding to each time step is predicted according to the acoustic characteristics of the reference speech, so as to ensure that the language style and timbre and other acoustic characteristics of the text reading speech at each time step can match the acoustic characteristics of the reference speech, thereby improving the conversion effect of text to speech. On the other hand, the embodiment of the present application uses an autoregressive model to realize speech prediction processing, that is, the embodiment of the present application essentially proposes an autoregressive text to speech technology starting from continuous speech tokens. Compared with non-autoregressive text conversion technology, relying on the autoregressive model can capture the advantages of time correlation, flexibility and high prediction accuracy, mine the correlation between text embedding vectors and multi-dimensional speech embedding vectors (such as previous text embedding vectors and subsequent multi-dimensional speech embedding vectors), and the correlation between predicted speech information between time steps, thereby improving the accuracy of predicted speech information at each time step, and facilitating to improve the matching degree between the subsequently generated text reading speech and the reference speech.

[0139] S305: converting the predicted speech sequence to generate the text reading speech.

[0140] Based on the foregoing steps, in the text part, the text is represented as discrete text tokens by a text tokenizer, and converted into a text embedding vector by an embedding layer; and in the speech part, a continuous speech tokenizer is used to directly convert the reference speech into a multi-dimensional speech embedding vector without discretizing the vector into a single number (i.e., discrete tokens). The text embedding vector and the multi-dimensional speech embedding vector are input into a language model for autoregressive generation, and the output is a predicted speech sequence. The predicted speech sequence is the result of encoding the reference speech and the text, and the encoded result (i.e., predicted speech information) needs to be decoded to restore the text reading speech according to the embodiment of the present application; specifically, the predicted speech sequence is converted to obtain a text reading speech with acoustic characteristics similar to the acoustic characteristics of the reference speech in the time domain.

[0141] In the embodiment of the present application, the predicted speech sequence can be converted into the text reading speech corresponding to the text by using a speech decoder. The speech decoder mainly decodes and processes the predicted speech sequence by using built-in convolutional layers and feedforward neural network (FNN) modules, etc., to recover the text reading speech corresponding to the text. Specifically, the speech decoder uses different size convolution kernels to perform convolution processing on each predicted speech information in the input predicted speech sequence, to obtain the key speech features (such as key acoustic features) corresponding to each predicted speech information. Then, the speech decoder generates the predicted speech corresponding to each predicted speech information based on the key speech features corresponding to each predicted speech information, and uses flow-matching technology to help adjust the rhythm and intonation of the predicted speech during the conversion process, to ensure that the synthesized predicted speech has better fluency and naturalness. After obtaining the predicted speech corresponding to each predicted speech information, the speech decoder connects each predicted speech in the order of time steps, to generate the text reading speech corresponding to the text. The calculation formula for converting the predicted speech sequence into the text reading speech by using the speech decoder is as follows:

[0142]

[0143] wherein AudioDecoder is the speech decoder, is the predicted speech sequence input into the speech decoder, A out is the text reading speech output by the speech decoder.

[0144] In summary, the speech processing scheme provided in the embodiments of the present application supports continuous speech representation of the reference speech, specifically representing the continuous acoustic features of the reference speech in the order of speech playback during the speech feature extraction process of the reference speech, and representing the continuous acoustic features in the form of a multi-dimensional speech embedding vector, thereby realizing maximum retention of the speech information of the reference speech. Compared with discrete speech markers that only represent the acoustic features of the reference speech as a single numerical value, the speech information loss of the reference speech in the text-to-speech technology can be reduced. On the other hand, the speech processing scheme is a self-recurrent text-to-speech technology starting from continuous speech markers; compared with non-self-recurrent text conversion technology, the correlation between the text embedding vector and the multi-dimensional speech embedding vector, and the correlation between the predicted speech information between time steps can be mined, thereby improving the accuracy of the predicted speech information at each time step, and facilitating the improvement of the matching degree between the subsequently generated text reading speech and the reference speech. Based on the advantages described above, the text-to-speech technology starting from continuous speech representation provided in the embodiments of the present application is superior to the text-to-speech technology based on discrete markers in multiple evaluation indicators. The evaluation indicator comparison results are shown in Table 1 as follows:

[0145] Table 1

[0146]

[0147] The CLVP (Contrastive Listening Value Perception) score is mainly used to evaluate the consistency and authenticity of the generated text-to-speech voice and the text content; the lower the CLVP score, the higher the consistency of the text-to-speech voice and the text content. The wav2vec2 intelligibility score is mainly used to evaluate the speech intelligibility of the generated text-to-speech voice; the lower the wav2vec2 intelligibility score, the higher the intelligibility of the text-to-speech voice. The Stoi score is mainly used to evaluate the intelligibility of the generated text-to-speech voice (i.e., the degree of intelligibility to the user); the Stoi score is between 0 and 1, and the larger the Stoi score, the higher the intelligibility of the text-to-speech voice. The predicted speech quality is mainly used to evaluate the speech quality of the generated text-to-speech voice, such as speech intelligibility, whether there is interference, and whether there is signal loss, etc.; the higher the predicted speech quality score, the better the speech quality of the text-to-speech voice. The text-to-speech credibility is mainly used to evaluate the accuracy of the generated text-to-speech voice, such as whether the text content of the text-to-speech voice is consistent with the input text content, and the matching degree between the acoustic features of the text-to-speech voice and the acoustic features of the reference voice, etc.; the higher the text-to-speech credibility score, the higher the credibility of the text-to-speech voice. The low noise quality is mainly used to evaluate the level of low noise in the generated text-to-speech voice; the higher the low noise quality score, the less unnecessary low noise in the text-to-speech voice. The continuity is mainly used to evaluate the continuity of the speech signal of the generated text-to-speech voice; the higher the continuity score, the more continuous the speech signal of the text-to-speech voice. The loudness is mainly used to evaluate the volume of the generated text-to-speech voice; the larger the loudness score, the more appropriate the volume of the text-to-speech voice (i.e., the volume is not too small or too large). As can be seen, the text-to-speech scheme based on continuous marking provided by the embodiments of the present application is compared with the text-to-speech technology based on discrete marking through multiple evaluation indicators, and the scheme achieves higher predicted speech quality, and is superior to the text-to-speech technology based on discrete marking in terms of low noise, speech credibility, etc.

[0148] As described in the foregoing, the speech processing method provided by the embodiments of the present application is implemented through a text-to-speech model, which includes a target text marking module (i.e., a trained text marker), a target continuous speech marking module (i.e., a trained continuous speech marker), a target speech prediction module (i.e., a trained language model), and a target speech decoding module (i.e., a trained speech decoder).

[0149] The following describes the specific implementation process of the speech processing method provided by the embodiments of the present application from the dimensions of model inference and model training. As shown in Figure 9 , Figure 9 , Figure 9 Fig. 2 shows a flowchart of another speech processing method provided by an example embodiment of the present application; the speech processing method can be executed by the aforementioned computer device, such as a terminal; the speech processing method can include but is not limited to steps S901-S905:

[0150] S901: input the obtained text to be converted and reference speech into a text-to-speech model.

[0151] By way of example, the model structure diagram of the text-to-speech model can refer to Figure 10 ; as shown in Figure 10 , the text-to-speech model includes a target text tagging module, a target continuous speech tagging module, a target speech prediction module, a target speech decoding module, and an embedding layer. In the model inference stage, after the computer device receives the text to be converted and the reference audio, the computer device can input the text to be converted and the reference audio into the trained text-to-speech model. In this way, the text-to-speech model can call each module to perform text-to-speech on the text and the reference speech; specifically, the target text tagging module can be called to perform feature extraction processing on the text (specifically, the text is first discretely tagged, and then the discretely tagged text is input into the embedding layer) to obtain a text embedding vector of the text, and the target continuous speech tagging module can be called to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector of the reference speech; the target speech prediction module can be called to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence; and the target speech decoding module can be called to perform conversion processing on the predicted speech sequence to generate a text-to-speech audio corresponding to the text.

[0152] S902: the target text tagging module in the text-to-speech model, configured to perform feature extraction processing on the text to obtain a text embedding vector.

[0153] S903: the target continuous speech tagging module in the text-to-speech model, configured to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector.

[0154] S904: the target speech prediction module in the text-to-speech model, configured to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence.

[0155] S905: a target speech decoding module in the text-to-speech model, configured to perform conversion processing on the predicted speech sequence to generate text-to-speech reading voice corresponding to the text.

[0156] It should be noted that the specific implementation processes of each of steps S902-S905 and the foregoing Figure 3 The specific implementation processes of steps S302-S305 in the illustrated embodiment are the same, and thus are not described herein.

[0157] For example, the process in which the target text labeling module in step S902 performs feature extraction processing on the text to obtain a text embedding vector can be referred to in relation to the foregoing description of the specific implementation process of the feature extraction processing performed on the text to obtain a text embedding vector in step S302 in the illustrated embodiment. Figure 3 For example, the process in which the target text labeling module in step S902 performs feature extraction processing on the text to obtain a text embedding vector can be referred to in relation to the foregoing description of the specific implementation process of the feature extraction processing performed on the text to obtain a text embedding vector in step S302 in the illustrated embodiment.

[0158] Similarly, the process in which the target continuous speech labeling module in step S903 performs continuous speech labeling processing on the reference speech to obtain a multi-dimensional speech embedding vector can be referred to in relation to the foregoing description of the specific implementation process of the continuous speech labeling processing performed on the reference speech to obtain a multi-dimensional speech embedding vector in step S303 in the illustrated embodiment. Figure 3 Similarly, the process in which the target continuous speech labeling module in step S903 performs continuous speech labeling processing on the reference speech to obtain a multi-dimensional speech embedding vector can be referred to in relation to the foregoing description of the specific implementation process of the continuous speech labeling processing performed on the reference speech to obtain a multi-dimensional speech embedding vector in step S303 in the illustrated embodiment.

[0159] Similarly, the process in which the target speech prediction module in step S904 performs speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence can be referred to in relation to the foregoing description of the specific implementation process of the speech prediction processing performed on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence in step S304 in the illustrated embodiment. Figure 3 Similarly, the process in which the target speech prediction module in step S904 performs speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence can be referred to in relation to the foregoing description of the specific implementation process of the speech prediction processing performed on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence in step S304 in the illustrated embodiment.

[0160] Similarly, the process in which the target speech decoding module in step S905 performs conversion processing on the predicted speech sequence to generate text-to-speech reading voice corresponding to the text can be referred to in relation to the foregoing description of the specific implementation process of the conversion processing performed on the predicted speech sequence to generate text-to-speech reading voice in step S305 in the illustrated embodiment. Figure 3 Similarly, the process in which the target speech decoding module in step S905 performs conversion processing on the predicted speech sequence to generate text-to-speech reading voice corresponding to the text can be referred to in relation to the foregoing description of the specific implementation process of the conversion processing performed on the predicted speech sequence to generate text-to-speech reading voice in step S305 in the illustrated embodiment.

[0161] The above mainly introduces the speech processing method provided by the embodiments of the present application from the model inference dimension; the training process of the text-to-speech model in the embodiments of the present application is introduced from the model training dimension.

[0162] Specifically, the model training for the text-to-speech model in the embodiments of the present application is roughly divided into two steps, which are the model training for the initial continuous speech tagging module and the initial speech decoding module to be trained, and the model training for the entire text-to-speech model. In other words, the embodiments of the present application support the parameters of the preset target text tagging module, that is, the parameters of the target text tagging module are preset fixed; in this case, the initial continuous speech tagging module and the initial speech decoding module are first trained jointly / synchronously, and after the target continuous speech tagging module and the target speech decoding module are trained, the parameters of the target continuous speech tagging module and the target speech decoding module are fixed in the subsequent training of the entire text-to-speech model, and the parameter optimization for the initial speech prediction module to be trained is focused on, to obtain the trained target speech prediction module.

[0163] The above-described step-by-step training process for the text-to-speech model can be referred to Figure 11 ; the step-by-step training process can be summarized as steps s11-s14, wherein:

[0164] s11: obtaining a first data set. The first data set includes training data audio and corresponding text labels; the text labels are obtained by converting the corresponding training data audio into text.

[0165] The first data set is a speech data set used for synchronous optimization of the initial continuous speech tagging module and the initial speech decoding module; the first data set includes training data audio and text labels converted from the training data audio using a speech translator, and the text labels mainly include the text content included in the training data audio. The embodiments of the present application do not limit the number and source of the training data audio in the first data set.

[0166] s12: optimizing the initial continuous speech tagging module and the initial speech decoding module using the first data set to obtain a target continuous speech tagging module and a target speech decoding module.

[0167] After obtaining the first data set in the model training phase, the initial continuous speech tagging module and the initial speech decoding module can be trained synchronously on the first data set to obtain the trained target continuous speech tagging module and the target speech decoding module. The connection structure between the initial continuous speech tagging module and the initial speech decoding module used in the model training phase can be referred to Figure 12 ; as shown in Figure 12 , the output of the initial continuous speech tagging module is used as the input of the initial speech decoding module, and the training target for the synchronous training of the initial continuous speech tagging module and the initial speech decoding module is to reduce the loss between the predicted text information converted by the initial speech decoding module and the text labels corresponding to the training data audio.

[0168] In a specific implementation, a training data audio and a corresponding text label are obtained from the first data set; the initial continuous speech tokenization module is used to perform continuous speech tokenization processing on the training data audio, specifically, the training data audio is resampled by a resampling module in the initial continuous speech tokenization module, and then the resampled training data audio is subjected to multi-level feature extraction processing by using a convolution layer, to obtain a training multi-dimensional speech embedding vector. Then, the training multi-dimensional speech embedding vector is input into the initial speech decoding module, and the initial speech decoding module performs conversion processing on the training multi-dimensional speech embedding vector through feature recovery by a deep encoder and a multi-layer convolution layer, to obtain a predicted text reading speech corresponding to the training data audio. The predicted text reading speech is then converted into a predicted text information in text form; in this way, the initial continuous speech tokenization module and the initial speech decoding module can be optimized in a direction of reducing the difference between the predicted text information and the text label corresponding to the training data audio, specifically, a difference value between the predicted text information and the text label corresponding to the training data audio is calculated, and the parameters of the initial continuous speech tokenization module and the initial speech decoding module are adjusted in a direction of reducing the difference value, to obtain a target continuous speech tokenization module and a target speech decoding module. Figure 12

[0169] Exemplarily, given a training data audio A and a text label Y corresponding to the training data audio A, the calculation formula for optimizing the initial continuous speech tokenization module and the initial speech decoding module using the training data audio A and the text label Y is as follows:

[0170]

[0171]

[0172]

[0173] ContTokenizer(A) represents that the initial continuous speech tokenization module is used to perform continuous speech tokenization processing on the training data audio A. AudioDecoder(ContTokenizer(A)) represents that the initial speech decoding module is used to perform conversion processing / decoding processing on the training multi-dimensional speech embedding vector of the training data audio A, to obtain a predicted text reading speech.

[0174] to convert the predicted text reading speech into a text form, to obtain a predicted text information in text form.

[0175] ​represents the difference value (or difference value) between the predicted text information calculated using the CTC loss loss function (which can effectively deal with loss calculation under the condition that there are various speech speeds in the training data audio) and the text label Y corresponding to the training data audio. represents the synchronous adjustment of the parameters of the initial continuous speech tagging module and the initial speech decoding module in the direction of reducing the difference value, so as to obtain the trained target continuous speech tagging module and the target speech decoding module. ContTokenizer is the loss value of the initial continuous speech tagging module; θ AudioDecoder is the loss value of the initial speech decoding module.

[0176] s13: Obtain a second data set. The second data set includes training text embedding vectors corresponding to training texts, training multi-dimensional speech embedding vectors corresponding to training speeches, and audio labels corresponding to the training speeches.

[0177] Based on the trained target continuous speech tagging module obtained in the foregoing step s12, the target continuous speech tagging module can be used to perform continuous speech tagging processing on the training speeches to obtain training multi-dimensional speech embedding vectors corresponding to the training speeches, and the target text tagging module with fixed parameters can be used to perform feature extraction processing on the training texts to obtain training text embedding vectors corresponding to the training texts. In this way, the training multi-dimensional speech embedding vectors corresponding to the training speeches, the training text embedding vectors corresponding to the training texts, and the audio labels corresponding to the training speeches can constitute a training data pair in the second data set; wherein the audio labels corresponding to the training speeches can be manually or obtained by using an algorithm to perform speech tagging on the training speeches, such as acoustic features of the training speeches.

[0178] s14: Use the second data set to optimize the initial speech prediction module to obtain a target speech prediction module.

[0179] After the second data set is constructed, the initial speech prediction module can be trained using the second data set. The model training method follows the training method of an autoregressive model, that is, given an input sequence (that is, the training multi-dimensional speech embedding vectors corresponding to the training speeches and the training text embedding vectors corresponding to the training texts), the training target is to reduce the distance or similarity between the predicted label output by the initial speech prediction module and the next predicted label in the second data set, so as to enable the trained target speech prediction model to learn the correlation between the front and rear data in the input sequence, such as the correlation between the preceding training text embedding vectors and the following training multi-dimensional speech embedding vectors.

[0180] In a specific implementation, a training text embedding vector corresponding to a training text and a training multi-dimensional speech embedding vector corresponding to a training speech matching the training text are obtained from the second dataset. The training text and the training speech are matched, which means that the text content included in the training audio is the training text. Then, the initial speech prediction module is used to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector, to obtain a training predicted speech sequence. The training predicted speech sequence includes training predicted speech information referenced at each time step when the training audio is played. Furthermore, the audio label corresponding to the training audio is encoded into a predicted speech sequence label, which includes the real speech information at each time step after the training speech is encoded. Furthermore, the initial speech prediction module is optimized in the direction of reducing the difference between the training predicted speech sequence and the predicted speech sequence label to obtain a target speech prediction module.

[0181] It is worth noting that the "training predicted speech information" at each time step predicted by the initial speech prediction module is included in the "training predicted speech sequence" described above. For example, the training predicted speech information at time step 0, the training predicted speech information at time step 1, the training predicted speech information at time step 2, and so on. As described above in relation to the calculation logic of the training predicted speech information in formula (4), when t≥1, the input of the generation process of the training predicted speech information at time step t includes the training text embedding vector corresponding to the training text, the training multi-dimensional speech embedding vector corresponding to the training speech, and the speech information before time step t. In order to increase the accuracy and stability of the model prediction, the present application uses a teacher-driven training method to construct the input data of each time step input into the initial speech prediction module, and uses the real speech information at each time step in the predicted speech sequence label converted based on the audio label to construct the input data. The construction process of the input data specifically includes: connecting the training speech embedding vector corresponding to the training speech, the training text embedding vector corresponding to the training text, and the real speech information at each time step before time step t in the predicted speech sequence label corresponding to the audio label (i.e., the aforementioned speech information before time step t) to obtain a training connected vector (i.e., the input data input into the initial speech prediction module). In this way, the initial speech prediction module is used to perform regression processing on the training connected vector to obtain the training predicted speech information at time step t. As can be seen, by constructing the training connected vector using the real speech information at each time step, compared with constructing the training connected vector using the training predicted speech information output by the initial speech prediction module, the error caused by poor model performance can be reduced, the prediction accuracy of the initial speech prediction module can be improved, and a target speech prediction module with better performance can be trained.

[0182] Further, as described above, the embodiments of the present application support using the real speech information of the time step to construct the training connection vector, i.e., using the real speech information of each time step before the time step t in the training speech corresponding training speech embedding vector, the training text corresponding training text embedding vector, and the predicted speech sequence label corresponding audio label to construct the training connection vector. This allows the embodiments of the present application to allow pre-processing of the training connection vector of each time step, so that only the pre-processed training connection vector of each time step needs to be called during model training; this further improves the training speed and efficiency of the initial speech prediction module to some extent. In this implementation, the second data set constructed can not include the training text corresponding training text embedding vector, the training multi-dimensional speech embedding vector corresponding to the training speech, and the audio label corresponding to the training speech, but directly includes the training connection vector of each time step and the audio label corresponding to the training speech.

[0183] Exemplarily, given the training predicted speech sequence output by the initial speech prediction module and the audio label M corresponding to the training speech; then, the audio label M is encoded into continuous speech tokens (referred to as predicted speech sequence label in the embodiments of the present application), and the loss function is calculated by the distance between the predicted speech sequence label and the initial speech prediction module output. The calculation formula of the loss function is as follows:

[0184]

[0185]

[0186] wherein, ContTokenizer Audio (M) represents encoding the audio label M into continuous speech tokens using the target continuous speech token module represents calculating the continuous speech tokens using the MSE loss loss function and the training predicted speech sequence output by the initial speech prediction module . represents adjusting the parameters of the initial speech prediction module in the direction of reducing the difference between the continuous speech tokens and the training predicted speech sequence output by the initial speech prediction module .

[0187] To sum up, through the step-by-step training strategy shown in steps s11-s14, on the one hand, the initial continuous speech tagging module and the initial speech decoding module can be trained synchronously in a targeted manner, effectively improving the continuous speech tagging performance of the trained target continuous speech tagging module on the reference speech, and also improving the conversion performance of the trained target speech decoding module on the predicted speech sequence. On the other hand, the trained target continuous speech tagging module can be used to preprocess the training speech used for model training, that is, the training speech is preprocessed by the target continuous speech tagging module to obtain a training multi-dimensional speech embedding vector. Similarly, the target text tagging module is used to pre-process the training text used for model training to obtain a training text embedding vector. In this way, when the initial speech prediction model is trained, the preprocessed training multi-dimensional speech embedding vector and the training text embedding vector can be directly called, thereby improving the model training efficiency to a certain extent.

[0188] The above describes the method of the embodiments of the present application in detail. In order to facilitate better implementation of the above-mentioned scheme of the embodiments of the present application, the device of the embodiments of the present application is provided accordingly. In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0189] Figure 13 A structure diagram of a speech processing device provided by an example embodiment of the present application is shown; the speech processing device can be used to execute part or all of the steps in the method embodiments shown in Figure 3 and Figure 9 . Please refer to Figure 13 , the device includes the following units:

[0190] The acquisition unit 1301 is configured to acquire text to be converted and reference speech.

[0191] The processing unit 1302 is configured to perform feature extraction processing on the text to obtain a text embedding vector, the text embedding vector being used to represent the semantics of the text.

[0192] The processing unit 1302 is further configured to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector; the multi-dimensional speech embedding vector being used to represent the acoustic features of the reference speech.

[0193] The processing unit 1302 is further configured to perform voice prediction processing on the text based on the multi-dimensional voice embedding vector and the text embedding vector to obtain a predicted voice sequence, wherein the predicted voice sequence comprises a plurality of predicted voice information referenced when the text reading voice corresponding to the text is read.

[0194] The processing unit 1302 is further configured to perform conversion processing on the predicted voice sequence to generate the text reading voice, wherein an acoustic feature of the text reading voice matches an acoustic feature of the reference voice.

[0195] In an implementation manner, when the processing unit 1302 is configured to perform continuous voice labeling processing on the reference voice to obtain the multi-dimensional voice embedding vector, the processing unit 1302 is specifically configured to:

[0196] perform segmentation processing on the reference voice to obtain at least one voice segment;

[0197] perform continuous voice labeling processing on each voice segment to obtain a continuous voice label of each voice segment, wherein the continuous voice label of the voice segment is expressed as a multi-dimensional vector, the continuous voice label comprises a plurality of continuous values, and the plurality of continuous values can continuously represent acoustic features of the voice segment in a time step order; and the acoustic features comprise at least one of the following: pitch, intensity, timbre, and duration.

[0198] combine the continuous voice labels of the at least one voice segment to obtain the multi-dimensional voice embedding vector according to a playing position of the at least one voice segment in the reference voice; and a vector dimension of the multi-dimensional voice embedding vector is the same as a total number of continuous values included in the continuous voice labels of the at least one voice segment.

[0199] In an implementation manner, when the processing unit 1302 is configured to perform continuous voice labeling processing on each voice segment to obtain a continuous voice label of each voice segment, the processing unit 1302 is specifically configured to:

[0200] perform resampling processing on each voice segment to obtain a resampled voice segment;

[0201] perform multi-level feature extraction processing on the resampled voice segment to obtain the continuous voice label of the resampled voice segment.

[0202] In an implementation manner, the text reading voice is read in a time step order, and the time step is represented as t, t is a non-negative integer; when the processing unit 1302 is configured to perform voice prediction processing on the text based on the multi-dimensional voice embedding vector and the text embedding vector to obtain a predicted voice sequence, the processing unit 1302 is specifically configured to:

[0203] When the time step t=0, the multi-dimensional speech embedding vector and the text embedding vector are subjected to speech prediction processing to obtain predicted speech information of time step 0;

[0204] When the time step t>0, the multi-dimensional speech embedding vector, the text embedding vector, and the predicted speech information of each time step before the time step t are subjected to speech prediction processing to obtain predicted speech information of time step t;

[0205] Until the time step t reaches a predicted time length required for the text reading speech to be played;

[0206] The predicted speech information of each time step t constitutes a predicted speech sequence.

[0207] In an implementation manner, when the processing unit 1302 is used for subjecting the multi-dimensional speech embedding vector, the text embedding vector, and the predicted speech information of each time step before the time step t to speech prediction processing to obtain predicted speech information of time step t, the processing unit 1302 is specifically used for:

[0208] The multi-dimensional speech embedding vector, the text embedding vector, and the predicted speech information of each time step before the time step t are subjected to vector connection to obtain a connected vector;

[0209] The connected vector is subjected to regression processing to generate the predicted speech information of time step t.

[0210] In an implementation manner, when the processing unit 1302 is used for subjecting the predicted speech sequence to conversion processing to generate the text reading speech corresponding to the text, the processing unit 1302 is specifically used for:

[0211] Each predicted speech information in the predicted speech sequence is subjected to convolution processing to obtain key speech features corresponding to each predicted speech information;

[0212] Based on the key speech features corresponding to each predicted speech information, a predicted speech corresponding to each predicted speech information is generated;

[0213] Each predicted speech is connected in a time step sequence to generate the text reading speech corresponding to the text.

[0214] In an implementation manner, the speech processing method is implemented through a text-to-speech model, and the text-to-speech model includes a target text marking module, a target continuous speech marking module, a target speech prediction module, and a target speech decoding module;

[0215] The target text marking module is used for subjecting the text to feature extraction processing to obtain a text embedding vector;

[0216] The target continuous speech marking module is used for subjecting the reference speech to continuous speech marking processing to obtain a multi-dimensional speech embedding vector;

[0217] The target speech prediction module is configured to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector, to obtain a predicted speech sequence.

[0218] The target speech decoding module is configured to perform conversion processing on the predicted speech sequence, to generate text reading speech corresponding to the text.

[0219] In an implementation manner, parameters of the target text labeling module are preset; and the training process of the text-to-speech model comprises:

[0220] obtaining a first data set, the first data set comprising training data audio and corresponding text labels; the text labels being obtained by converting the corresponding training data audio into text;

[0221] optimizing the initial continuous speech labeling module and the initial speech decoding module using the first data set, to obtain the target continuous speech labeling module and the target speech decoding module;

[0222] obtaining a second data set, the second data set comprising training text embedding vectors corresponding to training texts, training multi-dimensional speech embedding vectors corresponding to training speeches, and audio labels corresponding to the training speeches; the training multi-dimensional speech embedding vectors being obtained by performing continuous speech labeling processing on the training speeches using the target continuous speech labeling module;

[0223] optimizing the initial speech prediction module using the second data set, to obtain the target speech prediction module.

[0224] In an implementation manner, when the processing unit 1302 is used to optimize the initial continuous speech labeling module and the initial speech decoding module using the first data set, to obtain the target continuous speech labeling module and the target speech decoding module, the processing unit 1302 is specifically configured to:

[0225] performing continuous speech labeling processing on the training data audio using the initial continuous speech labeling module, to obtain the training multi-dimensional speech embedding vector;

[0226] performing conversion processing on the training multi-dimensional speech embedding vector using the initial speech decoding module, to generate predicted text reading speech corresponding to the training data audio;

[0227] converting the predicted text reading speech into predicted text information in a text form;

[0228] optimizing the initial continuous speech labeling module and the initial speech decoding module in a direction of reducing a difference between the predicted text information and the text labels corresponding to the training data audio, to obtain the target continuous speech labeling module and the target speech decoding module.

[0229] In an implementation manner, the processing unit 1302 is configured to optimize the initial speech prediction module to obtain the target speech prediction module by using the second data set, and specifically configured to:

[0230] perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector by using the initial speech prediction module to obtain a training predicted speech sequence;

[0231] encode the audio label into a predicted speech sequence label, wherein the predicted speech sequence label comprises the real speech information of each time step of the training speech after being encoded;

[0232] optimize the initial speech prediction module in a direction of reducing the difference between the training predicted speech sequence and the predicted speech sequence label to obtain the target speech prediction module.

[0233] In an implementation manner, the training predicted speech sequence comprises the training predicted speech information of each time step, and the time step is represented as t, when t is an integer greater than zero, the processing unit 1302 is configured to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector by using the initial speech prediction module to obtain a training predicted speech sequence, and specifically configured to:

[0234] concatenate the training speech embedding vector, the training text embedding vector, and the real speech information of each time step before the time step t in the predicted speech sequence label to obtain a training concatenated vector;

[0235] perform regression processing on the training concatenated vector by using the initial speech prediction module to obtain the training predicted speech information of the time step t.

[0236] According to an embodiment of the present application, Figure 13 The units in the speech processing apparatus shown can be combined into one or several other units respectively or all, or some of the units can be further split into a plurality of units with smaller functions to constitute, which can realize the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above units are divided based on logical functions, and in actual application, the functions of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, the speech processing apparatus can also include other units, and in actual application, these functions can also be realized by other units, and can be realized by multiple units. According to another embodiment of the present application, the speech processing apparatus can be implemented by running the program capable of performing the above functions on a general-purpose computing device such as a computer including a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM), and the like processing elements and storage elements. Figure 3 and the like.Figure 9 a computer program (including program codes) of each step involved in the corresponding method shown above is constructed to configure a voice processing device as shown in Figure 13 the voice processing device shown in the above embodiment, and to implement the voice processing method of the present application. The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above computing device through the computer readable recording medium, and run therein.

[0237] In the present application, after obtaining the text to be converted and the reference voice, the text is subjected to feature extraction processing to obtain a text embedding vector for representing the semantics of the text; at the same time, the reference voice is subjected to continuous speech tagging to obtain a multi-dimensional speech embedding vector; since the continuous speech tagging method is used for the reference voice, the continuous speech tagging method can continuously tag the speech information of the reference voice in the form of a multi-dimensional vector according to the speech playing order, thereby retaining the speech information (or acoustic features) of the reference voice to the maximum extent. In this way, based on the multi-dimensional speech embedding vector of the reference voice obtained by continuous tagging and the text embedding vector of the text, the text is subjected to speech prediction processing, and a predicted speech sequence can be obtained, which includes a plurality of predicted speech information to which the text corresponding text reading voice is referred when being read; considering that a large amount of acoustic features of the reference voice are retained by a plurality of continuous values in the multi-dimensional speech embedding vector, a large amount of acoustic features of the reference voice are also retained in the predicted speech sequence obtained based on the multi-dimensional speech embedding vector. In this way, the predicted speech sequence is subjected to conversion processing, and the text corresponding text reading voice can be generated, and the acoustic features of the text reading voice can be ensured to match the acoustic features of the reference voice. In a popular way, the voice style of the text reading voice matches the voice style of the reference voice, i.e., the text reading voice and the reference voice are like the voice signals emitted by the same object (such as a person or an animal) in terms of auditory effect. Through the above aspects, the present application supports continuous speech representation of the reference voice, specifically, representing the continuous acoustic features of the reference voice according to the speech playing order in the speech feature extraction process of the reference voice, and representing the continuous acoustic features as a multi-dimensional speech embedding vector, thereby realizing the maximum retention of the speech information of the reference voice. Compared with the discrete speech tagging which only represents the acoustic features of the reference voice as a single numerical value, the loss of speech information of the reference voice in the text-to-speech technology can be reduced. In this way, the text-to-speech based on the multi-dimensional speech embedding vector with rich acoustic features can ensure that the acoustic features of the generated text reading voice match the acoustic features of the reference voice (such as the style and timbre of the text reading voice are close to the reference voice), thereby achieving a better conversion effect in the text-to-speech technology.

[0238] Figure 14A structural diagram of a computer device provided by an example embodiment of the present application is shown. Please refer to Figure 14 The computer device includes a processor 1401, a communication interface 1402, and a computer readable storage medium 1403. The processor 1401, the communication interface 1402, and the computer readable storage medium 1403 can be connected by a bus or other means. The communication interface 1402 is used to receive and send data. The computer readable storage medium 1403 can be stored in the memory of the computer device, and is used to store computer programs. The processor 1401 is used to execute the computer programs stored in the computer readable storage medium 1403. The processor 1401 (or CPU (Central Processing Unit, central processor)) is the computing core and control core of the computer device, which is suitable for implementing one or more computer programs, and is particularly suitable for loading and executing one or more computer programs to implement corresponding method processes or corresponding functions.

[0239] The example embodiments of the present application also provide a computer readable storage medium (Memory). The computer readable storage medium is a memory device in the computer device, and is used to store programs and data. It can be understood that the computer readable storage medium herein can include an internal storage medium in the computer device, and of course can also include an extended storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores the processing system of the computer device. In the storage space, one or more computer programs suitable for being loaded and executed by the processor 1401 are also stored. It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory; optionally, it can also be at least one computer readable storage medium located away from the aforementioned processor.

[0240] In an embodiment, the computer readable storage medium stores one or more computer programs; the processor 1401 loads and executes the one or more computer programs stored in the computer readable storage medium to implement the corresponding steps in the above voice processing method embodiments; in a specific implementation, the one or more computer programs in the computer readable storage medium are loaded and executed by the processor 1401 to implement the steps of the embodiments of the present application:

[0241] Obtaining text to be converted and reference speech;

[0242] Performing feature extraction processing on the text to obtain a text embedding vector, the text embedding vector being used to represent the semantics of the text;

[0243] The reference speech is processed by continuous speech tagging to obtain a multi-dimensional speech embedding vector; the multi-dimensional speech embedding vector is used to represent acoustic features of the reference speech;

[0244] Based on the multi-dimensional speech embedding vector and the text embedding vector, the text is processed by speech prediction to obtain a predicted speech sequence; the predicted speech sequence includes a plurality of predicted speech information referenced when the text reading speech corresponding to the text is read;

[0245] The predicted speech sequence is processed by conversion to generate a text reading speech, and the acoustic features of the text reading speech match the acoustic features of the reference speech.

[0246] In an implementation manner, one or more instructions in the computer readable storage medium are loaded by the processor 1401 and executed when the reference speech is processed by continuous speech tagging to obtain a multi-dimensional speech embedding vector, and the following steps are specifically executed:

[0247] The reference speech is processed by segmentation to obtain at least one speech segment;

[0248] Each speech segment is processed by continuous speech tagging to obtain continuous speech tagging of each speech segment; the continuous speech tagging of the speech segment is expressed as a multi-dimensional vector, and the continuous speech tagging includes a plurality of continuous values, which can be sequentially represented according to the time steps of the speech segment. Acoustic features of the speech segment; the acoustic features include at least one of the following: pitch, intensity, timbre and duration;

[0249] According to the playing position of the at least one speech segment in the reference speech, the continuous speech tagging of the at least one speech segment is combined to obtain a multi-dimensional speech embedding vector; the vector dimension of the multi-dimensional speech embedding vector is the same as the total number of continuous values included in the continuous speech tagging of the at least one speech segment.

[0250] In an implementation manner, one or more instructions in the computer readable storage medium are loaded by the processor 1401 and executed when each speech segment is processed by continuous speech tagging to obtain continuous speech tagging of each speech segment, and the following steps are specifically executed:

[0251] Each speech segment is processed by resampling to obtain each resampled speech segment;

[0252] Each resampled speech segment is processed by multi-level feature extraction to obtain continuous speech tagging of each resampled speech segment.

[0253] In an implementation manner, the text reading voice is read in a time step sequence, and the time step is represented as t, and t is a non-negative integer; when the one or more instructions in the computer readable storage medium are loaded by the processor 1401 and executed to perform the voice prediction processing on the text based on the multi-dimensional voice embedding vector and the text embedding vector, the predicted voice sequence is obtained, the following steps are specifically performed:

[0254] When the time step t = 0, the multi-dimensional voice embedding vector and the text embedding vector are subjected to voice prediction processing to obtain the predicted voice information of the time step 0;

[0255] When the time step t > 0, the multi-dimensional voice embedding vector, the text embedding vector and the predicted voice information of each time step before the time step t are subjected to voice prediction processing to obtain the predicted voice information of the time step t;

[0256] Until the time step t reaches the predicted length of time required for the text reading voice to be played;

[0257] Wherein, the predicted voice information of each time step t constitutes the predicted voice sequence.

[0258] In an implementation manner, when the one or more instructions in the computer readable storage medium are loaded by the processor 1401 and executed to perform the voice prediction processing on the multi-dimensional voice embedding vector, the text embedding vector and the predicted voice information of each time step before the time step t to obtain the predicted voice information of the time step t, the following steps are specifically performed:

[0259] The multi-dimensional voice embedding vector, the text embedding vector and the predicted voice information of each time step before the time step t are subjected to vector connection to obtain a connected vector;

[0260] The connected vector is subjected to regression processing to generate the predicted voice information of the time step t.

[0261] In an implementation manner, when the one or more instructions in the computer readable storage medium are loaded by the processor 1401 and executed to perform the conversion processing on the predicted voice sequence to generate the text reading voice corresponding to the text, the following steps are specifically performed:

[0262] Each predicted voice information in the predicted voice sequence is subjected to convolution processing to obtain the key voice feature corresponding to each predicted voice information;

[0263] Based on the key voice feature corresponding to each predicted voice information, the predicted voice corresponding to each predicted voice information is generated;

[0264] Each predicted voice is connected in a time step sequence to generate the text reading voice corresponding to the text.

[0265] In an implementation manner, the speech processing method is implemented through a text-to-speech model, and the text-to-speech model comprises a target text labeling module, a target continuous speech labeling module, a target speech prediction module, and a target speech decoding module.

[0266] The target text labeling module is configured to perform feature extraction processing on the text to obtain a text embedding vector.

[0267] The target continuous speech labeling module is configured to perform continuous speech labeling processing on the reference speech to obtain a multi-dimensional speech embedding vector.

[0268] The target speech prediction module is configured to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence.

[0269] The target speech decoding module is configured to perform conversion processing on the predicted speech sequence to generate text reading speech corresponding to the text.

[0270] In an implementation manner, parameters of the target text labeling module are preset; and a training process of the text-to-speech model comprises:

[0271] obtaining a first data set comprising training data audio and corresponding text labels; the text labels are obtained by converting the corresponding training data audio into text;

[0272] optimizing the initial continuous speech labeling module and the initial speech decoding module using the first data set to obtain the target continuous speech labeling module and the target speech decoding module;

[0273] obtaining a second data set comprising training text embedding vectors corresponding to training texts, training multi-dimensional speech embedding vectors corresponding to training speeches, and audio labels corresponding to the training speeches; the training multi-dimensional speech embedding vectors are obtained by performing continuous speech labeling processing on the training speeches using the target continuous speech labeling module;

[0274] optimizing the initial speech prediction module using the second data set to obtain the target speech prediction module.

[0275] In an implementation manner, one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and executed to specifically perform the following steps when optimizing the initial continuous speech labeling module and the initial speech decoding module using the first data set to obtain the target continuous speech labeling module and the target speech decoding module:

[0276] performing continuous speech labeling processing on the training data audio using the initial continuous speech labeling module to obtain the training multi-dimensional speech embedding vectors;

[0277] The initial speech decoding module is used to convert the training multi-dimensional speech embedding vector, to generate predicted text reading speech corresponding to the training data audio;

[0278] The predicted text reading speech is converted into predicted text information in the form of text;

[0279] The initial continuous speech tagging module and the initial speech decoding module are optimized in a direction of reducing the difference between the predicted text information and the text label corresponding to the training data audio, to obtain a target continuous speech tagging module and a target speech decoding module.

[0280] In an implementation manner, when the initial speech prediction module is optimized using the second data set to obtain a target speech prediction module, the one or more instructions in the computer readable storage medium are loaded by the processor 1401 and specifically perform the following steps:

[0281] The initial speech prediction module is used to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector, to obtain a training predicted speech sequence;

[0282] The audio label is encoded into a predicted speech sequence label; the predicted speech sequence label includes the real speech information of each time step of the training speech after being encoded;

[0283] The initial speech prediction module is optimized in a direction of reducing the difference between the training predicted speech sequence and the predicted speech sequence label, to obtain a target speech prediction module.

[0284] In an implementation manner, the training predicted speech sequence includes the training predicted speech information of each time step; when t is an integer greater than zero, the one or more instructions in the computer readable storage medium are loaded by the processor 1401 and specifically perform the following steps when performing speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector using the initial speech prediction module to obtain a training predicted speech sequence:

[0285] The training speech embedding vector, the training text embedding vector, and the real speech information of each time step before the time step t in the predicted speech sequence label are vector connected to obtain a training connected vector;

[0286] The initial speech prediction module is used to perform regression processing on the training connected vector, to obtain the training predicted speech information of the time step t.

[0287] Based on the same inventive concept, the computer device provided in the embodiments of the present application has similar principles and beneficial effects to the principles and beneficial effects of the speech processing method provided in the method embodiments of the present application. For brevity, the principles and beneficial effects of the method embodiments are not described again here.

[0288] The embodiments of the present application also provide a computer program product, which comprises a computer program. When the computer program is executed by a processor, the speech processing method described above is implemented.

[0289] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software mode depends on the specific application and design constraints of the technical solutions. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0290] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of a computer program product. The computer program product comprises one or more computer programs. When the computer program is loaded and executed on a computer device, the computer program executes the processes or functions described above in the embodiments of the present application. The computer device can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer program can be stored in a computer readable storage medium or transmitted by a computer readable storage medium. The computer program can be transmitted from one website, computer device, server or data center to another website, computer device, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium accessible by the computer device or a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0291] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A voice processing method, characterized by, The method comprises the following steps: obtaining text to be converted and reference speech; performing feature extraction processing on the text to obtain a text embedding vector, the text embedding vector being used to represent the semantics of the text; performing continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector; the multi-dimensional speech embedding vector is used to represent the acoustic characteristics of the reference speech; based on the multi-dimensional speech embedding vector and the text embedding vector, performing speech prediction processing on the text to obtain a predicted speech sequence; the predicted speech sequence includes a plurality of predicted speech information referenced when the text reading speech corresponding to the text is read; performing convolution processing on each of the predicted speech information in the predicted speech sequence to obtain a key speech feature corresponding to each of the predicted speech information; based on the key speech feature corresponding to each of the predicted speech information, generating a predicted speech corresponding to each of the predicted speech information; connecting each of the predicted speech in the time step sequence to generate the text reading speech corresponding to the text, and the acoustic characteristics of the text reading speech match the acoustic characteristics of the reference speech.

2. The method of claim 1, wherein, The method comprises the following steps: performing segmentation processing on the reference speech to obtain at least one speech segment; performing continuous speech tagging processing on each of the speech segments to obtain a continuous speech tag of each of the speech segments; the continuous speech tag of the speech segment is expressed as a multi-dimensional vector, and the continuous speech tag includes a plurality of continuous values, which can continuously represent the acoustic characteristics of the speech segment in the order of the time steps of the speech segment; the acoustic characteristics include at least one of the following: pitch, intensity, timbre, and duration; combining the continuous speech tags of at least one of the speech segments according to the playing positions of at least one of the speech segments in the reference speech to obtain a multi-dimensional speech embedding vector; the vector dimension of the multi-dimensional speech embedding vector is the same as the total number of continuous values included in the continuous speech tags of at least one of the speech segments.

3. The method of claim 2, wherein, The method comprises the following steps: performing resampling processing on each of the speech segments to obtain each of the resampled speech segments; performing multi-level feature extraction processing on each of the resampled speech segments to obtain a continuous speech tag of each of the resampled speech segments.

4. The method of claim 1, wherein, The text reading speech is read in the order of time steps, and the time step is represented as t, t being a non-negative integer; the method comprises the following steps: when the time step t=0, performing speech prediction processing on the multi-dimensional speech embedding vector and the text embedding vector to obtain the predicted speech information of the time step 0; when the time step t > 0, performing speech prediction processing on the multi-dimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before the time step t to obtain the predicted speech information of the time step t; until the time step t reaches the predicted length of time required for the text reading speech to be played; wherein the predicted speech information of each time step t constitutes a predicted speech sequence.

5. The method of claim 4, wherein, The speech prediction processing on the multi-dimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before the time step t to obtain the predicted speech information of the time step t comprises: vector connection of the multi-dimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before the time step t to obtain a connected vector; regression processing on the connected vector to generate the predicted speech information of the time step t.

6. The method according to any one of claims 1 to 5, wherein, The method is implemented by a text-to-speech model, and the text-to-speech model comprises a target text marking module, a target continuous speech marking module, a target speech prediction module and a target speech decoding module; The target text marking module is configured to perform feature extraction processing on the text to obtain the text embedding vector; The target continuous speech marking module is configured to perform continuous speech marking processing on the reference speech to obtain a multi-dimensional speech embedding vector; The target speech prediction module is configured to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence; The target speech decoding module is configured to perform conversion processing on the predicted speech sequence to generate a text reading speech corresponding to the text.

7. The method of claim 6, wherein, Parameters of the target text marking module are preset; and a training process of the text-to-speech model comprises: obtaining a first data set comprising training data audio and corresponding text labels; the text labels are obtained by converting the corresponding training data audio into text; optimizing an initial continuous speech marking module and an initial speech decoding module using the first data set to obtain the target continuous speech marking module and the target speech decoding module; obtaining a second data set comprising training text embedding vectors corresponding to training texts, training multi-dimensional speech embedding vectors corresponding to training speeches, and audio labels corresponding to the training speeches; the training multi-dimensional speech embedding vectors are obtained by performing continuous speech marking processing on the training speeches using the target continuous speech marking module; optimizing an initial speech prediction module using the second data set to obtain the target speech prediction module.

8. The method of claim 7, wherein, The optimization of the initial continuous speech marking module and the initial speech decoding module using the first data set to obtain the target continuous speech marking module and the target speech decoding module comprises: performing continuous speech marking processing on the training data audio using the initial continuous speech marking module to obtain training multi-dimensional speech embedding vectors; The initial speech decoding module is used to convert the training multi-dimensional speech embedding vector, and a predicted text reading speech corresponding to the training data audio is generated; The predicted text reading speech is converted into predicted text information in the form of text; The initial continuous speech tagging module and the initial speech decoding module are optimized in the direction of reducing the difference between the predicted text information and the text label corresponding to the training data audio, and the target continuous speech tagging module and the target speech decoding module are obtained.

9. The method of claim 7, wherein, The second data set is used to optimize the initial speech prediction module, and the target speech prediction module is obtained. The initial speech prediction module is used to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector, and a training predicted speech sequence is obtained; The audio label is encoded into a predicted speech sequence label; the predicted speech sequence label includes the true speech information of each time step after the training speech is encoded; The initial speech prediction module is optimized in the direction of reducing the difference between the training predicted speech sequence and the predicted speech sequence label, and the target speech prediction module is obtained.

10. The method of claim 9, wherein, The training predicted speech sequence includes training predicted speech information of each time step; the time step is represented as t, and when t is an integer greater than zero, the initial speech prediction module is used to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector, and a training predicted speech sequence is obtained. The training speech embedding vector, the training text embedding vector, and the true speech information of each time step before time step t in the predicted speech sequence label are connected into a training connected vector; The initial speech prediction module is used to perform regression processing on the training connected vector, and training predicted speech information of time step t is obtained.

11. A speech processing device, characterized by It includes: An acquisition unit is configured to acquire text to be converted and reference speech; A processing unit is configured to perform feature extraction processing on the text to obtain a text embedding vector, which is used to represent the semantics of the text; The processing unit is further configured to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector; The multi-dimensional speech embedding vector is used to represent the acoustic characteristics of the reference speech; The processing unit is further configured to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence; the predicted speech sequence includes a plurality of predicted speech information referenced when a text reading speech corresponding to the text is read. The processing unit is further configured to perform convolution processing on each of the predicted speech information in the predicted speech sequence to obtain a key speech feature corresponding to each of the predicted speech information; generate a predicted speech corresponding to each of the predicted speech information based on the key speech feature corresponding to each of the predicted speech information; and connect each of the predicted speech in a time step sequence to generate a text reading speech corresponding to the text, wherein an acoustic feature of the text reading speech matches an acoustic feature of the reference speech.

12. A computer device, comprising: The computer program product comprises computer instructions, and the computer instructions are executed by the processor to implement the speech processing method according to any one of claims 1-10. The computer program product comprises computer instructions, and the computer instructions are executed by the processor to implement the speech processing method according to any one of claims 1-10. The computer program product comprises computer instructions, and the computer instructions are executed by the processor to implement the speech processing method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer program product comprises computer instructions, and the computer instructions are executed by the processor to implement the speech processing method according to any one of claims 1-10.

14. A computer program product, characterised in that, ​

Images