Voice processing method, device, equipment, medium and program product

Abstract

The embodiment of the invention discloses a voice processing method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring a to-be-converted text and a reference voice; performing feature extraction processing on the text to obtain a text embedding vector; performing continuous voice marking processing on the reference voice to obtain a multi-dimensional voice embedding vector; performing voice prediction processing on the text based on the multi-dimensional voice embedding vector and the text embedding vector to obtain a predicted voice sequence; and performing conversion processing on the predicted voice sequence to generate text reading voice. By adopting the embodiment of the invention, the acoustic details of the reference voice can be reserved to the maximum extent in the text-to-voice process, so that the text-to-voice conversion effect is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of artificial intelligence, and specifically to a speech processing method, a speech processing device, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Text-to-speech technology is a technology that takes text and reference speech as input and outputs text-reading speech that reads the text in the style of the reference speech.

[0003] At present, text-to-speech technology is based on discrete speech markers to discretely mark reference speech to extract acoustic details of the reference speech. However, discrete speech markers discretize the input reference speech into a single value, which will lose a lot of acoustic details of the reference speech, resulting in loss of speech feature extraction for the reference speech during the text-to-speech process, reducing the conversion effect of text-to-speech. Summary of the invention

[0004] The embodiments of the present application provide a speech processing method, apparatus, device, medium and program product, which can retain the acoustic details of the reference speech to the greatest extent during the text-to-speech process, thereby improving the text-to-speech conversion effect.

[0005] On the one hand, an embodiment of the present application provides a speech processing method, the method comprising:

[0006] Obtain the text to be converted and the reference speech;

[0007] Perform feature extraction on the text to obtain a text embedding vector, which is used to represent the semantics of the text;

[0008] Performing continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector; the multi-dimensional speech embedding vector is used to characterize the acoustic features of the reference speech;

[0009] Based on the multi-dimensional speech embedding vector and the text embedding vector, the text is subjected to speech prediction processing to obtain a predicted speech sequence; the predicted speech sequence includes a plurality of predicted speech information referenced when the text reading speech corresponding to the text is read aloud;

[0010] The predicted speech sequence is converted to generate text-reading speech, and the acoustic features of the text-reading speech match the acoustic features of the reference speech.

[0011] On the other hand, an embodiment of the present application provides a speech processing device, the device comprising:

[0012] An acquisition unit, used for acquiring the text to be converted and the reference speech;

[0013] A processing unit, used for performing feature extraction processing on the text to obtain a text embedding vector, where the text embedding vector is used to represent the semantics of the text;

[0014] The processing unit is further used to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector; the multi-dimensional speech embedding vector is used to characterize the acoustic features of the reference speech;

[0015] The processing unit is further used to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence; the predicted speech sequence includes a plurality of predicted speech information referenced when the text reading speech corresponding to the text is read aloud;

[0016] The processing unit is also used to convert the predicted speech sequence to generate text reading speech, and the acoustic features of the text reading speech match the acoustic features of the reference speech.

[0017] In one implementation, the processing unit is used to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector, specifically for:

[0018] Segmenting the reference speech to obtain at least one speech segment;

[0019] Performing continuous speech tag processing on each speech segment to obtain a continuous speech tag of each speech segment; the continuous speech tag of the speech segment is expressed as a multidimensional vector, the continuous speech tag includes a plurality of continuous values, and the plurality of continuous values ​​can continuously represent the acoustic features of the speech segment according to the sequence of time steps of the speech segment; the acoustic features include at least one of the following: pitch, intensity, timbre and duration;

[0020] According to the playback position of at least one speech segment in the reference speech, the continuous speech tags of at least one speech segment are combined to obtain a multidimensional speech embedding vector; the vector dimension of the multidimensional speech embedding vector is the same as the sum of the number of continuous values ​​included in the continuous speech tags of at least one speech segment.

[0021] In one implementation, the processing unit is used to perform continuous speech mark processing on each speech segment to obtain the continuous speech mark of each speech segment, specifically for:

[0022] Resampling each speech segment to obtain each resampled speech segment;

[0023] A multi-level feature extraction process is performed on each resampled speech segment to obtain a continuous speech tag of each resampled speech segment.

[0024] In one implementation, the text reading speech is read aloud in the order of time steps, where the time step is represented by t, and t is a non-negative integer; the processing unit is used to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector, and when the predicted speech sequence is obtained, it is specifically used to:

[0025] When time step t=0, the multi-dimensional speech embedding vector and the text embedding vector are processed for speech prediction to obtain the predicted speech information of time step 0;

[0026] When time step t>0, the multidimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before time step t are processed for speech prediction to obtain the predicted speech information of time step t;

[0027] Until the time step t reaches the predicted time length required for the text reading speech to be played;

[0028] Among them, the predicted speech information at each time step t constitutes a predicted speech sequence.

[0029] In one implementation, the processing unit is used to perform speech prediction processing on the multidimensional speech embedding vector, the text embedding vector and the predicted speech information at each time step before the time step t to obtain the predicted speech information at the time step t, specifically for:

[0030] Perform vector concatenation on the multidimensional speech embedding vector, the text embedding vector and the predicted speech information at each time step before time step t to obtain a concatenated vector;

[0031] The concatenated vector is regressed to generate the predicted speech information at time step t.

[0032] In one implementation, the processing unit is used to convert the predicted speech sequence to generate a text reading speech corresponding to the text, specifically for:

[0033] Perform convolution processing on each predicted speech information in the predicted speech sequence to obtain the key speech features corresponding to each predicted speech information;

[0034] Generate predicted speech corresponding to each predicted speech information based on the key speech features corresponding to each predicted speech information;

[0035] Connect each predicted speech in the order of time steps to generate the text reading speech corresponding to the text.

[0036] In one implementation, the speech processing method is implemented by a text-to-speech model, which includes a target text marking module, a target continuous speech marking module, a target speech prediction module, and a target speech decoding module;

[0037] The target text tagging module is used to perform feature extraction on the text to obtain a text embedding vector;

[0038] The target continuous speech tagging module is used to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector;

[0039] The target speech prediction module is used to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence;

[0040] The target speech decoding module is used to convert the predicted speech sequence and generate the text reading speech corresponding to the text.

[0041] In one implementation, the parameters of the target text tagging module are preset; the training process of the text-to-speech model includes:

[0042] Obtain a first data set, the first data set including training data audio and corresponding text labels; the text labels are obtained by converting the corresponding training data audio into text;

[0043] Using the first data set, the initial continuous speech marking module and the initial speech decoding module are optimized to obtain a target continuous speech marking module and a target speech decoding module;

[0044] Obtain a second data set, the second data set including a training text embedding vector corresponding to the training text, a training multidimensional speech embedding vector corresponding to the training speech, and an audio tag corresponding to the training speech; the training multidimensional speech embedding vector is obtained by performing continuous speech tagging processing using a target continuous speech tagging module;

[0045] The initial speech prediction module is optimized using the second data set to obtain a target speech prediction module.

[0046] In one implementation, the processing unit is used to optimize the initial continuous speech marking module and the initial speech decoding module using the first data set to obtain the target continuous speech marking module and the target speech decoding module, specifically for:

[0047] Use the initial continuous speech tagging module to perform continuous speech tagging on the training data audio to obtain a training multi-dimensional speech embedding vector;

[0048] Use the initial speech decoding module to convert the training multi-dimensional speech embedding vector to generate the predicted text reading speech corresponding to the training data audio;

[0049] Converting predictive text reading speech into predictive text information in text form;

[0050] In the direction of reducing the difference between the predicted text information and the text label corresponding to the training data audio, the initial continuous speech labeling module and the initial speech decoding module are optimized to obtain the target continuous speech labeling module and the target speech decoding module.

[0051] In one implementation, the processing unit is used to optimize the initial speech prediction module using the second data set to obtain the target speech prediction module, specifically to:

[0052] Using the initial speech prediction module to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector, to obtain a training predicted speech sequence;

[0053] Encode the audio label into a predicted speech sequence label; the predicted speech sequence label includes the real speech information of each time step after the training speech is encoded;

[0054] The initial speech prediction module is optimized in the direction of reducing the difference between the training predicted speech sequence and the predicted speech sequence label to obtain the target speech prediction module.

[0055] In one implementation, the training predicted speech sequence includes training predicted speech information at each time step; the time step is represented by t, and when t is an integer greater than zero, the processing unit is used to use the initial speech prediction module to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector to obtain the training predicted speech sequence, specifically for:

[0056] Concatenate the training speech embedding vector, the training text embedding vector, and the real speech information of each time step before time step t in the predicted speech sequence label to obtain a training concatenated vector;

[0057] The initial speech prediction module is used to regress the training concatenated vector to obtain the training predicted speech information at time step t.

[0058] On the other hand, an embodiment of the present application provides a computer device, the device comprising:

[0059] A processor for loading and executing computer programs;

[0060] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned speech processing method is implemented.

[0061] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned speech processing method.

[0062] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and when the computer instructions are executed by a processor, the above-mentioned speech processing method is implemented.

[0063] In an embodiment of the present application, after obtaining the text to be converted and the reference speech, feature extraction processing is performed on the text to obtain a text embedding vector for characterizing the semantics of the text; at the same time, continuous speech marking is performed on the reference speech to obtain a multidimensional speech embedding vector; since a continuous speech marking method is used for the reference speech, the continuous speech marking method can continuously mark the speech information of the reference speech in the form of a multidimensional vector in the order of speech playback, thereby retaining the speech information (or acoustic features) of the reference speech to the greatest extent. In this way, based on the multidimensional speech embedding vector of the reference speech obtained by continuous marking and the text embedding vector of the text, speech prediction processing is performed on the text to obtain a predicted speech sequence, which includes multiple predicted speech information referenced when the text reading speech corresponding to the text is read aloud; considering that a large number of acoustic features of the reference speech are retained through multiple continuous values ​​in the multidimensional speech embedding vector, a large number of acoustic features of the reference speech are also retained in the predicted speech sequence obtained based on the multidimensional speech embedding vector. In this way, by converting the predicted speech sequence, the text reading speech corresponding to the text can be generated, and it can be ensured that the acoustic features of the text reading speech match the acoustic features of the reference speech. In layman's terms, the speech style of the text reading speech matches the speech style of the reference speech. For example, from the auditory effect, the text reading speech and the reference speech sound like speech signals emitted by the same object (such as a person or an animal). Through the above-mentioned description, the embodiments of the present application support continuous speech characterization of the reference speech, specifically, characterizing the continuous acoustic features of the reference speech according to the speech playback order during the speech feature extraction process of the reference speech, and representing the continuous acoustic features in the form of a multi-dimensional speech embedding vector, thereby achieving the maximum retention of the speech information of the reference speech; compared with discrete speech tags that only discretize the acoustic features of the reference speech into a single numerical value to represent it, the speech information loss of the reference speech in the text-to-speech technology can be reduced. In this way, text-to-speech based on a multi-dimensional speech embedding vector with rich acoustic features can ensure that the acoustic features of the generated text reading speech match the acoustic features of the reference speech (such as the style and timbre of the text reading speech are close to the reference speech), thereby achieving better conversion effects in text-to-speech technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 is a schematic diagram of the structure of an existing discrete speech tagger;

[0066] Figure 2 is a schematic diagram of the architecture of a speech processing system provided by an exemplary embodiment of the present application;

[0067] Figure 3 It is a flowchart of a speech processing method provided by an exemplary embodiment of the present application;

[0068] Figure 4 is a schematic diagram of an information input interface for uploading text to be converted and reference speech provided by an exemplary embodiment of the present application;

[0069] Figure 5 It is a flowchart of a method of using a continuous speech marker to perform continuous speech marking processing on a reference speech, provided by an exemplary embodiment of the present application;

[0070] Figure 6 is a structural schematic diagram of a continuous speech marker provided by an exemplary embodiment of the present application;

[0071] Figure 7 It is a schematic diagram of an existing discrete marker;

[0072] Figure 8 It is a flowchart of performing speech prediction processing using a language model provided by an exemplary embodiment of the present application;

[0073] Fig. 9 is a flowchart of another speech processing method provided by an exemplary embodiment of the present application;

[0074] Fig.10 It is a schematic diagram of a text-to-speech model structure provided by an exemplary embodiment of the present application;

[0075] Fig.11 It is a schematic diagram of a training process for a text-to-speech model provided by an exemplary embodiment of the present application;

[0076] Fig.12 It is a schematic diagram of a connection structure between an initial continuous speech marking module and an initial speech decoding module provided by an exemplary embodiment of the present application;

[0077] Fig.13 is a structural schematic diagram of a speech processing device provided by an exemplary embodiment of the present application;

[0078] Fig.14 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0080] In the embodiment of the present application, a speech processing solution is proposed; the speech processing solution is specifically a solution that can convert text into text-reading speech that matches the acoustic details of a reference speech based on text-to-speech technology (Text To Speech, TTS). Through the speech processing solution, it is possible to achieve rapid text-to-speech conversion while ensuring that the style of the converted text-reading speech is similar to that of the reference speech. For users, they only need to specify a reference speech to obtain a text-reading speech of a similar style, thereby meeting the user's personalized text-to-speech needs.

[0081] Among them: ① Text, or text content, refers to the expression of natural language in written form; according to the length of the text, the text can be divided into a single sentence, paragraph or chapter. Text content of any length is composed of characters or strings. A string is composed of at least two characters. The characters may include at least one of the following: Chinese characters (i.e., Chinese characters), English characters (i.e., letters), numbers and punctuation marks (such as commas "," , periods ".", square brackets "【】"). The embodiments of the present application do not limit the number of characters included in the text. In some cases, the length of the text can be represented by the number of characters included in the text. The more characters there are, the longer the length of the text is determined to be.

[0082] ② Speech, or speech signal, is a subset of audio signal. Speech specifically refers to the sound made by people or animals, such as the sound produced when humans speak; it is mainly used for language communication, and it contains semantic information and emotional information, etc., which can be used to identify the identity of the speaker. In other words, speech has acoustic characteristics, and the acoustic characteristics of speech emitted by different speakers are different, so that the identity of the speaker can be effectively distinguished according to the acoustic characteristics of speech. Among them, the acoustic characteristics of speech can include at least one of the following: pitch, sound intensity, timbre and sound length. Pitch refers to the height of the sound, which is mainly determined by the sound wave frequency of the sound. The higher the sound wave frequency, the higher the pitch of the sound. Sound intensity refers to the strength of the sound, which is mainly determined by the sound wave vibration amplitude of the sound. The larger the sound wave vibration amplitude, the stronger the sound intensity. Timbre refers to the unique quality of the sound. According to the different characteristics of the vocal cords of the speaker, the characteristics of the sound produced by the speaker are different, which can be used to distinguish the sources of different sounds; for example, the timbre of a child's voice is usually more tender and crisp than that of an adult's voice. The duration of a sound refers to the length of the sound, which is determined by the length of time it is pronounced.

[0083] ③Text-to-speech technology is a technology that converts text content into speech signals; it supports converting text content into speech signals through processes such as speech synthesis, speech recognition, and speech conversion, and finally outputs natural and smooth speech signals. In order to achieve the purposeful guidance of the final generated speech to meet the acoustic characteristics required by the user in the process of text-to-speech conversion; it supports the use of a reference speech to guide the conversion of text into a speech signal that matches the acoustic characteristics of the reference speech. At this time, text-to-speech technology refers to a technology that takes text and reference speech as input and outputs a text reading speech that can read the text content of the text in the style of the reference speech. For example, the reference speech is a sound signal emitted by a 6-year-old girl, and the text to be converted is "Today's weather is good". In the process of converting text to speech, the acoustic characteristics of the reference speech are used to guide the generation of the speech signal corresponding to the text, ensuring that the acoustic characteristics of the text reading speech corresponding to the output text (such as the acoustic characteristics summarized as style and timbre, etc.) will be close to the acoustic characteristics of the 6-year-old girl.

[0084] In practical applications, the reference speech with guiding role will be speech-tagged (or called speech representation) in the text-to-speech process. The so-called speech tagging refers to the process of extracting features from the reference speech to extract the acoustic features of the reference speech. At present, in the text-to-speech technology, speech taggers are used to implement speech tagging for the reference speech. The speech tagger can be called a tokenizer; specifically, discrete speech taggers are used to implement speech tagging for the reference speech. Among them, the discrete speech tagger or discrete tokenizer can convert the reference speech into digital data (the digital data is represented as discrete tags, or discrete tags); in this way, the acoustic features of the reference speech are represented by a single number. The number of discrete tags is limited, which is called a vocabulary, and the size of the vocabulary is 1024; that is, the value of the discrete tag obtained by discretely representing the reference speech is a number in [0,1023], so that the acoustic features of the reference speech are represented by the single number. It has been found in practice that since the discrete marker uses a single number to represent the acoustic features of the reference speech, a large amount of speech information will be lost in the calculation process of extracting the acoustic features of the reference speech. Specifically, the higher-dimensional features are discretized into a single number, which greatly loses the speech information of the reference speech. This reduces the conversion effect of text-to-speech (that is, the acoustic features of the final generated text reading speech and the acoustic features of the reference speech have a low match).

[0085] In order to improve the conversion effect of text to speech, the speech processing scheme proposed in the embodiment of the present application specifically provides a text-to-speech technology starting from continuous speech representation. The so-called "starting from continuous speech representation" refers to the use of continuous speech tags of reference speech to realize text-to-speech technology. It can be seen that, unlike discrete speech markers, the embodiment of the present application uses a continuous speech marker to realize feature extraction for reference speech, greatly reducing the speech information loss of reference speech in text-to-speech technology, aiming to achieve better conversion effect in text-to-speech technology. Among them, the continuous speech marker, or continuous representation speech marker, supports encoding the reference speech into continuous tags (or continuous speech tags), which are expressed as multi-dimensional vectors and can be used to characterize the acoustic features of the reference speech; that is, the continuous speech marker is different from the discrete speech marker, and its output is a continuous representation rather than a discrete tag. For example, for reference speech signals of the same duration, the number of tags output by discrete speech tags and continuous speech tags (i.e., the number of tags used to characterize the acoustic features of the reference speech) is the same. The difference is that the continuous tags output by the continuous speech tagger are multidimensional vectors, while the discrete tags output by the discrete tagger are discrete values ​​within a limited range (i.e., numbers within the range of [0,1023]). In this way, using a continuous speech tagger to extract acoustic features of the reference speech can retain the speech information of the reference speech to the greatest extent, so that continuous speech tags based on rich speech information can achieve better text-to-speech effects.

[0086] The general process of the speech processing solution based on continuous speech characterization provided by the embodiment of the present application may include: a computer device obtains the text to be converted and the reference speech, the text is the text content that needs to be converted into a speech signal, and the reference speech is a reference signal used as a guide for acoustic features. The computer device performs feature extraction processing on the text to obtain a text embedding vector for representing the semantics of the text; similarly, the reference speech is subjected to continuous speech tagging processing to obtain a multidimensional speech embedding vector (i.e., a multidimensional vector), which can be used to represent the acoustic features of the reference speech. Then, based on the multidimensional speech embedding vector of the reference speech and the text embedding vector of the text, the computer device performs speech prediction processing on the text to obtain a predicted speech sequence; the speech prediction processing here is intended to use the multidimensional speech embedding vector to guide the generation of speech signals for the text embedding vector, thereby generating multiple predicted speech signals that are referenced when the text reading speech corresponding to the text is read aloud. In this way, the computer device can generate the text reading speech corresponding to the text by converting the predicted speech sequence, specifically the multiple predicted speech signals in the predicted speech sequence. Considering that most of the speech information of the reference speech is retained to guide the generation of the predicted speech sequence, it can ensure that the acoustic features of the text reading speech match the acoustic features of the reference speech.

[0087] It can be seen that the speech processing solution based on continuous speech representation provided by the embodiment of the present application has obvious advantages when converting text to speech. The following is an example of comparing the present application solution with the existing mainstream text-to-speech technology to illustrate the advantages of the embodiment of the present application, where:

[0088] The existing mainstream text-to-speech technology based on discrete speech markers includes an autoregressive text-to-speech technology based on discrete speech markers. The autoregressive text-to-speech technology based on discrete speech markers supports representing text as discrete markers, and generates discrete speech markers from reference speech through discrete speech markers; then the two modal inputs (i.e., discretely marked text and discretely marked reference speech) are unified into the semantic space of discrete markers, so as to generate the text reading speech corresponding to the text through the autoregressive large language model. There are three main types of commonly used discrete speech markers, namely: discrete speech markers based on residual quantization, discrete speech markers based on grouped residual quantization, and Hubert discrete speech markers. Among them, the discrete speech marker based on residual quantization can repeat multiple discrete speech marking operations. Each time the nearest distance vector index is selected, the residual of the speech embedding vector of the reference speech minus the vocabulary embedding vector is used as the input of the next discrete speech marking operation; after repeated multiple times, multiple discrete speech markers are obtained. The discrete speech marker based on grouped residual quantization is based on the discrete speech marker based on residual quantization. The speech embedding vector of the reference speech is grouped according to the feature dimension for operation. After the discrete speech marking operation is completed, each mark combination is used as the final output. The Hubert discrete speech marker uses the Hubert pre-trained model as the discrete speech marker. For example, the structural diagram of the discrete speech marker can be seen in Figure 1 ;like Figure 1 As shown, the discrete speech tagging mainly includes modules such as encoder, quantizer and decoder; the reference speech is input into the discrete speech tagger in the form of waveform and then resampled, and the resampled output is input into the encoder, which initially represents the reference speech as a speech embedding vector in the form of a vector, and then prepares the vector embedding of the vocabulary; in this way, after the speech embedding vector is input into the quantizer, for the speech embedding vector, the quantizer selects the index of the vocabulary vector that is closest to the speech embedding vector in the vocabulary as the tag, that is, selects a discrete value from 0 to 1023 numbers as the discrete tag of the speech embedding vector.

[0089] It is not difficult to find that in the process of feature extraction of reference speech using discrete speech markers, the loss of speech information of the reference speech mainly occurs in the quantizer part in the discrete speech marker operation (i.e., feature extraction process); that is, in the step from speech embedding vector to discrete marker, the semantic embedding vector with a higher dimension is discretized into a single numerical value (a number from 0 to 1023), thereby losing most of the speech information of the reference speech, thereby reducing the conversion effect of text-to-speech. Furthermore, in order to reduce the speech loss caused by the discrete speech tagger, the prior art usually combines residual quantization for design; specifically, the speech embedding vector is represented by multiple discrete tags through multiple discretizations (multiple layers of residual quantization generate multiple discrete tags, such as shallow residual quantization mainly retains semantic content, and deep layers mainly retain acoustic details; however, such multiple discretizations will bring computational complexity and redundancy. Therefore, due to the information loss of the reference speech during the discrete speech tagging operation, there is a loss in speech feature extraction in the text-to-speech conversion, which makes the text-to-speech technology based on discrete speech tags seriously insufficient in performance, including but not limited to: the quality of the generated text reading speech is poor (such as the mismatch between the acoustic features of the text reading speech and the acoustic features of the reference speech); the naturalness of the text reading speech is poor, and from the auditory effect, it is easy to be recognized as machine-converted speech; the low noise quality is poor, that is, there is more noise in the text reading speech; the signal continuity of the text reading speech is poor; and the sound loudness of the text reading speech is small.

[0090] The embodiment of the present application introduces a continuous speech marker in the text-to-speech technology to extract features from the reference speech, and it is expected that the continuous speech marker output by the continuous speech marker can retain the complete speech information of the reference speech, thereby improving the quality of the text-reading speech based on the continuous speech marker. Compared with the discrete speech marker that only discretizes the acoustic features of the reference speech into a single numerical value to represent, the speech information loss of the reference speech in the text-to-speech technology can be reduced. In this way, text-to-speech is performed based on a multidimensional speech embedding vector with rich acoustic features, which can ensure that the acoustic features of the generated text-reading speech match the acoustic features of the reference speech (such as the style and timbre of the text-reading speech are close to the reference speech), thereby achieving a better conversion effect in the text-to-speech technology.

[0091] The speech processing solution based on continuous speech representation provided in the embodiment of the present application is a general text-to-speech technology that can automatically convert text into text-reading speech that matches the acoustic characteristics of the reference speech; this makes the speech processing solution provided in the embodiment of the present application applicable to a variety of text-to-speech scenarios that require text-to-speech conversion. Text-to-speech scenarios may include but are not limited to:

[0092] (1) The text-to-speech scenario is a dubbing scenario. In the dubbing scenario, it is necessary to dub the characters in the video (such as movies, TV series, advertisements, and games) to achieve the effect of attracting the attention of listeners or viewers with unique sound effects and conveying information. Taking the dubbing scenario as a video dubbing scenario as an example, in the scenario of dubbing a character in the video, the user can provide a reference voice (a voice belonging to the user who provided the video, or a voice of another authorized user) and the text to be converted; the computer device uses the voice processing method provided in the embodiment of the present application to perform feature extraction processing on the text to obtain a text embedding vector, and performs continuous voice tagging processing on the reference voice to obtain a multi-dimensional voice embedding vector; the computer device then performs voice prediction processing on the text based on the text embedding vector and the multi-dimensional voice embedding vector to obtain a predicted voice sequence, and finally converts the predicted voice sequence, that is, it can generate a text reading voice, the acoustic details of the text reading voice and the acoustic details of the reference voice provided by the user are similar / matched, thereby achieving dubbing for the character in the video.

[0093] It can be seen that the speech processing scheme provided by the embodiment of the present application can be used in a variety of dubbing scenarios, which can improve the voice quality of dubbing. In addition, in some dubbing scenarios, it also helps to improve the efficiency of dubbing; for example, in dubbing scenarios such as advertising dubbing or game dubbing, only one reference voice is needed for the same character to realize text-to-speech conversion of multiple texts of the same character, thereby realizing batch text-to-speech conversion.

[0094] (2) The text-to-speech scenario is a speech generation scenario. The speech generation scenario may include but is not limited to: ① The scenario of speech generation during the video clip generation process; for example, the generation of video or movie clips needs to ensure the consistency of multiple modal information (such as the video picture and the video voice need to ensure consistency). The text-to-speech technology used in the embodiment of the present application is used to generate video or movie clips. It can be used to simplify the generation process of video clips containing multiple modalities such as continuous images, text and voice, and reduce the generation of voice modalities while obtaining high-quality video clips. ② The speech generation scenario of text during barrier-free reading; for example, for visually impaired people, they cannot see the text content. At this time, the speech processing solution provided in the embodiment of the present application can be used to generate text reading speech based on reference speech and text content; the text content is broadcast to the visually impaired in the form of playing text reading speech, which greatly improves the convenience of reading text content for the visually impaired.

[0095] (3) The text-to-speech scenario is an intelligent question-and-answer scenario. Intelligent question-and-answer refers to a human-computer interactive question-and-answer method that receives questions from users and answers them accurately and concisely in natural language. In an intelligent question-and-answer scenario (such as an intelligent question-and-answer application on a car system, or an intelligent question-and-answer function deployed in a hotel robot), a computer device receives questions input by a user, analyzes the text, and retrieves answers to the questions in text form. The voice processing solution provided in the embodiment of the present application can be used to convert the answers to the questions in text form into text-reading speech, and the text-reading speech can be played to answer the user's questions in natural language. This improves the simplicity of question-and-answering while ensuring the accuracy of text-to-speech conversion.

[0096] It should be understood that the above description is only an exemplary product performance and text-to-speech scenario given in the embodiments of the present application, and does not limit the product performance and text-to-speech scenario of the voice processing solution provided in the embodiments of the present application. The voice processing solution provided in the embodiments of the present application can provide efficient, accurate and convenient text-to-speech services in various text-to-speech scenarios, showing high value and practicality in various text-to-speech scenarios, and helping to improve user experience and satisfaction.

[0097] To facilitate understanding of the voice processing solution provided in the embodiment of the present application, Figure 2 The scenario diagram shown in the figure briefly introduces the text-to-speech scenario involved in the embodiment of the present application; Figure 2 As shown, the system includes a terminal 201 and a server 202. The embodiment of the present application does not limit the number and naming of the object 201, the terminal 201 and the server 202.

[0098] Among them, the terminal 201 may refer to a terminal device with a text-to-speech function; the user holds the terminal 201 and can perform user operations in the terminal 201. The terminal 201 may include, but is not limited to: a smartphone (such as a smartphone deploying an Android system, or a smartphone deploying an Internet working Operating System (IOS)), a tablet computer, a portable personal computer, a mobile Internet device (Mobile Internet Devices, MID), a vehicle-mounted device, a head-mounted device, an intelligent chat robot and an aircraft and other terminal devices. The embodiment of the present application does not limit the type of terminal device, which is explained here. The server 202 is a server corresponding to the terminal 201, which is used to interact with the terminal 201 for data to provide computing and application service support for the terminal 201. The server 202 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. Among them, each device included in the text-to-speech system can be directly or indirectly connected to each other through limited or wireless means, and the embodiment of the present application does not limit the communication method between devices; for example: the communication method between devices may include but is not limited to: using HTTP requests, remote procedure calls (RemoteProcedur e Call), sockets (Socket) and shared content.

[0099] The speech processing solution provided in the embodiment of the present application can be executed by a computer device, which is one or more devices in a text-to-speech system. Figure 2 The terminal 201 or server 202, or computer device in the system shown is Figure 2 The terminal 201 and the server 202 in the system are shown. In other words, the speech processing solution provided in the embodiment of the present application can be executed by one of the terminal 201 or the server 202, or by both the terminal 201 and the server 202; the embodiment of the present application does not limit the type and quantity of computer devices.

[0100] Taking the computer device including terminal 201 and server 202 as an example, Figure 2The process of the speech processing solution provided in the embodiment of the present application is introduced. In a specific implementation, the user can input the text to be converted and the reference speech through the terminal 201. If the terminal 201 is an accessible device, the user inputs the text and the reference speech that he wants to read aloud through the accessible device; at this time, the terminal 201 forwards the text to be converted and the reference speech to the server 202. In this way, after receiving the text to be converted and the reference speech, the server 202 performs feature extraction processing on the text to obtain a text embedding vector for representing the semantics of the text; similarly, the reference speech is subjected to continuous speech tagging processing to obtain a multidimensional speech embedding vector (i.e., a multidimensional vector), which can be used to represent the acoustic features of the reference speech. Then, based on the multidimensional speech embedding vector of the reference speech and the text embedding vector of the text, the server 202 performs speech prediction processing on the text to obtain a predicted speech sequence. The server 202 then performs conversion processing on the predicted speech sequence to generate a text reading speech with acoustic details similar to the input reference speech. The server 202 packages the text-reading voice and returns it to the terminal 201 so as to play the text-reading voice through the speaker of the terminal 201 .

[0101] Based on the above-described speech processing solution and system architecture, the following points need to be explained:

[0102] ① The above-mentioned embodiments of the present application Figure 2 The system shown is for more clearly illustrating the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. It is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems. For example, the above is based on the example that the execution subject "computer device" of the embodiment of the present application includes a terminal and a server, that is, the terminal and the server jointly execute the voice processing solution provided in the embodiment of the present application; it should be understood that in some cases, if the terminal 201 has strong computing power, then the voice processing solution can be executed by the terminal 201 alone.

[0103] ② The speech processing scheme provided in the embodiment of the present application is implemented through a text-to-speech model. Specifically, the trained text-to-speech model can be deployed in a computer device; in this way, when the computer device receives the text to be converted and the reference speech, the text-to-speech model can be directly called to execute the speech processing scheme, which not only improves the speed of text-to-speech, but also effectively ensures the accuracy of text-to-speech conversion through the model. Among them, if the computer device used to execute the speech processing scheme provided in the embodiment of the present application is a terminal, the trained text-to-speech model is deployed in the terminal. If the computer device used to execute the speech processing scheme provided in the embodiment of the present application is a server, then the trained text-to-speech model is deployed in the server; in this case, the terminal used by the user transmits the text to be converted and the reference speech to the server for text-to-speech operation, and the server pushes the converted text reading speech to the terminal.

[0104] Furthermore, when the trained text-to-speech model is deployed in the terminal, it can be deployed in the terminal in the form of a plug-in or an application. For example, the trained text-to-speech model is deployed in the terminal as a system-level plug-in, then any application deployed in the terminal can call the plug-in to implement text-to-speech. For another example, the trained text-to-speech model is deployed in an application, then after the application is started through the terminal, the trained text-to-speech model can be called in the application to implement text-to-speech; at this time Figure 2The server in the system shown is a server owned by the application provider of the application, which can interact with the application in the terminal to provide a text-to-speech server. Among them, the application may refer to a computer program for completing one or more specific tasks; by classifying the application according to different dimensions (such as the operation mode and function of the application), the types of the same application in different dimensions can be obtained. For example: according to the operation mode of the application, the application may include but is not limited to: a client installed in the terminal, a small program (as a subprogram of the client) that can be used without downloading and installing, a web (World Wide Web, Global Wide Area Network) application opened through a browser, etc. For another example: according to the functional type of the application, the application may include but is not limited to: IM (Instant Messaging) application, content interaction application, etc. Among them, the instant messaging application refers to an application for instant communication of messages and social interaction based on the Internet. The instant messaging application may include but is not limited to: a social application with communication functions, a map application with social interaction functions, a game application, etc. Content interaction application refers to an application that can realize content interaction, such as online banking, sharing platform, personal space, news and other applications. The embodiments of the present application do not limit the specific type of application running in the terminal and deployed with a trained text-to-speech model, which is explained here.

[0105] ③ The collection and processing of relevant data in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations. The acquisition of personal information requires the knowledge or consent of the individual subject (or the legal basis for obtaining information), and the subsequent use and processing of data shall be carried out within the scope of authorization of laws and regulations and the subject of personal information. For example, when the embodiments of this application are applied to specific products or technologies, such as reference speech, the permission or consent of the speaker of the reference speech is required, and the collection, use and processing of relevant data (such as the conversion of converted text into speech, etc.) shall comply with the relevant laws, regulations and standards of the relevant regions.

[0106] Based on the speech processing scheme described above, the embodiment of the present application proposes a more detailed speech processing method. The speech processing method proposed in the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings.

[0107] Figure 3 A flow chart of a speech processing method provided by an exemplary embodiment of the present application is shown; the speech processing method may be executed by the aforementioned computer device, such as the computer device is a terminal; the speech processing method may include but is not limited to steps S301-S305:

[0108] S301: Acquire the text to be converted and the reference speech.

[0109] The text to be converted is the text content that needs to be converted into speech, and the reference speech is a reference speech used to guide / direct the acoustic features of the speech to be generated in the text-to-speech process; the text-to-speech technology aims to convert the text into a text reading speech whose acoustic features match the acoustic features of the reference speech.

[0110] Depending on the different text-to-speech scenarios, the sources of the text to be converted and the reference voice (i.e., the acquisition method) will be different. As mentioned above, the text-to-speech scenario is a dubbing scenario - a video dubbing scenario; optionally, the text to be converted and the reference voice can be provided by the user, specifically, the user uploads the text to be converted and the reference voice through the information input interface provided by the terminal (such as the information input interface is a service interface 401 provided by an application with text-to-speech), so that the computer device can obtain the text to be converted and the reference voice; optionally, the text to be converted is the original lines of the film and television drama, provided by the video application, and the reference voice is provided by the user to help the user convert the original lines into a text reading voice that conforms to his or her own language style. For another example, the text-to-speech scenario is a dubbing scenario - a game dubbing scenario; at this time, the text to be converted and the reference voice are both set by the game developer. This method of providing the text to be converted and the reference voice by the game developer can unify the voice content and language style of the same game character in the game scenario, and meet the personalized needs of the game developer for the game character.

[0111] It should be noted that the way in which the user inputs the text to be converted and the reference speech in the service interface provided by the application can be the same or different. For example, the text can be manually edited and input directly in the information input interface, or it can be imported into the local text content or downloaded from the Internet; for another example, the reference speech can be imported from the local memory or the Internet, or the reference audio can be selected from the candidate audios provided by the application, etc.; the embodiment of the present application does not limit the way in which the user inputs the text to be converted and the reference speech in the information input interface. In addition, Figure 4 The interface content and style of the information input interface shown are examples and will not limit the specific implementation method of the user inputting the text to be converted and the reference voice in the embodiment of the present application, and are specially explained here.

[0112] S302: Perform feature extraction on the text to obtain a text embedding vector.

[0113] After the computer device obtains the text to be converted, it needs to perform feature extraction processing on the text, aiming to learn and extract the semantics of the text so that the extracted text semantics can be better used for subsequent tasks (such as text-to-speech). The feature extraction processing for the text here is to extract a set of feature vectors from the text to represent the semantics of the text; in the embodiment of the present application, a set of feature vectors extracted from the text is called a text embedding vector, which is used to represent the semantics of the text.

[0114] In the embodiment of the present application, it is supported to use a text marker to perform feature extraction processing on the text, and then obtain a text embedding vector represented by a vector through an embedding layer in a neural network. Among them, a text marker is a general term for a class of text processing tools that can effectively help users process and analyze text content; in the embodiment of the present application, a text marker can perform text marking on the input text, aiming to mark important strings (or words, characters, etc.) in the text, omit meaningless characters or strings in the text (such as removing stop words in the text, such as words or symbols that have no effect on semantic understanding, etc.), and segment the text to represent the text content as discrete text tags (such as some strings that appear frequently in the text or have important semantics); then use the embedding layer to convert the text tags output by the text marker into a text embedding vector in vector form. Exemplarily, the formula for feature extraction processing of text using a text marker can be found as follows:

[0115] x = Tokenizer Text (T) (1)

[0116]

[0117] Among them, Tokenizer Text is a text tagger; T is the input text; x is the preliminary vector representation obtained by feature extraction of the text; Embeds is the embedding layer; is the text embedding vector of the text, which can be expressed as [T0, T1, ...], where T i It is the sub-text embedding vector obtained by performing feature extraction on each text block after dividing the text into blocks, i = 0, 1, ...

[0118] The embedding layer mentioned above is a special layer in the neural network, which is mainly used to map discrete feature representations (i.e. text tags) to continuous vector space, and use vector form to represent the semantics of text tags, so as to facilitate the subsequent learning of the text embedding vector in the vector form; in other words, the embedding layer can map the high-dimensional discrete features of the text (such as text tags - words or characters) to vector representations in low-dimensional continuous space. This conversion enables the vector distance between different vector representations to reflect the semantic relationship between the corresponding high-dimensional discrete features, thereby preserving the semantic relationship between text tags in the text, such as the similarity and correlation between semantics. The neural network mentioned above belongs to the field of machine learning (ML) under artificial intelligence (AI) technology; the basic components of the neural network include input layer, hidden layer (one or more) and output layer. Each layer consists of neurons and weights between neurons. The input layer receives external input information and passes it to the hidden layer for processing (such as the embedding layer is a hidden layer), and finally the output layer outputs the result.

[0119] S303: Perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector.

[0120] After obtaining the reference speech, the computer device supports continuous speech tagging of the reference speech to obtain a multi-dimensional speech embedding vector of the reference speech; the multi-dimensional speech embedding vector is a multi-dimensional vector representation that can be used to completely and continuously characterize the acoustic features of the reference speech. In an embodiment of the present application, by characterizing the reference speech as a continuous multi-dimensional speech embedding vector, compared with the related art of discretizing the reference speech into a number to characterize the semantics, the acoustic features of the reference speech can be largely retained, such as the speech information of each time step in the speech time period of the reference speech (the time step refers to the time point distribution according to the speech time period, the time interval between the two previous and next time points, such as the time step is a time interval of 0 seconds to 1 second) is retained, so as to guide the generation of the text reading speech based on richer acoustic features, which can improve the matching between the acoustic features of the text reading speech and the acoustic features of the reference speech, that is, the output style and timbre of the text reading speech is similar to the reference speech.

[0121] In an embodiment of the present application, a continuous speech marker is provided to implement continuous speech marking processing for a reference speech. As described above, the continuous speech marker is different from a discrete speech marker, and can output a continuous acoustic representation (i.e., the acoustic features of the reference speech in the entire speech time period can be represented and retained), rather than a discrete mark (i.e., only the preliminary vector representation of the reference speech mapped to the vector space is quantized to a nearest number within [0,1023]), so more acoustic features of the reference speech can be retained for subsequent text-to-speech.

[0122] For ease of understanding, the following Figure 5 The process of using a continuous speech marker to mark the reference speech is introduced as an example. Figure 5 As shown, after the computer device obtains the reference speech, it takes into account that the speech duration of the reference speech is often long, such as the speech duration of the reference speech is several hours; if the entire reference speech is subjected to continuous speech marking processing, the long speech duration will lead to complex calculations and even reduce the marking accuracy. To this end, the embodiment of the present application supports segmenting the reference speech before performing continuous speech marking processing on the reference speech, such as using a speech segmenter to perform speech segmentation processing on the reference speech, so as to split the reference speech with a longer speech duration into at least one speech segment with a shorter speech duration for easy analysis. Among them, the speech segmenter involved in the embodiment of the present application is an algorithm or method for splitting a large section of reference speech into shorter speech durations; its goal is to create manageable speech segments that can be processed individually. As Figure 5 The speech duration of the reference speech shown is 30 seconds. The reference speech is split into three speech segments using a speech segmenter. The speech duration of speech segment 1 is 10 seconds, the speech duration of speech segment 2 is 8 seconds, and the speech duration of speech segment 3 is 12 seconds.

[0123] Then, the computer device uses a continuous speech marker to perform continuous speech marking processing on each speech segment in at least one speech segment obtained by segmentation, and obtains a continuous speech marker of each speech segment. The continuous speech marker of any speech segment is represented by a multidimensional vector (or a multidimensional vector), which is different from the discrete speech marker marking the speech segment as a discrete single number when marking the speech segment. The continuous speech marker can completely retain all acoustic features of the speech segment in the segment time period of the speech segment in the form of a multidimensional vector; specifically, the continuous speech marker includes multiple continuous values, which constitute a multidimensional vector, and the multiple continuous values ​​can continuously represent the acoustic features of the speech segment in the order of the time steps of the segment time period (i.e., the speech duration) of the speech segment. For example, the speech duration of speech segment 1 is 10 seconds and the time step is 1 second. Then the continuous speech mark obtained by performing continuous speech mark processing on the speech segment 1 can be represented as a vector with one row and multiple columns. The vector values ​​in the vector with one row and multiple columns (that is, the continuous values ​​mentioned above) can represent the acoustic features of each time step; for example, the multiple continuous values ​​in the continuous speech mark can represent the acoustic features of the speech segment 1 at the 1st second, the 2nd second, the 3rd second, ..., the 10th second in turn (such as the pitch, timbre and loudness at the 2nd second).

[0124] Among them, the exemplary structural diagram of the continuous speech marker can be found in Figure 6 ;like Figure 6As shown, the continuous speech marker includes a resampling module and a convolution module. In a specific implementation, after receiving at least one speech segment, the continuous speech marker first uses the resampling module to resample each speech segment to obtain each resampled speech segment; wherein, the resampling of each speech segment, on the one hand, by changing the sampling rate, sampling format and number of channels of each speech segment, the speech format (or audio format) of the speech segment input into the subsequent module for subsequent tasks is unified; on the other hand, the frequency range of each speech segment can be appropriately reduced, thereby reducing the amount of data required for calculation, which is convenient for data storage and transmission. Then, the convolution module is used to perform multi-level feature extraction processing on each resampled speech segment to obtain a continuous speech marker for each resampled speech segment. In detail, the convolution module includes a one-dimensional convolution layer, N convolution layers and a one-dimensional convolution layer in sequence, where N is a positive integer; the convolution layer can extract features from the input resampled speech segment, and these features are mainly acoustic features of semantic segments, such as timbre, pitch and sound intensity. The embodiment of the present application realizes multi-level or multi-level extraction of acoustic features of speech fragments by setting multiple convolutional layers in the continuous speech marker, which can not only extract the semantic content of the speech fragments but also the acoustic features, thereby avoiding the loss of semantic feature extraction for the reference speech in the text-to-speech process and improving the performance of the text-to-speech technology.

[0125] Finally, after performing continuous speech tagging processing on each speech segment of the reference speech based on the above description to obtain the continuous speech tag of each speech segment, the continuous speech tag of at least one speech segment is combined according to the playback position of at least one speech segment in the reference speech, so as to obtain the multi-dimensional speech embedding vector of the reference speech. The combination here means that the continuous speech tags of at least one speech segment are sequentially connected according to the playback position of each speech segment in the reference speech; Figure 5 As shown, the continuous speech mark of speech segment 2 is connected to the end of speech segment 1, and the continuous speech mark of speech segment 3 is connected to the end of the continuous speech mark of speech segment 2. The vector dimension of the connected multidimensional speech embedding vector is the same as the sum of the number of continuous values ​​included in the continuous speech mark of at least one speech segment. For example, if the number of continuous values ​​of the continuous speech mark of speech segment 1 is 100, the continuous value of the continuous speech mark of speech segment 2 is 80, and the continuous value of the continuous speech mark of speech segment 3 is 200, then the vector dimension of the multidimensional speech embedding vector is 100+80+200=380.

[0126] Exemplarily, the formula for performing continuous speech tagging processing on the reference speech using the continuous speech tagger can be seen as follows:

[0127]

[0128] Among them, ContTokenizer Audio is a continuous speech tagger, whose input A is the reference speech and whose output is the multi-dimensional speech embedding vector of the reference speech, which can be expressed as [A0, A1, ...], where A j It is a sub-multidimensional speech embedding vector obtained by segmenting the reference speech and performing continuous speech tagging on each speech segment, j = 0, 1, ... The vector dimension of the multidimensional speech embedding vector is the same as the feature dimension of the embedding layer of the language model used for subsequent speech prediction processing, so as to ensure that the output data format of the continuous speech tagger meets the input data format requirements of the embedding layer in the speech model.

[0129] It can be seen that the embodiment of the present application can maximize the retention of speech information (such as semantic information and acoustic features) of the reference speech by making the speech part continuous during the text-to-speech process, specifically using a continuous speech marker to continuously mark the reference speech to characterize the acoustic features of the reference speech, rather than using a discrete marker to discretely mark the reference speech to characterize the acoustic features, thereby improving the text-to-speech conversion effect. For example, the existing discrete marking process can be seen in Figure 7 ;like Figure 7 As shown, the discrete marker only performs a preliminary vector representation of the reference speech (or speech fragment) (such as using only one convolutional layer for feature extraction), and selects a number closest to the vector representation from the vocabulary (i.e., a limited range [0,1023]) as the discrete marker of the reference speech, that is, uses the single number to characterize the acoustic features of the reference speech; it is not difficult to find that the discrete speech marker, which selects a number from the vocabulary to represent the vector representation, will lose more acoustic details of the reference speech, resulting in poor text-to-speech performance. In contrast, the embodiment of the present application uses a continuous speech marker to perform continuous speech tagging on the reference speech (or speech fragment), which can not only continuously characterize the acoustic features of each time step in the speech time period of the reference speech, but also use a multi-dimensional speech embedding vector to not only fully characterize the acoustic features but also ensure the accuracy of the acoustic feature representation.

[0130] S304: Based on the multi-dimensional speech embedding vector and the text embedding vector, perform speech prediction processing on the text to obtain a predicted speech sequence.

[0131] Based on the above steps, a continuous speech marker is used to perform continuous speech marking processing on the reference speech to obtain a multi-dimensional speech embedding vector, and a text marker is used to perform feature extraction processing on the text. After the text embedding vector is obtained, the language model is used to support text speech prediction processing (or autoregressive processing) based on the multi-dimensional speech embedding vector and the text embedding vector, and the output is a predicted predicted speech sequence; the predicted speech sequence includes multiple predicted speech information referenced / used at each time step when the text reading speech corresponding to the text is read aloud, and one predicted speech information is referenced at one time step. In other words, the language model can be used to realize the predicted speech information corresponding to the text reading speech at each time step when the multi-dimensional speech embedding vector based on the reference speech and the text embedding vector of the text is converted into the text reading speech; the predicted speech information corresponding to any time step can be understood as the information used to generate the speech signal to be played corresponding to the time step.

[0132] Among them, the language model, or large language model, is a model based on machine learning and natural language processing technology, which learns to serve the ability of human language understanding and generation by training a large amount of text data. Specifically, the large language model is designed to understand and generate human language; it is trained on a large amount of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. The embodiment of the present application mainly relates to an autoregressive model (AR) under a language model; in the field of natural language processing, the autoregressive model has the ability to predict the next tag based on the previous series of tags, specifically the language model task of predicting the next (or previous) possible string based on the previous content (or the following content). The autoregressive model usually uses a stacked decoder to realize the prediction of the output tag, and has the characteristic of relying on its own historical value to predict the future value. Among them, the autoregressive model may include but is not limited to a model with autoregressive properties obtained by pre-training and fine-tuning, and the embodiment of the present application does not limit the model type of the autoregressive model.

[0133] Considering that the text reading speech to be generated is read in the order of time steps, the predicted speech signal of each time step is read in sequence; therefore, the embodiment of the present application supports using the autoregressive model to predict the predicted speech information of each time step in units of time steps, and then composes a predicted speech sequence based on the predicted speech information of all time steps. Based on this, the implementation process of using the autoregressive model based on the multi-dimensional speech embedding vector of the reference speech and the text embedding vector of the text to perform speech prediction processing on the text to obtain the predicted speech sequence is introduced below. In the specific implementation, the time step is represented as t, where t is a non-negative integer, i.e., t=0, 1, 2, ...; when the time step t=0, the multidimensional speech embedding vector and the text embedding vector are input into the autoregressive model for speech prediction processing / autoregressive processing to obtain the predicted speech information of time step 0; when the time step t>0, the multidimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before the time step t are subjected to speech prediction processing to obtain the predicted speech information of the time step t; until the time step t reaches the predicted time length required for the text reading speech to be played, that is, until the autoregressive model predicts that the time step t has reached the end time step of the text reading speech. In this way, the predicted speech information of each time step t predicted by the autoregressive model constitutes a predicted speech sequence.

[0134] In more detail, when time step t=0, the process of inputting the multidimensional speech embedding vector and the text embedding vector into the autoregressive model for speech prediction processing / autoregressive processing may include: firstly performing vector connection or concatenation on the multidimensional speech embedding vector and the text embedding vector to obtain a connected vector; then using a language model (specifically an autoregressive model) to regress the connected vector to generate predicted speech information at time step 0. Similarly, when time step t>0, the process of performing speech prediction processing on the multidimensional speech embedding vector, the text embedding vector and the predicted speech information at each time step before time step t may include: performing vector connection on the multidimensional speech embedding vector, the text embedding vector and the predicted speech information at each time step before time step t to obtain a connected vector; then using a language model (specifically an autoregressive model) to regress the connected vector to generate predicted speech information at time step t. The calculation formula for the predicted speech information at any time step t is as follows:

[0135]

[0136] Among them, LM is the language model; is the predicted speech information at time step t; is the text embedding vector of the text; is a multi-dimensional speech embedding vector of the reference speech; is the predicted speech information at each time t in all time steps t before time step t. Indicates that Perform vector connection, specifically embedding multidimensional speech into vectors Connect to text embedding vector After that, the predicted speech information of each time t in all time steps t before time step t is After connecting to the multidimensional speech embedding vector, by splicing the text tag for the text (i.e., the text embedding vector) in front and the continuous speech tag of the reference speech (i.e., the multidimensional speech embedding vector) in the back as the input condition of the language model, it helps the language model to output predicted speech information with a style and timbre similar to the reference speech according to this input condition format.

[0137] To facilitate understanding of the speech prediction process performed by the language model described above, Figure 8 The speech prediction processing performed by the language model at time steps t=0, 1, 2, and 3 is used as an example to introduce the process of obtaining the predicted speech information at each time step. Figure 8 As shown, when time step t = 0, the multidimensional speech embedding vector of the reference speech is first connected to the text embedding vector of the text to obtain the connected vector; as shown in the multidimensional speech embedding vector Text Embedding Vector Then the concatenated vector can be expressed as Then, the concatenated vector is input into a language model (such as an autoregressive model) for regression processing to generate the predicted speech information of time step 0; the predicted speech information of time step 0 can be expressed as Similarly, when time step t = 1, the multidimensional speech embedding vector of the reference speech is connected to the text embedding vector of the text, and the predicted speech information at time step 0 is connected to the multidimensional speech embedding vector to obtain the connected vector, which is represented as The concatenated vector is then input into the language model for regression processing to generate the predicted speech information at time step t = 1. The predicted speech information at time step t = 1 can be expressed as By analogy, we get the predicted speech information at time step t=2 and the predicted speech information at time step t=3 Then, the final predicted speech sequence includes

[0138] It can be seen that, on the one hand, the embodiment of the present application supports predicting the predicted voice information referenced by each time step when the text reading voice is read aloud from the time step dimension, which can ensure that the predicted voice information corresponding to each time step is predicted based on the acoustic features of the reference voice, thereby ensuring that the acoustic features such as the language style and timbre of the text reading voice at each time step can match the acoustic features of the reference voice, thereby improving the text-to-speech conversion effect. On the other hand, the embodiment of the present application uses an autoregressive model to implement speech prediction processing, that is, the embodiment of the present application essentially proposes an autoregressive text-to-speech technology based on continuous speech markers; compared with non-autoregressive text conversion technology, it relies on the autoregressive model to capture the advantages of time correlation, flexibility and high prediction accuracy, and mines the correlation between the text embedding vector and the multidimensional voice embedding vector (such as the previous text embedding vector and the subsequent multidimensional voice embedding vector), as well as the correlation between the predicted voice information between the time steps, thereby improving the accuracy of the predicted voice information at each time step, which is conducive to improving the matching degree between the subsequently generated text reading voice and the reference voice.

[0139] S305: converting the predicted speech sequence to generate text reading speech.

[0140] Based on the above steps, in the text part, it supports representing the text as discrete text tags through a text tagger, and converting it into a text embedding vector through an embedding layer; and, in the speech part, a continuous speech tagger is used to directly convert the reference speech into a multidimensional speech embedding vector without discretizing the vector into a single number (i.e., discrete tags). The text embedding vector and the multidimensional speech embedding vector are input into a language model for autoregressive generation, and the output is a predicted speech sequence. The predicted speech sequence is the result of encoding the reference speech and text. The embodiment of the present application also needs to decode the result of the encoding (i.e., the predicted speech information) to restore the text reading speech; specifically, the predicted speech sequence is converted to obtain a text reading speech whose acoustic features in the time domain are similar to those of the reference speech.

[0141] In an embodiment of the present application, a speech decoder can be used to convert a predicted speech sequence into a text-reading speech corresponding to the text. Among them, the speech decoder mainly decodes the predicted speech sequence through built-in convolutional layers and feedforward neural networks (FNN) and other modules to restore the text-reading speech corresponding to the text. Specifically, the speech decoder uses convolution kernels of different sizes to perform convolution processing on each predicted speech information in the input predicted speech sequence to obtain the key speech features (such as key acoustic features) corresponding to each predicted speech information; then, the speech decoder generates a predicted speech corresponding to each predicted speech information based on the key speech features corresponding to each predicted speech information, and uses flow-matching technology to help adjust the rhythm and intonation of the predicted speech during the conversion process to ensure that the synthesized predicted speech has better fluency and naturalness; after obtaining the predicted speech corresponding to each predicted speech information, the speech decoder connects each predicted speech in the order of time steps, that is, generates the text-reading speech corresponding to the text. Among them, the calculation formula for converting the predicted speech sequence into text-reading speech using a speech decoder is as follows:

[0142]

[0143] Among them, AudioDecoder is the voice decoder, is the predicted speech sequence input to the speech decoder, A out The text-to-speech output of the speech decoder.

[0144] In summary, the speech processing scheme provided in the embodiment of the present application, on the one hand, supports continuous speech characterization of the reference speech, specifically, characterizing the continuous acoustic features of the reference speech according to the speech playback order during the speech feature extraction process of the reference speech, and expressing the continuous acoustic features in the form of a multi-dimensional speech embedding vector, so as to achieve the maximum retention of the speech information of the reference speech; compared with the discrete speech mark only discretizing the acoustic features of the reference speech into a single numerical value to represent, it can reduce the speech information loss of the reference speech in the text-to-speech technology. On the other hand, the speech processing scheme is specifically an autoregressive text-to-speech technology starting from continuous speech mark; compared with the non-autoregressive text conversion technology, it can mine the correlation between the text embedding vector and the multi-dimensional speech embedding vector, as well as the correlation between the predicted speech information between the time steps, thereby improving the accuracy of the predicted speech information of each time step, which is conducive to improving the matching degree between the subsequently generated text reading speech and the reference speech. Based on the advantages described above, the text-to-speech technology starting from continuous speech characterization provided in the embodiment of the present application is superior to the text-to-speech technology based on discrete markup in multiple evaluation indicators; wherein, the comparison results of the evaluation indicators are shown in Table 1 below:

[0145] Table 1

[0146]

[0147] Among them, the CLVP (Contrastive Listening Value Perception) score is mainly used to evaluate the consistency and authenticity of the generated text reading speech and text content; the lower the CLVP score, the higher the consistency of the text reading speech and text content. The wav2vec2 clarity score is mainly used to evaluate the speech clarity of the generated text reading speech; the lower the wav2vec2 clarity score, the higher the clarity of the text reading speech. The Stoi score is mainly used to evaluate the intelligibility of the generated text reading speech (that is, the degree to which it can be understood by the user); the value of the Stoi score is between 0 and 1, and the larger the value of the Stoi score, the higher the intelligibility of the text reading speech. The predicted speech quality is mainly used to evaluate the speech quality of the generated text reading speech, such as speech clarity, whether there is interference, and whether there is signal loss; the higher the predicted speech quality score, the better the speech quality of the text reading speech. Text-to-speech credibility is mainly used to evaluate the accuracy of the generated text-reading speech, such as whether the text content of the text-reading speech is consistent with the input text content, and the matching degree between the acoustic features of the text-reading speech and the acoustic features of the reference speech. The higher the score of text-to-speech credibility, the higher the credibility of the text-reading speech. Low-noise quality is mainly used to evaluate the level of low noise in the generated text-reading speech. The higher the score of low-noise quality, the less unnecessary low noise in the text-reading speech. Continuity is mainly used to evaluate the continuity of the speech signal of the generated text-reading speech. The higher the score of continuity, the more continuous the speech signal of the text-reading speech. Loudness is mainly used to evaluate the volume of the generated text-reading speech. The larger the score of loudness, the more appropriate the volume of the text-reading speech (that is, the volume will not be too small or too large). It can be seen that the embodiment of the present application compares the text-to-speech solution based on continuous marking provided by the embodiment of the present application with the text-to-speech technology based on discrete marking through multiple evaluation indicators, and shows that the present solution achieves higher predicted speech quality, and is superior to the text-to-speech technology based on discrete marking in terms of indicators such as low noise and speech credibility.

[0148] As described above, the speech processing method provided in the embodiment of the present application is implemented through a text-to-speech model, which includes a target text marking module (i.e., a trained text marker), a target continuous speech marking module (i.e., a trained continuous speech marker), a target speech prediction module (i.e., a trained language model) and a target speech decoding module (i.e., a trained speech decoder).

[0149] Combine the following Fig. 9 The illustrated embodiment introduces the specific implementation process of the speech processing method provided by the embodiment of the present application from the dimensions of model reasoning and model training. Fig. 9 As shown, Fig. 9 A flow chart of another speech processing method provided by an exemplary embodiment of the present application is shown; the speech processing method may be executed by the aforementioned computer device, such as the computer device is a terminal; the speech processing method may include but is not limited to steps S901-S905:

[0150] S901: Input the acquired text to be converted and reference speech into a text-to-speech model.

[0151] For example, the model structure diagram of the text-to-speech model can be found in Fig.10 ;like Fig.10 As shown, the text-to-speech model includes a target text tagging module, a target continuous speech tagging module, a target speech prediction module, a target speech decoding module, and an embedding layer. In the model inference stage, after the computer device receives the text to be converted and the reference audio, the text to be converted and the reference audio can be input into the trained text-to-speech model. In this way, the text-to-speech model can call each module to perform text-to-speech on the text and the reference speech; specifically, through step S902, the target text tagging module can be called to perform feature extraction processing on the text (specifically including first discretely tagging the text, and then inputting the discretely marked text into the embedding layer) to obtain the text embedding vector of the text, and, through step S903, the target continuous speech tagging module is called to perform continuous speech tagging processing on the reference speech to obtain the multi-dimensional speech embedding vector of the reference speech; through step S904, the target speech prediction module is called to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence; and, through step S905, the target speech decoding module is called to convert the predicted speech sequence to generate the text reading speech corresponding to the text.

[0152] S902: The target text tagging module in the text-to-speech model is used to perform feature extraction processing on the text to obtain a text embedding vector.

[0153] S903: A target continuous speech tagging module in the text-to-speech model is used to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector.

[0154] S904: A target speech prediction module in the text-to-speech model is used to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence.

[0155] S905: The target speech decoding module in the text-to-speech model is used to convert the predicted speech sequence to generate a text reading speech corresponding to the text.

[0156] It should be noted that the specific implementation process of each step in steps S902-S905 is the same as the above Figure 3 The specific implementation process shown in steps S302-S305 in the illustrated embodiment is the same and will not be described in detail here.

[0157] For example, the process of the target text marking module shown in step S902 performing feature extraction processing on the text to obtain the text embedding vector can be referred to in the above Figure 3 The description of the specific implementation process of performing feature extraction processing on the text to obtain a text embedding vector as shown in step S302 in the illustrated embodiment.

[0158] Similarly, the process of the target continuous speech tagging module shown in step S903 performing continuous speech tagging on the reference speech to obtain a multi-dimensional speech embedding vector can be referred to in the above Figure 3 The description of the specific implementation process of performing continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector as shown in step S303 in the illustrated embodiment.

[0159] Similarly, the target speech prediction module shown in step S904 performs speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain the process of predicting the speech sequence, which can be referred to in the above Figure 3 In the illustrated embodiment, step S304 shows a description of a specific implementation process of performing speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence.

[0160] Similarly, the target speech decoding module shown in step S905 converts the predicted speech sequence to generate the text reading speech corresponding to the text, which can be seen in the above Figure 3 The description of the specific implementation process of converting the predicted speech sequence to generate text reading speech as shown in step S305 in the illustrated embodiment.

[0161] The above mainly introduces the speech processing method provided in the embodiment of the present application from the model reasoning dimension; the following introduces the training process of the text-to-speech model in the embodiment of the present application from the model training dimension.

[0162] Specifically, the model training for the text-to-speech model in the embodiment of the present application is roughly divided into two steps, namely, the model training for the initial continuous speech marking module and the initial speech decoding module to be trained, and the model training for the entire text-to-speech model. In other words, the embodiment of the present application supports the preset parameters of the target text marking module, that is, the parameters of the target text marking module are preset and fixed; in this case, the initial continuous speech marking module and the initial speech decoding module are first jointly / synchronously trained, and after the target continuous speech marking module and the target speech decoding module are obtained through training, in the subsequent training of the entire text-to-speech model, the parameters of the target continuous speech marking module and the target speech decoding module are kept fixed, and the parameters of the initial speech prediction module to be trained are focused on to optimize the parameters to obtain a trained target speech prediction module.

[0163] The step-by-step training process for the text-to-speech model described above can be found in Fig.11 ; The step-by-step training process can be summarized as steps s11-s14, where:

[0164] s11: Obtain a first data set. The first data set includes training data audio and corresponding text labels; the text labels are obtained by converting the corresponding training data audio into text.

[0165] The first data set is a speech data set used to synchronously optimize the initial continuous speech tagging module and the initial speech decoding module; the first data set includes training data audio and text labels converted from the training data audio using a speech translator, and the text labels mainly include the text content included in the training data audio. The embodiment of the present application does not limit the number and source of the training data audio in the first data set.

[0166] s12: using the first data set to optimize the initial continuous speech marking module and the initial speech decoding module to obtain a target continuous speech marking module and a target speech decoding module.

[0167] After the first data set is obtained in the model training phase, the initial continuous speech mark module and the initial speech decoding module can be synchronously trained on the first data set to obtain the trained target continuous speech mark module and the target speech decoding module. The connection structure diagram between the initial continuous speech mark module and the initial speech decoding module used in the model training phase can be found in Fig.12 ;like Fig.12 As shown, the output of the initial continuous speech tag module is used as the input of the initial speech decoding module. The training goal of the synchronous training of the initial continuous speech tag module and the initial speech decoding module is to reduce the loss between the predicted text information converted by the initial speech decoding module and the text label corresponding to the training data audio.

[0168] In a specific implementation, a training data audio and a corresponding text label are obtained from the first data set; the training data audio is processed by continuous speech labeling using an initial continuous speech labeling module, specifically according to Fig.12 The resampling module in the initial continuous speech marking module shown resamples the training data audio, and then uses the convolution layer to perform multi-level feature extraction processing on the resampled training data audio to obtain a training multi-dimensional speech embedding vector. Then, the training multi-dimensional speech embedding vector is input into the initial speech decoding module, and the initial speech decoding module performs feature recovery through a deep encoder and a multi-layer convolution layer to achieve conversion processing for the training multi-dimensional speech embedding vector, and obtains the predicted text reading speech corresponding to the training data audio. Then, the predicted text reading speech is converted into predicted text information in text form; in this way, the initial continuous speech marking module and the initial speech decoding module can be optimized in the direction of reducing the difference between the predicted text information and the text label corresponding to the training data audio, specifically, the difference value between the predicted text information and the text label corresponding to the training data audio is calculated, and the parameters of the initial continuous speech marking module and the initial speech decoding module are adjusted in the direction of reducing the difference value to obtain the target continuous speech marking module and the target speech decoding module.

[0169] Exemplarily, given that the training data audio is A and the text label corresponding to the training data audio is Y, the calculation formula for optimizing the initial continuous speech labeling module and the initial speech decoding module using the training data audio A and the text label Y is as follows:

[0170]

[0171]

[0172]

[0173] Among them, ContTokenizer(A) indicates that the initial continuous speech tagging module is used to perform continuous speech tagging on the training data audio A. AudioDecoder(ContTokenizer(A)) indicates that the initial speech decoding module is used to perform conversion processing / decoding processing on the training multi-dimensional speech embedding vector of the training data audio A. Speaks speech for the decoded predictive text.

[0174] Indicates that the predictive text is spoken using the voice translator Convert to text form, The predicted text information is in the converted text form.

[0175] It means that the difference value (or difference value) between the predicted text information and the text label Y corresponding to the training data audio is calculated using the CTC loss function (which can effectively cope with the loss calculation in the case of various speech speeds in the training data audio). Indicates that the parameters of the initial continuous speech labeling module and the initial speech decoding module are adjusted synchronously in the direction of reducing the difference value to obtain the trained target continuous speech labeling module and the target speech decoding module. ContTokenizer is the loss value of the initial continuous speech tagging module; θ AudioDecoder is the loss value of the initial speech decoding module.

[0176] s13: Obtain a second data set. The second data set includes a training text embedding vector corresponding to the training text, a training multi-dimensional speech embedding vector corresponding to the training speech, and an audio tag corresponding to the training speech.

[0177] When the trained target continuous speech labeling module is obtained based on the aforementioned step s12, the target continuous speech labeling module can be used to perform continuous speech labeling processing on the training speech to obtain a training multi-dimensional speech embedding vector corresponding to the training speech; and a target text labeling module with fixed parameters can be used to perform feature extraction processing on the training text to obtain a training text embedding vector corresponding to the training text. In this way, the training multi-dimensional speech embedding vector corresponding to the training speech, the training text embedding vector corresponding to the training text, and the audio label corresponding to the training speech can form a training data pair in the second data set; wherein the audio label corresponding to the training speech can be a label obtained by manually or by using an algorithm to label the training speech, such as labeling the acoustic features of the training speech.

[0178] s14: using the second data set to optimize the initial speech prediction module to obtain a target speech prediction module.

[0179] After constructing the second data set, the second data set can be used to perform model training on the initial speech prediction module. The model training method follows the training method of the autoregressive model, that is, given an input sequence (that is, a training multidimensional speech embedding vector corresponding to the training speech and a training text embedding vector corresponding to the training text), the training goal is to reduce the distance or similarity between the prediction tag output by the initial speech prediction module and the next prediction tag in the second data set, so as to enable the trained target speech prediction model to learn the correlation between the previous and next data in the input sequence, such as the correlation between the previous training text embedding vector and the subsequent training multidimensional speech embedding vector.

[0180] In a specific implementation, a training text embedding vector corresponding to a training text and a training multi-dimensional speech embedding vector corresponding to a training speech matching the training text are obtained from the second data set; the so-called matching of the training text and the training speech means that the text content included in the training audio is the training text. Then, the initial speech prediction module is used to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector to obtain a training predicted speech sequence; the training predicted speech sequence includes the training predicted speech information referenced at each time step when the training audio is played. In addition, the audio label corresponding to the training audio is encoded as a predicted speech sequence label, and the predicted speech sequence label includes the real speech information at each time step after the training speech is encoded. Further, the initial speech prediction module is optimized in the direction of reducing the difference between the training predicted speech sequence and the predicted speech sequence label to obtain the target speech prediction module.

[0181] It is worth noting that the "training predicted speech sequence" in the above description includes the "training predicted speech information" of each time step predicted by the initial speech prediction module, for example, the training predicted speech information of time step 0, the training predicted speech information of time step 1, and the training predicted speech information of time step 2, etc. As can be seen from the relevant description of the calculation logic of the training prediction voice information in the above formula (4), in the process of generating the training prediction voice information when the time step t≥1, the input is the training text embedding vector corresponding to the training text, the training multidimensional voice embedding vector corresponding to the training voice, and the voice information before the time step t; in order to increase the accuracy and stability of the model prediction, the embodiment of the present application uses a teacher-driven training method when constructing the input data input to the initial voice prediction module at each time step, and uses the real voice information of each time step in the predicted voice sequence label obtained by conversion based on the audio label to construct the input data. The construction process of the input data specifically includes: vector connection of the training voice embedding vector corresponding to the training voice, the training text embedding vector corresponding to the training text, and the real voice information of each time step before the time step t in the predicted voice sequence label corresponding to the audio label (that is, the voice information before the time step t mentioned above), to obtain the training connection vector (that is, the input data input to the initial voice prediction module). In this way, the initial voice prediction module is used to regress the training connection vector to obtain the training prediction voice information of time step t. It can be seen that constructing the training concatenated vector through the real speech information of the time step can reduce the error caused by poor model performance and improve the prediction accuracy of the initial speech prediction module, compared with constructing the training concatenated vector through the training predicted speech information output by the initial speech prediction module, thereby training a target speech prediction module with better performance.

[0182] Further, as described above, the embodiment of the present application supports the use of the real speech information of the time step to construct the training post-connection vector, that is, the training speech embedding vector corresponding to the training speech, the training text embedding vector corresponding to the training text, and the real speech information of each time step before time step t in the predicted speech sequence label corresponding to the audio label to construct the training post-connection vector. This allows the embodiment of the present application to pre-process the training post-connection vector for each time step, so that when training the model, it is only necessary to call the pre-processed training post-connection vector for each time step; this further improves the training speed and efficiency of the initial speech prediction module to a certain extent. Under this implementation, the constructed second data set may not include the training text embedding vector corresponding to the training text, the training multi-dimensional speech embedding vector corresponding to the training speech, and the audio label corresponding to the training speech, but directly includes the training post-connection vector for each time step and the audio label corresponding to the training speech.

[0183] For example, given the training prediction speech sequence output by the initial speech prediction module The audio tag M corresponding to the training speech; then, the audio tag M is encoded as a continuous speech tag (referred to as a predicted speech sequence tag in the embodiment of the present application), and the loss function is calculated by the distance between the predicted speech sequence tag and the output of the initial speech prediction module. The calculation formula is as follows:

[0184]

[0185]

[0186] Among them, ContTokenizer Audio (M) indicates that the audio tag M is encoded as a continuous speech tag using the target continuous speech tag module Indicates that the continuous speech mark is calculated using the MSE loss function And the training prediction speech sequence output by the initial speech prediction module The difference between. Indicates that the continuous speech mark is reduced And the training prediction speech sequence output by the initial speech prediction module The parameters of the initial speech prediction module are adjusted according to the direction of the difference between the two, so as to optimize the initial speech prediction module.

[0187] In summary, through the step-by-step training strategy shown in the above steps s11-s14, on the one hand, the initial continuous speech marking module and the initial speech decoding module can be trained synchronously in a targeted manner, effectively improving the continuous speech marking performance of the trained target continuous speech marking module for the reference speech, and can also improve the conversion performance of the trained target speech decoding module for the predicted speech sequence. On the other hand, the preferentially trained target continuous speech marking module can pre-process the training speech used for model training, that is, the target continuous speech marking module is used to pre-process the training speech for continuous speech marking into a training multi-dimensional speech embedding vector; similarly, the target text marking module is used to pre-process the training text used for model training for feature extraction to obtain the training text embedding vector; in this way, when the model training is performed for the initial speech prediction model, the pre-processed training multi-dimensional speech embedding vector and the training text embedding vector can be directly called, which improves the model training efficiency to a certain extent.

[0188] The method of the embodiment of the present application is described in detail above. In order to facilitate the above-mentioned scheme of the embodiment of the present application to be better implemented, the device of the embodiment of the present application is provided below accordingly. In the embodiment of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the module or unit function.

[0189] Fig.13 A schematic diagram of the structure of a speech processing device provided by an exemplary embodiment of the present application is shown; the speech processing device can be used to perform Figure 3 and Fig. 9 Some or all of the steps in the method embodiment shown. Fig.13 , the device comprises the following units:

[0190] An acquisition unit 1301 is used to acquire the text to be converted and the reference speech;

[0191] The processing unit 1302 is used to perform feature extraction processing on the text to obtain a text embedding vector, where the text embedding vector is used to represent the semantics of the text;

[0192] The processing unit 1302 is further used to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector; the multi-dimensional speech embedding vector is used to characterize the acoustic features of the reference speech;

[0193] The processing unit 1302 is further used to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence; the predicted speech sequence includes a plurality of predicted speech information referenced when the text reading speech corresponding to the text is read aloud;

[0194] The processing unit 1302 is further configured to convert the predicted speech sequence to generate a text-reading speech, wherein the acoustic features of the text-reading speech match the acoustic features of the reference speech.

[0195] In one implementation, the processing unit 1302 is used to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector, specifically for:

[0196] Segmenting the reference speech to obtain at least one speech segment;

[0197] Performing continuous speech tag processing on each speech segment to obtain a continuous speech tag of each speech segment; the continuous speech tag of the speech segment is expressed as a multidimensional vector, the continuous speech tag includes a plurality of continuous values, and the plurality of continuous values ​​can continuously represent the acoustic features of the speech segment according to the sequence of time steps of the speech segment; the acoustic features include at least one of the following: pitch, intensity, timbre and duration;

[0198] According to the playback position of at least one speech segment in the reference speech, the continuous speech tags of at least one speech segment are combined to obtain a multidimensional speech embedding vector; the vector dimension of the multidimensional speech embedding vector is the same as the sum of the number of continuous values ​​included in the continuous speech tags of at least one speech segment.

[0199] In one implementation, the processing unit 1302 is configured to perform continuous speech mark processing on each speech segment to obtain the continuous speech mark of each speech segment, specifically for:

[0200] Resampling each speech segment to obtain each resampled speech segment;

[0201] A multi-level feature extraction process is performed on each resampled speech segment to obtain a continuous speech tag of each resampled speech segment.

[0202] In one implementation, the text reading speech is read aloud in the order of time steps, where the time step is represented by t, and t is a non-negative integer; the processing unit 1302 is used to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector, and when the predicted speech sequence is obtained, it is specifically used to:

[0203] When time step t=0, the multi-dimensional speech embedding vector and the text embedding vector are processed for speech prediction to obtain the predicted speech information of time step 0;

[0204] When time step t>0, the multidimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before time step t are processed for speech prediction to obtain the predicted speech information of time step t;

[0205] Until the time step t reaches the predicted time length required for the text reading speech to be played;

[0206] Among them, the predicted speech information at each time step t constitutes a predicted speech sequence.

[0207] In one implementation, the processing unit 1302 is used to perform speech prediction processing on the multidimensional speech embedding vector, the text embedding vector, and the predicted speech information at each time step before the time step t to obtain the predicted speech information at the time step t, specifically for:

[0208] Perform vector concatenation on the multidimensional speech embedding vector, the text embedding vector and the predicted speech information at each time step before time step t to obtain a concatenated vector;

[0209] The concatenated vector is regressed to generate the predicted speech information at time step t.

[0210] In one implementation, the processing unit 1302 is used to convert the predicted speech sequence to generate a text reading speech corresponding to the text, specifically for:

[0211] Perform convolution processing on each predicted speech information in the predicted speech sequence to obtain the key speech features corresponding to each predicted speech information;

[0212] Generate predicted speech corresponding to each predicted speech information based on the key speech features corresponding to each predicted speech information;

[0213] Connect each predicted speech in the order of time steps to generate the text reading speech corresponding to the text.

[0214] In one implementation, the speech processing method is implemented by a text-to-speech model, which includes a target text marking module, a target continuous speech marking module, a target speech prediction module, and a target speech decoding module;

[0215] The target text tagging module is used to perform feature extraction on the text to obtain a text embedding vector;

[0216] The target continuous speech tagging module is used to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector;

[0217] The target speech prediction module is used to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence;

[0218] The target speech decoding module is used to convert the predicted speech sequence and generate the text reading speech corresponding to the text.

[0219] In one implementation, the parameters of the target text tagging module are preset; the training process of the text-to-speech model includes:

[0220] Obtain a first data set, the first data set including training data audio and corresponding text labels; the text labels are obtained by converting the corresponding training data audio into text;

[0221] Using the first data set, the initial continuous speech marking module and the initial speech decoding module are optimized to obtain a target continuous speech marking module and a target speech decoding module;

[0222] Obtain a second data set, the second data set including a training text embedding vector corresponding to the training text, a training multidimensional speech embedding vector corresponding to the training speech, and an audio tag corresponding to the training speech; the training multidimensional speech embedding vector is obtained by performing continuous speech tagging processing using a target continuous speech tagging module;

[0223] The initial speech prediction module is optimized using the second data set to obtain a target speech prediction module.

[0224] In one implementation, the processing unit 1302 is configured to optimize the initial continuous speech marking module and the initial speech decoding module using the first data set to obtain the target continuous speech marking module and the target speech decoding module, specifically to:

[0225] Use the initial continuous speech tagging module to perform continuous speech tagging on the training data audio to obtain a training multi-dimensional speech embedding vector;

[0226] Use the initial speech decoding module to convert the training multi-dimensional speech embedding vector to generate the predicted text reading speech corresponding to the training data audio;

[0227] Converting predictive text reading speech into predictive text information in text form;

[0228] In the direction of reducing the difference between the predicted text information and the text label corresponding to the training data audio, the initial continuous speech labeling module and the initial speech decoding module are optimized to obtain the target continuous speech labeling module and the target speech decoding module.

[0229] In one implementation, the processing unit 1302 is configured to optimize the initial speech prediction module using the second data set to obtain the target speech prediction module, specifically to:

[0230] Using the initial speech prediction module to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector, to obtain a training predicted speech sequence;

[0231] Encode the audio label into a predicted speech sequence label; the predicted speech sequence label includes the real speech information of each time step after the training speech is encoded;

[0232] The initial speech prediction module is optimized in the direction of reducing the difference between the training predicted speech sequence and the predicted speech sequence label to obtain the target speech prediction module.

[0233] In one implementation, the training predicted speech sequence includes training predicted speech information at each time step; the time step is represented by t, and when t is an integer greater than zero, the processing unit 1302 is used to use the initial speech prediction module to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector to obtain the training predicted speech sequence, specifically for:

[0234] Concatenate the training speech embedding vector, the training text embedding vector, and the real speech information of each time step before time step t in the predicted speech sequence label to obtain a training concatenated vector;

[0235] The initial speech prediction module is used to regress the training concatenated vector to obtain the training predicted speech information at time step t.

[0236] According to one embodiment of the present application, Fig.13 The various units in the speech processing device shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the speech processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, it can be executed by running on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 3 and Fig. 9 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Fig.13 The speech processing device shown in the embodiment of the present application is implemented by the speech processing method. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0237] In an embodiment of the present application, after obtaining the text to be converted and the reference speech, feature extraction processing is performed on the text to obtain a text embedding vector for characterizing the semantics of the text; at the same time, continuous speech marking is performed on the reference speech to obtain a multidimensional speech embedding vector; since a continuous speech marking method is used for the reference speech, the continuous speech marking method can continuously mark the speech information of the reference speech in the form of a multidimensional vector in the order of speech playback, thereby retaining the speech information (or acoustic features) of the reference speech to the greatest extent. In this way, based on the multidimensional speech embedding vector of the reference speech obtained by continuous marking and the text embedding vector of the text, speech prediction processing is performed on the text to obtain a predicted speech sequence, which includes multiple predicted speech information referenced when the text reading speech corresponding to the text is read aloud; considering that a large number of acoustic features of the reference speech are retained through multiple continuous values ​​in the multidimensional speech embedding vector, a large number of acoustic features of the reference speech are also retained in the predicted speech sequence obtained based on the multidimensional speech embedding vector. In this way, by converting the predicted speech sequence, the text reading speech corresponding to the text can be generated, and it can be ensured that the acoustic features of the text reading speech match the acoustic features of the reference speech. In layman's terms, the speech style of the text reading speech matches the speech style of the reference speech. For example, from the auditory effect, the text reading speech and the reference speech sound like speech signals emitted by the same object (such as a person or an animal). Through the above-mentioned description, the embodiments of the present application support continuous speech characterization of the reference speech, specifically, characterizing the continuous acoustic features of the reference speech according to the speech playback order during the speech feature extraction process of the reference speech, and representing the continuous acoustic features in the form of a multi-dimensional speech embedding vector, thereby achieving the maximum retention of the speech information of the reference speech; compared with discrete speech tags that only discretize the acoustic features of the reference speech into a single numerical value to represent it, the speech information loss of the reference speech in the text-to-speech technology can be reduced. In this way, text-to-speech based on a multi-dimensional speech embedding vector with rich acoustic features can ensure that the acoustic features of the generated text reading speech match the acoustic features of the reference speech (such as the style and timbre of the text reading speech are close to the reference speech), thereby achieving better conversion effects in text-to-speech technology.

[0238] Fig.14FIG. 1 shows a schematic diagram of a computer device provided by an exemplary embodiment of the present application. Fig.14 , the computer device includes a processor 1401, a communication interface 1402 and a computer-readable storage medium 1403. The processor 1401, the communication interface 1402 and the computer-readable storage medium 1403 may be connected via a bus or other means. The communication interface 1402 is used to receive and send data. The computer-readable storage medium 1403 may be stored in the memory of the computer device, the computer-readable storage medium 1403 is used to store computer programs, and the processor 1401 is used to execute the computer programs stored in the computer-readable storage medium 1403. The processor 1401 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more computer programs, and is specifically suitable for loading and executing one or more computer programs to implement corresponding method flows or corresponding functions.

[0239] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, one or more computer programs suitable for being loaded and executed by the processor 1401 are also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0240] In one embodiment, one or more computer programs are stored in a computer-readable storage medium; the processor 1401 loads and executes the one or more computer programs stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned speech processing method embodiment; in a specific implementation, the one or more computer programs in the computer-readable storage medium are loaded by the processor 1401 and execute the steps of each embodiment of the present application:

[0241] Obtain the text to be converted and the reference speech;

[0242] Perform feature extraction on the text to obtain a text embedding vector, which is used to represent the semantics of the text;

[0243] Performing continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector; the multi-dimensional speech embedding vector is used to characterize the acoustic features of the reference speech;

[0244] Based on the multi-dimensional speech embedding vector and the text embedding vector, the text is subjected to speech prediction processing to obtain a predicted speech sequence; the predicted speech sequence includes a plurality of predicted speech information referenced when the text reading speech corresponding to the text is read aloud;

[0245] The predicted speech sequence is converted to generate text-reading speech, and the acoustic features of the text-reading speech match the acoustic features of the reference speech.

[0246] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and when performing continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector, the following steps are specifically performed:

[0247] Segmenting the reference speech to obtain at least one speech segment;

[0248] Performing continuous speech tag processing on each speech segment to obtain a continuous speech tag of each speech segment; the continuous speech tag of the speech segment is expressed as a multidimensional vector, the continuous speech tag includes a plurality of continuous values, and the plurality of continuous values ​​can continuously represent the acoustic features of the speech segment according to the sequence of time steps of the speech segment; the acoustic features include at least one of the following: pitch, intensity, timbre and duration;

[0249] According to the playback position of at least one speech segment in the reference speech, the continuous speech tags of at least one speech segment are combined to obtain a multidimensional speech embedding vector; the vector dimension of the multidimensional speech embedding vector is the same as the sum of the number of continuous values ​​included in the continuous speech tags of at least one speech segment.

[0250] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and when performing continuous speech tagging processing on each speech segment to obtain a continuous speech tag for each speech segment, the following steps are specifically performed:

[0251] Resampling each speech segment to obtain each resampled speech segment;

[0252] A multi-level feature extraction process is performed on each resampled speech segment to obtain a continuous speech tag of each resampled speech segment.

[0253] In one implementation, the text reading speech is read in sequence according to the time step sequence, where the time step is represented by t, and t is a non-negative integer; one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and are executed based on the multi-dimensional speech embedding vector and the text embedding vector to perform speech prediction processing on the text to obtain a predicted speech sequence, and the following steps are specifically performed:

[0254] When time step t=0, the multi-dimensional speech embedding vector and the text embedding vector are processed for speech prediction to obtain the predicted speech information of time step 0;

[0255] When time step t>0, the multidimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before time step t are processed for speech prediction to obtain the predicted speech information of time step t;

[0256] Until the time step t reaches the predicted time length required for the text reading speech to be played;

[0257] Among them, the predicted speech information at each time step t constitutes a predicted speech sequence.

[0258] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and when executing the multi-dimensional speech embedding vector, the text embedding vector and the predicted speech information at each time step before the time step t, the following steps are specifically performed:

[0259] Perform vector concatenation on the multidimensional speech embedding vector, the text embedding vector and the predicted speech information at each time step before time step t to obtain a concatenated vector;

[0260] The concatenated vector is regressed to generate the predicted speech information at time step t.

[0261] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and when the predicted speech sequence is converted to generate a text reading speech corresponding to the text, the following steps are specifically performed:

[0262] Perform convolution processing on each predicted speech information in the predicted speech sequence to obtain the key speech features corresponding to each predicted speech information;

[0263] Generate predicted speech corresponding to each predicted speech information based on the key speech features corresponding to each predicted speech information;

[0264] Connect each predicted speech in the order of time steps to generate the text reading speech corresponding to the text.

[0265] In one implementation, the speech processing method is implemented by a text-to-speech model, which includes a target text marking module, a target continuous speech marking module, a target speech prediction module, and a target speech decoding module;

[0266] The target text tagging module is used to perform feature extraction on the text to obtain a text embedding vector;

[0267] The target continuous speech tagging module is used to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector;

[0268] The target speech prediction module is used to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence;

[0269] The target speech decoding module is used to convert the predicted speech sequence and generate the text reading speech corresponding to the text.

[0270] In one implementation, the parameters of the target text tagging module are preset; the training process of the text-to-speech model includes:

[0271] Obtain a first data set, the first data set including training data audio and corresponding text labels; the text labels are obtained by converting the corresponding training data audio into text;

[0272] Using the first data set, the initial continuous speech marking module and the initial speech decoding module are optimized to obtain a target continuous speech marking module and a target speech decoding module;

[0273] Obtain a second data set, the second data set including a training text embedding vector corresponding to the training text, a training multidimensional speech embedding vector corresponding to the training speech, and an audio tag corresponding to the training speech; the training multidimensional speech embedding vector is obtained by performing continuous speech tagging processing using a target continuous speech tagging module;

[0274] The initial speech prediction module is optimized using the second data set to obtain a target speech prediction module.

[0275] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and when the processor 1401 optimizes the initial continuous speech mark module and the initial speech decoding module using the first data set to obtain the target continuous speech mark module and the target speech decoding module, the following steps are specifically performed:

[0276] Use the initial continuous speech tagging module to perform continuous speech tagging on the training data audio to obtain a training multi-dimensional speech embedding vector;

[0277] Use the initial speech decoding module to convert the training multi-dimensional speech embedding vector to generate the predicted text reading speech corresponding to the training data audio;

[0278] Converting predictive text reading speech into predictive text information in text form;

[0279] In the direction of reducing the difference between the predicted text information and the text label corresponding to the training data audio, the initial continuous speech labeling module and the initial speech decoding module are optimized to obtain the target continuous speech labeling module and the target speech decoding module.

[0280] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and when the processor 1401 optimizes the initial speech prediction module using the second data set to obtain the target speech prediction module, the following steps are specifically performed:

[0281] Using the initial speech prediction module to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector, to obtain a training predicted speech sequence;

[0282] Encode the audio label into a predicted speech sequence label; the predicted speech sequence label includes the real speech information of each time step after the training speech is encoded;

[0283] The initial speech prediction module is optimized in the direction of reducing the difference between the training predicted speech sequence and the predicted speech sequence label to obtain the target speech prediction module.

[0284] In one implementation, the training predicted speech sequence includes training predicted speech information at each time step; the time step is represented by t, and when t is an integer greater than zero, one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and executed using the initial speech prediction module to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector to obtain the training predicted speech sequence, and specifically perform the following steps:

[0285] Concatenate the training speech embedding vector, the training text embedding vector, and the real speech information of each time step before time step t in the predicted speech sequence label to obtain a training concatenated vector;

[0286] The initial speech prediction module is used to regress the training concatenated vector to obtain the training predicted speech information at time step t.

[0287] Based on the same inventive concept, the principles and beneficial effects of solving the problems provided by the computer device in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems provided by the speech processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0288] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned speech processing method is implemented.

[0289] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0290] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes a computer program (one or more). When a computer program is loaded and executed on a computer device, the computer program executes the above-mentioned process or function of the embodiment of the present application. The computer device can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer program can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer program can be transmitted from a website site, a computer device, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer device, server or data center. The computer-readable storage medium can be any available medium that a computer device can access or a data storage device such as a server or a data center that contains one or more available media integrations. Available media can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (Solid State Disk, SSD)) and the like.

[0291] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speech processing method, characterized in that: include: Obtain the text to be converted and the reference speech; Performing feature extraction processing on the text to obtain a text embedding vector, wherein the text embedding vector is used to represent the semantics of the text; Performing continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector; The multidimensional speech embedding vector is used to characterize the acoustic features of the reference speech; Based on the multi-dimensional speech embedding vector and the text embedding vector, the text is subjected to speech prediction processing to obtain a predicted speech sequence; the predicted speech sequence includes a plurality of predicted speech information referenced when the text reading speech corresponding to the text is read aloud; The predicted speech sequence is converted to generate the text-reading speech, and the acoustic features of the text-reading speech match the acoustic features of the reference speech.

2. The method according to claim 1, characterized in that The step of performing continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector comprises: Segmenting the reference speech to obtain at least one speech segment; Performing continuous speech tag processing on each of the speech segments to obtain a continuous speech tag of each of the speech segments; the continuous speech tag of the speech segment is expressed as a multidimensional vector, the continuous speech tag includes a plurality of continuous values, and the plurality of continuous values ​​can continuously represent the acoustic features of the speech segment according to the sequence of the time steps of the speech segment; the acoustic features include at least one of the following: pitch, intensity, timbre and duration; According to the playback position of at least one of the speech segments in the reference speech, the continuous speech tags of at least one of the speech segments are combined to obtain a multidimensional speech embedding vector; the vector dimension of the multidimensional speech embedding vector is the same as the sum of the number of continuous values ​​included in the continuous speech tags of at least one of the speech segments.

3. The method according to claim 2, characterized in that The step of performing continuous speech mark processing on each of the speech segments to obtain a continuous speech mark of each of the speech segments includes: Resampling each of the speech segments to obtain each of the speech segments after resampling; A multi-level feature extraction process is performed on each of the resampled speech segments to obtain a continuous speech tag of each of the resampled speech segments.

4. The method according to claim 1, characterized in that The text reading speech is read aloud in sequence according to the order of time steps, where the time step is represented by t, and t is a non-negative integer; the text is subjected to speech prediction processing based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence, including: When the time step t=0, the multidimensional speech embedding vector and the text embedding vector are subjected to speech prediction processing to obtain predicted speech information of the time step 0; When the time step t>0, the multidimensional speech embedding vector, the text embedding vector and the predicted speech information of each time step before the time step t are subjected to speech prediction processing to obtain the predicted speech information of the time step t; Until the time step t reaches the predicted time length required for the text reading voice to be played; The predicted speech information at each time step t constitutes a predicted speech sequence.

5. The method according to claim 4, characterized in that The step of performing speech prediction processing on the multidimensional speech embedding vector, the text embedding vector and the predicted speech information at each time step before the time step t to obtain the predicted speech information at the time step t includes: Performing vector concatenation on the multidimensional speech embedding vector, the text embedding vector and the predicted speech information at each time step before the time step t to obtain a concatenated vector; Regression processing is performed on the concatenated vector to generate predicted speech information at the time step t.

6. The method according to claim 1, characterized in that The step of converting the predicted speech sequence to generate a text reading speech corresponding to the text includes: Performing convolution processing on each of the predicted speech information in the predicted speech sequence to obtain key speech features corresponding to each of the predicted speech information; Based on the key speech features corresponding to each of the predicted speech information, generating predicted speech corresponding to each of the predicted speech information; Each of the predicted speech is connected in the order of time steps to generate the text reading speech corresponding to the text.

7. The method according to any one of claims 1 to 6, characterized in that: The method is implemented by a text-to-speech model, which includes a target text marking module, a target continuous speech marking module, a target speech prediction module and a target speech decoding module; The target text marking module is used to perform feature extraction processing on the text to obtain the text embedding vector; The target continuous speech labeling module is used to perform continuous speech labeling processing on the reference speech to obtain a multi-dimensional speech embedding vector; The target speech prediction module is used to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence; The target speech decoding module is used to convert the predicted speech sequence to generate a text reading speech corresponding to the text.

8. The method according to claim 7, characterized in that The parameters of the target text marking module are preset; the training process of the text-to-speech model includes: Acquire a first data set, wherein the first data set includes training data audio and corresponding text labels; the text labels are obtained by converting the corresponding training data audio into text; Using the first data set to optimize the initial continuous speech marking module and the initial speech decoding module, to obtain the target continuous speech marking module and the target speech decoding module; Acquire a second data set, wherein the second data set includes a training text embedding vector corresponding to the training text, a training multidimensional speech embedding vector corresponding to the training speech, and an audio tag corresponding to the training speech; the training multidimensional speech embedding vector is obtained by performing continuous speech labeling processing using the target continuous speech labeling module; The initial speech prediction module is optimized using the second data set to obtain the target speech prediction module.

9. The method according to claim 8, characterized in that The step of optimizing the initial continuous speech marking module and the initial speech decoding module using the first data set to obtain the target continuous speech marking module and the target speech decoding module includes: Perform continuous speech tagging processing on the training data audio using an initial continuous speech tagging module to obtain a training multi-dimensional speech embedding vector; Using an initial speech decoding module to convert the training multi-dimensional speech embedding vector to generate a predicted text reading speech corresponding to the training data audio; Converting the predicted text reading speech into predicted text information in text form; The initial continuous speech labeling module and the initial speech decoding module are optimized in the direction of reducing the difference between the predicted text information and the text label corresponding to the training data audio to obtain the target continuous speech labeling module and the target speech decoding module.

10. The method according to claim 8, characterized in that The step of optimizing the initial speech prediction module using the second data set to obtain the target speech prediction module includes: Using an initial speech prediction module, based on the training text embedding vector and the training multi-dimensional speech embedding vector, the training text is subjected to speech prediction processing to obtain a training predicted speech sequence; Encoding the audio tag into a predicted speech sequence tag; the predicted speech sequence tag includes the real speech information of each time step after the training speech is encoded; The initial speech prediction module is optimized in the direction of reducing the difference between the training predicted speech sequence and the predicted speech sequence label to obtain the target speech prediction module.

11. The method according to claim 10, characterized in that The training predicted speech sequence includes training predicted speech information of each time step; the time step is represented by t, and when t is an integer greater than zero, the initial speech prediction module is used to perform speech prediction processing on the training text based on the training text embedding vector and the training multi-dimensional speech embedding vector to obtain the training predicted speech sequence, including: Performing vector connection on the training speech embedding vector, the training text embedding vector, and the real speech information of each time step before time step t in the predicted speech sequence label to obtain a training connection vector; The training concatenated vector is regressed using an initial speech prediction module to obtain training predicted speech information at time step t.

12. A speech processing device, characterized in that: include: An acquisition unit, used for acquiring the text to be converted and the reference speech; A processing unit, used for performing feature extraction processing on the text to obtain a text embedding vector, wherein the text embedding vector is used to represent the semantics of the text; The processing unit is further used to perform continuous speech tagging processing on the reference speech to obtain a multi-dimensional speech embedding vector; The multidimensional speech embedding vector is used to characterize the acoustic features of the reference speech; The processing unit is further used to perform speech prediction processing on the text based on the multi-dimensional speech embedding vector and the text embedding vector to obtain a predicted speech sequence; the predicted speech sequence includes a plurality of predicted speech information referenced when the text reading speech corresponding to the text is read aloud; The processing unit is further used to perform conversion processing on the predicted speech sequence to generate the text reading speech, and the acoustic features of the text reading speech match the acoustic features of the reference speech.

13. A computer device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the speech processing method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the speech processing method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the speech processing method according to any one of claims 1 to 11 is implemented.

Images