A data processing method, device, apparatus, and computer-readable storage medium

Abstract

This application discloses a data processing method, apparatus, device, and computer-readable storage medium. The method includes: inputting a first training sample set into a speech recognition initial model; the first training sample set includes first sample speech carrying International Phonetic Alphabet (IPA) sequence tags; in an initial speech tagger, performing speech tag prediction processing on the first sample speech to obtain a speech tag estimation sequence; in an initial tagger decoder, performing tag decoding processing on the speech tag estimation sequence to obtain an IPA estimation sequence; adjusting the model parameters in the initial speech tagger based on the IPA estimation sequence and the IPA sequence tags to obtain a speech tagger; the speech tagger is used to generate a first speech tag sequence for a second sample speech, and the first speech tag sequence is used as training tags for the initial speech synthesis model. Using this application can improve the accuracy and applicability of multilingual and cross-lingual speech recognition.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the popularization of artificial intelligence technology and the development of computer internet, intelligent applications are ubiquitous. Among them, Automatic Speech Recognition (ASR) is one of the hottest areas of intelligent applications today.

[0003] Mainstream automatic speech recognition primarily relies on machine learning methods. For example, an initial speech recognition model is trained using Chinese speech samples to obtain Speech Recognition Model 1 for recognizing Chinese speech, and an initial model is trained using English speech samples to obtain Speech Recognition Model 2 for recognizing English speech. Understandably, Speech Recognition Model 1 can accurately recognize Chinese speech but is not suitable for recognizing English or other languages; similarly, Speech Recognition Model 2 can accurately recognize English speech but is not suitable for recognizing Chinese or other languages. Therefore, existing technologies suffer from both low applicability and low accuracy in speech recognition. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and computer-readable storage medium that can improve the accuracy and applicability of multilingual and cross-lingual speech recognition.

[0005] One embodiment of this application provides a data processing method, including:

[0006] Obtain the first training sample set and input it into the initial speech recognition model; the first training sample set includes first sample speech carrying International Phonetic Alphabet (IPA) sequence labels; the IPA in the IPA sequence labels is used to describe the pronunciation of multiple languages ​​using unified phonetic symbols;

[0007] In the initial speech tagger of the initial speech recognition model, the first sample speech is processed for speech tag prediction to obtain the speech tag estimation sequence;

[0008] In the initial tag decoder of the initial speech recognition model, the speech tag estimation sequence is processed by tag decoding to obtain the International Phonetic Alphabet (IPA) estimation sequence;

[0009] Based on the IPA estimation sequence and IPA sequence labels, the model parameters in the initial speech tagger are adjusted to obtain the speech tagger. The speech tagger is used to generate the first speech tag sequence of the second sample speech when training the initial speech synthesis model. The first speech tag sequence is used as the training label of the initial speech synthesis model.

[0010] One embodiment of this application provides a data processing apparatus, including:

[0011] The acquisition module is used to acquire a first training sample set and input the first training sample set into the speech recognition initial model; the first training sample set includes first sample speech carrying International Phonetic Alphabet (IPA) sequence labels; the IPA in the IPA sequence labels is used to describe the pronunciation of multiple languages ​​through unified phonetic symbols;

[0012] The processing module is used to perform speech tag prediction processing on the first sample speech in the initial speech tagger of the initial speech recognition model to obtain the speech tag estimation sequence.

[0013] The processing module is also used to perform tag decoding processing on the speech tag estimation sequence in the initial tag decoder of the initial speech recognition model to obtain the International Phonetic Alphabet (IPA) estimation sequence;

[0014] The adjustment module is used to adjust the model parameters in the initial speech tagger according to the IPA estimation sequence and IPA sequence labels to obtain the speech tagger; the speech tagger is used to generate the first speech tag sequence of the second sample speech when training the initial speech synthesis model, and the first speech tag sequence is used as the training label of the initial speech synthesis model.

[0015] In one possible implementation, the initial speech tagger of the initial speech recognition model includes a convolutional encoder, an initial tag encoder, and an initial vector quantizer.

[0016] In the initial speech tagger of the initial speech recognition model, the processing module performs speech tag prediction processing on the first sample speech to obtain a speech tag estimation sequence, which is used to perform the following operations:

[0017] Acoustic features are extracted from the first sample speech using a convolutional encoder to obtain continuous semantic features, which are then input into the initial label encoder.

[0018] The continuous semantic features are fused with contextual features by an initial label encoder to obtain continuous semantic enhancement features, which are then input into an initial vector quantizer.

[0019] The continuous semantic enhancement features are processed by an initial vector quantizer to obtain discrete speech tags. The discrete speech tags are then combined into a speech tag estimation sequence according to the audio frame order of the first sample speech.

[0020] In one possible implementation, the processing module performs feature recognition processing on the continuous semantic enhancement features through an initial vector quantizer to obtain discrete speech tokens, which are then used to perform the following operations:

[0021] By using an initial vector quantizer, the continuous semantic enhancement features are compared with at least two vectors in the codebook to obtain the similarity values ​​corresponding to the at least two vectors respectively.

[0022] The maximum similarity value is obtained from at least two similarity values, and the speech tag corresponding to the vector with the maximum similarity value among at least two vectors is determined as the discrete speech tag of continuous semantic enhancement feature.

[0023] In one possible implementation, the initial tag decoder includes an initial optimization encoder and an initial decoder;

[0024] In the initial token decoder of the initial speech recognition model, the processing module performs token decoding on the speech token estimation sequence to obtain the International Phonetic Alphabet (IPA) estimation sequence, which is then used to perform the following operations:

[0025] The initial optimized encoder performs semantic enhancement processing on the speech tag estimation sequence to obtain continuous semantic optimization features, which are then input into the initial decoder.

[0026] The continuous semantic optimization features are decoded using the initial decoder to obtain the IPA estimation sequence.

[0027] In one possible implementation, the adjustment module adjusts the model parameters in the initial speech tokenizer based on the IPA estimation sequence and the IPA sequence labels to obtain a speech tokenizer, which is used to perform the following operations:

[0028] Based on the IPA estimation sequence and IPA sequence labels, determine the model loss value of the initial speech recognition model;

[0029] Based on the model loss value, the model parameters of the initial speech tokenizer and the initial token decoder in the initial speech recognition model are adjusted;

[0030] When the initial speech recognition model, which includes the adjusted initial speech tokenizer and the adjusted initial tokenizer decoder, satisfies the model convergence condition, the adjusted initial speech tokenizer is determined as the speech tokenizer.

[0031] In one possible implementation, the acquisition module is also used to perform the following operations:

[0032] Obtain the second training sample set; the second training sample set includes sample text and the corresponding second sample speech.

[0033] The second sample speech is input into the speech tagger, and the speech tagger performs speech tag prediction processing on the second sample speech to obtain the first speech tag sequence.

[0034] The sample text and the first speech tag sequence are input into the speech synthesis initial model. In the speech prediction initial network layer of the speech synthesis initial model, the sample text is processed for speech tag prediction based on the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text.

[0035] Based on the predicted speech tag sequence and the first speech tag sequence, the model parameters in the initial speech prediction network layer are adjusted to obtain the speech prediction network layer.

[0036] The initial speech synthesis model, which includes a speech prediction network layer, is defined as the speech synthesis model; the speech synthesis model is used to generate speech data corresponding to the input text.

[0037] In one possible implementation, the initial network layer for speech prediction includes an initial text encoder and a speech tagging initial network layer;

[0038] In the initial speech prediction network layer of the initial speech synthesis model, the acquisition module performs speech tag prediction processing on the sample text based on the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text, which is used to perform the following operations:

[0039] The sample text is encoded by the initial text encoder to obtain the text encoding estimation vector, and the text encoding estimation vector is input into the initial speech tag network layer.

[0040] In the initial speech tag network layer, the text encoding estimation vector is processed by speech tag prediction based on the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text.

[0041] In one possible implementation, the first speech tag sequence includes at least two sequentially arranged first speech tags; the predicted speech tag sequence includes at least two sequentially arranged second speech tags;

[0042] The acquisition module adjusts the model parameters in the initial speech prediction network layer based on the predicted speech tag sequence and the first speech tag sequence to obtain the speech prediction network layer, which is used to perform the following operations:

[0043] Loss processing is performed on the first speech tag sequence and the predicted speech tag sequence, where the first speech tag and the second speech tag have the same arrangement position, to obtain at least two speech tag loss values;

[0044] Summing at least two speech tag loss values ​​yields the network layer loss value of the initial speech prediction network layer.

[0045] Based on the network layer loss value, the model parameters in the initial network layer of speech prediction are adjusted to obtain the speech prediction network layer.

[0046] In one possible implementation, the acquisition module is also used to perform the following operations:

[0047] The input text is obtained and fed into the speech synthesis model. The speech synthesis model refers to the initial speech synthesis model that meets the convergence conditions. The speech synthesis model includes a speech prediction network layer and a stream matching network layer.

[0048] In the text encoder of the speech prediction network layer, the input text is encoded to obtain a text encoding vector, and the text encoding vector is input into the speech tag network layer of the speech prediction network layer.

[0049] In the speech tagging network layer, speech tag prediction processing is performed on the text encoding vector to obtain the second speech tag sequence;

[0050] The second speech tag sequence is input into the stream matching network layer. In the stream matching network layer, the first random noise is denoised according to the second speech tag sequence to obtain the Mel spectrogram.

[0051] The waveform reconstruction process is performed on the Mel spectrogram to obtain the synthesized speech corresponding to the input text.

[0052] In one possible implementation, the acquisition module is also used to perform the following operations:

[0053] The third sample speech is obtained and input into the timbre converter. The timbre of the third sample speech is modified by the timbre converter to obtain the timbre-modified speech.

[0054] The timbre-modified speech is input into the speech tagger, and the speech tagger performs speech tag prediction processing on the timbre-modified speech to obtain the third speech tag sequence.

[0055] Obtain the timbre vector corresponding to the third sample speech, and input the timbre vector and the third speech tag sequence into the initial network layer of the stream matching network;

[0056] In the initial network layer of the stream matching, the third sample speech with added second random noise is denoised based on the third speech tag sequence and timbre vector to obtain the estimated Mel spectrogram;

[0057] Based on the estimated Mel spectrogram and the third sample speech, the model parameters in the initial network layer of the stream matching network are adjusted to obtain the stream matching network layer.

[0058] In one possible implementation, the acquisition module adjusts the model parameters in the initial network layer of the stream matching network based on the estimated Mel spectrogram and the third sample speech, resulting in a stream matching network layer used to perform the following operations:

[0059] The third sample speech is converted into a sample Mel spectrogram, the sample Mel spectrogram is determined, and the Mel spectrum difference between the Mel spectrograms is estimated.

[0060] The network layer loss value of the initial flow matching network layer is determined based on the Mel spectrum difference. The model parameters in the initial flow matching network layer are then adjusted based on the network layer loss value to obtain the flow matching network layer.

[0061] This application provides a computer device, including: a processor, a memory, and a network interface;

[0062] The processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the methods in the embodiments of this application.

[0063] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the method described in this application.

[0064] One aspect of this application provides a computer program product, which includes a computer program stored in a computer-readable storage medium; a processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method described in this application.

[0065] In this embodiment, the computer device inputs a first training sample set into the speech recognition initial model. The first training sample set includes first sample speech carrying International Phonetic Alphabet (IPA) sequence labels. The IPA symbols in the IPA sequence labels are used to describe the pronunciation of multiple languages ​​using unified phonetic symbols. In the initial speech tagger of the speech recognition initial model, speech tag prediction processing is performed on the first sample speech to obtain a speech tag estimation sequence. In the initial tag decoder of the speech recognition initial model, tag decoding processing is performed on the speech tag estimation sequence to obtain an IPA estimation sequence. Based on the IPA estimation sequence and the IPA sequence labels, the model parameters in the initial speech tagger are adjusted to obtain a speech tagger. The speech tagger is used to generate a first speech tag sequence of the second sample speech when training the speech synthesis initial model. The first speech tag sequence is used as the training label for the speech synthesis initial model. As described above, the embodiments of this application propose using the International Phonetic Alphabet (IPA) as the training target for the speech tokenizer. This allows the speech tokenizer to fully utilize the consistency of pronunciation across multiple languages ​​(e.g., English, Chinese) to learn language-independent semantic information (i.e., speech tokens). By focusing on unified pronunciation (i.e., IPA), it is more suitable for training multilingual and cross-lingual speech tokens in speech recognition, thus improving the applicability and accuracy of multilingual and cross-lingual speech tokens, and consequently improving the accuracy and applicability of multilingual and cross-lingual speech recognition. By training a speech tokenizer with high accuracy and high applicability, the accuracy and applicability of the speech token sequence output by the speech tokenizer can be improved. By using the high-accuracy and high-applicability speech token sequence as training labels to train the initial speech synthesis model, the model training efficiency and model accuracy of the initial speech synthesis model can be improved. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application;

[0068] Figure 2 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 1 ;

[0069] Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 ;

[0070] Figure 4 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 2 ;

[0071] Figure 5 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 3 ;

[0072] Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 ;

[0073] Figure 7 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 4 ;

[0074] Figure 8 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 5 ;

[0075] Figure 9 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 6 ;

[0076] Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0077] Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0078] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0079] Please see Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application. For example... Figure 1 As shown, the system may include a business server 100 and a terminal device cluster. The terminal device cluster may include: terminal device 200a, terminal device 200b, terminal device 200c, ..., terminal device 200n. It is understood that the above system may include one or more terminal devices, and this application does not limit the number of terminal devices.

[0080] The terminal devices in the cluster can have communication connections with each other. For example, there is a communication connection between terminal devices 200a and 200b, and between terminal devices 200a and 200c. Simultaneously, any terminal device in the cluster can have a communication connection with the service server 100. For example, there is a communication connection between terminal device 200a and the service server 100. The communication connection method is not limited; it can be established directly or indirectly through wired communication, wireless communication, or other methods. This application does not impose any restrictions on this method.

[0081] It should be understood that, such as Figure 1 Each terminal device in the terminal device cluster shown can have an application client installed. When the application client runs on each terminal device, it can interact with the aforementioned... Figure 1 The business server 100 shown interacts with the data, i.e., the communication connection mentioned above. The application client can be a video application, a convenient lifestyle application, a payment application, a financial management application, a game application, a shopping application, a novel application, a navigation application, a browser, or other application client with voice recognition capabilities.

[0082] The application client can be a standalone client or an embedded sub-client integrated into another client (e.g., a video client, an educational client, or a multimedia client), without limitation. Taking a browser as an example, the business server 100 can be a collection of multiple servers, including the browser's corresponding backend server and a data processing server. Therefore, each terminal device can transmit data with the business server 100 through the application client corresponding to that browser. For example, each terminal device can upload its input voice to the business server 100 through the browser's application client. The business server 100 can recognize and process the input voice to obtain the question text, answer the question text, generate the answer text, and return the answer text to the application client. Alternatively, the business server 100 can convert the answer text into speech to obtain the answer speech and return the answer speech to the application client.

[0083] For ease of subsequent understanding and explanation, the embodiments of this application may be... Figure 1 Select one terminal device example from the terminal device cluster shown, for example, terminal device 200a. When the application client receives the model training instruction for the initial speech recognition model, terminal device 200a can generate a model training request corresponding to the model training instruction in the application client, and send the model training request to the business server 100 through the application client.

[0084] The business server 100 obtains the initial speech recognition model and the first training sample set for training the initial speech recognition model according to the model training request. This embodiment does not limit the way the business server 100 obtains the initial speech recognition model, and it can be set according to the actual application scenario. One feasible way is that the model training request carries the initial speech recognition model, and the business server 100 can obtain the initial speech recognition model from the model training request. Another feasible way is that the model training request carries a model identifier for indicating the initial speech recognition model, and the business server 100 can obtain the model identifier from the model training request and obtain the initial speech recognition model from local memory or a storage device that stores the initial speech recognition model according to the model identifier. Yet another feasible way is that the model training request carries the storage path of the initial speech recognition model, and the business server 100 can obtain the storage path from the model training request, access the storage path, and obtain the initial speech recognition model.

[0085] Similarly, this application embodiment does not limit the method by which the service server 100 obtains the first training sample set, and can be set according to the actual application scenario. The first training sample set includes first sample speech carrying International Phonetic Alphabet (IPA) sequence tags, wherein the IPA sequence tags refer to tags generated by the IPA sequence, and the IPA symbols in the IPA sequence tags are used to describe the pronunciation of multiple languages ​​using unified phonetic symbols. For ease of understanding, example IPA symbols are provided. The name is voiceless postalveolar fricative, explained using the following language examples:

[0086] English: "sh" in "ship" middle.

[0087] French: "ch" in middle.

[0088] Mandarin Chinese: "x" in "west" (Note: Chinese) It is a voiceless alveolar fricative, and... The articulation points are slightly different, but the sound is similar.

[0089] The International Phonetic Alphabet (IPA) uses a standardized symbol system to accurately describe the sounds in human languages, avoiding confusion between spelling and pronunciation. For example, the English "sh," the French "ch," and the German "sch" may be spelled differently, but they are all pronounced the same.

[0090] The service server 100 inputs the first training sample set into the initial speech recognition model; the initial speech recognition model can be divided into an initial speech tagger and an initial tag decoder. Through the initial speech tagger, the service server 100 can perform speech tag prediction processing on the first sample speech to obtain a speech tag estimation sequence, which is a set of predicted discrete speech tags.

[0091] The service server 100 inputs the speech tag estimation sequence into the initial tag decoder. Through the initial tag decoder, the service server 100 can perform tag decoding processing on the speech tag estimation sequence to obtain the International Phonetic Alphabet (IPA) estimation sequence, which is a pre-estimated set of IPA symbols. Understandably, in the early stages of training the initial speech recognition model, the IPA estimation sequence may have a large error compared to the IPA symbol sequence labels.

[0092] Based on the IPA estimation sequence and IPA sequence labels, the service server 100 adjusts the model parameters in the initial speech recognition model, which includes an initial speech tokenizer and an initial token decoder. When the initial speech recognition model converges, a speech recognition model is obtained, which includes the parameter-adjusted initial speech tokenizer, and the parameter-adjusted initial speech tokenizer is called the speech tokenizer.

[0093] Speech recognition models can process speech to be recognized, such as recognizing an audio file and generating the corresponding text, such as "Hello, how have you been lately?".

[0094] Furthermore, the trained speech tagger can accurately output the speech tag sequence, thus generating the first speech tag sequence of the second sample speech during the training of the initial speech synthesis model. This first speech tag sequence is used as the training label for the initial speech synthesis model. This application does not describe the process of training the initial speech synthesis model in detail; please refer to the following text. Figure 6 The description in the corresponding embodiments.

[0095] The service server 100 can return the speech recognition model to the terminal device 200a. Therefore, the terminal device 200a can input speech into the speech recognition model, which then generates the answer text or performs speech conversion on the answer text to obtain the answer speech. Alternatively, the terminal device 200a can send the input speech to the service server 100. The service server 100 can then input the input speech into the speech recognition model, which then generates the answer text or performs speech conversion on the answer text to obtain the answer speech, and returns the answer text or answer speech to the terminal device 200a.

[0096] Optionally, if the terminal device 200a locally stores the aforementioned initial speech recognition model and has offline computing capabilities, it can train the initial speech recognition model locally using the first training sample set to obtain a speech recognition model containing the trained speech tokenizer. It is understood that the process of the terminal device 200a training the initial speech recognition model using the first training sample set is the same as the process of the business server 100 training the initial speech recognition model using the first training sample set, and therefore will not be elaborated upon here.

[0097] As described above, the embodiments of this application propose using the International Phonetic Alphabet (IPA) as the training target for the speech tokenizer. This allows the speech tokenizer to fully utilize the consistency of pronunciation across multiple languages ​​(e.g., English, Chinese) to learn language-independent semantic information (i.e., speech tokens). By focusing on unified pronunciation (i.e., IPA), it is more suitable for training multilingual and cross-lingual speech tokens in speech recognition, thus improving the applicability and accuracy of multilingual and cross-lingual speech tokens, and consequently improving the accuracy and applicability of multilingual and cross-lingual speech recognition. By training a speech tokenizer with high accuracy and high applicability, the accuracy and applicability of the speech token sequence output by the speech tokenizer can be improved. By using the high-accuracy and high-applicability speech token sequence as training labels to train the initial speech synthesis model, the model training efficiency and model accuracy of the initial speech synthesis model can be improved.

[0098] It is understood that the methods provided in this application embodiment can be executed by computer devices, including but not limited to terminal devices or business servers. The business server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud databases, cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The terminal devices and business servers can be directly or indirectly connected via wired or wireless means, and this application embodiment does not impose any limitations on this connection.

[0099] Further, please see Figure 2 , Figure 2 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 1. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, audio and video, etc. The embodiments of the present application are applicable to service scenarios such as speech recognition scenarios, text generation scenarios, text recognition scenarios, speech synthesis scenarios, etc., and specific service scenarios will not be enumerated one by one here. The implementation process of the data processing scenario may be performed in a service server, in a terminal device, or through interaction between a terminal device and a service server, which is not limited herein. The terminal device may be the above-mentioned Figure 1 any one of the terminal devices in the terminal device cluster of the corresponding embodiment, and the service server may be the above-mentioned Figure 1 the service server 100 of the corresponding embodiment. For the convenience of description and understanding, the embodiments of the present application are described by taking the implementation in the service server as an example, that is, the service server serves as a computer device to implement the data processing scenario.

[0100] As shown in Figure 2 , the service server acquires a first training sample set 20a, where the first training sample set 20a includes one or more first sample speeches. The embodiments of the present application do not limit the number of the first sample speeches, which may be set according to actual application scenarios. It can be understood that if the first training sample set includes a plurality of first sample speeches, the process of adjusting the model parameters of the initial speech recognition model through each first sample speech is the same, which can also be understood as the process of processing each first sample speech through the initial speech recognition model respectively is the same. Therefore, the embodiments of the present application are described by taking one first sample speech as an example, such as Figure 2 the first sample speech 20b illustrated in the example.

[0101] The first sample speech 20b is a piece of time-series audio. For the convenience of understanding and description, Figure 2 the audio content 201b of the exemplary first sample speech 20b is "the weather is really nice today". Corresponding to the audio content 201b, the International Phonetic Alphabet sequence label 202b of the first sample speech 20b is

[0102] The International Phonetic Alphabet (not limited to the International Phonetic Alphabet in the International Phonetic Alphabet sequence label 202b) is used to describe the pronunciation of multiple languages through unified phonetic symbols. Taking the International Phonetic Alphabet t in the International Phonetic Alphabet sequence label 202b as an example h , it is applied in multiple languages, including but not limited to the following:

[0103] 1. Chinese (Mandarin), such as "tian" in tiān;

[0104] 2. English, such as top[t h pp];

[0105] 3. French, for example, thé[t] h e]

[0106] International Phonetic Alphabet t h Its application varies in different languages, but the pronunciation is the same: it is an aspirated voiceless alveolar stop.

[0107] The service server obtains the initial speech recognition model, which consists of two parts: an initial speech tagger and an initial tagger decoder. This embodiment does not describe the specific structure of the initial speech tagger and the initial tagger decoder in detail; please refer to the following text. Figure 3 The description in the corresponding embodiments. Please refer again. Figure 2 The service server inputs the first sample speech 20b into the initial speech tagger. Through the initial speech tagger, the service server performs speech tag prediction processing on the first sample speech 20b to obtain the estimated speech tag sequence 20c. Figure 2 An example is [T42, T87, T87, T12, T20], where an estimated speech tag in the speech tag estimation sequence 20c represents a vector of the codebook in the initial speech tagger.

[0108] Furthermore, the service server inputs the speech tag estimation sequence 20c into the initial tag decoder. The initial tag decoder then performs tag decoding processing on the speech tag estimation sequence 20c to obtain the International Phonetic Alphabet (IPA) estimation sequence 20d. Figure 2 Example is Clearly, the speech tag estimation sequence 20c obtained by the initial speech recognition model for the first sample speech 20b differs from the IPA sequence label 202b carried by the first sample speech 20b. Based on this, the service server adjusts the model parameters in the initial speech tagger according to the IPA estimation sequence 20d and the IPA sequence label 202b. Simultaneously, it adjusts the model parameters in the initial tag decoder according to the IPA estimation sequence 20d and the IPA sequence label 202b. When both the parameter-adjusted initial speech tagger and the parameter-adjusted initial tag decoder meet the convergence condition, the speech tagger and tag decoder are obtained. This indicates that the initial speech recognition model consists of two parts: the initial speech tagger and the initial tag decoder. Therefore, the speech tagger and the tag decoder can form a trained speech recognition model.

[0109] A well-trained speech recognition model can be applied to many scenarios, including but not limited to human-computer dialogue and autonomous driving. Its included speech tagger can also serve as the training foundation for an initial speech synthesis model. During training, the speech tagger generates a first speech tag sequence from the second sample speech, which can then be used as the training labels for the initial speech synthesis model. The training process for the initial speech synthesis model is described below. Figure 6 The description in the corresponding embodiments.

[0110] As described above, the embodiments of this application propose using the International Phonetic Alphabet (IPA) as the training target for the speech tokenizer. This allows the speech tokenizer to fully utilize the consistency of pronunciation across multiple languages ​​(e.g., English, Chinese) to learn language-independent semantic information (i.e., speech tokens). By focusing on unified pronunciation (i.e., IPA), it is more suitable for training multilingual and cross-lingual speech tokens in speech recognition, thus improving the applicability and accuracy of multilingual and cross-lingual speech tokens, and consequently improving the accuracy and applicability of multilingual and cross-lingual speech recognition. By training a speech tokenizer with high accuracy and high applicability, the accuracy and applicability of the speech token sequence output by the speech tokenizer can be improved. By using the high-accuracy and high-applicability speech token sequence as training labels to train the initial speech synthesis model, the model training efficiency and model accuracy of the initial speech synthesis model can be improved.

[0111] Further, please see Figure 3 , Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, assisted driving, audio and video, etc. The embodiments of this application are applicable to business scenarios such as speech recognition, text generation, text recognition, and speech synthesis; specific business scenarios will not be listed here.

[0112] This data processing method can be provided by a business server (e.g., the one mentioned above). Figure 1 The business server 100 shown can execute the service, or it can be executed by a terminal device (e.g., the one described above). Figure 1 The terminal device 200a) shown can execute the method, but it can also be executed interactively by the business server and the terminal device. For ease of understanding, this embodiment of the application illustrates the method as being executed by the business server, i.e., the business server, acting as a computer device, executes the data processing method. Figure 3 As shown, the data processing method may include at least the following steps S101-S104.

[0113] Step S101: Obtain the first training sample set and input the first training sample set into the speech recognition initial model; the first training sample set includes first sample speech carrying International Phonetic Alphabet (IPA) sequence labels; the IPA in the IPA sequence labels is used to describe the pronunciation of multiple languages ​​through unified phonetic symbols.

[0114] Specifically, speech synthesis models based on Large Language Models (LLMs) exhibit high naturalness and zero-shot learning capabilities. In this paradigm, discrete speech tokens are crucial. Building upon the use of Automatic Speech Recognition (ASR) tasks to enhance the semantic information in speech tokens, this application proposes using the International Phonetic Alphabet (IPA) as the training target for ASR to learn language-independent speech tokens.

[0115] The embodiments of this application do not limit the source and content of the first training sample set, which can be determined according to the needs of the actual application scenario. The Common Voice dataset contains data in 40 languages, including Arabic (ar), Bashkir (ba), Bulgarian (bg), Catalan (ca), Chinese (zh), Czech (cs), Welsh (cy), Dutch (nl), English (en), Estonian (et), Basque (eu), Persian (fa), Finnish (fi), French (fr), Georgian (ka), German (de), Hungarian (hu), Indonesian (id), Italian (it), Latvian (lv), Lithuanian (lt), Marathi (mr), Polish (pl), Portuguese (pt), Romanian (ro), Russian (ru), Slovak (sk), Slovenian (sl), Spanish (es), Swahili (sw), Swedish (sv), Tamil (ta), Turkish (tr), Ukrainian (uk), Urdu (ur), Uzbek (uz), Chuvash (cv), and Tatar (tt). The data duration for each language ranges from 50 to 200 hours, totaling 4,879 hours, with each speech segment accompanied by a corresponding text transcription.

[0116] Step S102: In the initial speech tagger of the initial speech recognition model, the first sample speech is subjected to speech tag prediction processing to obtain the speech tag estimation sequence.

[0117] Specifically, the initial speech tagger of the initial speech recognition model includes a convolutional encoder, an initial tag encoder, and an initial vector quantizer. The convolutional encoder extracts acoustic features from the first sample speech to obtain continuous semantic features, which are then input into the initial tag encoder. The initial tag encoder performs contextual feature fusion processing on the continuous semantic features to obtain continuous semantic enhancement features, which are then input into the initial vector quantizer. The initial vector quantizer performs feature recognition processing on the continuous semantic enhancement features to obtain discrete speech tags. The discrete speech tags are then combined into a speech tag estimation sequence according to the audio frame order of the first sample speech.

[0118] The specific process of obtaining discrete speech tags by performing feature recognition processing on continuous semantic enhancement features through an initial vector quantizer may include: using an initial vector quantizer to perform similarity processing on the continuous semantic enhancement features with at least two vectors in the codebook to obtain similarity values ​​corresponding to the at least two vectors respectively; obtaining the maximum similarity value among the at least two similarity values; and determining the speech tag corresponding to the vector with the maximum similarity value among the at least two vectors as the discrete speech tag of the continuous semantic enhancement features.

[0119] Please see also Figure 4 , Figure 4 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 2 For ease of understanding and narration, Figure 4 The first sample speech 30a is taken as a 1-second segment of speech data. This speech data has a sampling rate of 16000 Hz, meaning 16000 amplitude values ​​are collected per second, with each amplitude value called a sampling point. It is important to emphasize that... Figure 4 All data in the examples are for illustrative purposes only and do not represent actual data values ​​in real-world application scenarios.

[0120] The service server inputs the first sample speech 30a into the initial speech tagger, which includes a convolutional encoder, an initial tag encoder, and an initial vector quantizer. The convolutional encoder can be a convolutional neural network (CNN), a type of feedforward neural network that includes convolutional computation and has a deep structure; it is one of the representative algorithms of deep learning. The convolutional encoder (CNN Encoder) can be a pre-trained network. Therefore, in this embodiment, when adjusting the parameters of the initial speech tagger, the parameters of its convolutional encoder are not adjusted; that is, the parameters of the components of the initial speech tagger other than the convolutional encoder are adjusted.

[0121] The core function of the CNN encoder is to convert the first sample speech 30a (a speech waveform) which is a time-domain signal into higher-level acoustic features. The specific process is as follows:

[0122] Step 1: Framing

[0123] Speech signals are time-varying and need to be processed in segments. Typically, continuous sampling points are divided into small segments (e.g., each segment is 20-40ms), called a frame. Figure 4 As shown, the segmented data processed by the convolutional encoder includes frame length and frame shift. Frame length refers to the length of each audio frame. Figure 4 The example is 25 milliseconds (ms), therefore each frame contains 0.025s × 16000 samples / s = 400 samples. Frame shift refers to the distance moved between adjacent frames. Figure 4 The example is 10ms, meaning one frame is taken every 10ms, so adjacent frames overlap by 15ms. By using frame shifting, it can be determined that the first sample speech 30a in 1 second can be divided into 100 audio frames, that is, 100 frames per second.

[0124] Step 2: CNN extracts frame features

[0125] Each frame is processed using a CNN. Figure 4 (Example: 400 sampling points) Extract acoustic features to obtain continuous semantic vectors for each frame. Figure 4 For example, a continuous semantic vector has a dimension of 256. By extracting acoustic features from 100 audio frames, the business server can obtain 100 continuous semantic vectors, each with a dimension of 256. In this embodiment, these 100 continuous semantic vectors with a dimension of 256 are referred to as continuous semantic features, such as... Figure 4 The example continuous semantic feature 30b can be understood as a 100×256 matrix.

[0126] Furthermore, the business server inputs the continuous semantic features 30b into the initial token encoder. The core function of the initial token encoder is to enhance the contextual information of the CNN features, overcome the limitations of local receptive fields, and improve the robustness of subsequent quantization. This application does not limit the network structure of the initial token encoder; it can be set according to the actual application scenario, including but not limited to a Transformer Encoder composed of multiple layers of self-attention and feedforward networks (FFN), convolutional neural networks, and bidirectional long short-term memory networks (BiLSTM).

[0127] Please see again. Figure 4 The initial tag encoder performs context modeling on the 256-dimensional vector of each frame (e.g., using a Transformer to analyze the relationship between adjacent frames), outputting an enhanced 256-dimensional semantic vector, specifically the first consecutive semantic vector (e.g., ...). Figure 4 Example of a continuous semantic enhancement vector [0.2 0.5 1.2...0.7], such as Figure 4 The example [-1.2 02.2...-0.7] incorporates information from its preceding and following frames; the second continuous semantic vector (such as...) Figure 4 Example of a continuous semantic enhancement vector ([0.5 0.4 1.1... 0.8]) such as Figure 4 The example [0.3 0.2-1.1...1.8] incorporates information from its preceding and following frames; the third continuous semantic vector (such as...) Figure 4 Example of a continuous semantic enhancement vector [2.5 1.2 1.6...1], such as Figure 4 The example [2.1 2 1.1...-1] incorporates information from its preceding and following frames; the 100th consecutive semantic vector (such as...) Figure 4 Example of a continuous semantic enhancement vector [3.1 2.5 2.1...0.2], such as Figure 4 The example [2.1-2 1.1...-0.1] incorporates information from its preceding and following frames.

[0128] Based on the above processing, the business server can obtain 100 continuous semantic enhancement vectors, each with 256 dimensions. In this embodiment, these 100 continuous semantic enhancement vectors with 256 dimensions are referred to as continuous semantic enhancement features, such as... Figure 4 Example of continuous semantic enhancement feature 30c.

[0129] The business server inputs the continuous semantic enhancement feature 30c into the initial vector quantizer, which includes a codebook 30d containing a finite number of vectors with the same dimensions as the continuous semantic vectors. For example... Figure 4 As shown in the example, the codebook 30d in the initial vector quantizer contains 512 256-dimensional vectors, each with a vector identifier (also called a vector index), meaning that a vector identifier uniquely identifies a vector. For example... Figure 4In the example codebook 30d, the first vector [1.2 2.5 1.1... 0.6] corresponds to vector identifier T1, the second vector [1.5 0.2 -1.1... 0] corresponds to vector identifier T2, the third vector [-0.5 0.2 -1.6... 1] corresponds to vector identifier T3, and the 512th vector [2.1 2.2 2.6... -0.2] corresponds to vector identifier T512.

[0130] In the initial vector quantizer, the business server performs similarity processing between the continuous semantic enhancement feature 30c and the 512 vectors in the codebook 30d, respectively. The specific process is as follows: Figure 4 The example describes how the business server performs similarity processing on the first 256-dimensional continuous semantic augmentation vector (-1.2 0 2.2...-0.7) and the first 256-dimensional vector (1.2 2.5 1.1...0.6) in codebook 30d, resulting in a similarity value of 1.1. It then performs similarity processing on the first 256-dimensional continuous semantic augmentation vector (-1.2 0 2.2...-0.7) and the second 256-dimensional vector (1.5 0.2 -1.1...0) in codebook 30d, resulting in a similarity value of 1.2. Finally, the business server performs similarity processing on the first 256-dimensional continuous semantic augmentation vector (-1.2 0 2.2...-0.7) and the third 256-dimensional vector (-0.5...-0.6) in codebook 30d. Similarity processing is performed on the vectors from 0.2 to 1.6...1, resulting in a similarity value of 1.3 for the two vectors. The first 256-dimensional continuous semantic enhancement vector (-1.2 0 2.2...-0.7) is then compared with the 512th 256-dimensional vector in codebook 30d (2.1 2.2 2.6...-0.2), resulting in a similarity value of 1.512. The business server selects the maximum similarity value from the 512 similarity values ​​between 1.1 and 1.512, and determines the vector corresponding to this maximum similarity value in codebook 30d, for example, the 42nd vector. The business server then identifies the vector identifier T42, uniquely identifying the 42nd vector in codebook 30d, as the discrete speech tag for the first 256-dimensional continuous semantic enhancement vector (-1.2 0 2.2...-0.7).

[0131] Similarly, the business server can determine the discrete speech markers corresponding to each continuous speech enhancement vector in the continuous semantic enhancement feature 30c from the codebook 30d, such as... Figure 4As shown in the example, the discrete speech label of the second 256-dimensional continuous semantic enhancement vector (0.3 0.2-1.1...1.8) is vector identifier T87, which indicates that the 87th vector in codebook 30d has the highest similarity to the continuous semantic enhancement vector (0.3 0.2-1.1...1.8); the discrete speech label of the third 256-dimensional continuous semantic enhancement vector (2.1 21.1...-1) is vector identifier T83, which indicates that the 83rd vector in codebook 30d has the highest similarity to the continuous semantic enhancement vector (2.1 2 1.1...-1); the discrete speech label of the 100th 256-dimensional continuous semantic enhancement vector (2.1-2 1.1...-0.1) is vector identifier T20, which indicates that the 20th vector in codebook 30d has the highest similarity to the continuous semantic enhancement vector (2.1-2 1.1...-0.1).

[0132] This application does not limit the similarity processing method, including but not limited to cosine similarity, Euclidean distance, Pearson correlation coefficient, and Manhattan distance.

[0133] In summary, the service server combines 100 discrete speech tags into a speech tag estimation sequence 30e according to the audio frame order of the first sample speech 30a, as follows: Figure 4 Example [T42, T87, T83, ..., T20].

[0134] Step S103: In the initial tag decoder of the initial speech recognition model, the speech tag estimation sequence is processed by tag decoding to obtain the International Phonetic Alphabet (IPA) estimation sequence.

[0135] Specifically, the initial token decoder includes an initial optimization encoder and an initial decoder; the initial optimization encoder performs semantic enhancement processing on the speech token estimation sequence to obtain continuous semantic optimization features, which are then input into the initial decoder; the initial decoder decodes the continuous semantic optimization features to obtain the International Phonetic Alphabet (IPA) estimation sequence.

[0136] The initial speech recognition model consists of an initial speech tokenizer and an initial token decoder; please refer to [link / reference needed]. Figure 4 as well as Figure 5 , Figure 5 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 3 The above step S102 is combined Figure 4The process of generating the estimated speech tag sequence 30e of the first sample speech 30a through the initial speech tagger is described in detail. This step will describe in detail the process of generating the estimated IPA sequence 30g of the first sample speech 30a through the initial tagger decoder.

[0137] The speech tag estimation sequence 30e is discrete data. Directly inputting it into the initial decoder will cause gradient propagation to fail (due to its discrete non-differentiability). Therefore, as... Figure 5 As shown, the business server inputs the speech tag estimation sequence 30e into the initial refine encoder. The refine encoder remaps the discrete speech tag estimation sequence 30e into continuous vectors, restoring differentiability. The core function of the refine encoder includes the re-continuum transformation of the discrete tags described above. Specifically, this can be achieved through an embedding layer, which converts each speech tag into a dense vector. Since each frame's speech tag is processed independently, temporal dependencies (such as tonal coherence and coarticulation) may be ignored. Therefore, a neural network (such as a Transformer or a Recurrent Neural Network (RNN)) is used in the initial vector quantizer to analyze the contextual relationships of the speech tag estimation sequence 30e to capture long-distance dependencies (such as the intonation of Chinese spanning multiple phonemes) and eliminate local inconsistencies caused by quantization errors, i.e., to achieve contextual information fusion and obtain continuous semantic optimization features, such as... Figure 5 The example uses an initial optimized encoder to first re-continuum the discrete tokens of the estimated speech token sequence 30e, and then fuses contextual information to obtain continuous semantic optimized features 30f.

[0138] Corresponding to Figure 4 The example's continuous semantic enhancement feature 30c and continuous semantic optimization feature 30f are also 100 continuous semantic optimization vectors, each with 256 dimensions, such as... Figure 5 Examples of continuous semantic optimization vectors: (-1 0.1-2.2...0.5), (1.3 2.2 1.1...-0.8), (3.1 0 2.1...1.3), and (1.8 2-1.1...0.1).

[0139] Through the initial optimization encoder, the business server can input the continuous and semantically rich continuous semantic optimization features 30f into the initial decoder. The initial decoder can then decode the continuous semantic optimization features 30f to obtain the IPA estimation sequence 30g, such as... Figure 5 Example

[0140] Step S104: Adjust the model parameters in the initial speech tagger according to the IPA estimation sequence and IPA sequence labels to obtain the speech tagger; the speech tagger is used to generate the first speech tag sequence of the second sample speech when training the initial speech synthesis model, and the first speech tag sequence is used as the training label of the initial speech synthesis model.

[0141] Specifically, based on the IPA estimation sequence and IPA sequence labels, the model loss value of the initial speech recognition model is determined; based on the model loss value, the model parameters of the initial speech tokenizer and the initial token decoder in the initial speech recognition model are adjusted; when the initial speech recognition model containing the adjusted initial speech tokenizer and the adjusted initial token decoder satisfies the model convergence condition, the adjusted initial speech tokenizer is determined as the speech tokenizer.

[0142] Please see again. Figure 5 The first sample speech 30a carries the International Phonetic Alphabet sequence tag 30h, such as Figure 5 Example The business server calculates the error (loss) of the IPA estimation sequence 30g and the IPA sequence label 30h. This application embodiment does not limit the method of calculating the error (i.e., loss), including but not limited to cross-entropy (CE) loss.

[0143] Cross-entropy loss, as a key objective function of the initial speech recognition model, measures the difference between the decoder's predicted IPA sequence and the true label. Figure 5 The error value 30i between the example IPA estimation sequence 30g and the IPA sequence label 30h is 0.7. The error value 30i of 0.7 can be used as the model loss value of the initial speech recognition model to drive model optimization. The core task of cross-entropy loss is to minimize the "information distance" between the probability distribution predicted by the model and the true distribution, so as to ensure that the speech mark can accurately correspond to the linguistic phoneme (IPA). The cross-entropy formula is shown in the following formula (1):

[0144]

[0145] Wherein, y in formula (1) i p represents the probability distribution of the IPA classification of the i-th phoneme in the real label (in this embodiment, the IPA sequence label), for example, a one-hot value (0 or 1). i Let represent the probability distribution of the IPA symbol classification of the i-th phoneme predicted by the model, and n represent the total number of phonemes.

[0146] The CE loss is derived from the model parameters in the initial speech recognition model (including the model parameters in the initial speech tagger). The gradient is backpropagated through the initial decoder, initial optimized encoder, initial vector quantizer, and initial tag encoder, so that the model's prediction probability for the real phonemes approaches 1, and other phonemes approaches 0.

[0147] In this embodiment, the computer device inputs a first training sample set into the speech recognition initial model. The first training sample set includes first sample speech carrying International Phonetic Alphabet (IPA) sequence labels. The IPA symbols in the IPA sequence labels are used to describe the pronunciation of multiple languages ​​using unified phonetic symbols. In the initial speech tagger of the speech recognition initial model, speech tag prediction processing is performed on the first sample speech to obtain a speech tag estimation sequence. In the initial tag decoder of the speech recognition initial model, tag decoding processing is performed on the speech tag estimation sequence to obtain an IPA estimation sequence. Based on the IPA estimation sequence and the IPA sequence labels, the model parameters in the initial speech tagger are adjusted to obtain a speech tagger. The speech tagger is used to generate a first speech tag sequence of the second sample speech when training the speech synthesis initial model. The first speech tag sequence is used as the training label for the speech synthesis initial model. As described above, the embodiments of this application propose using the International Phonetic Alphabet (IPA) as the training target for the speech tokenizer. This allows the speech tokenizer to fully utilize the consistency of pronunciation across multiple languages ​​(e.g., English, Chinese) to learn language-independent semantic information (i.e., speech tokens). By focusing on unified pronunciation (i.e., IPA), it is more suitable for training multilingual and cross-lingual speech tokens in speech recognition, thus improving the applicability and accuracy of multilingual and cross-lingual speech tokens, and consequently improving the accuracy and applicability of multilingual and cross-lingual speech recognition. By training a speech tokenizer with high accuracy and high applicability, the accuracy and applicability of the speech token sequence output by the speech tokenizer can be improved. By using the high-accuracy and high-applicability speech token sequence as training labels to train the initial speech synthesis model, the model training efficiency and model accuracy of the initial speech synthesis model can be improved.

[0148] Please see Figure 6 , Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 This method can be implemented by a business server (e.g., the one mentioned above). Figure 1 The business server 100 shown can execute the service, or it can be executed by a terminal device (e.g., the one described above). Figure 1 The terminal device 200a) shown can execute the method, but it can also be executed interactively by the business server and the terminal device. For ease of understanding, this embodiment uses the execution by the business server as an example, where the business server acts as a computer device to execute the data processing method. Figure 6 As shown, the data processing method includes steps S201-S209.

[0149] Step S201: Obtain the first training sample set and input the first training sample set into the speech recognition initial model; the first training sample set includes first sample speech carrying International Phonetic Alphabet (IPA) sequence labels; the IPA in the IPA sequence labels is used to describe the pronunciation of multiple languages ​​through unified phonetic symbols.

[0150] Specifically, text-to-speech (TTS) models, also known as speech synthesis models, can generate high-fidelity speech that is almost indistinguishable from human speech for a specific speaker. With the development of audio representation learning techniques, such as self-supervised learning (SSL) models and speech codec models, continuous or discrete speech representations are integrated into speech synthesis models as intermediate representations.

[0151] Clearly, speech tags are crucial in TTS models based on Large Language Models (LLMs) or Large Models (LMs), determining the model's performance. Previous studies typically did not consider the relationships between different languages ​​and used unsupervised speech tags. Because ASR models struggle to learn shared language knowledge, the extracted speech tags lack language-independent information, limiting the performance of multilingual LM-based TTS models, especially in cross-lingual scenarios.

[0152] To address this issue, embodiments of this application propose using the International Phonetic Alphabet (IPA) as the training target for the ASR model (also referred to as a speech recognition model in this application). IPA is a unified phonetic notation system used to represent the pronunciation of all languages. This allows the ASR model to fully leverage the consistency of pronunciation across languages ​​and learn language-independent semantic information. To obtain a language-independent speech segmenter, embodiments of this application insert a vector quantization layer into the IPA-targeted ASR model.

[0153] Step S202: In the initial speech tagger of the initial speech recognition model, the first sample speech is subjected to speech tag prediction processing to obtain the speech tag estimation sequence.

[0154] Specifically, the automatic speech recognition (ASR) task is to transcribe speech into text. Previous research has proposed using ASR tasks to enhance the semantic information in speech representations and their alignment with text. This application proposes a language-independent speech segmenter, which is also trained through an ASR task.

[0155] The initial speech recognition model refers to the model whose parameters need to be adjusted. Therefore, the speech recognition model after parameter adjustment can be called a speech recognition model. Similarly, the initial speech tagger refers to the model or network whose parameters need to be adjusted. The trained initial speech tagger is the initial speech tagger after parameter adjustment, and can be called a speech tagger.

[0156] In this embodiment, the initial speech tagger includes a convolutional encoder, which is a trained network, such as HuBERT. HuBERT (Hidden-Unit BERT) is a speech representation learning model based on self-supervised learning, which aims to solve the problems of multiple sound units, lack of dictionary, and unclear unit boundaries in speech signals. Therefore, when adjusting the model parameters in the initial speech recognition model based on the model loss value, the parameters in the convolutional encoder are not included.

[0157] To reduce computational and storage resources, this embodiment uses the same CNN encoder as HuBERT to downsample the speech waveform (i.e., the first sample speech mentioned above) to a 50Hz representation. For better performance, in one feasible implementation, the original HuBERT model has a sampling rate of 50Hz; this embodiment compresses it to 25Hz without adding additional parameters. Since HuBERT uses convolutional positional embeddings, this embodiment changes the stride of the positional convolution to 2 before the first encoder layer and performs average pooling on the 50Hz representation to obtain the 25Hz speech token.

[0158] The tag encoder can consist of six Transformer layers. A vector quantizer is inserted after the tag encoder to obtain discrete speech tags. This application embodiment does not limit the source of the codebook in the vector quantizer; it can be set according to the actual application scenario. This application embodiment uses a vector quantizer with a Euclidean codebook updated via Exponential Moving Average (EMA). After quantization, this application embodiment can obtain discrete speech tags for the first sample speech.

[0159] Step S203: In the initial tag decoder of the initial speech recognition model, the speech tag estimation sequence is processed by tag decoding to obtain the International Phonetic Alphabet (IPA) estimation sequence.

[0160] Specifically, the initial label decoder refers to the model whose parameters need to be adjusted; therefore, the initial label decoder with adjusted parameters can be called a label decoder.

[0161] Before inputting the speech tag estimation sequence into the initial optimizer of the initial tag decoder, the business server can add additional positional embeddings to the speech tag estimation sequence, and then input the speech tag estimation sequence with positional embeddings into the initial optimizer. The initial optimizer consists of subsequent layers, and during training, its output is input into the initial decoder in a teacher-forced mode, which can use cross-entropy (CE) loss to predict the target.

[0162] Step S204: Adjust the model parameters in the initial speech tagger according to the IPA estimation sequence and IPA sequence labels to obtain the speech tagger; the speech tagger is used to generate the first speech tag sequence of the second sample speech when training the initial speech synthesis model, and the first speech tag sequence is used as the training label of the initial speech synthesis model.

[0163] Specifically, in order to generate IPA training targets, this application embodiment uses the Phonemizer tool (an open-source project mainly used to convert text into phonemes) to convert the first sample speech into an IPA sequence, and uses the IPA sequence as the label of the first sample speech.

[0164] To improve the accuracy of Chinese character conversion, the business server uses the pypinyin1 tool to convert Chinese characters to Pinyin. pypinyin1 is a Python library (a collection of modules and functions) widely used in Chinese character processing and natural language processing. Based on a Chinese character Pinyin database, pypinyin1 provides a simple yet powerful Application Programming Interface (API) to meet various Chinese character to Pinyin conversion needs. pypinyin1 supports multiple Pinyin styles and output formats, including standard style, style with tones, and initial letter style, and also supports custom dictionaries and operation on multiple platforms.

[0165] Furthermore, the business server uses the pinyin-to-ipa tool (a tool for converting Chinese Pinyin to International Phonetic Alphabet (IPA)) to convert the Pinyin from Chinese characters into IPA with tones. As mentioned above, the initial speech recognition model in this embodiment is initialized using a pre-trained multilingual model called mHuBERT-147, which is compressed to a labeling rate of 25Hz without introducing any additional parameters. After training, the speech recognition model has a character error rate (CER) of 3.57% on a validation set of 40 languages.

[0166] To verify the effectiveness of the language-independent speech tag proposed in the embodiments of this application, the inventors also trained another speech recognition model with the same architecture and data, but using Byte-Pair Encoding (BPE) tagging, which is not the IPA proposed in the embodiments of this application. The size of the BPE tag set is 43319, and the character error rate (CER) on the validation set of 40 languages ​​is 10.24%.

[0167] The embodiments of this application employ IPA, which focuses only on pronunciation, making it more suitable for training multilingual and cross-lingual tokenizers. Experiments show that in zero-sample speech synthesis, the content of this application has a lower character error rate and greater timbre similarity compared to the baseline. Subjective listening evaluations also demonstrate the superiority of the proposed solution.

[0168] Step S205: Obtain the second training sample set; the second training sample set includes sample text and the second sample speech corresponding to the sample text.

[0169] Specifically, this application embodiment does not limit the source of the second training sample set, which can be obtained according to the actual application scenario, nor does it limit the number of sample texts in the second training sample set. If there are multiple sample texts, each sample text corresponds to a second sample speech.

[0170] Step S206: Input the second sample speech into the speech tagger, and perform speech tag prediction processing on the second sample speech through the speech tagger to obtain the first speech tag sequence.

[0171] Specifically, the processing procedure of the speech tagger for the second sample speech is the same as that of the initial speech tagger for the first sample speech, and can include the following steps: The speech tagger includes a convolutional encoder, a tag encoder, and a vector quantizer; the convolutional encoder extracts acoustic features from the second sample speech to obtain continuous semantic features corresponding to the second sample speech, and inputs the continuous semantic features into the tag encoder; the tag encoder performs contextual feature fusion processing on the continuous semantic features corresponding to the two sample speech to obtain continuous semantic enhancement features corresponding to the two sample speech, and inputs the continuous semantic enhancement features into the vector quantizer; the vector quantizer performs feature recognition processing on the continuous semantic enhancement features corresponding to the two sample speech to obtain discrete speech tags corresponding to the two sample speech, and combines the discrete speech tags into a first speech tag sequence according to the audio frame order of the second sample speech.

[0172] The specific process of obtaining discrete speech tags corresponding to the continuous semantic enhancement features of two-sample speech by performing feature recognition processing on the continuous semantic enhancement features corresponding to the two-sample speech through a vector quantizer may include: using a vector quantizer, performing similarity processing on the continuous semantic enhancement features corresponding to the two-sample speech with at least two vectors in the codebook to obtain similarity values ​​corresponding to at least two vectors respectively; obtaining the maximum similarity value among the at least two similarity values, and determining the speech tag corresponding to the vector with the maximum similarity value among the at least two vectors as the discrete speech tag of the continuous semantic enhancement features corresponding to the two-sample speech.

[0173] Please refer to the above process as well. Figure 7 , Figure 7 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 4 .like Figure 7 As shown, the service server inputs the second sample speech 60b into the trained speech tagger. The speech tagger then performs speech tag prediction processing on the second sample speech 60b to obtain the first speech tag sequence 60e. Figure 7 The first speech tag sequence 60e is exemplified as [T42, T87, T83, ..., T20].

[0174] Step S207: Input the sample text and the first speech tag sequence into the speech synthesis initial model. In the speech prediction initial network layer of the speech synthesis initial model, perform speech tag prediction processing on the sample text according to the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text.

[0175] Specifically, the initial speech prediction network layer includes an initial text encoder and a speech tag initial network layer. The initial text encoder encodes the sample text to obtain a text encoding estimation vector, which is then input into the speech tag initial network layer. In the speech tag initial network layer, the text encoding estimation vector is processed for speech tag prediction based on the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text.

[0176] This application's embodiments extract language-independent speech tokens from the proposed speech tokenizer (also known as a speech tokenizer), i.e., the aforementioned first speech token sequence. The first speech token sequence is then integrated into an LLM-based multilingual TTS model (referred to as a speech synthesis model in this application embodiment). This model uses a language model to autoregressively predict speech tokens. When the service server integrates the first speech token sequence into the LLM-based multilingual speech synthesis model, it can effectively improve intelligibility, naturalness, and zero-shot cross-lingual performance.

[0177] Please see again. Figure 7The initial speech synthesis model refers to the model whose parameters need adjustment; therefore, the speech synthesis initial model with adjusted parameters is called the speech synthesis model. The initial speech synthesis model consists of... Figure 7 The diagram shows the initial network layer for speech prediction and the initial network layer for stream matching. The training processes of the initial network layer for speech prediction and the initial network layer for stream matching are independent of each other, meaning that the training processes of the initial network layer for speech prediction and the initial network layer for stream matching are unrelated.

[0178] This application does not limit the training execution order of the initial network layer for speech prediction and the initial network layer for stream matching. The initial network layer for speech prediction can be trained first, followed by the initial network layer for stream matching; or the initial network layer for stream matching can be trained first, followed by the initial network layer for speech prediction. The description of step S209 is that determining the initial speech synthesis model containing the speech prediction network layer as the speech synthesis model is this training order, that is, step S209 sets the initial network layer for stream matching to be trained; or the initial network layer for speech prediction and the initial network layer for stream matching can be trained simultaneously.

[0179] When the initial speech prediction network layer is trained, the business server generates the speech prediction network layer. When the initial flow matching network layer is trained, the business server generates the flow matching network layer. The initial speech synthesis model, which includes the speech prediction network layer and the flow matching network layer, is determined as the speech synthesis model.

[0180] This step, along with step S208, mainly describes the training process of the initial network layer for speech prediction. Please refer to [link / reference needed]. Figure 7 The initial network layer for speech prediction includes an initial text encoder and a speech tag initial network layer. Using the initial text encoder, the service server encodes the sample text 60a to obtain the text encoding estimation vector 60d. Figure 7 The sample text 60a carries a Chinese language tag and its content is "The weather is really nice today".

[0181] This application does not limit the structure of the initial text encoder. It can be set according to the actual application scenario, including but not limited to the Universal Sentence Encoder (USE, a pre-trained model based on Transformer or Deep Averaging Network (DAN), supporting multiple languages, directly encoding sentences into fixed-length vectors, suitable for tasks such as semantic similarity calculation and clustering), Bidirectional Encoder Representations from Transformers (BERT) and its variants (a pre-trained model based on Transformer, context-aware, suitable for generating dynamic word vectors. Vectors labeled [CLS] can be directly used as sentence encoding), and lightweight word vector models (static word vectors based on sub-words, supporting out-of-vocabulary (OOV) words, suitable for multiple languages ​​and morphologically rich languages, such as Chinese and Turkish).

[0182] To better control language, this embodiment of the application randomly adds a language prefix, i.e., a language label, to the sample text during the training of the initial network layer for speech prediction with a first probability (a value ranging from 0 to 1). The specific process is as follows: Obtaining text carrying language labels (e.g., ...) Figure 7 The example uses Chinese speech tags) to represent first candidate sample text and second candidate sample text; language tags are used to indicate the language to which the first candidate sample text belongs; based on the first sampling probability of the first candidate sample text and the second sampling probability of the second candidate sample text, sampling is performed between the first candidate sample text and the second candidate sample text, and the collected candidate sample text is determined as sample text; the second sample speech corresponding to the sample text is obtained, and the sample text and the second sample speech are determined as the second training sample set.

[0183] The first sampling probability mentioned above is the same as the first probability, and the sum of the first sampling probability and the second sampling probability is 1.

[0184] like Figure 7 As shown, the business server marks the start of the sequence as 601c (Start of Sequence). Figure 7 (abbreviated as S), text encoding estimation vector 60d, speech conversion tag 602c ( Figure 7(Abbreviated as T), and the first speech tag sequence 60e are sequentially input into the speech tag initialization network layer. The sequence start marker 601c is a special marker used to inform the speech tag initialization network layer that the input sequence has begun, helping the speech tag initialization network layer identify the starting point of the input sequence and thus better process the input data. The speech conversion marker 602c is also a special marker used to instruct the speech tag initialization network layer to convert from content embedding (i.e., text encoding estimation vector 60d) to speech tag (i.e., the first speech tag sequence 60e), helping the speech tag initialization network layer understand that the current task is the conversion from text content to speech tags.

[0185] The initial network layer for speech tagging can be understood as a pre-trained large model (LM) or large language model (LLM). This application does not limit the structure of the initial network layer for speech tagging, and it can be set according to the actual application scenario.

[0186] In the initial speech tagging network layer, speech tag prediction processing is performed on the text encoding estimation vector 60d based on the first speech tag sequence 60e to obtain the predicted speech tag sequence 60f corresponding to the sample text 60a. Figure 7 The predicted speech tag sequence 60f is exemplified as [T4, T47, T35, ..., T2]. The initial speech tagging network layer also outputs a sequence end marker 603c, End of Sequence. Figure 7 Abbreviated as E, its function is to terminate the generation process, prevent the model from looping indefinitely, and help the model learn when to stop outputting (such as the end of a speech frame) during the initial training of the speech tag network layers.

[0187] Step S208: Adjust the model parameters in the initial speech prediction network layer according to the predicted speech tag sequence and the first speech tag sequence to obtain the speech prediction network layer.

[0188] Specifically, the first speech tag sequence includes at least two sequentially arranged first speech tags; the predicted speech tag sequence includes at least two sequentially arranged second speech tags; loss processing is performed on the first and second speech tags in the first and predicted speech tag sequences that are in the same position to obtain at least two speech tag loss values; the at least two speech tag loss values ​​are summed to obtain the network layer loss value of the initial speech prediction network layer; the model parameters in the initial speech prediction network layer are adjusted according to the network layer loss value to obtain the speech prediction network layer.

[0189] Please see again. Figure 7The first speech marker sequence 60e includes at least two sequentially arranged first speech markers, such as... Figure 7 The example first speech mark T42 is the first speech mark among at least two first speech marks, the first speech mark T87 is the second speech mark among at least two first speech marks, the first speech mark T83 is the third speech mark among at least two first speech marks, and the first speech mark T20 is the last speech mark among at least two first speech marks.

[0190] The predicted speech tag sequence 60f includes at least two sequentially arranged second speech tags, such as... Figure 7 The example second speech mark T4 is the first speech mark among at least two second speech marks, the second speech mark T47 is the second speech mark among at least two second speech marks, the second speech mark T35 is the third speech mark among at least two second speech marks, and the second speech mark T2 is the last speech mark among at least two second speech marks.

[0191] The service server performs loss processing on the first speech markers and second speech markers with the same arrangement position in the first speech marker sequence 60e and the predicted speech marker sequence 60f. Specifically, loss processing is performed on the first speech marker T42 and the second speech marker T4, which are both in the first position, to obtain a speech identifier loss value 1 between the first speech marker T42 and the second speech marker T4; loss processing is performed on the first speech marker T87 and the second speech marker T47, which are both in the second position, to obtain a speech identifier loss value 2 between the first speech marker T87 and the second speech marker T47; loss processing is performed on the first speech marker T83 and the second speech marker T35, which are both in the third position, to obtain a speech identifier loss value 3 between the first speech marker T83 and the second speech marker T35; and loss processing is performed on the first speech marker T20 and the second speech marker T2, which are both in the last position, to obtain a speech identifier loss value between the first speech marker T20 and the second speech marker T2.

[0192] In summary, the business server obtains at least two speech tag loss values. These two loss values ​​are then summed to obtain the network layer loss value for the initial network layer of speech prediction. Figure 7 The example network layer loss value 60g is 0.8. Based on the network layer loss value, the business server adjusts the model parameters in the initial speech prediction network layer to obtain the speech prediction network layer.

[0193] Step S209: The initial speech synthesis model containing the speech prediction network layer is determined as the speech synthesis model; the speech synthesis model is used to generate speech data corresponding to the input text.

[0194] Specifically, to evaluate the performance of the language-independent speech tagging-based LM-based TTS model (referred to as the speech synthesis model in this application) proposed in the embodiments of this application, the inventors used a multilingual dataset containing six languages. The Chinese data comes from Wenetspeech4TTS, a Chinese speech dataset specifically designed for training and evaluating large-scale speech generation models, with a duration of 945 hours. The English data comes from LibriTTS, a multilingual English language dataset, with approximately 585 hours of English speech read at a 24kHz sampling rate. For other languages, the inventors used German, French, Japanese, and Korean data from the Emilia dataset, with durations of 1590 hours, 1381 hours, 1715 hours, and 217 hours, respectively. The Emilia dataset is a multilingual speech generation dataset containing over 101,000 hours of speech data, covering six languages: Chinese, English, German, French, Japanese, and Korean. The Emilia dataset primarily consists of authentic natural speech from the internet, including talk shows, interviews, debates, sports commentary, and audiobooks, ensuring that the dataset captures a wide range of authentic human speaking styles. To demonstrate that speech tags are language-independent, Japanese and Korean in the second training sample set are languages ​​not encountered in the ASR task.

[0195] When evaluating the speech synthesis model, the inventors used the same model configuration as the Cosyvoice model. Furthermore, to prevent information leakage, the present application embodiments eliminated speaker embeddings from the language model (LM). For the timbre converter, the present application embodiments used OpenVoice (an open-source, high-efficiency multilingual voice cloning and text-to-speech (TTS) system). Regarding speaker embeddings (hereinafter referred to as timbre vectors), the present application embodiments used the campplus model (Context-Aware Multi-Scale Phoneme-Level Prosody Model, an advanced model designed for high-quality text-to-speech (TTS) that significantly improves the naturalness and expressiveness of synthesized speech by combining context awareness and multi-scale prosodic modeling, especially excelling in handling complex intonation, emotional expression, and multilingual scenarios) to extract them from the speech waveform. In the inventors' experiments, the inventors compared the following three models:

[0196] • Token: A TTS model trained using speech tokens extracted from an ASR model targeting BPE tokens.

[0197] • Language-Independent Speech Token (LIST): The model proposed in this application uses speech tokens extracted from an ASR model targeting IPA.

[0198] •LIST-Timbre Converter (tc): The optimized model proposed in this application uses a timbre converter to decouple and enhance the speaker.

[0199] To test the performance of the multilingual TTS model, the inventors prepared an objective evaluation set and a subjective evaluation set for each language. The objective evaluation set was used to test various objective metrics of the model. For Chinese and English, the inventors used the open-source benchmark from SEED, which contained 2000 and 1000 samples respectively. For the other four languages, the inventors extracted 1000 samples from the test set of the Common Voice dataset for each language. The subjective evaluation set was used for subjective listening tests because it has higher expressiveness and more diverse styles. It contains approximately 100 samples for each language. Furthermore, to test the model's performance in cross-language scenarios, the inventors also constructed a cross-language test set using test sets from different languages.

[0200] The inventor used three objective indicators:

[0201] 1) Character Error Rate (CER) is used to assess intelligibility. For languages ​​other than Chinese, the inventors used whisper-large-v3 (an open-source multilingual automatic speech recognition model, which has become one of the benchmark models in academia and industry due to its strong multilingual support, high accuracy, and robustness) for transcription of the generated speech. For Chinese, Paraformer-zh (a Chinese speech recognition model mainly used for processing Chinese speech data) was used. Then, the CER was calculated.

[0202] 2) SIM (Speaker Similarity) is obtained by calculating the cosine similarity between the speaker embeddings (referred to as timbre vectors in this application) of the generated speech and the reference speech. These embeddings are extracted using a speaker verification model finely tuned on WavLM3 (a novel speech pre-training model developed based on the Denoising Masked Speech Modeling (DMSM) framework, which is pre-trained on 94,000 hours of English speech data).

[0203] 3) UTMOS4, used to evaluate the objective quality of speech. UTMOS4 stands for Unified Test-to-MOSPrediction System Version 4. It is an advanced model for objectively evaluating the output quality of speech synthesis (TTS) or speech conversion systems. It quantifies the naturalness and clarity of speech by predicting the Mean Opinion Score (MOS).

[0204] The objective evaluation results for zero-shot and cross-lingual zero-shot TTS are shown below: In terms of CER, LIST and LIST-tc outperform Token, indicating that speech tokens extracted with IPA as the target contain more accurate semantic information. LIST-tc has a slightly higher CER than LIST, which may be because timbre conversion affects content information.

[0205] From the perspective of the SIM metric, LIST shows a significant improvement over Token. Furthermore, LIST-tc can further improve speaker similarity, which also demonstrates the superiority of the speech tagging proposed in this application. Similarly, the UTMOS metric shows the same trend as the SIM metric.

[0206] The inventors used a paired t-test to measure the significance of the differences between any two of the three models. LIST significantly outperformed Token in all cases (p<0.05). This indicates that the method proposed in this application fully utilizes the unified phonetic features of different languages ​​in IPA, enabling phonetic tokens to contain more language-independent semantic information, thereby improving performance. Meanwhile, LIST-tc also generally outperformed Token (p<0.05). Compared to LIST, LIST-tc typically showed a slight but insignificant improvement (p>0.05).

[0207] The inventors conducted subjective listening tests to evaluate the mean opinion score (MOS) of the synthesized speech. For each language, the inventors randomly selected 20 samples from the subjective evaluation set for zero-shot synthesis. The inventors then conducted subjective listening tests on the crowdsourcing platform Amazon Mechanical Turk5, with approximately 15 native speakers participating in each language group. These native speakers first listened to the prompt audio, and then rated the synthesized speech on a scale of 1 to 5 (in increments of 0.5) based on speech quality, expressiveness, and similarity to the speaker in the prompt audio.

[0208] Specifically, the input text is obtained and fed into the speech synthesis model. The speech synthesis model refers to the initial speech synthesis model that meets the model convergence condition. The speech synthesis model includes a speech prediction network layer and a stream matching network layer. In the text encoder of the speech prediction network layer, the input text is encoded to obtain a text encoding vector, which is then fed into the speech tagging network layer of the speech prediction network layer. In the speech tagging network layer, the text encoding vector is processed for speech tag prediction to obtain a second speech tag sequence. The second speech tag sequence is then fed into the stream matching network layer, where the first random noise is denoised based on the second speech tag sequence to obtain a Mel spectrogram. The Mel spectrogram is then processed for waveform reconstruction to obtain the synthesized speech corresponding to the input text.

[0209] Please see also Figure 8 , Figure 8 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 5 .like Figure 8 As shown, the business server obtains the input text 80z. This embodiment does not limit the language and content of the input text 80z; these should be determined based on the actual application scenario. Figure 8 Let's take the input text 80z as an example, "Let's go eat." The business server inputs the input text 80z into the speech synthesis model, which includes a speech prediction network layer and a stream matching network layer. In the text encoder of the speech prediction network layer, the business server encodes the input text 80z to obtain the text encoding vector 80x. The sequence start marker 801y and the text encoding vector 80x are then input into the speech tagging network layer of the speech prediction network layer. In the speech tagging network layer, the business server performs speech tag prediction processing on the text encoding vector 80x to obtain the second speech tag sequence 80w and the sequence end marker 802y. Figure 8 The second speech marker sequence 80w is exemplified as [T4, T47, T35, ..., T2].

[0210] Furthermore, the service server inputs the second speech tag sequence 80w into the stream matching network layer. In the stream matching network layer, the first random noise is denoised based on the second speech tag sequence 80w to obtain a Mel spectrogram. This embodiment does not limit the generation method of the random noise (including the first random noise and the second random noise described below), and can be set according to the actual application scenario. The service server performs waveform reconstruction processing on the Mel spectrogram to obtain the synthesized speech 80p corresponding to the input text 80z. It can be understood that the synthesized speech 80p is the Chinese phrase "Let's go eat."

[0211] This step may further include the following: acquiring a third sample speech, inputting the third sample speech into a timbre converter, modifying the timbre of the third sample speech through the timbre converter to obtain timbre-modified speech; inputting the timbre-modified speech into a speech tagger, performing speech tag prediction processing on the timbre-modified speech through the speech tagger to obtain a third speech tag sequence; acquiring the timbre vector corresponding to the third sample speech, inputting the timbre vector and the third speech tag sequence into the initial network layer of the stream matching; in the initial network layer of the stream matching, performing denoising processing on the third sample speech with added second random noise based on the third speech tag sequence and the timbre vector to obtain an estimated Mel spectrogram; adjusting the model parameters in the initial network layer of the stream matching based on the estimated Mel spectrogram and the third sample speech to obtain the stream matching network layer.

[0212] The specific process of adjusting the model parameters in the initial network layer of the stream matching network layer based on the estimated Mel spectrogram and the third sample speech to obtain the stream matching network layer may include: converting the third sample speech into a sample Mel spectrogram, determining the Mel spectrum difference between the sample Mel spectrogram and the estimated Mel spectrogram; determining the network layer loss value of the initial network layer of the stream matching network layer based on the Mel spectrum difference; and adjusting the model parameters in the initial network layer of the stream matching network layer based on the network layer loss value to obtain the stream matching network layer.

[0213] The training process of the initial network layer for speech prediction in the initial speech synthesis model has been described above. It is independent of the training process of the initial network layer for stream matching. The training process of the initial network layer for speech prediction has been described in detail above. The training process of the initial network layer for stream matching will be described below.

[0214] This application does not limit the number of third sample speech instances, which can be determined according to the actual application scenario. It is understood that if there are multiple third sample speech instances, the training process of the initial network layer for each third sample speech instance is the same. Similarly, the processing process for each third sample speech instance through the initial network layer for the flow matching is the same. Therefore, this description uses a single third sample speech instance as an example.

[0215] Please see also Figure 9 , Figure 9 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. Figure 6 .like Figure 9As shown, the business server obtains the third sample voice 80a and inputs it into a voice converter. The voice converter modifies the voice of the third sample voice 80a to obtain voice-modified voice 80b. It can be understood that the content expressed by voice-modified voice 80b is the same as that expressed by the third sample voice 80a, for example, both are "Let's go eat," but the two audios correspond to different voices. For example, the third sample voice 80a is the voice of person 1, while voice-modified voice 80b is the voice of person 2, which is different from person 1.

[0216] The service server inputs the timbre-modified speech 80b into the speech tagger, which then performs speech tag prediction processing on the timbre-modified speech 80b to obtain the third speech tag sequence 80c. Figure 9 The third speech tag sequence 80c is exemplified as [T42, T87, T83, ..., T20]. The service server obtains the timbre vector corresponding to the third sample speech 80b, and inputs the timbre vector and the third speech tag sequence 80c into the stream matching initial network layer. In the stream matching initial network layer, based on the third speech tag sequence 80c and the timbre vector, the service server performs denoising processing on the third sample speech 80a with added second random noise to obtain the estimated Mel spectrogram, as shown below. Figure 9 As shown, the service server converts the third sample speech 80a into a sample Mel spectrogram, determines the Mel spectrum difference between the sample Mel spectrogram and the estimated Mel spectrum, determines the network layer loss value of the initial network layer for stream matching based on the Mel spectrum difference, and adjusts the model parameters in the initial network layer for stream matching based on the network layer loss value to obtain the stream matching network layer.

[0217] In summary, this application proposes a method to help decouple speakers during the training of a stream matching model. Before extracting speech tags, this method modifies the timbre attributes of sample speech using a timbre converter, extracting the Mel spectrogram from the unmodified speech (i.e., the timbre vector). In this way, the initial network layer of the stream matching is forced to obtain speaker information from the provided speaker embeddings rather than from speech tags, which can achieve higher speaker similarity and naturalness.

[0218] As described above, this application proposes using the International Phonetic Alphabet (IPA) as the training target for a speech tokenizer. This allows the speech tokenizer to fully utilize the consistency of pronunciation across multiple languages ​​(e.g., English, Chinese) to learn language-independent semantic information (i.e., speech tokens). By focusing on unified pronunciation (i.e., IPA), it is more suitable for training multilingual and cross-lingual speech tokens in speech recognition, thus improving the applicability and accuracy of multilingual and cross-lingual speech tokens, and consequently improving the accuracy and applicability of multilingual and cross-lingual speech recognition. By training a highly accurate and highly applicable speech tokenizer, the accuracy and applicability of the speech token sequence output by the speech tokenizer can be improved. By using the highly accurate and highly applicable speech token sequence as training labels to train the initial speech synthesis model, the model training efficiency and model accuracy of the initial speech synthesis model can be improved. Furthermore, through a highly accurate and highly applicable speech synthesis model, the accuracy and applicability of the synthesized speech can be improved during speech synthesis. This application also proposes a speaker decoupling method during the stream matching training process. Specifically, before extracting speech tags, a timbre converter is used to modify the timbre attributes of the sample speech, predicting the Mel spectrogram extracted from the unchanged speech. In this way, stream matching is forced to obtain speaker information from the provided speaker embedding (i.e., timbre vector) and the sample Mel spectrogram, rather than from the speech tags, which can improve the speaker similarity and naturalness of the synthesized speech.

[0219] Further, please see Figure 10 , Figure 10 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. The data processing apparatus 1 described above can be used to execute the corresponding steps in the method provided in the embodiment of this application. For example... Figure 10 As shown, the data processing device 1 may include: an acquisition module 11, a processing module 12, and an adjustment module 13.

[0220] The acquisition module 11 is used to acquire a first training sample set and input the first training sample set into the speech recognition initial model; the first training sample set includes first sample speech carrying international phonetic symbol sequence labels; the international phonetic symbols in the international phonetic symbol sequence labels are used to describe the pronunciation of multiple languages ​​through unified phonetic symbols;

[0221] Processing module 12 is used to perform speech tag prediction processing on the first sample speech in the initial speech tagger of the initial speech recognition model to obtain a speech tag estimation sequence.

[0222] The processing module 12 is also used to perform tag decoding processing on the speech tag estimation sequence in the initial tag decoder of the initial tag of the speech recognition initial model to obtain the International Phonetic Alphabet estimation sequence;

[0223] The adjustment module 13 is used to adjust the model parameters in the initial speech tagger according to the IPA estimation sequence and the IPA sequence label to obtain the speech tagger; the speech tagger is used to generate the first speech tag sequence of the second sample speech when training the initial speech synthesis model, and the first speech tag sequence is used as the training label of the initial speech synthesis model.

[0224] In one possible implementation, the initial speech tagger of the initial speech recognition model includes a convolutional encoder, an initial tag encoder, and an initial vector quantizer.

[0225] In the initial speech tagger of the initial speech recognition model, the processing module 12 performs speech tag prediction processing on the first sample speech to obtain a speech tag estimation sequence, which is used to perform the following operations:

[0226] Acoustic features are extracted from the first sample speech using a convolutional encoder to obtain continuous semantic features, which are then input into the initial label encoder.

[0227] The continuous semantic features are fused with contextual features by an initial label encoder to obtain continuous semantic enhancement features, which are then input into an initial vector quantizer.

[0228] The continuous semantic enhancement features are processed by an initial vector quantizer to obtain discrete speech tags. The discrete speech tags are then combined into a speech tag estimation sequence according to the audio frame order of the first sample speech.

[0229] In one possible implementation, the processing module 12 performs feature recognition processing on the continuous semantic enhancement features through an initial vector quantizer to obtain discrete speech tokens, which are used to perform the following operations:

[0230] By using an initial vector quantizer, the continuous semantic enhancement features are compared with at least two vectors in the codebook to obtain the similarity values ​​corresponding to the at least two vectors respectively.

[0231] The maximum similarity value is obtained from at least two similarity values, and the speech tag corresponding to the vector with the maximum similarity value among at least two vectors is determined as the discrete speech tag of continuous semantic enhancement feature.

[0232] In one possible implementation, the initial tag decoder includes an initial optimization encoder and an initial decoder;

[0233] In the initial tag decoder of the initial speech recognition model, processing module 12 performs tag decoding processing on the speech tag estimation sequence to obtain the International Phonetic Alphabet (IPA) estimation sequence, which is used to perform the following operations:

[0234] The initial optimized encoder performs semantic enhancement processing on the speech tag estimation sequence to obtain continuous semantic optimization features, which are then input into the initial decoder.

[0235] The continuous semantic optimization features are decoded using the initial decoder to obtain the IPA estimation sequence.

[0236] In one possible implementation, the adjustment module 13 adjusts the model parameters in the initial speech tokenizer based on the IPA estimation sequence and the IPA sequence labels to obtain a speech tokenizer for performing the following operations:

[0237] Based on the IPA estimation sequence and IPA sequence labels, determine the model loss value of the initial speech recognition model;

[0238] Based on the model loss value, the model parameters of the initial speech tokenizer and the initial token decoder in the initial speech recognition model are adjusted;

[0239] When the initial speech recognition model, which includes the adjusted initial speech tokenizer and the adjusted initial tokenizer decoder, satisfies the model convergence condition, the adjusted initial speech tokenizer is determined as the speech tokenizer.

[0240] In one possible implementation, module 11 is also used to perform the following operations:

[0241] Obtain the second training sample set; the second training sample set includes sample text and the corresponding second sample speech.

[0242] The second sample speech is input into the speech tagger, and the speech tagger performs speech tag prediction processing on the second sample speech to obtain the first speech tag sequence.

[0243] The sample text and the first speech tag sequence are input into the speech synthesis initial model. In the speech prediction initial network layer of the speech synthesis initial model, the sample text is processed for speech tag prediction based on the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text.

[0244] Based on the predicted speech tag sequence and the first speech tag sequence, the model parameters in the initial speech prediction network layer are adjusted to obtain the speech prediction network layer.

[0245] The initial speech synthesis model, which includes a speech prediction network layer, is defined as the speech synthesis model; the speech synthesis model is used to generate speech data corresponding to the input text.

[0246] In one possible implementation, the initial network layer for speech prediction includes an initial text encoder and a speech tagging initial network layer;

[0247] In the initial speech prediction network layer of the initial speech synthesis model, module 11 performs speech tag prediction processing on the sample text based on the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text, which is used to perform the following operations:

[0248] The sample text is encoded by the initial text encoder to obtain the text encoding estimation vector, and the text encoding estimation vector is input into the initial speech tag network layer.

[0249] In the initial speech tag network layer, the text encoding estimation vector is processed by speech tag prediction based on the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text.

[0250] In one possible implementation, the first speech tag sequence includes at least two sequentially arranged first speech tags; the predicted speech tag sequence includes at least two sequentially arranged second speech tags;

[0251] The acquisition module 11 adjusts the model parameters in the initial speech prediction network layer based on the predicted speech tag sequence and the first speech tag sequence to obtain the speech prediction network layer, which is used to perform the following operations:

[0252] Loss processing is performed on the first speech tag sequence and the predicted speech tag sequence, where the first speech tag and the second speech tag have the same arrangement position, to obtain at least two speech tag loss values;

[0253] Summing at least two speech tag loss values ​​yields the network layer loss value of the initial speech prediction network layer.

[0254] Based on the network layer loss value, the model parameters in the initial network layer of speech prediction are adjusted to obtain the speech prediction network layer.

[0255] In one possible implementation, module 11 is also used to perform the following operations:

[0256] The input text is obtained and fed into the speech synthesis model. The speech synthesis model refers to the initial speech synthesis model that meets the convergence conditions. The speech synthesis model includes a speech prediction network layer and a stream matching network layer.

[0257] In the text encoder of the speech prediction network layer, the input text is encoded to obtain a text encoding vector, and the text encoding vector is input into the speech tag network layer of the speech prediction network layer.

[0258] In the speech tagging network layer, speech tag prediction processing is performed on the text encoding vector to obtain the second speech tag sequence;

[0259] The second speech tag sequence is input into the stream matching network layer. In the stream matching network layer, the first random noise is denoised according to the second speech tag sequence to obtain the Mel spectrogram.

[0260] The waveform reconstruction process is performed on the Mel spectrogram to obtain the synthesized speech corresponding to the input text.

[0261] In one possible implementation, module 11 is also used to perform the following operations:

[0262] The third sample speech is obtained and input into the timbre converter. The timbre of the third sample speech is modified by the timbre converter to obtain the timbre-modified speech.

[0263] The timbre-modified speech is input into the speech tagger, and the speech tagger performs speech tag prediction processing on the timbre-modified speech to obtain the third speech tag sequence.

[0264] Obtain the timbre vector corresponding to the third sample speech, and input the timbre vector and the third speech tag sequence into the initial network layer of the stream matching network;

[0265] In the initial network layer of the stream matching, the third sample speech with added second random noise is denoised based on the third speech tag sequence and timbre vector to obtain the estimated Mel spectrogram;

[0266] Based on the estimated Mel spectrogram and the third sample speech, the model parameters in the initial network layer of the stream matching network are adjusted to obtain the stream matching network layer.

[0267] In one possible implementation, the acquisition module 11 adjusts the model parameters in the initial network layer of the stream matching network based on the estimated Mel spectrogram and the third sample speech, to obtain the stream matching network layer, which is used to perform the following operations:

[0268] The third sample speech is converted into a sample Mel spectrogram, the sample Mel spectrogram is determined, and the Mel spectrum difference between the Mel spectrograms is estimated.

[0269] The network layer loss value of the initial flow matching network layer is determined based on the Mel spectrum difference. The model parameters in the initial flow matching network layer are then adjusted based on the network layer loss value to obtain the flow matching network layer.

[0270] In this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of a larger module or unit that includes the functionality of that module or unit.

[0271] As described above, this application proposes using the International Phonetic Alphabet (IPA) as the training target for a speech tokenizer. This allows the speech tokenizer to fully utilize the consistency of pronunciation across multiple languages ​​(e.g., English, Chinese) to learn language-independent semantic information (i.e., speech tokens). By focusing on unified pronunciation (i.e., IPA), it is more suitable for training multilingual and cross-lingual speech tokens in speech recognition, thus improving the applicability and accuracy of multilingual and cross-lingual speech tokens, and consequently improving the accuracy and applicability of multilingual and cross-lingual speech recognition. By training a highly accurate and highly applicable speech tokenizer, the accuracy and applicability of the speech token sequence output by the speech tokenizer can be improved. By using the highly accurate and highly applicable speech token sequence as training labels to train the initial speech synthesis model, the model training efficiency and model accuracy of the initial speech synthesis model can be improved. Furthermore, through a highly accurate and highly applicable speech synthesis model, the accuracy and applicability of the synthesized speech can be improved during speech synthesis. This application also proposes a speaker decoupling method during the stream matching training process. Specifically, before extracting speech tags, a timbre converter is used to modify the timbre attributes of the sample speech, predicting the Mel spectrogram extracted from the unchanged speech. In this way, stream matching is forced to obtain speaker information from the provided speaker embedding (i.e., timbre vector) and the sample Mel spectrogram, rather than from the speech tags, which can improve the speaker similarity and naturalness of the synthesized speech.

[0272] Further, please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be... Figure 1 At least one of the terminal devices or service servers shown. For example... Figure 11 As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components.

[0273] In some embodiments, the user interface 1003 may include a display screen and a keyboard, and the network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001.

[0274] like Figure 11 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0275] exist Figure 11 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0276] Obtain the first training sample set and input it into the initial speech recognition model; the first training sample set includes first sample speech carrying International Phonetic Alphabet (IPA) sequence labels; the IPA in the IPA sequence labels is used to describe the pronunciation of multiple languages ​​using unified phonetic symbols;

[0277] In the initial speech tagger of the initial speech recognition model, the first sample speech is processed for speech tag prediction to obtain the speech tag estimation sequence;

[0278] In the initial tag decoder of the initial speech recognition model, the speech tag estimation sequence is processed by tag decoding to obtain the International Phonetic Alphabet (IPA) estimation sequence;

[0279] Based on the IPA estimation sequence and IPA sequence labels, the model parameters in the initial speech tagger are adjusted to obtain the speech tagger. The speech tagger is used to generate the first speech tag sequence of the second sample speech when training the initial speech synthesis model. The first speech tag sequence is used as the training label of the initial speech synthesis model.

[0280] It should be understood that the computer device 1000 described in the embodiments of this application can perform the data processing methods or apparatus described in the preceding embodiments, and will not be repeated here. Furthermore, the beneficial effects of using the same methods will also not be repeated.

[0281] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the data processing methods or apparatus described in the preceding embodiments, which will not be repeated here. Furthermore, the beneficial effects of using the same methods will also not be repeated.

[0282] The aforementioned computer-readable storage medium may be the data processing apparatus provided in any of the foregoing embodiments or the internal storage unit of the aforementioned computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device.

[0283] Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0284] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, enabling the computer device to perform the data processing methods or apparatus described in the preceding embodiments, which will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0285] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0286] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0287] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the first training sample set and input the first training sample set into the initial speech recognition model; The first training sample set includes first sample speech carrying International Phonetic Alphabet sequence labels; The International Phonetic Alphabet (IPA) symbols in the sequence label are used to describe the pronunciation of multiple languages ​​using standardized phonetic symbols; In the initial speech tagger of the initial speech recognition model, the first sample speech is subjected to speech tag prediction processing to obtain a speech tag estimation sequence; In the initial tag decoder of the initial speech recognition model, the speech tag estimation sequence is processed by tag decoding to obtain the International Phonetic Alphabet (IPA) estimation sequence; Based on the estimated International Phonetic Alphabet (IPA) sequence and the IPA sequence label, the model parameters in the initial speech tagger are adjusted to obtain a speech tagger; the speech tagger is used to generate a first speech tag sequence of the second sample speech when training the initial speech synthesis model, and the first speech tag sequence is used as the training label of the initial speech synthesis model.

2. The method according to claim 1, characterized in that, The initial speech tagger of the initial speech recognition model includes a convolutional encoder, an initial tag encoder, and an initial vector quantizer; In the initial speech tagger of the initial speech recognition model, the first sample speech is subjected to speech tag prediction processing to obtain a speech tag estimation sequence, including: The first sample speech is subjected to acoustic feature extraction by the convolutional encoder to obtain continuous semantic features, and the continuous semantic features are input into the initial label encoder. The continuous semantic features are fused with contextual features by the initial label encoder to obtain continuous semantic enhancement features, and the continuous semantic enhancement features are input into the initial vector quantizer; The continuous semantic enhancement features are processed by the initial vector quantizer to obtain discrete speech tags. The discrete speech tags are then combined into a speech tag estimation sequence according to the audio frame order of the first sample speech.

3. The method according to claim 2, characterized in that, The step of performing feature recognition processing on the continuous semantic enhancement features through the initial vector quantizer to obtain discrete speech tags includes: The initial vector quantizer is used to perform similarity processing on the continuous semantic enhancement features with at least two vectors in the codebook to obtain the similarity values ​​corresponding to the at least two vectors respectively. The maximum similarity value is obtained from at least two similarity values, and the speech tag corresponding to the vector with the maximum similarity value among the at least two vectors is determined as the discrete speech tag of the continuous semantic enhancement feature.

4. The method according to claim 1, characterized in that, The initial tag decoder includes an initial optimized encoder and an initial decoder; In the initial tag decoder of the initial speech recognition model, the speech tag estimation sequence is subjected to tag decoding processing to obtain the International Phonetic Alphabet (IPA) estimation sequence, including: The initial optimized encoder performs semantic enhancement processing on the speech tag estimation sequence to obtain continuous semantic optimization features, and inputs the continuous semantic optimization features into the initial decoder; The continuous semantic optimization features are decoded using the initial decoder to obtain the International Phonetic Alphabet (IPA) estimation sequence.

5. The method according to claim 1, characterized in that, The step of adjusting the model parameters in the initial speech tokenizer based on the estimated IPA sequence and the IPA sequence labels to obtain the speech tokenizer includes: Based on the estimated International Phonetic Alphabet sequence and the International Phonetic Alphabet sequence label, determine the model loss value of the initial speech recognition model; Based on the model loss value, the model parameters of the initial speech tokenizer and the initial token decoder in the initial speech recognition model are adjusted; When the initial speech recognition model, which includes the adjusted initial speech tokenizer and the adjusted initial tokenizer decoder, satisfies the model convergence condition, the adjusted initial speech tokenizer is determined as the speech tokenizer.

6. The method according to claim 1, characterized in that, The method further includes: Obtain a second training sample set; the second training sample set includes sample text and the second sample speech corresponding to the sample text; The second sample speech is input into the speech tagger, and the speech tagger performs speech tag prediction processing on the second sample speech to obtain the first speech tag sequence. The sample text and the first speech tag sequence are input into the speech synthesis initial model. In the speech prediction initial network layer of the speech synthesis initial model, the sample text is processed for speech tag prediction based on the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text. Based on the predicted speech tag sequence and the first speech tag sequence, the model parameters in the initial speech prediction network layer are adjusted to obtain the speech prediction network layer. The initial speech synthesis model containing the speech prediction network layer is determined as the speech synthesis model; the speech synthesis model is used to generate speech data corresponding to the input text.

7. The method according to claim 6, characterized in that, The initial network layer for speech prediction includes an initial text encoder and a speech tagging initial network layer; In the initial speech prediction network layer of the initial speech synthesis model, the sample text is processed for speech tag prediction based on the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text, including: The sample text is encoded by the initial text encoder to obtain a text encoding estimation vector, and the text encoding estimation vector is input into the initial speech tag network layer. In the initial speech tag network layer, the text encoding estimation vector is processed by speech tag prediction based on the first speech tag sequence to obtain the predicted speech tag sequence corresponding to the sample text.

8. The method according to claim 6, characterized in that, The first speech tag sequence includes at least two sequentially arranged first speech tags; the predicted speech tag sequence includes at least two sequentially arranged second speech tags; The step of adjusting the model parameters in the initial speech prediction network layer based on the predicted speech tag sequence and the first speech tag sequence to obtain the speech prediction network layer includes: Loss processing is performed on the first speech tag sequence and the predicted speech tag sequence, where the first speech tag and the second speech tag have the same arrangement position, to obtain at least two speech tag loss values; The at least two speech tag loss values ​​are summed to obtain the network layer loss value of the initial speech prediction network layer. Based on the network layer loss value, the model parameters in the initial speech prediction network layer are adjusted to obtain the speech prediction network layer.

9. The method according to claim 1, characterized in that, The method further includes: The input text is obtained and fed into the speech synthesis model; the speech synthesis model refers to the initial speech synthesis model that meets the model convergence condition; the speech synthesis model includes a speech prediction network layer and a stream matching network layer. In the text encoder of the speech prediction network layer, the input text is encoded to obtain a text encoding vector, and the text encoding vector is input to the speech tagging network layer of the speech prediction network layer. In the speech tag network layer, the text encoding vector is subjected to speech tag prediction processing to obtain a second speech tag sequence; The second speech tag sequence is input into the stream matching network layer. In the stream matching network layer, the first random noise is denoised according to the second speech tag sequence to obtain the Mel spectrogram. The Mel spectrogram is subjected to waveform reconstruction processing to obtain the synthesized speech corresponding to the input text.

10. The method according to claim 9, characterized in that, The method further includes: A third sample speech is obtained, and the third sample speech is input into a timbre converter. The timbre of the third sample speech is modified by the timbre converter to obtain timbre-modified speech. The timbre-modified speech is input into the speech tagger, and the speech tagger performs speech tag prediction processing on the timbre-modified speech to obtain a third speech tag sequence. Obtain the timbre vector corresponding to the third sample speech, and input the timbre vector and the third speech tag sequence into the stream matching initial network layer; In the initial network layer of the stream matching, the third sample speech with added second random noise is denoised according to the third speech tag sequence and the timbre vector to obtain the estimated Mel spectrogram; Based on the estimated Mel spectrogram and the third sample speech, the model parameters in the initial network layer of the stream matching are adjusted to obtain the stream matching network layer.

11. The method according to claim 10, characterized in that, The step of adjusting the model parameters in the initial network layer of the stream matching network based on the estimated Mel spectrogram and the third sample speech to obtain the stream matching network layer includes: The third sample speech is converted into a sample Mel spectrogram, and the Mel spectrum difference between the sample Mel spectrogram and the estimated Mel spectrogram is determined. The network layer loss value of the initial flow matching network layer is determined based on the Mel spectrum difference. The model parameters in the initial flow matching network layer are adjusted based on the network layer loss value to obtain the flow matching network layer.

12. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire a first training sample set and input the first training sample set into the speech recognition initial model. The first training sample set includes first sample speech carrying International Phonetic Alphabet (IPA) sequence labels; the IPA symbols in the IPA sequence labels are used to describe the pronunciation of multiple languages ​​using standardized phonetic symbols. The processing module is used to perform speech tag prediction processing on the first sample speech in the initial speech tagger of the initial speech recognition model to obtain a speech tag estimation sequence. The processing module is further configured to perform tag decoding processing on the speech tag estimation sequence in the initial tag decoder of the initial speech recognition model to obtain the International Phonetic Alphabet (IPA) estimation sequence; An adjustment module is used to adjust the model parameters in the initial speech tagger according to the estimated International Phonetic Alphabet sequence and the International Phonetic Alphabet sequence label to obtain a speech tagger; the speech tagger is used to generate a first speech tag sequence of the second sample speech when training the initial speech synthesis model, and the first speech tag sequence is used as the training label of the initial speech synthesis model.

13. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-11.

15. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, the computer program being adapted to be read and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-11.

Images