Model training method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The application provides a model training method and device, electronic equipment, computer readable storage medium and computer program product; the method comprises: obtaining a training sample, and constructing an input sequence based on the training sample; extracting features of the input sequence through an audio synthesis model to be trained to obtain a shared hidden state sequence; performing audio prediction on the shared hidden state sequence to obtain an audio prediction result, and performing phoneme prediction on the shared hidden state sequence to obtain a phoneme prediction result; determining an audio loss value based on the audio prediction result and an audio label sequence corresponding to the training sample, and determining a phoneme loss value based on the phoneme prediction result and a phoneme label sequence corresponding to the training sample; updating model parameters of the audio synthesis model to be trained based on the audio loss value and the phoneme loss value to obtain a trained audio synthesis model. Through the application, the alignment accuracy of audio and phonemes output by the audio synthesis model can be improved.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a model training method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In the field of intelligent voice and multimodal interaction, audio synthesis models are widely used to convert input information into corresponding audio data.

[0003] In related technologies, in practical applications such as lip-syncing or pronunciation evaluation, downstream tasks often require extracting phoneme sequences from input information and audio data output by audio synthesis models using alignment tools. However, the method of extracting phoneme sequences using alignment tools is limited by the matching error between the pronunciation structure and acoustic features at the time boundary, resulting in low alignment accuracy between the final obtained phoneme sequence and the audio data. Summary of the Invention

[0004] This application provides a model training method, apparatus, electronic device, computer-readable storage medium, and computer program product that can improve the alignment accuracy of audio and phonemes output by an audio synthesis model.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a model training method, the method comprising:

[0007] Obtain training samples and construct input sequences based on the training samples;

[0008] The input sequence is used to extract features through the audio synthesis model to be trained, thereby obtaining a shared hidden state sequence.

[0009] Audio prediction is performed on the shared hidden state sequence to obtain audio prediction results, and phoneme prediction is performed on the shared hidden state sequence to obtain phoneme prediction results;

[0010] Based on the audio prediction results and the audio label sequence corresponding to the training samples, the audio loss value is determined, and based on the phoneme prediction results and the phoneme label sequence corresponding to the training samples, the phoneme loss value is determined.

[0011] Based on the audio loss value and the phoneme loss value, the model parameters of the audio synthesis model to be trained are updated to obtain the trained audio synthesis model.

[0012] This application provides a model training apparatus, including:

[0013] An acquisition module is used to acquire training samples and construct an input sequence based on the training samples;

[0014] The extraction module is used to extract features from the input sequence using the audio synthesis model to be trained, to obtain a shared hidden state sequence;

[0015] The prediction module is used to perform audio prediction on the shared hidden state sequence to obtain audio prediction results, and to perform phoneme prediction on the shared hidden state sequence to obtain phoneme prediction results.

[0016] The determination module is used to determine the audio loss value based on the audio prediction result and the audio label sequence corresponding to the training sample, and to determine the phoneme loss value based on the phoneme prediction result and the phoneme label sequence corresponding to the training sample;

[0017] The update module is used to update the model parameters of the audio synthesis model to be trained based on the audio loss value and the phoneme loss value, so as to obtain the trained audio synthesis model.

[0018] This application provides an electronic device, the electronic device comprising:

[0019] Memory is used to store executable instructions or computer programs.

[0020] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the model training method provided in the embodiments of this application.

[0021] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the model training method provided in this application when executed by a processor.

[0022] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the model training method provided in this application.

[0023] The embodiments of this application have the following beneficial effects:

[0024] The input sequence is constructed based on training samples. The audio synthesis model to be trained is used to extract features from the input sequence to obtain a shared hidden state sequence. This allows subsequent audio prediction and phoneme prediction to be processed based on the same shared hidden state sequence to obtain audio prediction results and phoneme prediction results. The audio loss value and phoneme loss value are determined by combining the audio prediction result with the corresponding audio label sequence and the phoneme prediction result with the corresponding phoneme label sequence, respectively. The model parameters of the audio synthesis model to be trained are updated based on the audio loss value and the phoneme loss value. This allows the model parameters to learn the feature rules of audio and phonemes simultaneously during the update process, effectively improving the alignment accuracy of the trained audio synthesis model when outputting audio and phonemes. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the architecture of the model training system provided in the embodiments of this application;

[0026] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0027] Figure 3 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 1 ;

[0028] Figure 4 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 2 ;

[0029] Figure 5 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 3 ;

[0030] Figure 6 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 4 ;

[0031] Figure 7 This is a schematic diagram illustrating the principle of the phoneme loss value calculation process provided in the embodiments of this application;

[0032] Figure 8 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 5 ;

[0033] Figure 9 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 6 ;

[0034] Figure 10 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 7 ;

[0035] Figure 11 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 8 ;

[0036] Figure 12 This is a schematic diagram of the architecture of the audio synthesis system provided in the embodiments of this application;

[0037] Figure 13 This is a schematic diagram of the architecture of the phoneme prediction decoder extension mechanism provided in the embodiments of this application;

[0038] Figure 14 This is a flowchart illustrating a two-stage alignment strategy provided in an embodiment of this application;

[0039] Figure 15 This is a flowchart illustrating the hybrid training strategy provided in an embodiment of this application;

[0040] Figure 16 This is a schematic diagram of the dual-output and standardized storage mechanism for the inference stage provided in the embodiments of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0044] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0045] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0046] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0047] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0048] 1) Training samples: In the fields of machine learning and deep learning, these are samples used to optimize the parameters of an audio synthesis model. Training samples can include a continuous text data and audio data of real human pronunciation that corresponds to the text data.

[0049] 2) Input sequence: In the field of multimodal data processing, this refers to a one-dimensional tensor data structure obtained by concatenating multiple discrete feature representations according to a preset physical arrangement order. In this embodiment, the input sequence can be a sequence formed by concatenating the text feature tag sequence, speaker features, audio prompt sequence, and preset control tags obtained after extracting the above-mentioned training samples in a fixed arrangement order.

[0050] 3) Audio synthesis model: refers to a computer program entity built on a deep neural network architecture that can receive a specific input sequence and map the input sequence into a prediction result of the corresponding modality. In the embodiments of this application, the audio synthesis model can be a multi-branch network topology assembled from a large language model as the backbone of feature extraction, and audio prediction networks and phoneme prediction networks used for different modality prediction outputs.

[0051] 4) Ignore flag: In the field of computer data sequence processing and tensor computation, an ignore flag is a mask parameter or placeholder symbol with special numerical attributes pre-configured for data elements at a specific location. In this embodiment, the ignore flag is used to physically isolate non-audio regions in the target phoneme label sequence that do not have corresponding audio information when constructing the target phoneme label sequence for a phoneme prediction task. This triggers a masking mechanism in the loss calculation process, ensuring that the location configured with the ignore flag does not participate in the calculation of the phoneme loss value.

[0052] 5) Large Language Model (LLM): A deep neural network model based on a transformer architecture, used to model the context of input sequences and output a shared hidden state sequence. The LLM is the core backbone network of audio synthesis models.

[0053] 6) Shared hidden state sequence: refers to the intermediate feature representations output by the large language model at each layer or time step, which are used to carry the contextual information related to text and speech.

[0054] 7) Phoneme: The smallest unit of pronunciation in speech, used to describe the pronunciation structure of speech. Phonemes are represented in the form of discrete categories.

[0055] 8) Phone Decoder: A neural network module connected to the output of the hidden state of a large language model, used to map the shared hidden state sequence to a phoneme category probability distribution.

[0056] 9) Discrete audio features (Speech Tokens): These are discrete representations extracted from audio data through a speech discretization model, used for speech generation or speech modeling, and include multiple speech feature tokens.

[0057] 10) Multi-task joint training: refers to a training method that simultaneously introduces speech prediction task and phoneme prediction task in the same training process and jointly optimizes model parameters.

[0058] 11) Dual Prediction Head Collaborative Architecture: This refers to a joint architecture in which an audio prediction network and a phoneme prediction network are set up in parallel on the output side of a large language model. The two networks are based on a shared hidden state sequence and synchronously complete speech generation and phoneme prediction.

[0059] 12) Factory pattern extension mechanism: refers to the technical method of achieving non-intrusive extension of phoneme prediction network through registry, configuration creation and backward compatibility mechanism, so that the new phoneme prediction network structure does not need to modify the main training or inference process.

[0060] In related technologies, phoneme information is typically used only as a labeling aid during the training phase or as a latent variable within the model. Optimization efforts often focus on improving the naturalness of the final output audio data. Under this conventional processing logic, the inference phase often does not explicitly predict and output the structured phoneme sequence. Therefore, when facing downstream application requirements such as lip-syncing and pronunciation evaluation, it is often necessary to deploy an additional independent phoneme prediction module, thereby increasing the overall application architecture's deployment complexity and data processing latency. Secondly, when it is necessary to acquire both phoneme sequences and audio data simultaneously, a separate architecture is commonly adopted, deploying the phoneme prediction model and the audio generation model independently. In this separate processing architecture, different models operate independently at the feature extraction level. Due to the lack of unified feature correlation and constraints between the prediction of pronunciation rules and the generation of acoustic representations, temporal feature misalignment can easily occur between the final output phoneme sequence and the audio data, thus affecting the alignment accuracy of the multidimensional output results. Furthermore, related phoneme prediction modules often employ fixed network structures and are deeply coupled with the model training or inference process. This rigid coupling design often leads to invasive modifications to the overall code flow when switching network structures to meet diverse business scenarios. It also makes it difficult to achieve backward compatibility and reuse of model parameters from previous versions, increasing maintenance costs during model iteration.

[0061] To address the aforementioned problems, embodiments of this application provide a model training method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the alignment accuracy of the trained audio synthesis model when outputting audio and phonemes. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or it can be implemented as a server. The following will describe exemplary applications when the electronic device is implemented as a terminal or server.

[0062] See Figure 1 , Figure 1This is a schematic diagram of the architecture of a model training system provided in this application embodiment. To support a model training application, the model training system 100 includes at least a terminal 400, a network 300, and a server 200. The terminal 400 is connected to the server 200 through the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both. In some embodiments, the server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, which is not limited in this application embodiment.

[0063] In some embodiments, the present application embodiments can be implemented by the terminal 400 alone. For example, the terminal 400 runs a model training program (such as an audio synthesis model training framework), obtains training samples and the corresponding audio tag sequences and phoneme tag sequences, and constructs an input sequence based on the training samples; the terminal 400 extracts features from the input sequence using the audio synthesis model to be trained to obtain a shared hidden state sequence; the terminal 400 performs audio prediction on the shared hidden state sequence to obtain an audio prediction result, and performs phoneme prediction on the shared hidden state sequence to obtain a phoneme prediction result; the terminal 400 determines an audio loss value based on the audio prediction result and the audio tag sequence corresponding to the training samples, and determines a phoneme loss value based on the phoneme prediction result and the phoneme tag sequence corresponding to the training samples; the terminal 400 updates the model parameters of the audio synthesis model to be trained based on the audio loss value and the phoneme loss value to obtain a trained audio synthesis model. The terminal 400 can also use the trained audio synthesis model for inference, directly outputting audio waveforms and corresponding phonetic symbol sequences, and saving the phonetic symbol sequences as text files for direct reading by downstream applications.

[0064] In some embodiments, this application embodiment can be implemented collaboratively by a terminal 400 and a server 200. The user configures model training through the terminal 400. The terminal 400 receives the user-triggered model training instruction and selected training data, encapsulates the instruction and data into a model training request, and transmits it to the server 200 via the network 300. In response to the received model training request, the server 200 extracts features from the input sequence using the audio synthesis model to be trained, obtaining a shared hidden state sequence. The server 200 performs audio prediction on the shared hidden state sequence, obtaining an audio prediction result, and performs phoneme prediction on the same sequence, obtaining a phoneme prediction result. Based on the audio prediction result and the audio label sequence corresponding to the training samples, the server 200 determines the audio loss value, and based on the phoneme prediction result and the phoneme label sequence corresponding to the training samples, determines the phoneme loss value. Based on the audio loss value and the phoneme loss value, the server 200 updates the model parameters of the audio synthesis model to be trained, obtaining the trained audio synthesis model. The server 200 sends the trained audio synthesis model to the terminal 400, which receives and stores the trained audio synthesis model.

[0065] The model training method provided in this application can be applied to any scenario requiring high-precision audio generation and structured pronunciation feature extraction from multimodal interaction data. This significantly improves the alignment accuracy and consistency of multidimensional output results while reducing the complexity of the model deployment architecture. Specific application scenarios include:

[0066] 1) Game Character Interaction Scenarios. During game development, developers need to generate unique voices for non-player characters (NPCs) based on the game script text, and simultaneously generate animation data that drives the facial lip movements of the 3D model of the character. The server obtains training samples containing the game character's pronunciation materials, as well as audio label sequences and phoneme label sequences corresponding to the training samples, and constructs an input sequence based on the training samples; the server extracts features from the input sequence using the audio synthesis model to be trained, obtaining a shared hidden state sequence; the server performs audio prediction and phoneme prediction based on the shared hidden state sequence, obtaining audio prediction results and phoneme prediction results respectively; the server updates the model parameters of the audio synthesis model to be trained based on the audio prediction results and the audio label sequence determined by the audio loss value, and the phoneme loss value determined by the phoneme prediction results and the phoneme label sequence. When the terminal calls the trained audio synthesis model to process the game script text, it can simultaneously output highly natural audio waveforms and highly aligned phoneme sequences. The game engine receives this phoneme sequence and can directly and automatically generate mouth-shaped animations that are strictly synchronized with the voice for the game character.

[0067] 2) Intelligent Speech Teaching and Pronunciation Assessment Scenarios. In language learning applications, the platform needs to provide users with standard pronunciation examples and corresponding pronunciation structure comparisons. The server acquires standard teaching corpora as training samples and prepares corresponding audio label sequences and phoneme label sequences. After constructing the input sequence, the server uses the audio synthesis model to be trained to extract the shared hidden state sequence of the input sequence. The server synchronously outputs audio prediction results and phoneme prediction results for the same shared hidden state sequence, and combines the audio loss value and phoneme loss value to update the model parameters of the audio synthesis model to be trained, thus obtaining the trained audio synthesis model. After the terminal downloads the trained audio synthesis model, it can generate standard pronunciation data locally and use the output phoneme sequence as a comparison benchmark to accurately match the user's pronunciation to output pronunciation assessment results.

[0068] 3) Platform-level iterative training scenario for large-scale speech foundation models. When constructing a large-scale cloud-based speech model supporting multiple languages, continuous optimization using massive corpora is necessary. The server acquires massive multilingual corpora as training samples and generates corresponding audio label sequences and phoneme label sequences. After constructing the input sequence based on the training samples, the server performs unified feature extraction on the input sequence using the audio synthesis model to be trained, generating a shared hidden state sequence. The server then performs audio prediction and phoneme prediction based on the shared hidden state sequence, obtaining the audio prediction results and phoneme prediction results respectively. Next, the server combines the audio loss value and the phoneme loss value to jointly update the model parameters of the audio synthesis model to be trained. This training method effectively improves the utilization of server computational resources during multi-task joint training, resulting in a trained audio synthesis model with high-precision output capabilities.

[0069] 4) Automated Audio / Video Dubbing and Precise Subtitle Alignment Scenarios. In the automated video generation process, it's necessary not only to convert script text into dubbing but also to generate subtitle timelines aligned with the audio. The terminal acquires film and television dubbing materials as training samples, along with corresponding audio tag sequences and phoneme tag sequences, and constructs an input sequence based on the training samples. The terminal calls the audio synthesis model to be trained to process the input sequence, obtaining a shared hidden state sequence, and simultaneously acquires audio prediction results and phoneme prediction results based on the shared hidden state sequence. The terminal calculates audio loss values ​​and phoneme loss values ​​based on the prediction results and the corresponding audio tag sequences and phoneme tag sequences, thereby updating the model parameters of the audio synthesis model to be trained. Video editing applications, by calling the trained audio synthesis model, can simultaneously acquire the generated dubbing waveform and high-time-precision phoneme sequences, thus achieving automatic frame-level alignment between dubbing and subtitles.

[0070] In some embodiments, the electronic device implementing the model training method provided in this application may be Figure 1 Terminal 400 in the middle. See also Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components of the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0071] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0072] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0073] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0074] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0075] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0076] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0077] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0078] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.).

[0079] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0080] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A model training apparatus 455 stored in memory 450 is shown. This apparatus can be software in the form of programs and plugins, and includes the following software modules: acquisition module 4551, extraction module 4552, prediction module 4553, determination module 4554, and update module 4555. These modules are logically linked and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0081] The model training method provided in the embodiments of this application will be described below. As mentioned above, the electronic device implementing the model training method of the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0082] See Figure 3 , Figure 3 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 1 , will combine Figure 3 The steps shown are explained as follows: Figure 3 As shown, the method includes steps 101 to 105.

[0083] In step 101, training samples are obtained, and input sequences are constructed based on the training samples.

[0084] Here, training samples refer to the basic data units used to train the audio synthesis model. Training samples can include text data and the corresponding audio data. For example, training samples can include text data containing the Chinese word "pinyin" and the corresponding audio recording of a real person pronouncing the word. The audio synthesis model can be a generative model built on a deep learning network, used to convert input conditional information (such as text, timbre features, etc.) into audio data. The input sequence refers to the multimodal feature vector sequence obtained after preprocessing the data included in the training samples; this input sequence serves as the direct input to the audio synthesis model.

[0085] In some embodiments, the input sequence can be constructed based on training samples by: extracting text feature sequences, speaker features, and audio cue sequences from the training samples; obtaining preset control markers, wherein the control markers are used to characterize sequence boundary information and task category information in the input sequence to be generated; and fusing the text feature sequences, speaker features, audio cue sequences, and control markers to obtain the input sequence.

[0086] Here, the text feature sequence refers to the embedded representation sequence containing discrete semantic information obtained by mapping the text data in the training samples after word segmentation. Speaker features (or voiceprint features) refer to the static vector representations extracted from the audio data of the training samples, representing global timbre attributes or acoustic style. The audio cue sequence refers to the discrete acoustic feature label sequence formed after feature extraction from the audio data of the training samples. Sequence boundary information is used to characterize the start or end position of the input sequence in the physical tensor space. Control tags include boundary control characters corresponding to the sequence boundary information (e.g., start of sequence (SOS) and end of sequence (EOS)), used to trigger the initialization or termination logic of the autoregressive generation process of the large language model. Task category information is used to characterize the specific prediction task type that needs to be performed in the current multimodal data stream (e.g., audio prediction task). Control tags include task control characters (Task ID) corresponding to the task category information, used to guide the audio synthesis model to distinguish different task modes within a unified weight space, thereby performing the correct attention routing calculation.

[0087] In some embodiments, the text data in the training samples is sliced ​​based on a word segmenter to obtain multiple discrete text feature tokens. The obtained multiple text feature tokens are input into a pre-trained embedding layer, and a dense semantic feature tensor is obtained through embedding mapping operations. This dense semantic feature tensor is then determined as a text feature sequence.

[0088] In some embodiments, acoustic features are extracted from audio data in training samples based on a voiceprint extraction network. The voiceprint extraction network extracts static feature vectors that represent global timbre attributes or acoustic style by performing convolution and pooling operations on the audio waveforms of the audio data. These static feature vectors are then used as the speaker features (speaker embedding) corresponding to the training sample.

[0089] In some embodiments, a neuroacoustic codec converts the audio waveforms of the audio data in the training samples into one-dimensional discrete acoustic feature tokens (speech tokens) by a fixed time frame shift. These one-dimensional discrete acoustic feature tokens are then used as the audio cue sequence. Alternatively, the first half of the one-dimensional discrete acoustic feature token or a specific historical time window is extracted in chronological order, and the resulting segment sequence is used as the audio cue sequence corresponding to the training sample.

[0090] In some embodiments, the input sequence is obtained by fusing text feature sequences, speaker features, audio cue sequences, and control symbols. This can be achieved by concatenating boundary control symbols, speaker features, text feature sequences, task control symbols, and audio cue sequences in a preset order.

[0091] For example, a training sample containing the text "Pinyin" and its corresponding pronunciation recording is obtained. The text data "Pinyin" is segmented to obtain two text feature tokens (let's assume text token1 and text token2), and a text feature sequence with a length of 2 is extracted. Voiceprint extraction is performed on the pronunciation recording to obtain speaker features (i.e., speaker embedding) with a length of 1. The pronunciation recording is discretized and historical segments are extracted to obtain an audio cue sequence containing 50 discrete acoustic feature tokens (let's assume speech token1 to speech token50). A start control character (i.e., [SOS / EOS]) with a length of 1 and an audio synthesis task control character (i.e., [TaskID]) with a length of 1 are obtained. Subsequently, the input sequence is sequentially concatenated and merged along the time series length dimension of the tensor, resulting in an input sequence arranged as [SOS / EOS]+[speaker embedding]+[text token1]+[text token2]+[Task ID]+[voice token1]+[voice token2]+...+[voice token50]. Based on the above total length calculation formula, the total length of this input sequence is... The resulting input sequence is then used as a single tensor and fed into the audio synthesis model to be trained for feature extraction.

[0092] In step 102, the input sequence is feature extracted using the audio synthesis model to be trained to obtain a shared hidden state sequence.

[0093] Here, the audio synthesis model to be trained is a deep neural network framework used to simultaneously perform audio generation and phoneme prediction tasks. This audio synthesis model includes a backbone network and two parallel prediction branches. The backbone network is a Large Language Model (LLM), and the two parallel prediction branches are an audio prediction network and a phoneme prediction network, respectively. Feature extraction refers to the process of using the self-attention mechanism included in the LLM to correlate and compute features at each time step in the input sequence to output a fused feature vector sequence. The shared hidden state sequence refers to the feature vector sequence output by the LLM after completing the feature extraction, which is used as input to the subsequent parallel audio prediction network and phoneme prediction network.

[0094] In some embodiments, the shared hidden state sequence is obtained by extracting features from the input sequence using the audio synthesis model to be trained. This can be achieved by: inputting the input sequence into the large language model of the audio synthesis model to be trained; and performing context modeling on the input sequence using the large language model to obtain the shared hidden state sequence.

[0095] When performing context modeling, the large language model does not perform additional acoustic network encoding on the input sequence. Instead, it directly utilizes its own built-in Transformer network structure to extract features from the dependencies between various feature elements in the input sequence. Assume the mathematical expression corresponding to this context modeling process is: ,in, Represents the input sequence. This indicates the multi-head self-attention computation and feedforward network processing operations involved in the large language model. This represents the shared hidden state sequence output by the large language model. In specific configurations, the vector dimension of the shared hidden state sequence can be set according to the network size of the large language model, for example, 4096 or 768. This feature extraction method directly reuses the feature extraction capability of the large language model for sequence data, providing a unified data foundation for subsequent parallel audio prediction and phoneme prediction without introducing additional acoustic coding network structures, thereby reducing the complexity of the network architecture and computational overhead.

[0096] For example, an input sequence of length 55 is obtained, consisting of boundary control characters, speaker features, text feature sequences, task control characters, and audio cue sequences concatenated sequentially. This input sequence is then fed into the large language model within the audio synthesis model to be trained. The large language model does not perform additional acoustic coding transformations on the input sequence; instead, it directly performs feature extraction calculations based on its internal transformer network structure. After feature extraction, the large language model generates a corresponding feature vector for each feature representation contained in the input sequence, combining them to form a feature vector sequence of length 55. This feature vector sequence is then output as the shared hidden state sequence. Assuming the feature vector dimension of the large language model is configured to be 4096, the data shape of the extracted shared hidden state sequence is 55×4096. Subsequently, this 55×4096 shared hidden state sequence is simultaneously provided to the audio prediction network and phoneme prediction network within the audio synthesis model to be trained.

[0097] In step 103, audio prediction is performed on the shared hidden state sequence to obtain the audio prediction result, and phoneme prediction is performed on the shared hidden state sequence to obtain the phoneme prediction result.

[0098] Here, audio prediction refers to the process of mapping the input shared hidden state sequence to a discrete audio representation based on an audio prediction network. The audio prediction result refers to the discrete audio representation output by the audio prediction network. This discrete audio representation is used to characterize the audio content generated by the audio synthesis model. Phoneme prediction refers to the process of performing classification mapping calculations on the input shared hidden state sequence based on a phoneme prediction network. The phoneme prediction result refers to the phoneme category probability distribution output by the phoneme prediction network for a predefined set of phonetic symbols. The phoneme category probability distribution includes the probability value for each position in the input sequence belonging to each phonetic symbol in the phonetic symbol set.

[0099] In some embodiments, audio prediction is performed on the shared hidden state sequence to obtain an audio prediction result, and phoneme prediction is performed on the shared hidden state sequence to obtain a phoneme prediction result. This can be achieved by: inputting the shared hidden state sequence into the audio prediction network and the phoneme prediction network included in the audio synthesis model, respectively; performing feature mapping calculation on the shared hidden state sequence through the audio prediction network to output a discrete audio representation, and determining the discrete audio representation as the audio prediction result; performing classification mapping calculation on the shared hidden state sequence through the phoneme prediction network to output a phoneme category probability distribution, and determining the phoneme category probability distribution as the phoneme prediction result.

[0100] It should be noted that the embodiments of this application do not limit the network architecture of the audio prediction network and the phoneme prediction network.

[0101] For example, a shared hidden state sequence with a dimension of 55×4096 is obtained from the large language model. This shared hidden state sequence is simultaneously input into an audio prediction network and a phoneme prediction network. In the audio prediction network, the output dimension of the audio prediction result is configured to be 1024. The audio prediction network performs feature mapping calculation on the shared hidden state sequence, outputting a discrete audio representation with a dimension of 55×1024, which is determined as the audio prediction result. Meanwhile, in the phoneme prediction network, a multilayer perceptron network structure is adopted, and the number of phonetic symbol categories is configured to be 100 (e.g., including 100 specific phonetic symbols such as p and n). The phoneme prediction network performs classification mapping calculation on the shared hidden state sequence, and for a specific position in the input sequence, outputs a vector containing 100 probability values, where each probability value corresponds to one of the 100 phonetic symbols. For example, the probability value of the phonetic symbol p in this vector is 0.85, the probability value of the phonetic symbol n is 0.02, and the sum of the probabilities of the other symbols is 0.13. The final output is a phoneme category probability distribution with a dimension of 55×100, and this phoneme category probability distribution is determined as the phoneme prediction result. Finally, the audio prediction result and the phoneme prediction result are output synchronously.

[0102] In step 104, the audio loss value is determined based on the audio prediction results and the audio label sequence corresponding to the training samples, and the phoneme loss value is determined based on the phoneme prediction results and the phoneme label sequence corresponding to the training samples.

[0103] Here, the audio label sequence refers to the one-dimensional acoustic feature label sequence obtained after discretizing and encoding the audio data contained in the training samples. The audio label sequence can include multiple discretized integer labels (i.e., discrete acoustic features), and each discrete acoustic feature is accompanied by the start and end coordinates (i.e., timestamp information) representing the physical time period in which the discrete acoustic feature is located within the audio data. The phoneme label sequence refers to the category number sequence extracted by an offline alignment tool, used to characterize the actual pronunciation structure in the audio data contained in the training samples. This phoneme label sequence consists of integer numbers representing phonetic symbols (i.e., phoneme category numbers), and each phoneme category number is accompanied by absolute physical coordinates (i.e., time boundary information) representing the duration of pronunciation of the phoneme category number. The audio loss value refers to the difference between the audio prediction result output by the audio prediction network and the true audio label sequence, measured using a preset loss function, to reflect the prediction error of the audio synthesis model in the audio generation task. The phoneme loss value refers to the error comparison between the phoneme prediction result output by the phoneme prediction network and the true phoneme label sequence, to reflect the classification error of the audio synthesis model in the phoneme prediction task.

[0104] In some embodiments, the audio tag sequence corresponding to the training sample can be obtained by: performing feature quantization on the audio data corresponding to the training sample using a neural acoustic codec, extracting a one-dimensional discrete acoustic feature tag sequence according to a fixed time frame shift, and using this one-dimensional discrete acoustic feature tag sequence as the audio tag sequence corresponding to the training sample. Alternatively, the latter half of the one-dimensional discrete acoustic feature tag sequence can be extracted as the audio tag sequence corresponding to the training sample. The audio tag sequence includes multiple discrete acoustic features and timestamp information corresponding to each discrete acoustic feature.

[0105] In some embodiments, the phoneme label sequence corresponding to the training sample can be obtained by: performing forced alignment on the audio data corresponding to the training sample and the text data corresponding to the audio data based on a phoneme alignment tool (e.g., a forced alignment tool (Montreal Forced Aligner, MFA)) to obtain a phoneme label sequence containing multiple phoneme category numbers and time boundary information corresponding to each phoneme category number.

[0106] For example, a training sample containing the text "pinyin" and a corresponding 2-second recording of a human's pronunciation is obtained. When extracting the audio tag sequence, a pre-trained neural acoustic codec is used to discretize the 2-second pronunciation recording with a fixed frame shift of 20ms, generating a total audio sequence containing 100 acoustic discrete features. To facilitate the autoregressive training mechanism, the latter half of this total audio sequence (e.g., from the 51st to the 100th marker) is extracted, resulting in an audio tag sequence containing 50 acoustic discrete features. When extracting the phoneme tag sequence, the 2-second pronunciation recording and the text "pinyin" are input into a forced alignment tool. The forced alignment tool performs temporal boundary segmentation and forced alignment of phonemes such as p and n on the time axis with a fixed frame shift of 10ms, outputting 200 corresponding phoneme symbols frame by frame. Subsequently, according to a preset phoneme symbol category mapping table, these 200 phoneme symbols are converted one by one into corresponding integer category numbers, resulting in a phoneme tag sequence containing 200 phoneme category numbers.

[0107] In some embodiments, the audio loss value is determined based on the audio prediction result and the audio label sequence corresponding to the training samples. This can be achieved by: performing a time-step difference comparison between each discrete audio representation in the audio prediction result and the corresponding discrete acoustic features in the audio label sequence; and calculating the error of the time-step comparison result using a preset loss function to obtain the audio loss value. This application embodiment does not limit the preset audio loss function; for example, it can be the cross-entropy loss function.

[0108] When calculating the audio loss value, the cross-entropy loss function is mainly used to measure the difference between the discrete audio representation predicted by the audio synthesis model at each time step and the true discrete acoustic features. For example, the formula for calculating the audio loss value can satisfy the following formula (1).

[0109] Formula (1);

[0110] in, This represents the calculated audio loss value; This represents the total number of time steps in the time series representing the audio prediction results; This represents the current time step; This represents the input sequence that is fed into the audio synthesis model to be trained. Represents the time step At that time, in the audio prediction results, the sequence corresponding to the true audio label at time step... The acoustic discrete feature at that location (i.e. The predicted probability value of ). This represents the natural logarithm function. By calculating this cross-entropy loss function, the accuracy of the audio synthesis model being trained on the audio generation task can be quantitatively evaluated.

[0111] For example, suppose the audio cue sequence in the current input sequence is 50 units long. The audio prediction network predicts for this input sequence, outputting a probability distribution vector of size 1024 at each time step, ultimately outputting a discrete audio representation with a data shape of 50×1024. This discrete audio representation is determined as the audio prediction result. Simultaneously, an audio tag sequence of length 50 is obtained, where each time step contains a true discrete acoustic feature (e.g., an integer number between 0 and 1023). Then, the cross-entropy loss function is applied to process the data. At time step... At that time, the cross-entropy loss function reads the acoustic discrete features of the first position in the audio tag sequence (assuming the integer number is 256), then extracts the predicted probability value corresponding to the number 256 (assuming it is 0.8) from the 1024-dimensional probability distribution vector of the first time step of the audio prediction result, and calculates its negative logarithmic loss value. The cross-entropy loss function calculates the negative logarithmic loss value for 50 time steps sequentially, sums these 50 negative logarithmic loss values, and calculates the average. Assuming the final calculated average error value is 1.2, this value of 1.2 is determined as the current audio loss value.

[0112] In some embodiments, see Figure 4 , Figure 4The step 104, which determines the phoneme loss value based on the phoneme prediction results and the phoneme label sequence corresponding to the training samples, may include steps 1041 to 1042.

[0113] In step 1041, the phoneme label sequence corresponding to the training sample is resampled to obtain the resampled phoneme label sequence.

[0114] The resampled phoneme tag sequence has the same sequence length as the original audio tag sequence.

[0115] Here, resampling refers to the data processing process of changing the number of data points on the time axis of one of two sequences with different time resolutions using a preset numerical mapping algorithm, so as to achieve strict alignment of the two sequences in physical length. The resampled phoneme label sequence refers to the phoneme label sequence obtained after performing a length scaling operation on the phoneme label sequence corresponding to the training samples. The resampled phoneme label sequence has the same sequence length as the audio label sequence.

[0116] In some embodiments, resampling the phoneme label sequence corresponding to the training samples to obtain the resampled phoneme label sequence can be achieved in the following way: obtaining the timestamp information corresponding to each acoustic discrete feature in the audio label sequence; obtaining the time boundary information corresponding to each phoneme category number in the phoneme label sequence; based on the overlap relationship between the timestamp information and the time boundary information, mapping the phoneme category number in the phoneme label sequence to the sequence position where each acoustic discrete feature in the audio label sequence is located, to obtain the resampled phoneme label sequence.

[0117] For example, a training sample with a pronunciation duration of 2 seconds is obtained. When extracting the audio label sequence, the neuroacoustic codec outputs a sequence containing 100 discrete acoustic features, and records the timestamp information of the 10th discrete acoustic feature (sequence position index t=9) as 180 ms to 200 ms. When extracting the phoneme label sequence, the phoneme alignment tool outputs not only the phoneme category number but also the time boundary information. For example, the time boundary information for phoneme category number 45 corresponding to the phoneme p is 175 ms to 185 ms, and the time boundary information for phoneme category number 12 corresponding to the phoneme n is 185 ms to 210 ms. When performing resampling mapping processing, a physical time window of 180 ms to 200 ms is extracted, and comparison shows that within this 20 ms time window, phoneme category number "45" occupies 5 ms (180 ms to 185 ms), and phoneme category number "12" occupies 15 ms (185 ms to 200 ms). Since phoneme category number "12" has the longest duration within this physical time window, it is directly extracted and assigned to the 10th sequence position in the resampled phoneme tag sequence. Following this logic, the maximum duration projection is calculated by traversing the timestamp windows corresponding to the 100 acoustic discrete features contained in the audio tag sequence, ultimately generating a phoneme category number sequence with a length reduced to 100. This 100-length sequence, whose length alignment is entirely based on the overlap between timestamps and time boundaries, is identified as the resampled phoneme tag sequence.

[0118] In some embodiments, see Figure 5 , Figure 5 The step 1041 shows that resampling the phoneme label sequence corresponding to the training sample to obtain the resampled phoneme label sequence may include steps 10411 to 10413.

[0119] In step 10411, the first time resolution of the phoneme tag sequence and the second time resolution of the audio tag sequence are obtained.

[0120] Here, the first time resolution refers to the time sampling frequency used to extract the phoneme tag sequence (e.g., a 10ms frame shift). The second time resolution refers to the time sampling frequency used to extract the audio tag sequence (e.g., a 20ms frame shift). The difference between the first and second time resolutions is the fundamental reason for the physical mismatch in sequence length between the phoneme tag sequence and the audio tag sequence.

[0121] In some embodiments, obtaining the first temporal resolution of the phoneme tag sequence and the second temporal resolution of the audio tag sequence can be achieved by: querying the frame shift configuration parameters of the phoneme alignment tool used in the data preprocessing stage of the training samples, and determining the frame shift configuration parameters as the first temporal resolution; simultaneously, querying the frame shift configuration parameters of the neuroacoustic codec used in the continuous audio feature discretization extraction process, and determining the frame shift configuration parameters as the second temporal resolution.

[0122] During the above acquisition operation, the server or terminal can directly obtain the first temporal resolution of the phoneme tag sequence and the second temporal resolution of the audio tag sequence by reading a preset configuration file (such as a JSON-formatted dataset metadata file) or by calling the application programming interface (API) of the underlying feature extractor. Let the extracted first temporal resolution be denoted as... The second time resolution is denoted as And both satisfy The relationship.

[0123] For example, during the data preprocessing stage for a batch of training samples, the phoneme alignment tool performs pronunciation boundary segmentation with a fixed step size of 10ms. Subsequently, by reading the metadata log synchronously generated by the phoneme alignment tool when outputting the phoneme label sequence, the 10ms frame shift configuration parameter is obtained, and 10ms is determined as the first temporal resolution of the phoneme label sequence. Simultaneously, when processing continuous audio data corresponding to the same batch of training samples, the neural acoustic codec performs feature discretization with a fixed step size of 20ms. By calling the attribute query interface of the neural acoustic codec, the 20ms frame shift configuration parameter is obtained, and 20ms is determined as the second temporal resolution of the audio label sequence.

[0124] In step 10412, the target scaling ratio is determined based on the first time resolution and the second time resolution.

[0125] Here, the target scaling ratio refers to a numerical metric used to quantify the temporal resolution difference between the phoneme tag sequence and the audio tag sequence. This target scaling ratio determines the physical factor by which the original phoneme tag sequence should be stretched or compressed on the time axis during resampling. The target scaling ratio can be a fixed constant (e.g., a positive number greater than 1 when the temporal resolution of the audio tag sequence is lower than that of the phoneme tag sequence), used to guide the position mapping calculation in subsequent interpolation algorithms.

[0126] In some embodiments, the target scaling ratio is determined based on a first time resolution and a second time resolution, which can be achieved by determining the ratio of the second time resolution to the first time resolution as the target scaling ratio.

[0127] For example, assume that the first time resolution of the acquired sequence for extracting phoneme tags is 10ms. Simultaneously, the second time resolution of the acquired sequence for extracting audio tags is 20ms. Calculate the ratio of the second time resolution 20ms to the first time resolution 10ms, i.e. The ratio 2 is determined as the target scaling factor.

[0128] In step 10413, the phoneme tag sequence is interpolated according to the target scaling ratio to obtain the resampled phoneme tag sequence.

[0129] Here, interpolation refers to the data processing process of determining the mapping relationship between the position of the sequence to be generated and the position of the original sequence based on the target scaling ratio, and extracting the values ​​at the positions of the original sequence to fill the sequence to be generated.

[0130] This application does not limit the specific method of interpolation processing in its embodiments. For example, when performing the above interpolation processing, various interpolation processing methods such as linear interpolation or region pooling aggregation can be used. Alternatively, the nearest neighbor interpolation algorithm can be used for interpolation processing.

[0131] This application embodiment obtains a first temporal resolution of the phoneme tag sequence and a second temporal resolution of the audio tag sequence, and determines a target scaling ratio based on both, so that subsequent time alignment operations for the two tag sequences have a precise and quantifiable scaling basis; the phoneme tag sequence is interpolated according to the target scaling ratio, and finally the resampled phoneme tag sequence is obtained. This not only effectively eliminates the length mismatch problem caused by different feature extraction frequencies, but also achieves seamless scaling at the sequence level while preserving the original pronunciation structure features, thus laying the foundation for providing high-quality tags with fully time-step alignment for subsequent audio and phoneme dual prediction networks.

[0132] In some embodiments, the process of interpolating the phoneme tag sequence according to the target scaling ratio in step 10413 to obtain the resampled phoneme tag sequence can be implemented as follows: First, an initial resampled sequence is constructed, wherein the sequence length of the initial resampled sequence is the same as the sequence length of the audio tag sequence; then, based on the target scaling ratio, the mapping position of the target position index contained in the initial resampled sequence in the phoneme tag sequence is determined; then, from the phoneme tag sequence, the original position index that is closest to the mapping position is determined, and the original tag value corresponding to the original position index in the phoneme tag sequence is obtained; finally, the original tag value is filled into the target position index of the initial resampled sequence to obtain the resampled phoneme tag sequence.

[0133] Here, the initial resampled sequence refers to a blank sequence pre-constructed during interpolation processing, with the same length as the audio tag sequence. The target position index refers to the position number in the initial resampled sequence used to receive and store interpolated data. The mapped position refers to the corresponding position obtained by transforming the target position index to the time dimension of the phoneme tag sequence according to the target scaling ratio. The original position index refers to the position in the phoneme tag sequence closest to the mapped position. The original tag value refers to the phoneme category number recorded in the phoneme tag sequence at the original position index.

[0134] First, the length of the audio tag sequence is read, and an empty array of the same length is initialized in memory as the initial resampling sequence. Next, for each target position index in this initial resampling sequence, the target position index is multiplied by the obtained target scaling ratio to obtain a mapped position. Since this mapped position is a floating-point coordinate and cannot be directly used to extract data, it is rounded to the nearest integer, and the resulting integer coordinate is used as the original position index in the phoneme tag sequence. Then, the position of this original position index in the phoneme tag sequence is directly accessed, and the actual phoneme category number (i.e., the original tag value) recorded at that position is read. Finally, the read phoneme category number is filled into the target position index being processed in the initial resampling sequence.

[0135] For example, assume the obtained audio tag sequence has a length of 100, the obtained phoneme tag sequence has a length of 200, and the target scaling ratio is determined to be 2. During interpolation, a blank sequence with the same length of 100 is constructed and used as the initial resampled sequence. For the 5th position in this initial resampled sequence (i.e., target position index 4), the target position index 4 is multiplied by the target scaling ratio 2 to calculate the corresponding mapping position 8 in the phoneme tag sequence. Since the mapping position 8 is itself an integer, the nearest original position index to this mapping position is directly determined as 8 in the phoneme tag sequence. Subsequently, the original tag value recorded at the original position index 8 in the phoneme tag sequence (assuming the recorded phoneme category number is 45) is read, and this original tag value 45 is filled into the target position index 4 of the initial resampled sequence. Following this calculation logic, the 100 target position indices contained in the initial resampled sequence are traversed and processed sequentially. Once all target location indices have been filled, the sequence of these 100 values ​​will be determined as the resampled phoneme tag sequence.

[0136] This application constructs an initial resampled sequence with the same sequence length as the audio tag sequence, and accurately determines the nearest original position index in the phoneme tag sequence based on the target scaling ratio to extract the original tag values ​​for filling. This not only strictly achieves data length alignment between the phoneme tag sequence and the audio tag sequence in terms of physical structure, but also ensures from the underlying algorithm logic that each value in the resampled phoneme tag sequence originates from the original tag values ​​that actually exist in the phoneme tag sequence. This avoids the generation of illegal feature data during the interpolation process and effectively ensures the data reliability of the phoneme tag sequence after the sequence length is scaled.

[0137] In step 1042, the phoneme loss value is determined based on the phoneme prediction results and the resampled phoneme tag sequence.

[0138] In some embodiments, the phoneme loss value is determined based on the phoneme prediction result and the resampled phoneme label sequence. This can be achieved by comparing the phoneme category probability distribution contained in the phoneme prediction result with the phoneme category number contained in the resampled phoneme label sequence step by step; and using the cross-entropy loss function to calculate the error of the step-by-step comparison result to obtain the phoneme loss value.

[0139] For example, assume the length of the current resampled phoneme label sequence is 50. The phoneme prediction network predicts the input shared hidden state sequence, outputting a probability distribution vector of size 100 at each time step, ultimately outputting a phoneme category probability distribution with a data shape of 50×100. This phoneme category probability distribution is determined as the phoneme prediction result. Simultaneously, a resampled phoneme label sequence of length 50 is obtained, where each time step contains a true phoneme category number (e.g., an integer number between 0 and 99). Then, the cross-entropy loss function is called to process the data. At time step... At that time, the cross-entropy loss function reads the phoneme category number (assuming the integer number is 45) at the first position in the resampled phoneme label sequence, then extracts the predicted probability value (assuming it is 0.85) corresponding to the number 45 from the 100-dimensional probability distribution vector of the phoneme prediction result at the first time step, and calculates its negative logarithmic loss value. The cross-entropy loss function calculates the negative logarithmic loss value for 50 time steps sequentially, sums these 50 negative logarithmic loss values, and calculates the average. Assuming the final calculated average error value is 0.5, this value of 0.5 is determined as the current phoneme loss value.

[0140] This application embodiment resamples the phoneme tag sequences corresponding to the training samples to obtain resampled phoneme tag sequences, ensuring that the sequence length of the resampled phoneme tag sequences is the same as the sequence length of the audio tag sequences. This eliminates the length mismatch caused by the different time resolutions of the two tag sequences at the physical data structure level. Based on the phoneme prediction results and the resampled phoneme tag sequences, the phoneme loss value is determined, ensuring that the phoneme prediction results can be accurately compared with the resampled phoneme tag sequences with perfectly aligned lengths, effectively improving the accuracy of the calculated phoneme loss value. This provides a stable and reliable gradient signal for subsequent updates to the model parameters of the audio synthesis model based on the phoneme loss value.

[0141] In some embodiments, see Figure 6 , Figure 6 The step 1042, which determines the phoneme loss value based on the phoneme prediction result and the resampled phoneme tag sequence, may include steps 10421 to 10424.

[0142] In step 10421, the total length of the input sequence is obtained, and the first position of the non-audio region in the input sequence is determined.

[0143] Here, the total sequence length refers to the total number of feature elements contained in the input sequence on the time axis after performing feature concatenation on the modal features of the training samples. Non-audio regions refer to sequence segments in the input sequence that do not contain any real sound pronunciation information, such as sequence boundary control markers, speaker features, text feature marker sequences, and padding regions used for batch alignment. The first position refers to the position corresponding to these non-audio regions in the input sequence. There can be multiple first positions or only one.

[0144] In some embodiments, determining the first position corresponding to a non-audio region in the input sequence can be achieved by: extracting feature segments from the input sequence to represent non-sound pronunciation information; recording the start and end coordinates of each feature segment in the input sequence, and using all coordinate points between the start and end coordinates as the coordinate interval corresponding to that feature segment; and determining each position within the coordinate interval corresponding to all feature segments as the first position corresponding to the non-audio region.

[0145] During the process of concatenating text feature marker sequences, speaker features, audio cue sequences, and control markers to obtain the input sequence, the sequence length occupied by each feature component is recorded simultaneously. Summing these sequence lengths yields the total length of the input sequence. Since the primary goal of the phoneme prediction network is to predict pronunciation structure, and the control markers, speaker features, and text feature marker sequences in the input sequence are all instructive information lacking actual pronunciation process, these feature fragments are uniformly classified as non-audio regions. Finally, the coordinate intervals corresponding to these non-audio regions are determined, and each position within these coordinate intervals is designated as the first position corresponding to the non-audio region.

[0146] For example, suppose that when constructing the current input sequence, the following feature components were concatenated sequentially and their respective sequence lengths were recorded: boundary control characters in the control markers (length 1), speaker features (length 1), text feature marker sequence (length 5), task control characters in the control markers (length 1), and audio cue sequence (length 50). Adding these sequence lengths together (1+1+5+1+50), the total sequence length of the input sequence is calculated to be 58. Simultaneously, the boundary control characters, speaker features, text feature marker sequence, and task control characters are identified as feature segments that do not possess true pronunciation information. Next, the coordinate ranges of these feature segments are recorded: the start and end coordinates of the boundary control characters are both 0 (i.e., position 0); the start and end coordinates of the speaker features are both 1 (i.e., position 1); the start coordinate of the text feature marker sequence is 2, and the end coordinate is 6 (inclusive of positions 2, 3, 4, 5, and 6); the start and end coordinates of the task control characters are both 7 (i.e., position 7). Subsequently, all coordinate points between the start and end coordinates of these feature segments are taken as coordinate intervals, and the coordinate intervals corresponding to these four feature segments are traversed. The eight specific positions 0, 1, 2, 3, 4, 5, 6, and 7 contained in the coordinate intervals are determined as the first position corresponding to the non-audio region.

[0147] In step 10422, an initial phoneme tag sequence with a length equal to the total length of the sequence is constructed.

[0148] Here, the initial phoneme tag sequence refers to a blank sequence in memory with a length equal to the total length of the sequence.

[0149] In some embodiments, an initial phoneme tag sequence with a length equal to the total sequence length can be constructed by: initializing a one-dimensional blank sequence in memory with a sequence length equal to the total sequence length; and determining the one-dimensional blank sequence as the initial phoneme tag sequence.

[0150] Here, the sequence generation function is called, and the total length of the read sequence is passed to the sequence generation function as a dimension parameter. After receiving the dimension parameter, the sequence generation function allocates a contiguous space of the corresponding length in memory and generates an initial phoneme tag sequence consisting of initial default values ​​(such as all zeros).

[0151] For example, suppose the total length of the current input sequence is 58, obtained in the preceding steps. When constructing the initial phoneme tag sequence, the total sequence length of 58 is extracted, and the sequence generation function is called. This sequence generation function uses the total sequence length of 58 as the input length parameter, directly partitions a one-dimensional space in memory that can accommodate 58 data elements, and fills all 58 positions with the initial value 0. After the generation operation is completed, this one-dimensional sequence containing 58 initial values ​​is output, and this one-dimensional sequence is determined as the initial phoneme tag sequence.

[0152] In step 10423, an ignore flag is filled at the position corresponding to the first position in the initial phoneme tag sequence, and a resampled phoneme tag sequence is filled at the second position in the initial phoneme tag sequence, to obtain a target phoneme tag sequence filled with the ignore flag and the resampled phoneme tag sequence.

[0153] The second position is a position that is different from the first position.

[0154] Here, the ignore flag refers to a preset value configured in the initial phoneme label sequence to mask error interference at specific locations during subsequent phoneme loss calculation. For example, the ignore flag could be IGNORE_ID with a value of -100. The second position refers to the position of the valid audio region in the input sequence corresponding to the audio cue sequence. There can be one or more second positions. The target phoneme label sequence refers to the final one-dimensional complete sequence used for loss calculation after the ignore flag and the resampled phoneme label sequence are filled into specific positions in the initial phoneme label sequence.

[0155] In some embodiments, an ignore flag is filled at the position corresponding to the first position in the initial phoneme tag sequence, and a resampled phoneme tag sequence is filled at the second position in the initial phoneme tag sequence to obtain the target phoneme tag sequence. The target phoneme tag sequence filled with the ignore flag and the resampled phoneme tag sequence can be obtained by the following method: determining the coordinates that match the multiple first positions in the initial phoneme tag sequence and uniformly writing the preset ignore flag at these coordinates; determining the remaining positions in the initial phoneme tag sequence other than the multiple first positions and determining these remaining positions as second positions; obtaining the resampled phoneme tag sequence and writing the resampled phoneme tag sequence into the second position of the initial phoneme tag sequence in chronological order; and determining the initial phoneme tag sequence after completing the above writing operation as the target phoneme tag sequence.

[0156] For example, assume the total length of the initial phoneme tag sequence is 58 (with all initial values ​​being 0). Simultaneously, the first positions containing 8 specific coordinate points (index 0 to index 7) and a resampled phoneme tag sequence of length 50 are obtained. During the filling operation, the preset ignore flag "-100" is extracted first. The 8 first positions (index 0 to index 7) of the initial phoneme tag sequence are located, and the values ​​at these 8 positions are uniformly modified to "-100". Next, the remaining 50 positions from index 8 to index 57 in the initial phoneme tag sequence are determined as the second positions. Since these 50 second positions are exactly equal in number of elements to the resampled phoneme tag sequence of length 50, the first phoneme category number (e.g., 45) in the resampled phoneme tag sequence is directly filled into position 8, the second phoneme category number (e.g., 12) is filled into position 9, and so on, until the 50th phoneme category number is filled into position 57. Once all positions from index 0 to index 57 have been successfully written with the corresponding data, output a new sequence of length 58, consisting of 8 "-100" and 50 phoneme category numbers arranged in order, and determine this sequence as the final target phoneme tag sequence.

[0157] In step 10424, the phoneme loss value is determined based on the phoneme prediction results and the sequences in the target phoneme label sequence that are not configured with ignore flags.

[0158] In this case, the positions in the target phoneme label sequence that are configured with an ignore flag are not included in the calculation of the phoneme loss value.

[0159] In some embodiments, the phoneme loss value is determined based on the phoneme prediction result and the sequences in the target phoneme label sequence that are not configured with ignore labels. This can be achieved by: performing time-step alignment comparison between the phoneme category probability distribution contained in the phoneme prediction result and the values ​​at each position contained in the target phoneme label sequence; during the comparison process, identifying the sequences in the target phoneme label sequence that are configured with ignore labels and skipping the error calculation corresponding to the sequences; extracting the phoneme category numbers included in the sequences in the target phoneme label sequence that are not configured with ignore labels; and using the cross-entropy loss function to calculate the error between the phoneme category probability distribution at the corresponding position in the phoneme prediction result and the extracted phoneme category numbers to obtain the phoneme loss value.

[0160] For example, assume a phoneme prediction result of 58×100 is obtained (i.e., at each time step of a sequence with a total length of 58, the phoneme category probability distribution containing 100 probability values ​​is output). Simultaneously, a target phoneme label sequence with the same total length of 58 is obtained, where the first 8 positions are configured with the value "-100" (ignoring the identifier), and the last 50 positions are recorded as second positions containing phoneme category numbers (e.g., number 45, number 12, etc.). The phoneme prediction result is compared with the target phoneme label sequence time-step by time. When comparing the first 8 positions, the value "-100" is detected in the target phoneme label sequence, and the error calculation corresponding to these 8 first positions is skipped. When comparing the last 50 positions, the actual phoneme category numbers recorded in these 50 second positions are extracted. Subsequently, using the cross-entropy loss function, the error calculation is performed on the phoneme category probability distribution corresponding to these 50 second positions in the phoneme prediction result and the extracted 50 phoneme category numbers. The 50 calculated error values ​​are summed and averaged. Assuming the average error value is 0.5, this value of 0.5 is determined as the current phoneme loss value.

[0161] Figure 7 This is a schematic diagram illustrating the principle of the phoneme loss value calculation process provided in the embodiments of this application. See also... Figure 7 Based on the phoneme tag sequence (length 200) corresponding to the first time resolution of 10ms and the audio tag sequence (length 100) corresponding to the second time resolution of 20ms, the target scaling ratio is determined to be 2, and a resampled phoneme tag sequence (length 100) is obtained. Based on the total length of the input sequence = 58 and the first position: 0-7, the second position: 8-57, an initial phoneme tag sequence (length 58) is constructed, and the filler ignore flag (-100) and the filler resampled phoneme tag sequence are executed respectively to obtain the target phoneme tag sequence. On this basis, the positions configured with the ignore flag do not participate in the calculation of the phoneme loss value, and the phoneme loss value is obtained by combining the phoneme prediction result (58×100).

[0162] This application embodiment obtains the total length of the input sequence and determines the first position corresponding to the non-audio region in the input sequence, and then constructs an initial phoneme label sequence with a length equal to the total length of the sequence, ensuring that the target phoneme label sequence maintains structural consistency with the input sequence in the time dimension. An ignore flag is configured at the first position in the initial phoneme label sequence, and a resampled phoneme label sequence is configured at a second position different from the first position, which can accurately distinguish between non-audio regions without real pronunciation information and effective audio regions. Based on this, the phoneme loss value is determined based on the phoneme prediction results and the sequence in the target phoneme label sequence without the ignore flag, so that the position with the ignore flag does not participate in the calculation of the phoneme loss value, effectively shielding the prediction noise that non-audio regions may introduce during model training, and ensuring that the phoneme loss value is dominated only by the real resampled phoneme label sequence, thereby significantly improving the gradient update accuracy and training stability of the audio synthesis model to be trained on the phoneme prediction task.

[0163] In step 105, the model parameters of the audio synthesis model to be trained are updated based on the audio loss value and the phoneme loss value to obtain the trained audio synthesis model.

[0164] Here, model parameters refer to the weight matrices and bias vectors contained in each network layer of the audio synthesis model to be trained. These model parameters determine the specific feature mapping rules of the audio synthesis model's output audio prediction results and phoneme prediction results given an input sequence. Updating model parameters refers to the operation of adjusting the above weight matrices and bias vectors numerically using the audio loss value and phoneme loss value through the backpropagation algorithm and the optimizer during model training. The trained audio synthesis model refers to an audio synthesis model that, after multiple iterations of parameter updates, achieves convergence between the error of the output prediction result and the label sequence, thus possessing high-precision audio generation and phoneme prediction capabilities.

[0165] In some embodiments, the step 105, which updates the model parameters of the audio synthesis model to be trained based on the audio loss value and the phoneme loss value to obtain the trained audio synthesis model, can be implemented as follows: First, determine at least one training stage corresponding to the audio synthesis model to be trained. Then, in at least one training stage, update the model parameters of the audio synthesis model to be trained based on the audio loss value and the phoneme loss value to obtain the trained audio synthesis model.

[0166] Here, the training phase refers to a time interval with independent parameter update rules, defined during model training to achieve a specific parameter optimization objective. At least one training phase refers to the combination of all phases the audio synthesis model to be trained must undergo before completing parameter updates; this can be a single global joint optimization phase or multiple sequentially executed incremental optimization phases.

[0167] In some embodiments, determining at least one training stage corresponding to the audio synthesis model to be trained can be achieved by: reading the training configuration file of the audio synthesis model to be trained; extracting preset training strategy information from the training configuration file; and determining the number of training stages to be executed by the audio synthesis model to be trained and the specific training stages based on the training strategy information.

[0168] For example, suppose the current model training task is initiated, and the locally stored training configuration file is read. This configuration file is parsed to extract the pre-defined training strategy information (assuming this strategy is a phased freeze training strategy). Based on this strategy information, it is identified that the audio synthesis model to be trained needs to undergo a process of gradually unfreezing network parameters. Therefore, it is determined that the current model needs to perform multiple training phases, specifically including: a first training phase (training only the phoneme prediction network), a second training phase (training only the large language model and the audio prediction network), and a third training phase (joint training of the global network). These three specific training phases are collectively identified as at least one training phase corresponding to the audio synthesis model to be trained, to guide subsequent parameter update steps.

[0169] In some embodiments, during at least one training phase, the model parameters of the audio synthesis model to be trained are updated based on the audio loss value and the phoneme loss value to obtain the trained audio synthesis model. This can be achieved by: obtaining the calculated audio loss value and phoneme loss value within the execution cycle of the determined at least one training phase; using at least one of the audio loss value and phoneme loss value as the target error criterion based on the network state of the audio synthesis model to be trained in the current training phase; calculating the error gradient of the audio synthesis model to be trained based on the target error criterion using the backpropagation algorithm; and using an optimizer to numerically adjust the model parameters according to the error gradient until the preset training termination condition is met to obtain the trained audio synthesis model.

[0170] For example, suppose at least one training phase has been initiated, and in the current iteration step, an audio loss value of 1.2 and a phoneme loss value of 0.5 are calculated. During parameter updates, these 1.2 audio loss value and 0.5 phoneme loss value are extracted as the target error, and the backpropagation algorithm is invoked to calculate the error gradient of the entire audio synthesis model to be trained. Then, the optimizer, based on the calculated error gradient, makes small numerical adjustments to all model parameters included in the audio synthesis model to be trained (i.e., updating weights along the gradient descent direction). After tens of thousands (e.g., 100,000 steps) of iterative updates, when the sum of the audio loss value and the phoneme loss value decreases and stabilizes below a preset threshold (e.g., 0.1), the model is considered to have converged, and the update operation is stopped. At this point, the model entity with the adjusted parameters is permanently saved, and this saved model entity is identified as the trained audio synthesis model.

[0171] This application embodiment defines at least one training stage corresponding to the audio synthesis model to be trained, providing a clear global plan and execution stage division for the subsequent model parameter update process; in the defined at least one training stage, the model parameters of the audio synthesis model to be trained are updated based on the audio loss value and the phoneme loss value, so that the model can be simultaneously subjected to the dual error optimization of the audio generation task and the phoneme prediction task during the parameter optimization period, prompting the audio synthesis model to spontaneously learn the alignment relationship between the low-level acoustic features of audio and the high-level pronunciation structure features, thereby effectively improving the consistency of the performance and the prediction accuracy of the trained audio synthesis model on the multimodal joint generation task.

[0172] In some embodiments, during at least one training phase, the model parameters of the audio synthesis model to be trained are updated based on the audio loss value and the phoneme loss value to obtain the trained audio synthesis model. This can be achieved as follows: First, when the number of at least one training phase is one, the audio loss value and the phoneme loss value are weighted and summed in one training phase to obtain a first loss value; then, the model parameters of the audio synthesis model to be trained are updated based on the first loss value to obtain the trained audio synthesis model.

[0173] Here, the first loss value refers to the total error value used for global gradient update obtained by performing a linear weighted summation operation on the independently calculated audio loss value and phoneme loss value according to their respective set weight coefficients in the single-stage training mode.

[0174] In some embodiments, the calculation of the first loss value can satisfy the following formula (2).

[0175] Formula (2);

[0176] in, This is the first loss value; This represents the audio loss value. This represents the phoneme loss value. This is a preset first weighting parameter used to adjust the proportion of audio loss values; This is a preset second weighting parameter used to adjust the proportion of phoneme loss values. In specific configurations, and The value can be flexibly adjusted according to the focus of the actual task (for example, and ).

[0177] After calculating the unique first loss value using the weighted summation formula, the backpropagation algorithm directly uses this first loss value as the starting point for backpropagation to calculate the error gradients of the model parameters in the large language model, audio prediction network, and phoneme prediction network of the audio synthesis model to be trained, and then passes them to the optimizer to perform synchronous updates of the model parameters.

[0178] For example, suppose a model training task is initiated, and the number of at least one training stage corresponding to the audio synthesis model to be trained is determined to be one. Simultaneously, a preset first weight parameter of 0.7 and a second weight parameter of 0.3 are obtained. In this current training step, the calculated audio loss value is 1.2, and the phoneme loss value is 0.5. Before performing parameter updates, the audio loss value 1.2 is first multiplied by the first weight parameter 0.7 to obtain 0.84, and the phoneme loss value 0.5 is multiplied by the second weight parameter 0.3 to obtain 0.15; then, these two products are weighted and summed to obtain a first loss value of 0.99. Next, this first loss value of 0.99 is extracted, and the underlying deep learning optimization algorithm is invoked. This optimization algorithm uses 0.99 as the global error target, calculates the gradient of all model parameters within the audio synthesis model to be trained, and fine-tunes the weights and biases of each layer along the gradient descent direction. After multiple rounds (e.g., 100,000 rounds) of iterative updates, when the value of the first loss value decreases and stabilizes below the preset convergence threshold (e.g., 0.1), the parameter update operation is stopped, and the network entity that has completed parameter optimization at this time is identified as the trained audio synthesis model.

[0179] In this embodiment, when there is only one training stage, a first loss value is obtained by weighted summation of the audio loss value and the phoneme loss value. This enables the audio synthesis model to be trained to simultaneously consider audio generation quality and phoneme prediction accuracy within a single training stage. Based on this first loss value, the model parameters of the audio synthesis model to be trained are globally updated. This not only simplifies the complexity of model training process control, but also forces the model to learn the shared mapping rules between acoustic features and pronunciation structure under a unified error drive, effectively improving the consistency of the trained audio synthesis model in multimodal joint generation tasks.

[0180] In some embodiments, in at least one training phase, the model parameters of the audio synthesis model to be trained are updated based on the audio loss value and the phoneme loss value to obtain the trained audio synthesis model. This can also be achieved by: first, when there are multiple training phases, obtaining the execution order of the multiple training phases; then, according to the execution order, performing parameter update operations in each training phase in sequence based on the audio loss value and the phoneme loss value to obtain the trained audio synthesis model.

[0181] Here, execution order refers to the pre-configured order of execution for multiple training stages within the model training cycle. Following the execution order, based on audio loss values ​​and phoneme loss values, performing parameter update operations sequentially in each training stage means that during model training, according to the pre-defined execution order of multiple training stages, audio loss values, phoneme loss values, or a combination of both are selected in different training stages to perform local or global gradient update processes on specific model parameters within the audio synthesis model.

[0182] During the aforementioned training phases, according to a preset execution order, some network parameters are frozen in different training phases. Audio loss values ​​or phoneme loss values ​​are calculated only for the activated networks, and local gradient update operations are performed. This flexible parameter update strategy can gradually improve phoneme prediction accuracy while ensuring audio generation quality, alleviate gradient conflicts in the early stages of dual-task optimization, and accelerate the overall convergence of the model.

[0183] For example, suppose there are at least three training phases. When performing parameter updates, the execution order of these three training phases is first determined: the first training phase, the second training phase, and the third training phase are executed sequentially. In the current iteration step, assume the calculated audio loss value is 1.2 and the phoneme loss value is 0.5. Then, following the above execution order, the parameter update operation is performed in the first training phase; after the parameter update operation in the first training phase is completed, the parameter update operation in the second training phase is performed; after the parameter update operation in the second training phase is completed, the parameter update operation in the third training phase is performed. After multiple rounds (e.g., 100,000 rounds) of iterative updates through the above three training phases, the parameter update operation is stopped, and the network entity whose parameters have been adjusted at this point is identified as the trained audio synthesis model.

[0184] When there are multiple training stages in this embodiment, the execution order of the multiple training stages is obtained, and parameter update operations are performed sequentially in each training stage according to the execution order. This allows the audio synthesis model to be trained to take into account the training needs of audio generation tasks and phoneme prediction tasks separately or simultaneously in different training stages. Through this flexible update mechanism based on the division of training stages, the mutual interference of gradients between multiple tasks in the early stage of training can be effectively avoided, thereby improving the stability of the model parameters converging to the optimal solution and accelerating the overall training process of the model.

[0185] In some embodiments, the audio synthesis model to be trained includes a large language model, an audio prediction network, and a phoneme prediction network, and at least one training phase includes three training phases. Following the execution order, parameter update operations are performed sequentially in each training phase based on the audio loss value and the phoneme loss value to obtain the trained audio synthesis model. This can be achieved as follows: In the first training phase, the model parameters of the large language model and the audio prediction network are frozen, and the model parameters of the phoneme prediction network are updated based on the phoneme loss value; in the second training phase, the model parameters of the phoneme prediction network are frozen, and the model parameters of the large language model and the audio prediction network are updated based on the audio loss value; in the third training phase, the audio loss value and the phoneme loss value are weighted and summed to obtain a second loss value, and the model parameters of the large language model, the audio prediction network, and the phoneme prediction network are updated based on the second loss value to obtain the trained audio synthesis model.

[0186] Here, the second loss value refers to the global target error obtained by weighting and fusing the audio loss value and the phoneme loss value according to a preset ratio in the last training stage (such as the third training stage in the execution sequence). Freezing model parameters means forcibly turning off the gradient update mechanism of specific network layers (such as large language models or audio prediction networks) during backpropagation calculation, so that these network layers retain their original feature mapping capabilities in the current training stage without being changed by the loss value.

[0187] In the first training phase (i.e., the local update phase relying solely on phoneme loss values), gradient backpropagation of phoneme loss values ​​to the large language model is blocked. This allows the phoneme prediction network to quickly learn the classification ability to map shared hidden state sequences to correct pronunciation structures without interfering with the extraction of underlying acoustic features. In the second training phase (i.e., the local update phase relying solely on audio loss values), gradient backpropagation of audio loss values ​​to the phoneme prediction network is blocked, allowing the large language model and audio prediction network to focus on improving the acoustic fitting quality of the audio waveform. In the third training phase, all network layers in the audio synthesis model to be trained are fully unfrozen, and weight coefficients (such as...) are introduced. and The audio loss value and the phoneme loss value are fused into a second loss value. At this stage, this second loss value is used for end-to-end global parameter optimization, so that the large language model, which already has good audio and phoneme single-item prediction capabilities, can further enhance the temporal alignment and consistency of the two modal outputs within a unified error space.

[0188] For example, suppose the audio synthesis model to be trained is configured to undergo three training phases during the configuration phase, and the currently calculated audio loss value is 1.2, and the phoneme loss value is 0.5. In the first training phase (e.g., the first 1000 iterations), the parameters of the large language model and the audio prediction network are locked, and only the gradient of the phoneme loss value of 0.5 is backpropagated to the phoneme prediction network, fine-tuning the internal weights of the phoneme prediction network. In the second training phase (e.g., the next 5000 iterations), the parameters of the phoneme prediction network are locked, and only the gradient of the audio loss value of 1.2 is backpropagated to the large language model and the audio prediction network. In the third training phase (e.g., the remaining tens of thousands of iterations), all parameters are unlocked. First, the audio loss value of 1.2 is multiplied by the preset weight 0.7, and the phoneme loss value of 0.5 is multiplied by the preset weight 0.3. The products are then added together to calculate a second loss value of 0.99. Subsequently, the system uses the second loss value of 0.99 to simultaneously calculate the global error gradients of the large language model, the audio prediction network, and the phoneme prediction network, and performs joint updates on all model parameters of these three network structures. When the second loss value drops to a preset convergence threshold (e.g., 0.1), the network entity at this point is output and identified as the trained audio synthesis model.

[0189] In the embodiments of this application, some network parameters are frozen in the first and second training phases, respectively, to achieve independent capability ramping for phoneme prediction and audio generation tasks. This effectively avoids model parameter gradient conflicts caused by differences in multi-task objectives in the early stages of training. In the third training phase, a joint update of all parameters is performed by calculating a second loss value that includes audio loss and phoneme loss. This enables the large language model, audio prediction network, and phoneme prediction network, which already have certain single-item prediction capabilities, to achieve deep feature fusion under a unified error, significantly improving the generation quality and convergence stability of the finally trained audio synthesis model on multi-dimensional output tasks.

[0190] In some embodiments, the audio synthesis model to be trained includes a phoneme prediction network. See also Figure 8 , Figure 8 The embodiment of this application shows that before extracting features from the input sequence through the audio synthesis model to be trained to obtain the shared hidden state sequence, the model training method provided may further include the following steps 201 to 203.

[0191] In step 201, target configuration information is obtained, and target network type is extracted from the target configuration information.

[0192] Here, target configuration information refers to the information pre-defined for the phoneme prediction network in the audio synthesis model before starting the model training task, specifying the network structure and operating parameters. Target configuration information can be obtained from a configuration file or command-line arguments. The target network type refers to the specific neural network architecture classification (e.g., single-layer linear mapping network, multilayer perceptron network, or transformer-decoder network) parsed from the target configuration information, indicating the appropriate architecture for the phoneme prediction network. The target prediction network refers to the initial network entity structure dynamically instantiated in memory according to the target network type indication.

[0193] In some embodiments, obtaining target configuration information and extracting the target network type from the target configuration information can be achieved in the following ways: reading a configuration file preset on the local machine or in the cloud, or receiving input command-line parameters; determining the configuration file or command-line parameters as the target configuration information; parsing the key-value pairs contained in the target configuration information to identify the field values ​​used to define the network structure of the phoneme prediction network; classifying the specific network architecture corresponding to the field value and determining it as the target network type.

[0194] For example, suppose we need to start an audio synthesis model training task. First, we read the configuration file and load its contents into memory, identifying it as the target configuration information. We then call the configuration parser to retrieve and extract the value of the field with the key "network type" from this target configuration information. Assuming the value of this field is "MLP" (i.e., Multilayer Perceptron), we determine "Multilayer Perceptron Network" as the target network type required for the current training.

[0195] In step 202, a target prediction network belonging to the target network type is constructed.

[0196] Here, the target prediction network refers to a neural network entity with specific network layer connections that is initialized and generated in computer memory according to the indication of the target network type. The parameters of each network layer within the target prediction network can be default initial values ​​or random values ​​that have not been trained by any audio synthesis model.

[0197] In some embodiments, constructing a target prediction network corresponding to the target network type can be achieved by: obtaining the vector dimension of the shared hidden state sequence output by the large language model and the category dimension range of the probability distribution of the phoneme category to be predicted; determining the intermediate network hierarchy relationship that maps the vector dimension to the category dimension range according to the target network type; initializing the corresponding neural network layer entities in sequence according to the intermediate network hierarchy relationship, and determining the network architecture formed by combining these neural network layer entities as the target prediction network.

[0198] For example, assume the target network type is a multilayer perceptron network. The vector dimension of the shared hidden state sequence output by the current large language model is 4096, and the preset set of phonetic symbols has 100 categories (i.e., the category dimension range is 100). The multilayer perceptron network is assembled in memory: first, a first linear layer entity with an input feature dimension of 4096 is initialized; then, an activation function layer entity and a random deactivation layer entity are initialized; finally, a second linear layer entity with an output category dimension of 100 is initialized. The first linear layer entity, activation function layer entity, random deactivation layer entity, and second linear layer entity are concatenated sequentially according to the forward data flow to construct a complete network architecture. This network architecture, which includes input feature mapping, intermediate activation processing, and output classification mapping, is determined as the current target prediction network. At this point, the model parameters such as weights and biases within this target prediction network have not yet been assigned any historical model parameters.

[0199] In some embodiments, see Figure 9 , Figure 9 The construction of the target prediction network belonging to the target network type in step 202 is shown to include steps 2021 to 2022.

[0200] In step 2021, target prediction network classes belonging to the target network type are extracted from the target component library.

[0201] The target component library includes prediction network classes corresponding to various network types.

[0202] Here, the target component library refers to a set of class dictionary mapping tables pre-registered in the runtime environment of the audio synthesis model training program. The target component library can include prediction network classes corresponding to various network types for developers to call at any time. These prediction network classes corresponding to various network types refer to abstract code templates of network architectures, such as single-layer linear mapping network classes, multilayer perceptron network classes, and transformer-decoder network classes, respectively mapped in the target component library. The target prediction network class refers to the abstract code template of the network architecture uniquely corresponding to the target network type, retrieved from the target component library based on that target network type.

[0203] In some embodiments, extracting the target prediction network class belonging to the target network type from the target component library can be achieved in the following ways: obtaining a pre-initialized target component library, which may include multiple network types and their corresponding prediction network classes mapping key-value pairs; performing a matching query operation in the target component library using the target network type as the index key; extracting the code definition structure of the prediction network class corresponding to the target network type from the successfully matched key-value pairs, and determining the extracted code definition structure as the target prediction network class.

[0204] For example, assuming that before starting the model training task, the program framework has already registered the target component library in memory, this target component library contains single-layer linear mapping network classes corresponding to the "single-layer linear" network type, multilayer perceptron network classes corresponding to the "multilayer perceptron (MLP)" network type, and transformer decoder network classes corresponding to the "transformer" network type. The string "MLP" is parsed and extracted from the local configuration file and identified as the target network type. Then, using "MLP" as the query key, the target component library already loaded in memory is accessed. A matching query operation is performed in this target component library, successfully matching the code definition structure of the multilayer perceptron network class corresponding to the key name "MLP". The code definition structure of this multilayer perceptron network class is directly returned and identified as the target prediction network class to be instantiated.

[0205] In some embodiments, prior to step 2021, the model training method provided in this application embodiment can also be implemented in the following way: First, for each network type, the prediction network class to be registered is registered to obtain the prediction network class corresponding to the network type; then, the prediction network classes corresponding to multiple network types are stored in the target component library.

[0206] Here, "network type" refers to the specific neural network architecture classification (e.g., linear mapping network, multilayer perceptron network, etc.) used to indicate the phoneme prediction network. "Prediction network class" refers to the entity structure template written in the program code, used to dynamically instantiate the neural network architecture corresponding to the network type. "Prediction network class to be registered" refers to prediction network classes that have not yet been globally and uniformly identified and managed. "Target component library" refers to a data set built in computer memory specifically used for centrally storing and managing the prediction network classes corresponding to the various registered network types mentioned above.

[0207] In some embodiments, for each network type, the prediction network class corresponding to multiple network types is registered to obtain the prediction network class corresponding to the network type. This can be achieved in the following ways: obtaining a predefined network component registration interface; calling the network component registration interface when the program loads the code containing the prediction network class to be registered; passing the prediction network class corresponding to the network type as an input parameter to the network component registration interface; and using the network component registration interface to identify the prediction network class to be registered, and determining the prediction network class to be registered after the identification process is completed as the prediction network class corresponding to the network type.

[0208] Here, the network component registration interface refers to a predefined calling interface in the program, specifically used to centrally mark and collect prediction network classes scattered in different code modules.

[0209] For example, suppose a new decoder class based on a Transformer architecture is written to improve the alignment accuracy of pronunciation structures. This newly written decoder class has not yet been recognized; it is identified as the predictive network class to be registered, corresponding to the Transformer network type. In the code file containing the definition of this Transformer decoder class, the program calls the network component registration interface while declaring the class. When the code file is loaded at runtime, the network component registration interface is automatically triggered, capturing the Transformer decoder class to be registered as a parameter. This interface records the correspondence between the Transformer network type and the class object in an internal data structure. After recording, the interface returns the successfully marked Transformer decoder class, identified as the predictive network class corresponding to the Transformer network type.

[0210] Before constructing the target prediction network, this application embodiment performs a unified pre-registration process on prediction network classes containing various network types, enabling the automatic marking and identification of code modules for adding or modifying various network architectures. The prediction network classes after registration are centrally stored in the target component library, which not only establishes a well-structured and efficient component mapping set in memory, but also completely realizes the classification of phoneme prediction network structures and the physical isolation of the main training process code from the underlying engineering architecture. This greatly improves the backward compatibility and non-intrusive expansion efficiency of the audio synthesis model to be trained in terms of pronunciation structure prediction capabilities.

[0211] In step 2022, the target prediction network class is dynamically instantiated based on the target configuration information to obtain the target prediction network.

[0212] Here, dynamic instantiation refers to the object creation process during program execution, which involves allocating computational graph resources in computer memory and generating an actual executable neural network entity based on the extracted target prediction network and combined with specific configuration parameters passed in from the outside. The target prediction network refers to a specific neural network architecture with a feature input mapping to a category output mapping connection relationship generated after the above dynamic instantiation process.

[0213] In some embodiments, the target prediction network class is dynamically instantiated based on the target configuration information to obtain the target prediction network. This can be achieved by: obtaining the network hyperparameters contained in the target configuration information; passing the obtained network hyperparameters as input parameters to the constructor of the target prediction network class; calling the configuration creation interface to execute the constructor of the target prediction network class, dynamically generating a neural network object containing a specific network layer topology in memory; and determining the generated neural network object as the target prediction network.

[0214] For example, suppose the target prediction network class is a multilayer perceptron network class, and the network hyperparameters {hidden layer dimension ratio: 1.0, inactivation rate: 0.2} are read from the target configuration information. During dynamic instantiation, a unified configuration creation interface is called, and the above network hyperparameters are input as keyword parameters into the constructor of the multilayer perceptron network class. During the execution of the constructor, the first and second linear layer nodes with equal input and output dimensions are initialized according to the passed hidden layer dimension ratio parameter: 1.0; the random inactivation layer nodes with a dropout probability of 0.2 are initialized according to the passed inactivation rate parameter: 0.2; and the activation function layer nodes are connected to the above linear layers and inactivation layers in sequence, finally generating a complete multilayer perceptron network object in memory. This network object is output and identified as the target prediction network.

[0215] This application's embodiments accurately extract the target prediction network class corresponding to the target network type from a target component library containing prediction network classes for various network types, achieving decoupling and modular management of the phoneme prediction network architecture in the underlying code library. Based on target configuration information, the extracted target prediction network class is dynamically instantiated, thereby directly generating a target prediction network adapted to the current training requirements without modifying the original large language model or audio prediction network code flow. This component library-based and dynamically instantiated construction method significantly enhances the architectural scalability of the audio synthesis model to be trained on the audio prediction network.

[0216] In step 203, a phoneme prediction network is constructed based on the target prediction network.

[0217] In some embodiments, the construction of a phoneme prediction network based on a target prediction network can be achieved by: extracting each initial parameter node contained in the target prediction network; generating initial model parameters for each initial parameter node using a preset random numerical distribution generation strategy; writing the generated initial model parameters into the corresponding initial parameter nodes respectively; and determining the target prediction network after writing the initial model parameters as the phoneme prediction network.

[0218] For example, suppose a multilayer perceptron network consisting of a first linear layer, an activation function layer, a random deactivation layer, and a second linear layer has been instantiated and is designated as the target prediction network. When constructing the phoneme prediction network, the weight matrix nodes and bias vector nodes (i.e., initial parameter nodes) of the first and second linear layers within the target prediction network are first extracted. Then, a normal distribution random number generation function is called to generate small-amplitude floating-point random values ​​with a mean of 0 for the weight matrix nodes as initial model parameters, and simultaneously, all-zero values ​​are generated for the bias vector nodes as initial model parameters. These generated initial model parameters are then written into the corresponding nodes of the first and second linear layers one by one. After all initial parameter nodes have been assigned specific initial model parameters, the multilayer perceptron network that has completed the parameter assignment operation is formally established as the phoneme prediction network in the audio synthesis model to be trained.

[0219] Before extracting features from the input sequence of the audio synthesis model to be trained, this embodiment of the application obtains target configuration information and extracts the target network type from it, thereby achieving the decoupling configuration of the phoneme prediction network structure and the backbone structure of the large language model. Based on the dynamically extracted target network type, a target prediction network is constructed, and based on this target prediction network, a phoneme prediction network that actually participates in forward propagation is constructed. This not only enables the audio synthesis model to be trained to flexibly adapt to diverse phoneme prediction networks according to different task requirements, but also effectively avoids intrusive modifications to the original audio prediction process when switching the network type of the phoneme prediction network. This significantly improves the architectural scalability and engineering maintenance efficiency of the multimodal prediction model in the development and deployment phases.

[0220] In some embodiments, see Figure 10 , Figure 10 The step 203, which involves constructing a phoneme prediction network based on a target prediction network, may include steps 2031 to 2033.

[0221] In step 2031, historical model parameters are obtained, and the parameter mapping table is queried based on the historical model parameters.

[0222] The parameter mapping table includes the mapping relationship between different model parameters and different parameter positions.

[0223] Here, historical model parameters refer to the model parameters of the phoneme prediction network acquired before the start of this audio synthesis model training task. Historical model parameters may include the network weight matrix and bias vector data of the phoneme prediction network. The parameter mapping table is a set of mapping rules built in memory to indicate where historical model parameters should be assigned to target parameter positions within the target prediction network. Different model parameters refer to the relatively independent weight or bias data blocks stored in the historical model parameters. Different parameter positions refer to the specific parameter nodes within each network layer of the target prediction network that receive numerical assignments. The mapping relationship refers to the linking rules recorded in the parameter mapping table that correspond one-to-one with the different model parameters and their corresponding positions in the newly constructed target prediction network.

[0224] In some embodiments, obtaining historical model parameters and querying the parameter mapping table based on the historical model parameters can be achieved in the following ways: loading model parameter data pre-saved in the storage medium; determining the loaded model parameter data as historical model parameters; obtaining the parameter mapping table preset in the audio synthesis model configuration environment; traversing each extracted historical model parameter and inputting the extracted historical model parameters into the parameter mapping table to perform a matching query operation.

[0225] During the specific execution of the above acquisition and query operations, the network type of the phoneme prediction network in the audio synthesis model may change (e.g., from a single-layer linear mapping network to a multilayer perceptron network). Historical model parameters saved in older versions often cannot be directly loaded into the newly created target prediction network through simple structural alignment. To achieve backward compatibility and non-intrusive expansion, a parameter mapping table is introduced. This parameter mapping table pre-defines the mapping relationships between different model parameters and different parameter positions in the target prediction network. By querying each parsed historical model parameter in this parameter mapping table, it is possible to automatically identify which historical model parameters can be reused and transferred to the new target prediction network.

[0226] For example, suppose we are currently initiating a fine-tuning training task based on a multilayer perceptron network (i.e., an object prediction network). To accelerate model convergence, a set of weight data is loaded from a pre-saved file and identified as historical model parameters. Parsing this set of historical model parameters reveals a 4096×100 floating-point matrix (i.e., one of the different model parameters). Simultaneously, a pre-configured parameter mapping table is read, containing a clear mapping relationship: indicating that the 4096×100 floating-point matrix corresponds to a second linear layer node in the newly constructed multilayer perceptron network (i.e., one of the different parameter positions). Next, the 4096×100 floating-point matrix is ​​extracted and a lookup operation is performed in the parameter mapping table. Based on this, the parameter mapping table returns the corresponding query result according to the pre-defined mapping relationship, guiding the specific assignment position of the historical model parameter in subsequent steps.

[0227] In step 2032, when the parameter position corresponding to the historical model parameter is found, the found parameter position is determined as the target parameter position corresponding to the historical model parameter in the target prediction network.

[0228] Here, the target parameter location refers to the specific neural network layer node actually allocated within the target prediction network to receive the specific historical model parameters for numerical overlay updates, such as the weight matrix or bias vector of a specific linear layer.

[0229] In some embodiments, when a parameter position corresponding to a historical model parameter is found, the found parameter position is determined as the target parameter position corresponding to the historical model parameter in the target prediction network. This can be achieved in the following way: based on the input historical model parameter, retrieve the mapping relationship containing the historical model parameter in the parameter mapping table; in the successfully retrieved mapping relationship, extract a specific parameter position from the different parameter positions bound to the historical model parameter; and determine the extracted parameter position as the target parameter position corresponding to the historical model parameter in the target prediction network.

[0230] For example, suppose we are currently processing a historical model parameter (e.g., a set of 4096×100-dimensional floating-point weight matrices used in the old model to enhance pronunciation prediction). This floating-point weight matrix is ​​used as the search key and queried in a pre-acquired parameter mapping table. This parameter mapping table stores a predefined mapping relationship: "Old pronunciation enhancement weight matrix → Second linear layer weights of the new multilayer perceptron network". When this mapping relationship is found, the specific parameter position at the end pointed to by this mapping relationship (i.e., the network node corresponding to "Second linear layer weights of the new multilayer perceptron network") is extracted. The extracted parameter position is then formally determined as the target parameter position in the target prediction network corresponding to this 4096×100-dimensional historical model parameter.

[0231] In step 2033, the historical model parameters are assigned to the target parameter positions, and the assigned target prediction network is determined as the phoneme prediction network.

[0232] Here, assigning historical model parameters to the target parameter location refers to the data writing operation in memory, based on the extracted historical model parameter values, overwriting and replacing the original initial parameter values ​​(such as random numbers or all zero values) of the target prediction network at the target parameter location. The target prediction network after assignment refers to the neural network entity obtained after completing the writing operation of all successfully matched historical model parameters.

[0233] In some embodiments, assigning historical model parameters to the target parameter positions and determining the assigned target prediction network as a phoneme prediction network can be achieved in the following way: obtaining the specific floating-point numerical matrix or vector contained in the historical model parameters; locating the storage tensor inside the target prediction network corresponding to the target parameter positions; forcibly writing the specific floating-point numerical matrix or vector contained in the historical model parameters into the storage tensor, replacing the original initial values; after completing the writing operation of all historical model parameters, determining the target prediction network that has undergone parameter updates as a phoneme prediction network.

[0234] This application's embodiments acquire historical model parameters and query a parameter mapping table containing mapping relationships between different model parameters and different parameter positions, achieving a precise correspondence between historical model parameters and newly constructed target prediction network nodes. When a corresponding parameter position is found, it is determined as the target parameter position, and the historical model parameters are accurately assigned to that target parameter position, thereby establishing the assigned target prediction network as a phoneme prediction network. This parameter-oriented loading mechanism based on mapping relationships not only allows the newly constructed phoneme prediction network to seamlessly inherit the feature extraction experience accumulated in previous training tasks, significantly improving the starting performance and convergence efficiency of model joint training, but also completely solves the engineering bottleneck of backward incompatibility and reuse of old model parameters caused by network layer adjustments from the underlying architecture, enhancing the upgrade flexibility of audio synthesis models on audio prediction networks.

[0235] In some embodiments, after obtaining the trained audio synthesis model, see [link to documentation]. Figure 11 , Figure 11 The model training method provided in the embodiments of this application may further include steps 301 to 304.

[0236] In step 301, the input data sequence is obtained.

[0237] Here, the input data sequence refers to the multimodal sequence to be processed, which is received from external input information or automatically generated, when the audio synthesis model completes all training stages and is put into practical application (i.e., the inference stage). For example, the input data sequence is a concatenated sequence of input features containing text data, speaker timbre, and other conditions.

[0238] In some embodiments, input text and speaker timbre are acquired. Based on the input text and speaker timbre, an input data sequence is determined. The specific process for determining the input data sequence is similar to the process for determining the input sequence in the above embodiments, and will not be described again here.

[0239] In step 302, the trained audio synthesis model is used to predict the input data sequence to obtain audio prediction data and phoneme prediction data.

[0240] Here, audio prediction data refers to the discrete acoustic feature sequence (such as Speech Token) that represents the generated sound, output by the audio prediction network in the trained audio synthesis model after receiving the shared hidden state sequence extracted from the input data sequence by the large language model. Phoneme prediction data refers to the classification probability distribution of the pronunciation structure at each time step, synchronously output by the phoneme prediction network in the trained audio synthesis model.

[0241] In step 303, the audio prediction data is waveform converted to obtain the audio waveform.

[0242] Here, audio waveform refers to the continuous analog signal or digital audio file generated after acoustic decoding of audio prediction data, which can be directly output by an audio playback device.

[0243] In some embodiments, the audio prediction data is waveform-converted according to a preset sampling rate to obtain an audio waveform. The discrete acoustic feature sequence contained in the audio prediction data is obtained; a preset sampling rate (e.g., 16kHz or 24kHz) is obtained; the extracted discrete acoustic feature sequence is input into a pre-configured vocoder model; the vocoder model performs acoustic feature decoding and time-domain signal reconstruction on the discrete acoustic feature sequence according to the preset sampling rate, and outputs a continuous audio waveform.

[0244] For example, suppose the currently acquired audio prediction data is a sequence containing 500 integer discrete acoustic features. The preset sampling rate is 24kHz. The neural vocoder module converts the audio prediction data according to the 24kHz generation frequency standard, obtaining a continuous digital audio signal data with a duration of approximately 10 seconds and high-quality timbre and coherent intonation. This digital audio signal data is encapsulated into a standard audio file and defined as an audio waveform for subsequent voice playback or virtual human driving modules to use.

[0245] In step 304, the phoneme prediction data is converted to obtain a phoneme sequence.

[0246] Here, the phoneme sequence refers to the final output sequence obtained after decoding and formatting the phoneme prediction data, which is used to structurally represent the pronunciation features corresponding to the audio waveform.

[0247] In some embodiments, phoneme prediction data is converted into a phoneme sequence using a dual representation of phoneme IDs and phonetic symbols. The phoneme sequence may include a phoneme ID sequence and a phonetic symbol sequence. The phoneme ID sequence refers to the data sequence in the phoneme sequence that uses discrete integer category identifiers to represent each articulation structure. The phonetic symbol sequence refers to the data sequence in the phoneme sequence that uses readable, standardized phonetic symbols to represent each articulation structure. This phonetic symbol sequence can be used to meet the standardization requirements of manual pronunciation evaluation, mouth shape-driven rule mapping, or text system integration. The dual representation refers to a mechanism that simultaneously converts and outputs phoneme prediction data as a combination of the aforementioned phoneme ID sequence and phonetic symbol sequence.

[0248] In some embodiments, the phoneme prediction data is converted into a phoneme sequence according to a dual representation of phoneme numbers and phonetic symbols. This can be achieved by: obtaining the phoneme category probability distribution of the phoneme prediction data at each sequence position; extracting the target category with the highest probability value from the corresponding phoneme category probability distribution for each sequence position; combining the target categories extracted from each sequence position according to time sequence to obtain a phoneme number sequence; obtaining a pre-configured mapping dictionary containing the correspondence between phoneme numbers and phonetic symbols; replacing each target category in the phoneme number sequence with the corresponding phonetic symbol according to the mapping dictionary to obtain a phonetic symbol sequence; and jointly determining the phoneme sequence with the phoneme number sequence and the phonetic symbol sequence as a phoneme sequence in a dual representation.

[0249] For example, suppose the currently acquired phoneme prediction data is a matrix containing probability distributions for 100 phonetic symbol categories over a sequence with a time step length of 50. During the decoding conversion, these 50 time steps are traversed. In the first time step, the target category with the highest probability value in the probability distribution matrix (let's assume the target category number is 45) is extracted and determined as the first element of the phoneme number sequence. This process continues, obtaining a phoneme number sequence consisting of 50 integers such as 45 and 12. Next, a preset mapping dictionary is called, which records that the integer 45 corresponds to the phonetic symbol p, and the integer 12 corresponds to the International Phonetic Alphabet symbol I. The 45 in the phoneme number sequence is replaced with p, and the 12 is replaced with I, resulting in a phonetic symbol sequence consisting of 50 phonetic symbols such as P and I. This phoneme number sequence containing 50 integers and the phonetic symbol sequence containing 50 characters are packaged together, and this data set with dual representation is determined as the final inference output phoneme sequence.

[0250] In this embodiment, after obtaining the inference input sequence, the trained audio synthesis model predicts the inference input sequence, enabling simultaneous and efficient acquisition of audio prediction data and phoneme prediction data without relying on any external independent phoneme recognition model. The generated audio prediction data is waveform-converted to obtain a highly natural audio waveform, and the generated phoneme prediction data is also converted to obtain a structured inference phoneme sequence, greatly improving the generation efficiency and downstream adaptation flexibility of the audio synthesis model in multimodal interaction scenarios.

[0251] In some embodiments, a phonetic symbol sequence is extracted from the phoneme sequence. The phonetic symbol sequence is then split using a space delimiter to obtain the target text data. The target text data is then stored as a text file.

[0252] Here, the space separator refers to the whitespace character used in computer text processing to distinguish adjacent independent characters or word units (e.g., 0x20 in ASCII code). The target text data refers to a string of text formed by concatenating the individual phonetic symbols in a phonetic symbol sequence using space separators, conforming to conventional reading habits and downstream system parsing standards. The text file format refers to a file type that persistently stores the target text data on computer storage media in plain text format (e.g., with the .txt extension).

[0253] In some embodiments, the character array portion consisting of phonetic symbols in the phoneme sequence is identified, and the character array portion is determined as the phonetic symbol sequence; a space separator is inserted between every two adjacent phonetic symbols in the extracted phonetic symbol sequence, the character array after inserting the space separator is merged into a continuous string, and the string is determined as the target text data; a plain text file with a specified name is created on a local disk or cloud storage system; the target text data is written to the plain text file and a save operation is performed.

[0254] This application accurately extracts the phonetic symbol sequence from the phoneme sequence and performs clear separation processing on the phonetic symbol sequence based on the space separator, so that the originally high-dimensional and complex pronunciation structure prediction results are converted into structured and easily parsed target text data; the target text data is persistently stored in the form of a general text file, which not only fully preserves the high-precision phoneme sequence without being separated from the audio synthesis model running environment, but also provides a text file with cross-platform compatibility and high reading efficiency for downstream application systems such as lip-sync multimodal driving and automatic pronunciation evaluation.

[0255] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0256] In related technologies, audio synthesis systems typically aim to generate highly natural-looking speech signals. In these systems, phoneme information is primarily used in the pre-training data preparation phase, such as generating phoneme annotations using forced alignment tools to assist the model in learning the correspondence between speech and text. With the application of large language models in speech generation, some audio synthesis systems have begun to utilize the sequence modeling capabilities of large language models to generate speech tokens, but the output still mainly focuses on speech signals or discrete speech representations. During the inference phase, phoneme sequences are usually not explicitly output, making it difficult to directly use phoneme information for downstream tasks such as lip-syncing and pronunciation evaluation.

[0257] The related technologies suffer from at least the following problems: Lack of explicit modeling and standardized output of phoneme information: Phonemes are typically only used as pre-training annotations or implicit intermediate variables. The inference stage cannot directly output structured, standardized phoneme sequences, requiring the deployment of separate phoneme parsing modules for downstream tasks such as lip-syncing and pronunciation evaluation. This significantly increases overall complexity and inference latency (typically increasing latency by more than 50ms). Lack of structured intermediate representation in audio synthesis models: Audio synthesis models output only audio or speech tokens, making it difficult to directly support multimodal or analytical applications. A "dual-model separation" problem exists between phoneme prediction and speech generation: Phoneme prediction models (such as MFA-based alignment tools) or post-processing modules typically need to be deployed separately. This not only increases deployment costs (hardware resource consumption increases by approximately 40%), but also lacks unified contextual constraints between phoneme prediction and speech generation results, with an alignment error rate exceeding 15%, making it difficult to meet the requirements of high-precision applications. Decoder expansion suffers from an "intrusive modification" bottleneck: the phoneme prediction module has a fixed structure (mostly a single linear layer). When the decoder structure needs to be changed for different scenarios (such as fast training or high-quality prediction), it is often necessary to modify the main training or inference process code. The expansion cycle is long (average 1-2 weeks), and the compatibility is poor, making it difficult to reuse the parameters of the old model.

[0258] Therefore, a new phoneme prediction decoder architecture is needed that can be deeply integrated with large language models and has good scalability.

[0259] Based on this, this application provides a model training method, which is a dual-predictor phoneme decoding extension and training scheme based on a large language model. The core concept of this application is to overcome the structural bottleneck of "separation between phoneme prediction and speech generation, and low flexibility of decoder extension" in related technologies, and to build an integrated architecture of "dual-predictor collaboration + dynamic extension + hybrid training" on the output side of the large language model, so that the audio synthesis model can generate highly natural speech while simultaneously outputting a structured phoneme sequence that is strictly aligned with the speech.

[0260] The technical concept of the embodiments of this application includes the following aspects:

[0261] 1) Architectural collaboration: On the output side of the large language model, a dual prediction head parallel architecture of "speech token prediction head (corresponding to the audio prediction network in the above embodiment) - phoneme prediction decoder (corresponding to the phoneme prediction network in the above embodiment)" is designed. The two share the hidden state representation of the output of the large language model (LLM) (corresponding to the shared hidden state sequence in the above embodiment), without the need to introduce an additional acoustic coding network, thus realizing the deep integration of speech generation and phoneme prediction.

[0262] 2) Extension Mechanism: Based on the factory pattern, a three-layer decoder extension system of "registry - configuration creation - backward compatibility" is built, which supports zero-intrusion addition and switching of various types of phoneme prediction decoders such as single-layer linear, multilayer perceptron (MLP) and transformer.

[0263] 3) Data alignment: By adopting the strategy of "nearest neighbor interpolation resampling + ignoring the IGNORE_ID protection", the mismatch between the phoneme tag sequence and the speech token sequence (corresponding to the audio tag sequence in the above embodiment) in terms of frame shift and length is solved, ensuring the stability of dual-task joint training.

[0264] 4) Training optimization: Design a hybrid training paradigm of "phased freezing + course learning" to gradually improve phoneme prediction accuracy while ensuring the quality of speech generation.

[0265] 5) Structured output: During the inference stage, the speech waveform (corresponding to the audio waveform in the above embodiment) and the International Phonetic Alphabet (IPA) standard phoneme sequence (corresponding to the phonetic symbol sequence in the above embodiment) are output simultaneously to directly adapt to downstream application scenarios such as lip-syncing, pronunciation evaluation, and multimodal alignment.

[0266] This application provides an audio synthesis system that supports phoneme prediction, which can be deployed on a local server, in the cloud, or in a multi-machine, multi-GPU training / inference cluster. Figure 12 This is a schematic diagram of the architecture of the audio synthesis system provided in an embodiment of this application. See also... Figure 12 The audio synthesis system includes at least a large language model 1201, a speech prediction head 1202 (corresponding to the audio prediction network in the above embodiments), a phoneme prediction decoder 1203 (corresponding to the phoneme prediction network in the above embodiments), and a decoder management module 1204.

[0267] In terms of shared layer design, the Large Language Model 1201 (LLM Backbone) adopts a Transformer network structure to perform contextual modeling on the input sequence and output a shared hidden state sequence. The input sequence includes text tokens (which can constitute the text feature sequence in the above embodiments), speech tokens (which can constitute the audio cue sequence in the above embodiments), control tags, and speaker features (embedding). Speaker features are feature vectors used to control the generated speech timbre. A shared hidden state sequence is used. The dimension can be configured as a unified feature representation. For example, typical values ​​for the dimension include 4096 or 768. Shared hidden state sequence. Simultaneously inputting into two parallel prediction heads forms a dual-prediction-head collaborative architecture (sharing LLM hidden states). This architecture reuses the contextual modeling capabilities of large language models, completing multi-task prediction without introducing additional acoustic coding networks, achieving simultaneous modeling of speech generation and phoneme prediction, thereby avoiding additional computational overhead.

[0268] Control tokens refer to two special tokens used to construct the input sequence of the LLM. There are two types of control tokens: the first is the start-of-sequence / end-of-sequence (SOS / EOS) token, ID: 0, which marks the beginning and end of the input sequence and is placed at the very beginning. The second is the task token, ID: 1, which distinguishes the task type (e.g., text-to-speech) and is located between the text token and the speech token. For example, the input sequence is: [SOS / EOS] → [speaker embedding] → [text token] → [Task ID] → [speech token]. Specifically, the input sequence is [SOS / EOS] + [text token1] + [text token2] + ... + [text tokenN] + [Task ID] + [speech token1] + [speech token2]. The control tokens are converted into vectors through the LLM embedding layer and participate in the entire input sequence encoding process, affecting the output of the two prediction heads.

[0269] For the main task prediction head, the Speech Token Head 1202 is used based on the shared hidden state sequence. The system predicts a discrete speech representation (a sequence of speech tokens, corresponding to the audio prediction result in the above embodiments). The output dimension is configurable, typically 1024 or 2048 dimensions, to adapt to mainstream speech tokenizers. The predicted discrete speech representation is used by the subsequent acoustic / vocoder module to generate speech, i.e., as input to the vocoder 1205, to generate a high-naturalness speech waveform of 16kHz or 24kHz (corresponding to the audio waveform in the above embodiments), ultimately yielding a speech waveform output (16kHz / 24kHz).

[0270] For the auxiliary task prediction head, the Phone Decoder Head 1203 is based on a shared hidden state sequence. The phoneme category sequence is predicted, and the specific output is a phoneme category probability distribution (corresponding to the phoneme prediction result in the above embodiment). The category dimension ranges from 80 to 120 (e.g., the IPA phoneme category set, the number of categories is configurable, typically about 100 IPA phoneme categories), resulting in a phoneme sequence output (IPA standard). The phoneme prediction decoder 1203 supports multilingual phoneme configuration to adapt to different language scenarios.

[0271] In terms of extension features, the decoder management module 1204 (Decoder Factory / Registry) uses a dynamic extension mechanism based on the factory pattern to dynamically configure, create, register, and switch phoneme prediction decoders with different structures. The decoder structure is configurable (e.g., Linear, MLP, Transformer) and supports loading from configuration files or the command line, with backward compatibility for older parameters. This mechanism enables flexible switching and extension of different decoder structures without modifying the Large Language Model (LLM) backbone network structure and the main task head logic (speech token prediction head).

[0272] The audio synthesis system may also include a training controller module. Figure 12 (Not shown in the diagram) is used to select a multi-task joint training or phased freeze training strategy, and to control the trainable parameters of different modules (large language model, speech token prediction head, phoneme prediction decoder). Specific training strategies include, for example, phone-only training mode, large language model training mode (LLM-only), and normal mode.

[0273] At the application level, the audio synthesis system may also include an inference output module, which outputs speech results and phoneme sequences during the inference stage, and supports saving the phoneme sequences as text files for downstream use such as lip-syncing and pronunciation evaluation.

[0274] Specific application scenarios include: Scenario A: Multimodal driving of games / virtual humans: Outputting phoneme sequences during inference to provide direct input for lip-sync animation driving, achieving synchronization between speech and lip movements; Scenario B: Speech quality inspection and pronunciation evaluation: Comparing the output phoneme sequence with the standard phoneme sequence, calculating accuracy or F1 scores, etc., to evaluate the pronunciation consistency of synthesized speech.

[0275] The structure and expansion mechanism of the phoneme prediction decoder are described in detail below.

[0276] In one embodiment, the phoneme prediction decoder employs a single-layer linear mapping structure, and the phoneme prediction decoder processes the shared hidden state sequence output by the large language model. A linear transformation is performed, and the phoneme category probability distribution is output. The parameter size of this structure can be controlled within 1 million; when the input dimension input_dim=768, the number of parameters is approximately 770,000. This embodiment has a simple structure, low computational overhead, and fast convergence speed during training, making it suitable for rapid validation and low-resource deployment scenarios.

[0277] In another embodiment, the phoneme prediction decoder employs a multilayer perceptron (MLP) architecture. The network structure consists of a series of linear layers, activation functions, dropout layers, and linear layers. The hidden layer dimension ratio (hidden_dim_ratio) can be configured to 0.5–2.0, the dropout rate can be configured to 0.1–0.3, and the activation function is a Gaussian Error Linear Unit (GELU) activation function designed based on a Gaussian distribution probability. This embodiment has approximately 3–8 million parameters, demonstrating strong modeling capabilities in phoneme prediction tasks. The phoneme prediction accuracy can reach over 92%, making it suitable for pronunciation evaluation and multimodal driven applications requiring high prediction accuracy.

[0278] In an optional extended embodiment, the phoneme prediction decoder can employ a Transformer-Decoder architecture, comprising 2–4 layers of coding modules, with configurable multi-head attention heads of 8–16, used to model long-range dependencies between phoneme sequences. Compared to linear or MLP structures, this embodiment can further reduce phoneme alignment errors under complex sentence conditions, with an error reduction of approximately 20%.

[0279] For factory pattern extension, this application embodiment constructs a "three-layer dynamic extension mechanism", which realizes the non-intrusive extension of the phoneme prediction decoder through a three-layer dynamic extension mechanism of registration layer-creation layer-compatibility layer.

[0280] Figure 13 This is a schematic diagram of the architecture of the phoneme prediction decoder extension mechanism provided in an embodiment of this application. See also... Figure 13In the Registration Layer 1301, different types of phoneme prediction decoder classes are registered through a decorator (decoder_registry), which is essentially the decoder registry. For example, a declarative registration method can be used to incorporate phoneme prediction decoder classes into a unified component library (corresponding to the target component library in the above embodiments) for management, enabling the decoder type to be automatically identified and invoked during the initialization phase.

[0281] In Creation Layer 1302, during the decoder creation phase, dynamic instantiation of the decoder structure is supported via configuration files or runtime parameters. Specifically, this includes specifying the decoder type and parameter configuration via a YAML (Ain't a Markup Language) configuration file. First, configuration input is provided, for example, including decoder type (decoder_type): MLP; hidden layer dimension ratio (hidden_dim_ratio): 1.0; dropout: 0.2. Alternatively, configuration can be passed via command-line arguments. Then, dynamic instantiation occurs. The API is called to create the decoder object based on the configuration. The entire process requires no modification or additional code writing.

[0282] In Compatibility Layer 1303, to ensure backward compatibility, the extension mechanism uses a parameter mapping table (param_mapping) to automatically adapt the parameters between the old and new decoder versions. For example, it retrieves the parameters from the old decoder and maps them to the corresponding parameter positions in the new decoder, thus ensuring a 100% success rate when loading the new structure onto the old model.

[0283] After the above three layers of processing, the main training / inference flow of the model remains unchanged. Based on this, when a new decoder structure needs to be added, such as a residual decoder (ResNet-Decoder) structure, it only needs to implement the forward() method in the unified interface and complete the registration to be included in the training and inference flow, without modifying the main training or inference logic. Through this mechanism, the overall expansion cycle can be shortened to 1–2 days.

[0284] The model training of this application embodiment is described in detail below. Regarding the training objective, this application embodiment employs joint training of phone loss and speech token loss. During the training phase, two losses are calculated simultaneously: 1) Speech token loss: used for the main speech generation task; 2) Phone loss (a type of cross-entropy loss): used for the phoneme prediction task. The two are trained together to enhance the consistency between phoneme prediction and speech generation.

[0285] Regarding phoneme target resampling alignment strategies, this application addresses the inconsistency in temporal resolution (frame shift) and sequence length between the phoneme category number sequence (phone_id sequence) output by phoneme alignment tools (such as MFA) and the discrete speech representation (speech_token sequence) output by the speech tokenizer. This implementation proposes a two-stage alignment strategy to ensure the stability of joint training for the phoneme prediction and speech generation tasks. For example, the length of the speech_token sequence (frame shift 20ms) is 100, while the length of the phone_id sequence (frame shift 10ms) is 200. This results in an inconsistency between temporal resolution and sequence length.

[0286] Figure 14 This is a flowchart illustrating a two-stage alignment strategy provided in an embodiment of this application. See also... Figure 14 In the resampling stage (Stage 1) 1401, the phoneme category number sequence (phone_id sequence) output by the phoneme alignment tool uses a 10ms frame shift, while the speech discrete representation (speech_token sequence) uses a 20ms frame shift, resulting in a mismatch in sequence length. This embodiment employs nearest-neighbor interpolation resampling during the training data preparation stage to scale the length of the phone_id sequence to match the length of the speech_token sequence.

[0287] For example, when 100 speech_tokens correspond to 200 phone_ids, they are resampled into a phone_id sequence of length 100 through interpolation, thereby achieving target length alignment. In this process, the speech_token sequence is used as the "reference target" input to obtain the resampled phoneme category number sequence (corresponding to the resampled phoneme label sequence in the above embodiment).

[0288] In the protection phase (Stage 2) 1402, invalid regions are labeled. After length alignment, regions in the text segment without corresponding phonemes and padding regions are uniformly labeled with the ignore flag IGNORE_ID, resulting in the target phoneme label sequence (phone_target). The ignore flag has a value of -100. When calculating Phone Loss, the training process automatically masks the positions labeled IGNORE_ID, preventing these samples from participating in gradient calculation and thus avoiding interference from invalid alignment or missing labels on model training.

[0289] The regions without corresponding phonemes are considered text segments, specifically including: SOS / EOS control marker positions (1 position), all text token positions (text_token_len[i] positions), and sequence end marker positions (1 position). For example, in phone_target: the first (1+text_token_len[i]) positions are all marked as IGNORE_ID, 1 position is the SOS / EOS control symbol, and text_token_len[i] positions are text token positions. The text tokens themselves do not have corresponding phonemes; phonemes only correspond to the speech token portion. Therefore, text segments should be ignored in the phoneme prediction target. Furthermore, during model training, for data batches composed of variable-length sequences, the longest sequence in the batch is selected as the baseline, and shorter sequences are padded at the end dimensions. Specifically, a sequence padding function (such as the pad_sequence function) is used to pad the input sequence (lm_input), the audio tag sequence (lm_target), and the target phoneme tag sequence (phone_target), and the padding value used for padding operation is uniformly configured as the ignore flag (IGNORE_ID).

[0290] By employing the aforementioned two-stage alignment strategy, the alignment error rate between the phone_target and the speech_token target can be controlled below 3%, significantly improving the stability of the joint training process and enhancing the overall convergence stability by approximately 25%.

[0291] Figure 15 This is a flowchart illustrating the hybrid training strategy provided in an embodiment of this application. This embodiment designs a "three-mode + course learning" hybrid training strategy to achieve a stable balance between phoneme prediction accuracy and speech generation quality. The specific implementation is as follows:

[0292] See Figure 15The audio synthesis model provided in this application embodiment comprises, from top to bottom: a large language model 1501, a speech prediction head 1502, and a phoneme prediction decoder 1503. Based on this audio synthesis model, the specific implementation of the hybrid training strategy includes the following three modes:

[0293] 1) Phoneme Training Mode (Phone-only) 1504: In this training mode, "Freeze: Large Language Model" and "Freeze: Speech Prediction Head" are executed, i.e., the parameters of the Large Language Model (LLM) and the speech prediction head are frozen; simultaneously, "Train: Phoneme Prediction Decoder" is executed, i.e., only the phoneme prediction decoder is trained. This method can achieve a phoneme prediction accuracy of over 85% within 1–2 epochs. This mode is suitable for rapid model validation and the initial capability learning phase of the phoneme decoder.

[0294] 2) Large Language Model Training Mode (LLM-only) 1505: In this training mode, "Freeze: Phoneme Prediction Decoder" is executed, which freezes the parameters of the phoneme prediction decoder; simultaneously, "Train: Large Language Model" and "Train: Speech Prediction Head" are executed, meaning only the LLM and speech prediction head are trained to ensure the naturalness of speech generation. The phoneme loss value (PhoneLoss) does not participate in gradient backpropagation, thus avoiding interference with the main speech generation task. Under this optimization, the subjective naturalness score of the generated speech can reach ≥4.2.

[0295] 3) Normal Mode 1506: In this training mode, "Joint Training: LLM + Speech Prediction Head + Phoneme Prediction Decoder" is performed. This involves jointly training the LLM, speech prediction head, and phoneme prediction decoder, and using a weighted loss function for end-to-end optimization of the two tasks. The weighted loss function can be in the form of Loss = α × SpeechToken Loss + β × Phone Loss, where α = 0.7 and β = 0.3. This strategy also supports configuration adjustments based on training needs to achieve a balance between speech generation quality and phoneme prediction accuracy.

[0296] Building upon the three training modes described above, this application further introduces a course learning strategy, supporting progressive training in the order of Phone-only → LLM-only → Normal. This approach effectively alleviates the optimization conflict between the two tasks in the early stages of training, making the model training process smoother and improving the overall convergence speed by approximately 30%.

[0297] Figure 16This is a schematic diagram of the dual-output and standardized storage mechanism for the inference stage provided in this application embodiment. During the inference stage, this application embodiment employs a "dual-output + standardized storage" mechanism to simultaneously provide structured, reusable phoneme information while generating highly natural-sounding speech. The specific implementation is as follows:

[0298] Parallel output mechanism: see Figure 16 During the model inference stage 1601 (i.e., during the model inference process), the speech waveform 1602 and phoneme sequence 1603 are obtained through dual outputs. The speech waveform 1602 is output in WAV format with a sampling rate of 16kHz or 24kHz, and the phoneme sequence 1603 is output in a dual representation format of phone_id (corresponding to the phoneme number sequence in the above embodiment) and IPA symbol (corresponding to the phonetic symbol sequence in the above embodiment), thereby maintaining the consistency of internal identification while taking into account the readability and standardization requirements across systems.

[0299] Standardized storage method 1605: Phoneme sequences 1603 are stored in space-separated files (txt files), and the content consists of IPA phoneme symbols arranged in sequence, such as pɪn. ŋ This corresponds to the Chinese word "pinyin". This storage method supports batch export and batch parsing of phoneme sequences, facilitating direct reading and processing by downstream systems.

[0300] Downstream System Adaptation: Based on the aforementioned dual-output and standardized storage mechanism, the generated speech waveforms and phoneme sequences can be directly input into downstream application systems without additional preprocessing. Specifically, the phoneme sequences can be directly input into the lip-sync engine 1606, such as a virtual human system, to achieve synchronized speech and lip movement, with end-to-end latency controlled to ≤30ms. Furthermore, the phoneme sequences can be input into the pronunciation evaluation system 1607 to calculate phoneme prediction accuracy, achieving an F1 score ≥0.9.

[0301] This application embodiment achieves synchronous generation and direct reuse of speech and phoneme information during the inference stage through a dual-output and standardized storage mechanism.

[0302] The embodiments of this application have at least the following technical effects: 1) Explicit output of structured phonemes, significantly improving downstream adaptation efficiency: Standardized IPA phoneme sequences are directly output during the inference stage, eliminating the need for additional deployment of phoneme parsing or post-processing modules, enabling direct access for downstream tasks such as lip-syncing and pronunciation evaluation. This reduces the overall integration cycle of the downstream system from one week to two days, improving downstream adaptation efficiency by approximately 60%. 2) Deep collaboration between two tasks, significantly improving consistency: By sharing the LLM hidden state between the phoneme prediction task and the speech generation task, deep collaborative modeling between the two tasks is achieved, allowing the phoneme prediction results and speech generation results to be optimized synchronously under unified contextual constraints. This allows the alignment error rate to be controlled below 3%, significantly better than traditional schemes using independent models (whose alignment error rate is typically above 15%), improving overall consistency by approximately 30%. 3) Zero-intrusive expansion, significantly reducing maintenance costs: A three-layer factory pattern expansion mechanism enables zero-intrusive expansion of the phoneme prediction decoder. The newly added decoder structure requires no modification to the main training or inference process and is compatible with the parameters of the old model, shortening the decoder expansion cycle from the original 1-2 weeks to 1-2 days, reducing the overall maintenance cost by approximately 80%. 4) Hybrid training strategy for balanced performance optimization of dual tasks: Through a hybrid training strategy of "three modes + course learning," the phoneme prediction accuracy is improved while ensuring the naturalness of speech generation. The subjective naturalness score of speech generation can reach ≥4.2, and the phoneme prediction accuracy can reach ≥92%, thus simultaneously meeting the dual requirements of "generation quality" and "structured output." 5) Strong multilingual adaptability: Supports flexible configuration of 80-120 IPA phonemes, enabling rapid expansion to multilingual application scenarios such as English and Japanese, possessing excellent multilingual adaptability, and suitable for various speech generation and multimodal interaction products.

[0303] The following description continues to illustrate the exemplary structure of the model training device 455 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the model training device 455 in the memory 450 may include:

[0304] The acquisition module 4551 is used to acquire training samples and construct input sequences based on the training samples;

[0305] The extraction module 4552 is used to extract features from the input sequence through the audio synthesis model to be trained, and obtain the shared hidden state sequence.

[0306] The prediction module 4553 is used to perform audio prediction on the shared hidden state sequence to obtain the audio prediction result, and to perform phoneme prediction on the shared hidden state sequence to obtain the phoneme prediction result.

[0307] The determination module 4554 is used to determine the audio loss value based on the audio prediction result and the audio label sequence corresponding to the training sample, and to determine the phoneme loss value based on the phoneme prediction result and the phoneme label sequence corresponding to the training sample.

[0308] The update module 4555 is used to update the model parameters of the audio synthesis model to be trained based on the audio loss value and the phoneme loss value, so as to obtain the trained audio synthesis model.

[0309] In some embodiments, the update module 4555 is further configured to determine at least one training stage corresponding to the audio synthesis model to be trained; in the at least one training stage, the model parameters of the audio synthesis model to be trained are updated based on the audio loss value and the phoneme loss value to obtain the trained audio synthesis model.

[0310] In some embodiments, the update module 4555 is further configured to, when the number of at least one training phase is one, perform a weighted summation of the audio loss value and the phoneme loss value in one training phase to obtain a first loss value; update the model parameters of the audio synthesis model to be trained based on the first loss value to obtain the trained audio synthesis model.

[0311] In some embodiments, the update module 4555 is further configured to obtain the execution order of multiple training stages when there are multiple training stages; and according to the execution order, perform parameter update operations in each training stage in sequence based on the audio loss value and the phoneme loss value to obtain the trained audio synthesis model.

[0312] In some embodiments, the audio synthesis model to be trained includes a large language model, an audio prediction network, and a phoneme prediction network, and at least one training phase includes three training phases; the update module 4555 is further configured to, in the first execution sequence training phase, freeze the model parameters of the large language model and the model parameters of the audio prediction network, and update the model parameters of the phoneme prediction network based on the phoneme loss value; in the second execution sequence training phase, freeze the model parameters of the phoneme prediction network, and update the model parameters of the large language model and the model parameters of the audio prediction network based on the audio loss value; in the third execution sequence training phase, perform a weighted summation of the audio loss value and the phoneme loss value to obtain a second loss value, and update the model parameters of the large language model, the model parameters of the audio prediction network, and the model parameters of the phoneme prediction network based on the second loss value, thereby obtaining the trained audio synthesis model.

[0313] In some embodiments, the determining module 4554 is further configured to resample the phoneme label sequence corresponding to the training sample to obtain a resampled phoneme label sequence, wherein the sequence length of the resampled phoneme label sequence is the same as the sequence length of the audio label sequence before resampling; and to determine the phoneme loss value based on the phoneme prediction result and the resampled phoneme label sequence.

[0314] In some embodiments, the determining module 4554 is further configured to obtain a first time resolution of the phoneme tag sequence and a second time resolution of the audio tag sequence; determine a target scaling ratio based on the first time resolution and the second time resolution; and perform interpolation processing on the phoneme tag sequence according to the target scaling ratio to obtain a resampled phoneme tag sequence.

[0315] In some embodiments, the determining module 4554 is further configured to construct an initial resampling sequence, wherein the sequence length of the initial resampling sequence is the same as the sequence length of the audio tag sequence; determine the mapping position of the target position index contained in the initial resampling sequence in the phoneme tag sequence based on the target scaling ratio; determine the original position index that is closest to the mapping position from the phoneme tag sequence, and obtain the original tag value corresponding to the original position index in the phoneme tag sequence; fill the original tag value into the target position index of the initial resampling sequence to obtain the resampled phoneme tag sequence.

[0316] In some embodiments, the determining module 4554 is further configured to obtain the total length of the input sequence and determine the first position of the non-audio region in the input sequence; construct an initial phoneme tag sequence with a length equal to the total length of the sequence; fill the position corresponding to the first position in the initial phoneme tag sequence with an ignore flag, and fill the second position in the initial phoneme tag sequence with a resampled phoneme tag sequence, thereby obtaining a target phoneme tag sequence filled with the ignore flag and the resampled phoneme tag sequence, wherein the second position is a position different from the first position; and determine the phoneme loss value based on the phoneme prediction result and the sequence in the target phoneme tag sequence that is not configured with an ignore flag.

[0317] In some embodiments, the audio synthesis model to be trained includes a phoneme prediction network; the model training device 455 further includes a construction module for obtaining target configuration information and extracting a target network type from the target configuration information; constructing a target prediction network belonging to the target network type; and constructing a phoneme prediction network based on the target prediction network.

[0318] In some embodiments, the construction module is further configured to extract target prediction network classes belonging to the target network type from the target component library, wherein the target component library includes prediction network classes corresponding to various network types; and to dynamically instantiate the target prediction network classes based on the target configuration information to obtain the target prediction network.

[0319] In some embodiments, the model training device 455 further includes a registration module, which is used to register the prediction network class corresponding to each network type, obtain the prediction network class corresponding to the network type, and store the prediction network classes corresponding to multiple network types into the target component library.

[0320] In some embodiments, the construction module is further configured to obtain historical model parameters and query a parameter mapping table based on the historical model parameters, wherein the parameter mapping table includes the mapping relationship between different model parameters and different parameter positions; when a parameter position corresponding to a historical model parameter is found, the found parameter position is determined as the target parameter position corresponding to the historical model parameter in the target prediction network; the historical model parameter is assigned to the target parameter position, and the assigned target prediction network is determined as a phoneme prediction network.

[0321] In some embodiments, the model training device 455 further includes an output module for acquiring an input data sequence; predicting the input data sequence using the trained audio synthesis model to obtain audio prediction data and phoneme prediction data; performing waveform conversion on the audio prediction data to obtain an audio waveform; and converting the phoneme prediction data to obtain a phoneme sequence.

[0322] In some embodiments, the model training device 455 further includes a storage module for extracting a phonetic symbol sequence from the phoneme sequence; performing a separation process on the phonetic symbol sequence based on a space separator to obtain target text data; and storing the target text data in the form of a text file.

[0323] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform the model training method described above in this application.

[0324] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the model training method provided in this application. For example, ... Figure 3 The model training method is shown.

[0325] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0326] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0327] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0328] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0329] In summary, this application's embodiments construct a dual-branch prediction architecture with a shared hidden state sequence in the audio synthesis model. This achieves synchronous prediction of audio and phoneme data without introducing additional acoustic coding networks, effectively reducing the deployment complexity of multi-dimensional data generation. Furthermore, based on a unified feature extraction process, it significantly improves the alignment accuracy between the final output phoneme sequence and the audio data. Simultaneously, during model training, the resampling and ignore-label configuration mechanisms accurately shield the interference from non-audio regions in the input sequence, ensuring the stability of the dual-task joint training. A multi-stage parameter update strategy further improves the accuracy of phoneme prediction while maintaining audio generation quality. Moreover, the introduction of a dynamic instantiation and parameter mapping mechanism based on a target component library enables zero-intrusive expansion of the phoneme prediction network structure and backward compatibility with historical parameters, significantly reducing model maintenance and iteration costs. During the model inference stage, the simultaneous output of audio waveforms and inference phoneme sequences containing IPA symbols allows for direct adaptation to downstream tasks such as lip-syncing or pronunciation evaluation without additional preprocessing.

[0330] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method includes: Obtain training samples and construct input sequences based on the training samples; The input sequence is feature extracted using the audio synthesis model to be trained to obtain a shared hidden state sequence. The audio synthesis model to be trained includes a large language model, an audio prediction network, and a phoneme prediction network. The shared hidden state sequence refers to the intermediate feature representations output by the large language model at each layer or time step, which are used to carry contextual information related to text and speech. Audio prediction is performed on the shared hidden state sequence to obtain audio prediction results, and phoneme prediction is performed on the shared hidden state sequence to obtain phoneme prediction results; Based on the audio prediction results and the audio label sequences corresponding to the training samples, the audio loss value is determined; The phoneme tag sequence corresponding to the training sample is resampled to obtain the resampled phoneme tag sequence, wherein the sequence length of the resampled phoneme tag sequence is the same as the sequence length of the audio tag sequence before resampling; Obtain the total length of the input sequence and determine the first position of the non-audio region in the input sequence; Construct an initial phoneme tag sequence with a length equal to the total length of the given sequence; An ignore flag is filled at the position corresponding to the first position in the initial phoneme tag sequence, and a resampled phoneme tag sequence is filled at the second position in the initial phoneme tag sequence to obtain a target phoneme tag sequence filled with the ignore flag and the resampled phoneme tag sequence. The second position is a position different from the first position. The ignore flag is used to physically isolate non-audio regions in the target phoneme tag sequence that do not have corresponding audio information. Based on the phoneme prediction results and the sequences in the target phoneme tag sequence that do not have the ignore flag configured, the phoneme loss value is determined; Determine at least one training stage corresponding to the audio synthesis model to be trained; When the at least one training phase includes three training phases, in the first training phase of the execution order, the model parameters of the large language model and the model parameters of the audio prediction network are frozen, the gradient backpropagation of the phoneme loss value to the large language model is blocked, and the model parameters of the phoneme prediction network are updated based on the phoneme loss value. In the training phase of the second execution sequence, the model parameters of the phoneme prediction network are frozen, the gradient backpropagation of the audio loss value to the phoneme prediction network is blocked, and the model parameters of the large language model and the model parameters of the audio prediction network are updated based on the audio loss value. In the training phase of the third execution sequence, the audio loss value and the phoneme loss value are weighted and summed to obtain a second loss value. Based on the second loss value, the model parameters of the large language model, the model parameters of the audio prediction network, and the model parameters of the phoneme prediction network are updated to obtain the trained audio synthesis model.

2. The method according to claim 1, characterized in that, The method further includes: When the number of the at least one training phase is one, in one training phase, the audio loss value and the phoneme loss value are weighted and summed to obtain a first loss value; The model parameters of the audio synthesis model to be trained are updated based on the first loss value to obtain the trained audio synthesis model.

3. The method according to claim 1, characterized in that, The step of resampling the phoneme label sequence corresponding to the training sample to obtain the resampled phoneme label sequence includes: Obtain the first temporal resolution of the phoneme tag sequence and the second temporal resolution of the audio tag sequence; The target scaling ratio is determined based on the first time resolution and the second time resolution; The phoneme tag sequence is interpolated according to the target scaling ratio to obtain the resampled phoneme tag sequence.

4. The method according to claim 3, characterized in that, The step of interpolating the phoneme tag sequence according to the target scaling ratio to obtain the resampled phoneme tag sequence includes: Construct an initial resampling sequence, wherein the sequence length of the initial resampling sequence is the same as the sequence length of the audio tag sequence; Based on the target scaling ratio, determine the mapping position of the target position index contained in the initial resampled sequence in the phoneme tag sequence; From the phoneme tag sequence, determine the original position index that is closest to the mapped position, and obtain the original tag value corresponding to the original position index in the phoneme tag sequence; The original tag values ​​are filled into the target position index of the initial resampled sequence to obtain the resampled phoneme tag sequence.

5. The method according to claim 1, characterized in that, The audio synthesis model to be trained includes a phoneme prediction network; Before extracting features from the input sequence using the audio synthesis model to be trained to obtain the shared hidden state sequence, the method further includes: Obtain target configuration information and extract the target network type from the target configuration information; Construct a target prediction network belonging to the target network type; The phoneme prediction network is constructed based on the target prediction network.

6. The method according to claim 5, characterized in that, The construction of the target prediction network belonging to the target network type includes: Extract target prediction network classes belonging to the target network type from the target component library, wherein the target component library includes prediction network classes corresponding to multiple network types; The target prediction network class is dynamically instantiated based on the target configuration information to obtain the target prediction network.

7. The method according to claim 6, characterized in that, The method further includes: For each network type, a registration process is performed on the prediction network class to be registered, thereby obtaining the prediction network class corresponding to the network type. The prediction network classes corresponding to the various network types are stored in the target component library.

8. The method according to claim 5, characterized in that, The construction of the phoneme prediction network based on the target prediction network includes: Obtain historical model parameters, and based on the historical model parameters, query the parameter mapping table, wherein the parameter mapping table includes the mapping relationship between different model parameters and different parameter positions; When the parameter position corresponding to the historical model parameter is found, the found parameter position is determined as the target parameter position corresponding to the historical model parameter in the target prediction network; The historical model parameters are assigned to the target parameter positions, and the assigned target prediction network is determined as the phoneme prediction network.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Obtain the input data sequence; The trained audio synthesis model is used to predict the input data sequence to obtain audio prediction data and phoneme prediction data. The audio prediction data is subjected to waveform conversion to obtain the audio waveform; The phoneme prediction data is converted to obtain a phoneme sequence.

10. The method according to claim 9, characterized in that, The method further includes: Extract the phonetic symbol sequence from the phoneme sequence; The phonetic symbol sequence is segmented based on the space separator to obtain the target text data; The target text data is stored in the form of a text file.

11. A model training device, characterized in that, The device includes: An acquisition module is used to acquire training samples and construct an input sequence based on the training samples; The extraction module is used to extract features from the input sequence through the audio synthesis model to be trained, and obtain a shared hidden state sequence; wherein, the audio synthesis model to be trained includes a large language model, an audio prediction network and a phoneme prediction network, and the shared hidden state sequence refers to the intermediate feature representation output by the large language model at each layer or time step, which is used to carry the contextual information related to text and speech. The prediction module is used to perform audio prediction on the shared hidden state sequence to obtain audio prediction results, and to perform phoneme prediction on the shared hidden state sequence to obtain phoneme prediction results. A determination module is used to determine an audio loss value based on the audio prediction result and the audio label sequence corresponding to the training sample; resample the phoneme label sequence corresponding to the training sample to obtain a resampled phoneme label sequence, wherein the sequence length of the resampled phoneme label sequence is the same as the sequence length of the audio label sequence before resampling; obtain the total sequence length of the input sequence and determine the first position of the non-audio region in the input sequence; construct an initial phoneme label sequence with a length equal to the total sequence length; fill the position corresponding to the first position in the initial phoneme label sequence with an ignore flag, and fill the second position in the initial phoneme label sequence with the resampled phoneme label sequence to obtain a target phoneme label sequence filled with the ignore flag and the resampled phoneme label sequence, wherein the second position is a position different from the first position, and the ignore flag is used to physically isolate the non-audio region in the target phoneme label sequence that does not have corresponding audio information; and determine a phoneme loss value based on the phoneme prediction result and the sequence in the target phoneme label sequence that is not configured with the ignore flag. An update module is used to determine at least one training stage corresponding to the audio synthesis model to be trained. When the at least one training stage includes three training stages, in the first execution order of the training stage, the model parameters of the large language model and the model parameters of the audio prediction network are frozen, the gradient backpropagation of the phoneme loss value to the large language model is blocked, and the model parameters of the phoneme prediction network are updated based on the phoneme loss value. In the second execution order of the training stage, the model parameters of the phoneme prediction network are frozen, the gradient backpropagation of the audio loss value to the phoneme prediction network is blocked, and the model parameters of the large language model and the model parameters of the audio prediction network are updated based on the audio loss value. In the third execution order of the training stage, the audio loss value and the phoneme loss value are weighted and summed to obtain a second loss value, and the model parameters of the large language model, the model parameters of the audio prediction network, and the model parameters of the phoneme prediction network are updated based on the second loss value to obtain the trained audio synthesis model.

12. The apparatus according to claim 11, characterized in that, The update module is further configured to, when the number of the at least one training stage is one, perform a weighted summation of the audio loss value and the phoneme loss value in one training stage to obtain a first loss value; update the model parameters of the audio synthesis model to be trained based on the first loss value to obtain the trained audio synthesis model.

13. The apparatus according to claim 11, characterized in that, The determining module is further configured to obtain a first time resolution of the phoneme tag sequence and a second time resolution of the audio tag sequence; and determine a target scaling ratio based on the first time resolution and the second time resolution. The phoneme tag sequence is interpolated according to the target scaling ratio to obtain the resampled phoneme tag sequence.

14. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the model training method according to any one of claims 1 to 10.

15. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the model training method according to any one of claims 1 to 10.

16. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the model training method according to any one of claims 1 to 10 is implemented.

Images