Model training methods, devices, electronic equipment, computer-readable storage media, and computer program products

Abstract

This application provides a model training method, apparatus, electronic device, computer-readable storage medium, and computer program product. The method includes: extracting audio feature sequences from raw audio data; determining a first phoneme feature sequence based on the raw audio data and text data; aligning the first phoneme feature sequence based on the audio feature sequence to obtain a second phoneme feature sequence; configuring an omission flag in the second phoneme feature sequence based on text data to obtain a third phoneme feature sequence; structurally associating the audio feature sequence and the third phoneme feature sequence to obtain structured data; storing the structured data in a target storage file; and upon receiving a training instruction, reading the structured data from the target storage file and updating the model parameters of the audio synthesis model based on the data in the structured data that does not have an omission flag configured, thereby obtaining the trained audio synthesis model. This application can improve the training efficiency and accuracy of audio synthesis models.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a model training method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Audio synthesis models are widely used in human-computer interaction and audio generation. The training process involves acquiring multimodal media data, performing multi-dimensional feature extraction and sequence processing on the acquired media data, and then training the model based on the extracted features.

[0003] In model training scenarios involving large-scale data samples and multimodal joint modeling, as the feature dimensions increase, balancing the processing efficiency of multidimensional feature data and the accuracy of sequence feature representation during the model training phase places high demands on existing data processing and model training mechanisms. Summary of the Invention

[0004] This application provides a model training method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the training efficiency and accuracy of audio synthesis models.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a model training method, the method comprising:

[0007] Obtain raw audio data and corresponding text data; extract audio feature sequences from the raw audio data; and determine a first phoneme feature sequence based on the raw audio data and the text data.

[0008] Based on the audio feature sequence, the first phoneme feature sequence is aligned to obtain the second phoneme feature sequence;

[0009] Based on the text data, an ignore flag is configured in the second phoneme feature sequence to obtain the third phoneme feature sequence;

[0010] The audio feature sequence and the third phoneme feature sequence are structurally correlated to obtain structured data, and the structured data is stored in the target storage file;

[0011] Upon receiving a training instruction for the audio synthesis model to be trained, the structured data is read from the target storage file, and the model parameters of the audio synthesis model to be trained are updated based on the data in the read structured data that does not have the ignore flag configured, so as to obtain the trained audio synthesis model.

[0012] This application provides a model training apparatus, including:

[0013] The acquisition module is used to acquire raw audio data and text data corresponding to the raw audio data, extract audio feature sequences from the raw audio data, and determine a first phoneme feature sequence based on the raw audio data and the text data;

[0014] An alignment module is used to align the first phoneme feature sequence based on the audio feature sequence to obtain a second phoneme feature sequence.

[0015] The configuration module is used to configure an ignore flag in the second phoneme feature sequence based on the text data to obtain a third phoneme feature sequence;

[0016] The storage module is used to perform a structured association between the audio feature sequence and the third phoneme feature sequence to obtain structured data, and to store the structured data in a target storage file;

[0017] The training module is used to read the structured data from the target storage file upon receiving a training instruction for the audio synthesis model to be trained, and update the model parameters of the audio synthesis model to be trained based on the data in the read structured data that does not have the ignore flag configured, so as to obtain the trained audio synthesis model.

[0018] This application provides an electronic device, the electronic device comprising:

[0019] Memory is used to store executable instructions or computer programs.

[0020] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the model training method provided in the embodiments of this application.

[0021] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the model training method provided in this application when executed by a processor.

[0022] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the model training method provided in this application.

[0023] The embodiments of this application have the following beneficial effects:

[0024] The process involves extracting audio feature sequences from raw audio data, determining a first phoneme feature sequence based on the raw audio data and text data, aligning the first phoneme feature sequence with the audio feature sequence to obtain a second phoneme feature sequence, ensuring consistency in the temporal structure of features across different dimensions. A third phoneme feature sequence is obtained by configuring an ignore flag in the second phoneme feature sequence based on text data, allowing for the isolation of data that does not require computation during subsequent parameter updates. The audio feature sequence and the third phoneme feature sequence are then structurally associated and stored in a target storage file, decoupling the audio feature extraction process from the subsequent model training process in the temporal dimension. This allows for direct reading and processing of structured data from the target storage file upon receiving training instructions, eliminating the need to load the raw audio data and simplifying the data processing flow during training. The model parameters of the audio synthesis model under training are updated based on data without ignore flags in the read structured data, eliminating interference from invalid features (data with ignore flags) and thus improving the training efficiency and accuracy of the audio synthesis model. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the architecture of the model training system provided in the embodiments of this application;

[0026] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0027] Figure 3 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 1 ;

[0028] Figure 4 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 2 ;

[0029] Figure 5 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 3 ;

[0030] Figure 6 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 4 ;

[0031] Figure 7 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 5 ;

[0032] Figure 8 This is a block diagram illustrating the principle of writing structured data into a target storage file and performing on-demand reading during the training phase, as provided in the embodiments of this application.

[0033] Figure 9 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 6 ;

[0034] Figure 10 This is a block diagram illustrating the principle of structured data block allocation in a distributed computing cluster provided in this application embodiment;

[0035] Figure 11 This is a schematic diagram of the audio synthesis model training process provided in the embodiments of this application;

[0036] Figure 12 This is a schematic diagram of offline collaborative feature construction and incremental update provided in the embodiments of this application;

[0037] Figure 13 This is a schematic diagram of the structured storage and retrieval training mechanism provided in the embodiments of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0041] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0042] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0043] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0044] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0045] 1) Raw Audio Data: The basic sound waveform file that has not undergone feature extraction processing, such as a Waveform Audio File Format (WAV) file or a Free Lossless Audio Codec (FLAC) file. Raw audio data contains continuous analog or high-sampling-rate digital signals that record the physical acoustic information of the sound production.

[0046] 2) Text Data: Textual records that correspond to the original audio data in terms of content. They can represent the speech semantics contained in the original audio data in the form of natural language text or phonetic symbols.

[0047] 3) Audio Feature Sequence: A discrete symbol sequence extracted from the original audio data through a pre-trained speech discretization model, used to characterize the temporal structure information of the audio (i.e., speech feature sequence, which may include multiple speech feature tokens).

[0048] 4) Phoneme Feature Sequence: A discrete set of symbols (i.e., phoneme token sequence) representing the smallest sound unit of pronunciation, generated by a phoneme alignment tool based on the temporal correspondence between text data and original audio data.

[0049] 5) Ignore Identifier (IGNORE_ID): During the backpropagation calculation of the audio synthesis model, the feature label used to trigger the feature at a specific location does not participate in the gradient update calculation of the model parameters (e.g., a specific negative integer set in the deep learning framework, such as -100).

[0050] 6) Target Storage File: A physical file configured with a specific data storage format for persistent storage of structured data. This target storage file preferably uses a columnar storage format (e.g., Parquet format), supporting the writing of different characteristic fields contained in the structured data into physically independent storage columns.

[0051] 7) Speaker Embedding Vector: A high-dimensional discrete feature vector extracted from raw audio data to characterize the speaker's unique timbre attributes and vocal cord physical features.

[0052] 8) Audio synthesis model (Text-to-Speech, TTS): refers to a deep learning model that converts text information into corresponding speech signals. Audio synthesis models can include large language models, audio prediction models, phoneme prediction models, vocoder modules, etc.

[0053] 9) Preprocessed speech data: refers to structured data such as speech tokens, phoneme tokens, and voiceprint vectors extracted and stored from the raw audio data through an offline process before model training.

[0054] 10) Structured training data file: refers to training data file saved in a columnar storage format (such as Parquet format), which includes at least text data, speech tokens, phoneme tokens and speaker-related features.

[0055] 11) Structured feature decoupling training: refers to a method in which the original audio data is no longer loaded, decoded or processed during the model training stage, but the model training is completed by reading the structured training data obtained by offline collaborative preprocessing according to the fields.

[0056] In related technologies, the training process for audio synthesis models typically includes the following steps: real-time loading of raw audio data, such as Waveform Audio File Format (WAV), during the training phase; decoding, resampling, and feature extraction of the raw audio data; and inputting the extracted acoustic features or intermediate representations into the acoustic model or large language model for training. In distributed training or large-scale speech data training scenarios, the frequent random reading and decoding operations involving raw audio data reduce the training efficiency of the audio synthesis model.

[0057] To address the aforementioned problems, embodiments of this application provide a model training method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the training efficiency and accuracy of audio synthesis models. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or it can be implemented as a server. The following will describe exemplary applications when the electronic device is implemented as a terminal or server.

[0058] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of a model training system provided in this application embodiment. To support a model training application, the model training system 100 includes at least a terminal 400, a network 300, and a server 200. The terminal 400 is connected to the server 200 through the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both. In some embodiments, the server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, which is not limited in this application embodiment.

[0059] In some embodiments, the present application embodiments can be implemented by the terminal 400 alone. For example, in the offline stage, the terminal 400 acquires the original audio data and the corresponding text data, extracts audio feature sequences from the original audio data, and determines a first phoneme feature sequence based on the original audio data and text data; the terminal 400 aligns the first phoneme feature sequence based on the audio feature sequence to obtain a second phoneme feature sequence; the terminal 400 configures an ignore flag in the second phoneme feature sequence based on the text data to obtain a third phoneme feature sequence; the terminal 400 performs a structured association between the audio feature sequence and the third phoneme feature sequence to obtain structured data, and stores the structured data in a target storage file; upon receiving a training instruction for the audio synthesis model to be trained, the terminal 400 runs the model training program, reads the structured data from the target storage file, and updates the model parameters of the audio synthesis model to be trained based on the data in the read structured data that does not have an ignore flag configured, to obtain the trained audio synthesis model.

[0060] In some embodiments, the present application embodiments can be implemented collaboratively by terminal 400 and server 200. During the offline phase, the user performs offline data processing through terminal 400. Terminal 400 acquires the original audio data and the corresponding text data, extracts audio feature sequences from the original audio data, and determines a first phoneme feature sequence based on the original audio data and text data. Terminal 400 aligns the first phoneme feature sequence based on the audio feature sequence to obtain a second phoneme feature sequence. Terminal 400 configures an ignore flag in the second phoneme feature sequence based on the text data to obtain a third phoneme feature sequence. Terminal 400 performs a structured association between the audio feature sequence and the third phoneme feature sequence to obtain structured data, and stores the structured data in a target storage file. During the model training phase, the user trains the model through terminal 400. Terminal 400 receives training instructions triggered by the user, encapsulates the training instructions into a model training request, and transmits it to server 200 via network 300. In response to receiving a model training request, server 200 reads structured data from the target storage file and updates the model parameters of the audio synthesis model to be trained based on the data in the read structured data that does not have an ignore flag configured, thus obtaining the trained audio synthesis model. Server 200 can also send the trained audio synthesis model to terminal 400, which receives and stores the trained audio synthesis model.

[0061] The model training method provided in this application can be applied to scenarios that require offline feature construction of raw audio data and model training based on structured data, so as to decouple the model training process from the raw audio data processing process. Specific application scenarios may include:

[0062] 1) Virtual Character Voice Customization and Interaction Scenarios. In virtual character voice customization scenarios, such as non-player character voice generation in game development, interaction-driven interaction between digital humans and virtual anchors, and automated voice acting for film and animation characters, it is necessary to iterate and generate a dedicated audio synthesis model for the corresponding virtual character using the recording data of voice actors. For example, in game development, a dedicated audio synthesis model needs to be customized for non-player characters to generate interactive voices; in virtual anchor business, a dedicated audio synthesis model needs to be customized for virtual anchor characters to generate driving voices; and in film and animation production, a dedicated audio synthesis model needs to be customized for animated characters to generate voice-over data.

[0063] In the data preparation stage of the above-mentioned business scenarios, the terminal obtains the recording data of the voice actors as the raw audio data, and obtains the corresponding text data; the terminal extracts the audio feature sequence from the raw audio data, and determines the first phoneme feature sequence based on the raw audio data and text data; the terminal aligns the first phoneme feature sequence based on the audio feature sequence to obtain the second phoneme feature sequence, and configures the ignore flag in the second phoneme feature sequence based on the text data to obtain the third phoneme feature sequence; the terminal performs a structured association between the audio feature sequence and the third phoneme feature sequence to obtain structured data, and stores the structured data in the target storage file.

[0064] During the model training phase, upon receiving the training instruction, the server skips the loading and decoding of the raw audio data, directly reads structured data from the target storage file, and updates the model parameters of the audio synthesis model to be trained based on the data in the read structured data that is not configured with ignore flags, thus obtaining the trained audio synthesis model. This decoupling of data processing and training processes reduces the data loading load during model training and shortens the iteration cycle of the virtual character-specific audio synthesis model.

[0065] After model training is completed, the terminal can perform inference and prediction by calling the trained audio synthesis model, which can meet the customized needs of different business scenarios: In game development, the terminal can use the trained audio synthesis model to generate interactive voice that matches the non-player character settings in the game; in virtual anchor business, the terminal can use the trained audio synthesis model to output driving voice that matches the virtual anchor image; in film and animation production business, the terminal can use the trained audio synthesis model to output dubbing data with the unique timbre characteristics of the animation character.

[0066] 2) Intelligent Speech Teaching and Pronunciation Assessment Scenarios. In language learning applications, the platform needs to train a high-precision standard pronunciation model based on expert teachers' reading corpora. The server acquires the original audio and text data of the standard teaching materials, extracting audio feature sequences and first phoneme feature sequences offline. The server performs alignment processing to obtain the second phoneme feature sequence, and precisely configures ignore markers in the second phoneme feature sequence based on text data (such as punctuation or pauses) to obtain the third phoneme feature sequence. The server structures and associates the above features and stores them in the target storage file. During model training, the server directly reads the structured data from the target storage file and updates the model parameters based on the valid pronunciation data without configured ignore markers, obtaining the trained audio synthesis model. This scheme eliminates computational interference from non-pronunciation regions, ensuring the model's accurate representation of standard pronunciation features. The terminal can then use this model to provide users with extremely accurate pronunciation demonstrations and pronunciation assessment benchmarks.

[0067] 3) Platform-level iterative training scenario for large-scale speech foundation models. When building large-scale cloud-based speech models supporting multiple languages, the server typically faces a significant disk read bottleneck due to the massive amount of raw audio files. The server acquires massive amounts of raw audio and text data from multiple languages. In the centralized preprocessing stage, it extracts audio feature sequences and first phoneme feature sequences, aligns them to obtain second phoneme feature sequences, and generates third phoneme feature sequences after configuring ignore flags. Then, the server structurally correlates these features and stores them uniformly in the target storage file. During the large-scale distributed training stage, each computing node completely decouples itself from the massive amount of raw audio files and directly reads structured data from the target storage file. Each node jointly updates the model parameters of the audio synthesis model to be trained based on the read data without configured ignore flags. This decoupling training method effectively breaks through the disk loading bottleneck and significantly improves the model training iteration speed of the server under large-scale data.

[0068] 4) Automated Audio / Video Dubbing and Customized Voice Cloning Scenarios. In video creation platforms, users often upload their own materials for personalized voice customization. The video creation platform needs to respond extremely quickly and complete fine-tuning training. The terminal or server obtains the user-uploaded customized raw audio data and text data, quickly performs offline feature extraction to obtain audio feature sequences and first phoneme feature sequences; after alignment processing to obtain second phoneme feature sequences, and configuring ignore markers in non-pronunciation regions to obtain third phoneme feature sequences; the audio feature sequences and third phoneme feature sequences are structurally associated and saved to the target storage file. When receiving model training instructions, the time-consuming real-time audio loading and decoding process is bypassed, and structured data is directly read from the target storage file at high speed. The model parameters of the audio synthesis model to be trained are updated based on the efficient features without configured ignore markers. By calling the trained audio synthesis model, video editing applications can quickly generate automated dubbing for users with high timbre fidelity and accurate pronunciation.

[0069] In some embodiments, the electronic device implementing the model training method provided in this application may be Figure 1 Terminal 400 in the middle. See also Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components of the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2The general labeled all buses as Bus System 440.

[0070] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0071] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0072] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0073] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0074] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0075] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0076] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0077] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.).

[0078] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0079] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A model training device 455 stored in memory 450 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: acquisition module 4551, alignment module 4552, configuration module 4553, storage module 4554, and training module 4555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0080] The model training method provided in the embodiments of this application will be described below. As mentioned above, the electronic device implementing the model training method of the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0081] See Figure 3 , Figure 3 This is a flowchart illustrating the model training method provided in the embodiments of this application. Figure 1 , will combine Figure 3 The steps shown are explained as follows: Figure 3 As shown, the method includes steps 101 to 105.

[0082] In step 101, the original audio data and the corresponding text data are obtained, the audio feature sequence is extracted from the original audio data, and the first phoneme feature sequence is determined based on the original audio data and the text data.

[0083] Here, raw audio data refers to the original speech file used to train the audio synthesis model. The raw speech file can be a Waveform Audio File Format (WAV) file or a Free Lossless Audio Codec (FLAC) file. Text data refers to the textual record information that corresponds in content to the raw audio data. Audio feature sequence refers to a discrete set of symbols extracted from the raw audio data using an audio discretization model, consisting of multiple feature tokens, used to represent the temporal acoustic structure information of sound. The audio feature sequence can be an audio token sequence. The first phoneme feature sequence refers to a discrete set of symbols consisting of multiple feature tokens, used to represent the smallest sound unit of articulation. The first phoneme feature sequence can be a phoneme token sequence. The first phoneme feature sequence is generated using a phoneme alignment tool based on the temporal correspondence between the text data and the raw audio data. Feature tokens refer to discrete symbol sequence units extracted from continuous media data using a deep learning model, used to represent local structural information.

[0084] In some embodiments, extracting audio feature sequences from raw audio data and determining the first phoneme feature sequence based on the raw audio data and text data can be achieved through the following methods: Normalizing the raw audio data to obtain target audio data, wherein the normalization process includes at least one of resampling, loudness normalization, and silence clipping; inputting the target audio data into an audio discretization model, and performing acoustic dimension encoding on the target audio data through the audio discretization model to obtain the audio feature sequence; inputting the target audio data and text data together into a phoneme alignment model, and determining the timestamp information corresponding to each articulatory unit in the text data in the target audio data through the phoneme alignment model; and generating the first phoneme feature sequence corresponding to the target audio data and text data based on the timestamp information.

[0085] In some embodiments, the length of the extracted feature sequence depends on the frame shift parameter configured in the feature extraction model. Assume the total duration of the target audio data is... Seconds, the frame shift parameter configured for the feature extraction model is If the time interval is seconds, then the sequence length of the extracted discrete feature sequence is... The calculation formula is expressed as: In this formula, This represents the total number of feature labels contained in the output discrete feature sequence. Represents the total duration of the target audio data. The frame shift parameter represents the configuration of the feature extraction model. This represents the rounding up operation. Because the first frame shift parameter configured in the audio discretization model is not equal to the second frame shift parameter configured in the phoneme alignment model, they are used in the same total duration... The length of the extracted audio feature sequence is inconsistent with the length of the first phoneme feature sequence.

[0086] For example, in a game development scenario where an audio synthesis model is trained for a non-player character (NPC), the following raw audio data (WAV file format, total duration T=2 seconds) containing the message "Warrior, welcome to the magical world" and the corresponding text data "Warrior, welcome to the magical world" are obtained. First, the raw audio data is resampled and its loudness is normalized to obtain target audio data that meets the extraction criteria. Next, the target audio data is input into an audio discretization model with a first frame shift parameter of 0.02 seconds (i.e., 20 milliseconds) to calculate the sequence length of the audio feature sequence. This outputs an audio feature sequence containing 100 feature tags (e.g., represented as an array of 100 integers). Simultaneously, the target audio data and text data are input together into a phoneme alignment model (e.g., the Montreal Forced Aligner, MFA) with a second frame shift parameter of 0.01 seconds (10 milliseconds). The phoneme alignment model determines the start and end times of each phoneme's pronunciation using a forced alignment algorithm and outputs the sequence length. The first phoneme feature sequence (e.g., represented as an array of 200 integers).

[0087] In step 102, the first phoneme feature sequence is aligned based on the audio feature sequence to obtain the second phoneme feature sequence.

[0088] Here, alignment refers to the transformation operation that eliminates the misalignment of different modal feature sequences in terms of time axis and data length caused by differences in feature extraction parameters. The second phoneme feature sequence refers to the discrete symbol set composed of the smallest sound units that maintains consistency with the audio feature sequence in the sequence length dimension after length adjustment.

[0089] In some embodiments, see Figure 4 , Figure 4 The illustration shows that in step 102, the first phoneme feature sequence is aligned based on the audio feature sequence to obtain the second phoneme feature sequence, which may include steps 1021 to 1022.

[0090] In step 1021, the first time resolution of the first phoneme feature sequence and the second time resolution of the audio feature sequence are obtained.

[0091] Here, the first temporal resolution refers to the time interval parameter (e.g., frame shift parameter) between adjacent feature markers during the extraction of the first phoneme feature sequence. The second temporal resolution refers to the time interval parameter between adjacent feature markers during the acoustic dimension encoding of the audio feature sequence.

[0092] For example, in the training scenario of a non-player character voice synthesis model during game development, a first phoneme feature sequence extracted based on an alignment tool and an audio feature sequence extracted based on a discretization model are obtained. The first time resolution for obtaining the first phoneme feature sequence is 10 milliseconds, and the second time resolution for obtaining the audio feature sequence is 20 milliseconds.

[0093] In step 1022, the first phoneme feature sequence is resampled based on the first time resolution and the second time resolution to obtain the second phoneme feature sequence.

[0094] The second phoneme feature sequence has the same sequence length as the audio feature sequence.

[0095] Here, resampling refers to changing the temporal resolution of the first phoneme feature sequence so that the adjusted first phoneme feature sequence has a data length that matches the second temporal resolution while maintaining the same overall time span. The second phoneme feature sequence refers to the set of discrete symbols composed of the smallest sound units that, after resampling, maintain the same sequence length dimension as the audio feature sequence.

[0096] In some embodiments, resampling the first phoneme feature sequence based on a first time resolution and a second time resolution to obtain a second phoneme feature sequence can be achieved in the following way: determining the window step size and window size of a time sliding window based on the first time resolution and the second time resolution; applying the time sliding window to the first phoneme feature sequence according to the window step size to obtain multiple sequentially arranged feature sampling intervals; for each feature sampling interval, extracting a single target feature symbol from the candidate discrete symbols contained in the feature sampling interval according to a preset symbol aggregation rule; and combining the target feature symbols corresponding to each feature sampling interval according to the sliding order of the time sliding window to obtain the second phoneme feature sequence.

[0097] For example, in the training scenario of a non-player character voice synthesis model during game development, the first time resolution of the first phoneme feature sequence is 10 milliseconds, and the second time resolution of the audio feature sequence is 20 milliseconds. Based on the first time resolution (10 milliseconds) and the second time resolution (20 milliseconds), the window size of the time sliding window is determined. The window step size is set to 2. For the first phoneme feature sequence containing 200 feature markers, a time sliding window is applied to the first phoneme feature sequence with a step size of 2, resulting in 100 feature sampling intervals. For example, for the 0th feature sampling interval (covering feature markers at indices 0 and 1), according to the symbol aggregation rule of mode selection, if both indices 0 and 1 are values ​​of 15 representing the vowel 'a', then the value 15 is extracted as the target feature symbol. After traversing all 100 feature sampling intervals and extracting the target feature symbols sequentially, the extracted 100 target feature symbols are combined in chronological order to obtain a second phoneme feature sequence with a sequence length of 100. At this point, the sequence length of the second phoneme feature sequence is exactly the same as the sequence length of the audio feature sequence containing 100 feature markers.

[0098] This application embodiment obtains a first temporal resolution of the first phoneme feature sequence and a second temporal resolution of the audio feature sequence, and resamples the first phoneme feature sequence based on the first and second temporal resolutions to adjust the sequence length of the first phoneme feature sequence. This ensures that the sequence length of the second phoneme feature sequence obtained after resampling is the same as the sequence length of the audio feature sequence, achieving a strict match between the audio feature sequence and the second phoneme feature sequence in the sequence length dimension. This guarantees the data correspondence between each feature sequence when the audio feature sequence and the third phoneme feature sequence determined based on the second phoneme feature sequence are subsequently structured and associated, improving the effectiveness of updating the model parameters of the audio synthesis model to be trained based on structured data.

[0099] In some embodiments, in step 1022, the first phoneme feature sequence is resampled based on the first time resolution and the second time resolution to obtain the second phoneme feature sequence. This can be achieved by: first, determining the target scaling ratio based on the first time resolution and the second time resolution; then, interpolating the first phoneme feature sequence according to the target scaling ratio to obtain the second phoneme feature sequence.

[0100] Here, the target scaling ratio refers to the ratio between the second time resolution and the first time resolution. Interpolation processing refers to the data transformation process of calculating and extracting the position of feature markers contained in the first phoneme feature sequence according to the target scaling ratio, and generating a second phoneme feature sequence with the same sequence length as the audio feature sequence.

[0101] In some embodiments, the target scaling ratio is determined based on a first time resolution and a second time resolution, which can be achieved by dividing the second time resolution by the first time resolution to obtain the target scaling ratio.

[0102] For example, in the training scenario of a non-player character voice synthesis model during game development, a first phoneme feature sequence and an audio feature sequence are obtained. The first time resolution is set to 10 milliseconds, and the second time resolution is set to 20 milliseconds. Dividing the second time resolution (20 milliseconds) by the first time resolution (10 milliseconds) yields a target scaling factor of 2.

[0103] In some embodiments, interpolating the first phoneme feature sequence according to a target scaling ratio to obtain a second phoneme feature sequence can be achieved by: determining the time interval corresponding to each position in the audio feature sequence based on the target scaling ratio; extracting multiple candidate feature markers falling within the time interval from the first phoneme feature sequence; statistically analyzing the frequency data of each feature marker among the multiple candidate feature markers; determining the feature marker with the largest value in the frequency data as the target feature marker; and combining the determined target feature markers in chronological order to obtain the second phoneme feature sequence.

[0104] For example, based on a target scaling factor of 2, the time interval corresponding to the 50th position in the audio feature sequence (corresponding to target position index 49) is determined. From this first phoneme feature sequence, two candidate feature markers falling within this time interval are extracted (corresponding to feature markers with position indices 98 and 99 in the first phoneme feature sequence). Assuming that the two extracted candidate feature markers are both integer numbers "15" representing the vowel "a", the frequency data of each feature marker in these multiple candidate feature markers is counted, and the frequency of the integer number "15" is determined to be 2 times. The feature marker "15" with the largest value in the frequency data (i.e., appearing twice) is determined as the target feature marker. After traversing all 100 positions of the audio feature sequence and determining all target feature markers, the 100 determined target feature markers are combined in chronological order to obtain a second phoneme feature sequence of length 100.

[0105] This application's embodiments determine a target scaling ratio based on a first time resolution and a second time resolution, enabling precise quantification of the temporal ratio between the first phoneme feature sequence and the audio feature sequence. By interpolating the first phoneme feature sequence according to this target scaling ratio to obtain the second phoneme feature sequence, a proportional and precise adjustment of the first phoneme feature sequence in the sequence length dimension is achieved. This ensures a strict match in sequence length between the obtained second phoneme feature sequence and the audio feature sequence, thereby improving the accuracy of temporal alignment between different feature sequences and enhancing the reliability and accuracy of subsequent updates to the model parameters of the audio synthesis model under training based on structured data.

[0106] In some embodiments, interpolating the first phoneme feature sequence according to the target scaling ratio to obtain the second phoneme feature sequence can be achieved as follows: First, construct an initial resampled sequence, wherein the sequence length of the initial resampled sequence is the same as the sequence length of the audio feature sequence; then, based on the target scaling ratio, determine the mapping position of the target position index contained in the initial resampled sequence corresponding to the first phoneme feature sequence; next, determine the original position index that is closest to the mapping position from the first phoneme feature sequence, and obtain the original label value corresponding to the original position index in the first phoneme feature sequence; finally, fill the original label value into the target position index of the initial resampled sequence to obtain the second phoneme feature sequence.

[0107] Here, the initial resampled sequence refers to a blank sequence pre-constructed during interpolation, with the same length as the audio feature sequence. The target position index is the integer number within the initial resampled sequence used to identify the order of each feature marker. The mapped position refers to the position coordinates (e.g., floating-point coordinates containing decimals) of the target position index within the first phoneme feature sequence, calculated based on the target scaling ratio. The original position index is the integer number corresponding to the actual feature marker in the first phoneme feature sequence. The original label value refers to the value of the feature marker contained in the first phoneme feature sequence at that original position index.

[0108] First, the length of the audio feature sequence is read, and an empty array of the same length is initialized in memory as the initial resampling sequence. Next, for each target position index in this initial resampling sequence, the target position index is multiplied by the acquired target scaling ratio to obtain a mapped position. Since this mapped position is a floating-point coordinate and cannot be directly used to extract data, it is rounded to the nearest integer, and the resulting integer coordinate is used as the original position index in the first phoneme feature sequence. Then, the position of this original position index in the first phoneme feature sequence is directly accessed, and the actual original label value recorded at that position is read. Finally, the read original label value is filled into the target position index being processed in the initial resampling sequence.

[0109] For example, in the training scenario of a non-player character speech synthesis model during game development, a first phoneme feature sequence and an audio feature sequence are obtained. The sequence length of the audio feature sequence is determined to be 100. An initial resampled sequence is constructed such that its sequence length is 100 (i.e., the same as the sequence length of the audio feature sequence). For the target position index 49 contained in the initial resampled sequence, based on a target scaling ratio of 2, the corresponding mapping position of the target position index in the first phoneme feature sequence is calculated and determined to be 98. From the first phoneme feature sequence, the original position index closest to the mapping position 98 is determined to be 98. The original label value (e.g., the value 15 representing a certain articulation unit) corresponding to the original position index 98 in the first phoneme feature sequence is obtained. The original label value 15 is filled into the target position index 49 of the initial resampled sequence. After traversing all target position indices of the initial resampled sequence and completing the filling operation, a second phoneme feature sequence with a sequence length of 100 is obtained.

[0110] This application constructs an initial resampling sequence with the same sequence length as the audio feature sequence, establishing a data foundation for length alignment. Based on the target scaling ratio, the target position index contained in the initial resampling sequence is determined to correspond to the mapping position in the first phoneme feature sequence, enabling the establishment of position mapping relationships between feature sequences of different lengths. From the first phoneme feature sequence, the original position index closest to the mapping position is determined, and the original label value corresponding to the original position index in the first phoneme feature sequence is obtained, achieving accurate extraction of discrete feature labels. The original label value is filled into the target position index of the initial resampling sequence to obtain the second phoneme feature sequence, eliminating the difference in sequence length between the first phoneme feature sequence and the audio feature sequence, ensuring the accuracy of the alignment processing results, and improving the reliability of multi-granularity feature joint calculation in the subsequent model parameter update stage.

[0111] In step 103, based on the text data, an ignore flag is configured in the second phoneme feature sequence to obtain the third phoneme feature sequence.

[0112] Here, the ignore flag refers to a specific numerical marker (i.e., IGNORE_ID, which is set to a specific negative integer, such as -100, in deep learning frameworks) used to indicate that data at a specific location does not participate in the parameter update of the audio synthesis model. Configuration refers to the process of replacing the numerical value of the original feature marker at a specific location with the ignore flag, or adding the ignore flag at that specific location as a mask flag for the parallel dimension. The third phoneme feature sequence refers to a set of discrete symbols that, based on the second phoneme feature sequence, has undergone the ignore flag configuration operation, and whose sequence length remains consistent with the second phoneme feature sequence.

[0113] In some embodiments, see Figure 5 , Figure 5 The example shows that in step 103, based on text data, an ignore flag is configured in the second phoneme feature sequence to obtain the third phoneme feature sequence, which may include steps 1031 to 1032.

[0114] In step 1031, non-pronunciation regions contained in the second phoneme feature sequence are determined based on the text data.

[0115] Here, non-pronunciation regions refer to the positional span of segments in text data that do not produce actual pronunciation actions, mapped on the time axis to the second phoneme feature sequence, such as the discrete symbol set corresponding to punctuation marks or sentence pauses after feature extraction.

[0116] In some embodiments, determining the non-pronounceable regions contained in the second phoneme feature sequence based on text data can be achieved by: acquiring each text character contained in the text data; performing semantic feature analysis on each text character in the text data to extract target text characters that do not contain pronunciation attributes from the text data; acquiring the pronunciation time segment corresponding to the target text character in the text data; mapping the pronunciation time segment to the time dimension of the second phoneme feature sequence to determine the position interval corresponding to the pronunciation time segment in the second phoneme feature sequence; and determining the position interval as the non-pronounceable region contained in the second phoneme feature sequence.

[0117] For example, in the training scenario of a non-player character voice synthesis model during game development, the text data "Warrior, welcome to the magical world" is obtained, along with a second phoneme feature sequence containing 200 feature markers extracted from and aligned based on this text data. First, each text character in the text data is obtained, and semantic feature analysis is performed on each character to determine that "Warrior" and "Welcome to the magical world" are text characters containing pronunciation attributes, while the comma "," is a target text character that does not contain pronunciation attributes. Next, the pronunciation time segment corresponding to the comma "," in the text data is obtained (e.g., from 0.8 seconds to 1.2 seconds). Subsequently, the pronunciation time segment from 0.8 seconds to 1.2 seconds is mapped to the time dimension of the second phoneme feature sequence. Combining this with the time resolution of the second phoneme feature sequence (e.g., one feature marker every 10 milliseconds), the position interval corresponding to the pronunciation time segment in the second phoneme feature sequence is calculated to be between the 80th and 119th position indices. Then, the position interval determined by the indexes from the 80th position to the 119th position is taken as the non-pronounced region contained in the second phoneme feature sequence.

[0118] In step 1032, an ignore flag is configured in the non-pronunciation region of the second phoneme feature sequence to obtain the third phoneme feature sequence.

[0119] In some embodiments, configuring an ignore flag in the non-pronunciation region of the second phoneme feature sequence to obtain the third phoneme feature sequence can be achieved by: replacing the values ​​of each feature marker in the second phoneme feature sequence located in the non-pronunciation region with a preset specific value, and using the specific value as the ignore flag; retaining the values ​​of each feature marker in the second phoneme feature sequence located outside the non-pronunciation region to obtain the third phoneme feature sequence.

[0120] For example, in the pronunciation region of the second phoneme feature sequence, the ignore flag is configured, that is, the values ​​of all 40 feature markers from position index 80 to position index 119 are replaced with the value "-100" (i.e., the preset ignore flag), while retaining the original values ​​of the remaining feature markers representing the pronunciations of "warrior" and "welcome to the magical world". After the value replacement operation is completed, a third phoneme feature sequence with a length of 200 is obtained.

[0121] This application embodiment performs semantic mapping on the second phoneme feature sequence based on text data to determine non-pronunciation regions that do not produce actual pronunciation actions. It can accurately locate invalid pauses or silent segments in the second phoneme feature sequence using text data. Ignore markers are configured in the non-pronunciation regions of the second phoneme feature sequence, and the feature marker values ​​at specific positions are replaced with specific values ​​to obtain the third phoneme feature sequence. This provides a data isolation benchmark for subsequent model training to distinguish between effective pronunciation data and invalid pause data. When updating model parameters, data with ignore markers is shielded and does not participate in the calculation of model parameters, eliminating redundant calculation interference from invalid silent segments. This guides the audio synthesis model to focus parameter updates on learning effective pronunciation temporal features, thereby improving the accuracy of the audio synthesis model in representing pronunciation details.

[0122] In step 104, the audio feature sequence and the third phoneme feature sequence are structurally correlated to obtain structured data, and the structured data is stored in the target storage file.

[0123] Here, structured association refers to the process of establishing a logical mapping and binding relationship between an audio feature sequence and a third phoneme feature sequence based on a unique sample identifier. A sample identifier is a numerical or string encoding used to uniquely identify a segment of original audio data. Structured data refers to a data entry consisting of at least the sample identifier, the audio feature sequence, and the third phoneme feature sequence. The target storage file is a physical file configured with a specific data storage format for persistently storing the structured data; this target storage file can use a columnar storage format (e.g., Parquet format).

[0124] In some embodiments, see Figure 6 , Figure 6 The diagram shows that in step 104, the audio feature sequence and the third phoneme feature sequence are structurally correlated to obtain structured data, which may include steps 1041 to 1042.

[0125] In step 1041, the sample identifier corresponding to the original audio data is obtained.

[0126] For example, in the training scenario of a non-player character voice synthesis model during game development, for a piece of raw audio data, the sample identifier corresponding to the raw audio data is obtained, such as "Aria_01".

[0127] In step 1042, based on the sample identifier, the audio feature sequence and the third phoneme feature sequence are logically bound to obtain structured data containing the sample identifier.

[0128] Here, logical binding refers to the process of establishing a fixed mapping index and attribution relationship between the audio feature sequence and the third phoneme feature sequence based on the same sample identifier.

[0129] In some embodiments, the audio feature sequence and the third phoneme feature sequence are logically bound based on the sample identifier to obtain the structured data containing the sample identifier. This can be achieved by: establishing a first correspondence between the sample identifier and the audio feature sequence; establishing a second correspondence between the sample identifier and the third phoneme feature sequence; and aggregating the sample identifier, the audio feature sequence, and the third phoneme feature sequence based on the first and second correspondences to obtain the structured data.

[0130] For example, a first correspondence is established between the sample identifier "Aria_01" and the audio feature sequence containing 100 feature markers extracted from the original audio data. A second correspondence is established between the sample identifier "Aria_01" and a third phoneme feature sequence containing 100 feature markers, aligned and configured with an ignore marker. Based on the first and second correspondences, the sample identifier, the audio feature sequence, and the third phoneme feature sequence are aggregated to obtain a complete structured data.

[0131] This application embodiment obtains the sample identifier corresponding to the original audio data, and logically binds the audio feature sequence and the third phoneme feature sequence based on the sample identifier. This ensures the integrity of the feature sequences of different dimensions at the logical level and the accuracy of the correspondence, providing a logical index basis for subsequent field-by-field reading and efficient retrieval in the target storage file, and improving the data flow efficiency during the model training stage.

[0132] In some embodiments, see Figure 7 , Figure 7 The step 104, which involves storing structured data to a target storage file, may include steps 1043 to 1044.

[0133] In step 1043, the target storage file is divided into multiple independent storage columns.

[0134] Here, an independent storage column refers to a contiguous disk area within the physical architecture of the target storage file, dedicated to storing specific data dimensions. A data dimension refers to a feature category within structured data that possesses different physical meanings and attributes, such as sample identifiers, text data, audio feature sequences, third phoneme feature sequences, voiceprint vectors, raw audio data, and any of extended features. Dividing the target storage file into multiple independent storage columns refers to the process of pre-configuring non-overlapping physical addressing ranges for each data dimension within the target storage file's metadata structure.

[0135] In some embodiments, dividing a target storage file into multiple independent storage columns can be achieved by: obtaining the set of data dimensions contained in the structured data; obtaining the columnar organization protocol supported by the target storage file; configuring corresponding column identification information for each data dimension in the set of data dimensions according to the columnar organization protocol; defining multiple target column blocks in the target storage file that correspond one-to-one with each data dimension based on the amount of data contained in each data dimension, ensuring that the multiple target column blocks are isolated from each other in terms of physical storage address; associating the column identification information of each data dimension with the starting address of the corresponding target column block to obtain the multiple independent storage columns.

[0136] In some extended solutions, storage columns are partitioned through a dynamic field expansion mechanism. When the training task of the audio synthesis model requires the introduction of new data dimensions (such as sentiment features or speech rate features), new column definitions are appended to the metadata area of ​​the target storage file. A new free physical storage space is allocated, and this new free physical storage space is mounted as a newly added independent storage column to the target storage file. Through this dynamic partitioning method, the dimensions of training data can be flexibly expanded without changing the existing storage column structure, improving the compatibility of the target storage file with heterogeneous feature data.

[0137] For example, in the training scenario of a non-player character voice synthesis model during game development, it is necessary to store structured data containing five data dimensions (sample identifiers, text data, audio feature sequences, third phoneme feature sequences, and voiceprint vectors). Obtain the set of data dimensions containing these five dimensions. Define five corresponding column identifiers in the target storage file (e.g., a Parquet format file). Divide the target storage file into five isolated target column blocks. For example, define offsets 0-100 bytes as the sample identifier column block, 101-500 bytes as the text data column block, 501-2000 bytes as the audio feature sequence column block, 2001-3000 bytes as the phoneme feature sequence column block, and 3001-3500 bytes as the voiceprint vector column block. Finally, configure these five physically independent column blocks as independent storage columns in the target storage file.

[0138] In step 1044, the audio feature sequence and the third phoneme feature sequence in the structured data are written into the corresponding storage columns.

[0139] In some embodiments, the audio feature sequence and the third phoneme feature sequence in the structured data are written into corresponding storage columns, which can be achieved by: determining a first target column corresponding to the audio feature sequence and a second target column corresponding to the third phoneme feature sequence among the plurality of independent storage columns; extracting the audio feature sequence from the structured data and extracting the third phoneme feature sequence from the structured data; sequentially filling the audio feature sequence into the physical storage space of the first target column and sequentially filling the third phoneme feature sequence into the physical storage space of the second target column.

[0140] In some embodiments, when the structured data containing incremental samples is received, an append operation is performed on the feature columns. The tail offset information corresponding to each independent storage column already existing in the target storage file is obtained; for the structured data corresponding to the incremental sample, only the audio feature sequence and the third phoneme feature sequence are extracted; the extracted audio feature sequence is directly appended to the tail offset of the first target column, and the extracted third phoneme feature sequence is directly appended to the tail offset of the second target column.

[0141] For example, in a non-player character speech synthesis model training scenario during game development, structured data containing 1000 sample records is acquired. Each sample record contains a set of audio feature sequences and a set of third phoneme feature sequences. A series of pre-configured independent storage columns are determined in the target storage file. From these multiple storage columns, a first target column specifically for storing audio feature sequences (e.g., column identifier "speech_token") and a second target column specifically for storing third phoneme feature sequences (e.g., column identifier "align_token") are identified. The 1000 sets of audio feature sequences are then sequentially and collectively filled into the physical block of the first target column. Similarly, the 1000 sets of third phoneme feature sequences are sequentially and collectively filled into the physical block of the second target column.

[0142] This application embodiment divides the target storage file into multiple independent storage columns, providing a physically isolated storage architecture for different feature data. The audio feature sequence and the third phoneme feature sequence in the structured data are written into the corresponding storage columns, realizing the decoupling of the audio feature sequence and the third phoneme feature sequence in physical storage. This writing method of classifying and storing by independent storage columns enables the direct reading of the required audio feature sequence and the third phoneme feature sequence when receiving training instructions and reading structured data from the target storage file. This avoids the invalid input / output (I / O) overhead caused by reading other data that is not involved in the calculation, thereby improving the speed of model data loading.

[0143] In some embodiments, the structured data further includes a voiceprint vector extracted from the original audio data. In step 1044, the audio feature sequence and the third phoneme feature sequence in the structured data are written into their respective storage columns. This can be achieved as follows: First, the audio feature sequence, the third phoneme feature sequence, and the voiceprint vector in the structured data are each constructed as independent columns to be stored; then, the access frequency of each column to be stored is determined; next, based on the access frequency, each column to be stored is compressed to obtain compressed column data; finally, each compressed column data is written into its corresponding storage column.

[0144] Here, the voiceprint vector refers to the vector representation (i.e., speaker embedding vector) extracted from the original audio data to characterize the speaker's timbre. The column data to be stored refers to the collection of specific data dimensions extracted from the structured data and organized in memory as a single column. Access frequency refers to the number of times or the probability of accessing a specific data dimension when the audio synthesis model performs model parameter update operations. Compression processing refers to the process of reducing the physical storage space occupied by the column data to be stored using a specific encoding algorithm. Compressed column data refers to the data set with a smaller physical volume generated after the column data to be stored has undergone this compression processing.

[0145] In some embodiments, for each column of data to be stored, the average number of times the audio synthesis model calls each column of data to be stored within a single training cycle is calculated, and the average number of calls is determined as the access frequency of each column of data to be stored. When performing compression processing based on this access frequency, a differentiated compression strategy is configured. For each column of data to be stored, a corresponding compression algorithm is selected based on the access frequency corresponding to the column of data to be stored, and the selected compression algorithm is used to compress the column of data to be stored to obtain the corresponding compressed column of data.

[0146] For example, in the training scenario of a non-player character voice synthesis model during game development, the structured data includes an audio feature sequence, a third phoneme feature sequence, and a voiceprint vector. The extracted audio feature sequence is constructed as the first column of data to be stored, the extracted third phoneme feature sequence as the second column, and the extracted voiceprint vector as the third column. Statistics show that within a single training cycle, the average number of calls to the first and second columns is 1000, and the average number of calls to the third column is 100. The 1000 calls are determined as the access frequency of the first and second columns, and the 100 calls are determined as the access frequency of the third column. For each column, a corresponding compression algorithm is selected based on the access frequency. For example, for the first and second columns with an access frequency of 1000, a high-speed compression algorithm emphasizing read response speed is selected; for the third column with an access frequency of 100, an ultra-high compression ratio algorithm emphasizing storage space is selected. The selected compression algorithm is used to process the data in each column to be stored, resulting in the corresponding compressed column data. Finally, the resulting compressed column data is written to the corresponding storage column in the target storage file.

[0147] This application embodiment constructs audio feature sequences, third phoneme feature sequences, and voiceprint vectors into independent columns of data to be stored, thus decoupling the features at the storage level. It determines the access frequency of each column of data to be stored and compresses it based on this frequency to obtain compressed data columns. This provides matching storage optimization methods for the data characteristics and calling requirements of each column of data. Writing each compressed data column into its corresponding storage column reduces the physical space occupied by the target storage file and improves the data extraction efficiency when subsequently updating the model parameters of the audio synthesis model to be trained, thereby improving the training efficiency of the audio synthesis model.

[0148] In some embodiments, each column of data to be stored is compressed based on the access frequency to obtain compressed column data. This can be achieved in the following ways: for the column of data to be stored corresponding to the audio feature sequence or the third phoneme feature sequence, if the access frequency of the column of data to be stored is greater than or equal to a first preset threshold, the column of data to be stored is subjected to a first compression process to obtain compressed column data; for the column of data to be stored corresponding to the voiceprint vector, if the access frequency of the column of data to be stored is less than the first preset threshold, the column of data to be stored is subjected to a second compression process to obtain compressed column data, wherein the data compression ratio of the second compression process is higher than the data compression ratio of the first compression process.

[0149] Here, the first preset threshold refers to a reference numerical boundary used to measure the frequency with which the data to be stored is called during the parameter update process of the audio synthesis model. The first compression process refers to the encoding calculation process performed on the audio feature sequence or the third phoneme feature sequence, focusing on reducing the time required for data decompression and restoration. The second compression process refers to the encoding calculation process performed on the voiceprint vector, focusing on reducing the physical volume of the data. The data compression ratio refers to the ratio between the storage space occupied by the data to be stored after compression and the original storage space before compression.

[0150] In some embodiments, for each column of data to be stored, the access frequency of that column of data is compared with the first preset threshold. If the access frequency is greater than or equal to the first preset threshold, the column of data to be stored is subjected to a first compression process according to a preset first compression algorithm to generate corresponding compressed column data; if the access frequency is less than the first preset threshold, the column of data to be stored is subjected to a second compression process according to a preset second compression algorithm to generate corresponding compressed column data. The column of data to be stored with an access frequency greater than or equal to the first preset threshold can be column of data corresponding to an audio feature sequence or a third phoneme feature sequence. The column of data to be stored with an access frequency less than the first preset threshold can be column of data corresponding to a voiceprint vector.

[0151] For example, in the training scenario of a non-player character voice synthesis model during game development, data to be stored for the audio feature sequence, the third phoneme feature sequence, and the voiceprint vector are obtained. A first preset threshold is set to be accessed 100 times in a single training cycle. First, the access frequency of the data to be stored for both the audio feature sequence and the third phoneme feature sequence is determined to be 500 times. Since 500 times is greater than the first preset threshold (100 times), a first compression algorithm (e.g., a dictionary-based high-speed lossless data compression algorithm (Lempel–Ziv4, LZ4)) is used to perform the first compression process on the two data to be stored, resulting in compressed data with a faster decompression response. Next, the access frequency of the data to be stored for the voiceprint vector is determined to be 1 time. Since the number of compressions is less than the first preset threshold (100 times), a second compression algorithm (such as a lossless data compression algorithm supporting high compression ratios (Zstandard, ZSTD)) is used to perform the second compression process on the column data to be stored, resulting in compressed column data with a significantly reduced physical volume. In the above process, the data compression ratio corresponding to the second compression process (e.g., the compressed volume is 20% of the original volume) is higher than the data compression ratio corresponding to the first compression process (e.g., the compressed volume is 60% of the original volume).

[0152] This application embodiment performs differentiated first and second compression processes on the data to be stored for different access frequencies, achieving precise matching between storage strategies and data retrieval characteristics. The first compression process, with a lower data compression ratio but faster decompression speed, is used for high-frequency access audio feature sequences and third phoneme feature sequences, shortening the data decompression waiting time during the training phase. The second compression process, with a higher data compression ratio, is used for low-frequency access voiceprint vectors, maximizing storage resource savings while ensuring the integrity of model parameter update logic. At the physical level of the target storage file, storage economy and reading efficiency are balanced, thereby improving the training efficiency of the audio synthesis model.

[0153] In some embodiments, after each compressed column data is written to its corresponding storage column, the model training method provided in this application embodiment can also be implemented in the following way: First, for the compressed column data corresponding to the audio feature sequence or the third phoneme feature sequence, obtain the physical storage location of the compressed column data in the corresponding storage column; then, determine the multi-level index of the compressed column data based on the physical storage location; finally, write the multi-level index to the target storage file.

[0154] Here, physical storage location refers to the specific row number coordinates (i.e., "row number") of a particular compressed column of data within its corresponding storage column. A multi-level index refers to a hierarchical directory structure used to quickly locate the compressed column of data during the file reading phase. A multi-level index can include a first-level index and a second-level index. The first-level index points to the specific storage column (i.e., "column number") where the compressed column of data resides, and the second-level index points to the corresponding physical storage location (i.e., "row number") within that storage column.

[0155] In some embodiments, after the compressed column data completes the write operation to the corresponding storage column, the column number of the storage column to which the compressed column data was written is obtained; the column number is determined as the first-level index corresponding to the compressed column data; the specific row number generated after the compressed column data is written to the storage column is obtained, and the row number is used as the physical storage location; the physical storage location is determined as the second-level index corresponding to the compressed column data; a mapping relationship between the first-level index and the second-level index is established to obtain a multi-level index of the compressed column data; and the multi-level index is used as file addressing metadata and appended to the target storage file.

[0156] For example, in the training scenario of a non-player character voice synthesis model during game development, compressed column data corresponding to the audio feature sequence is obtained and written to the storage column numbered "Col_3" in the target storage file. After the write operation is completed, firstly, the column number "Col_3" of the storage column where the compressed column data is located is obtained, and "Col_3" is determined as the first-level index of the multi-level index (i.e., the data is located in the 3rd column). Next, the physical storage location of the compressed column data within "Col_3" is obtained. For example, if the compressed column data is assigned to be written to row 1500 of the target storage file, then the row number "1500" is obtained, and the row number "1500" is determined as the second-level index of the multi-level index (i.e., the data is located in row 1500). Subsequently, a hierarchical mapping relationship is established between the first-level index ("Col_3") and the second-level index ("1500"), and a multi-level index of the compressed column data is generated by combining them. Finally, the generated multi-level index is written to the metadata block at the end of the target storage file, which is specifically used to store the directory.

[0157] This application embodiment targets the compressed column data corresponding to audio feature sequences or third phoneme feature sequences, obtains the physical storage location of the compressed column data in the corresponding storage column, and determines the precise distribution location of the feature data within the storage column; based on the physical storage location, it determines a multi-level index and writes the multi-level index to the target storage file, realizing directory-based addressing of audio feature sequences and third phoneme feature sequences; thus, when receiving training instructions and reading data from the target storage file to perform model parameter updates, it can directly locate and extract the compressed column data at the corresponding physical storage location using the multi-level index, reducing the retrieval overhead during data reading, improving the data acquisition efficiency when updating the model parameters of the audio synthesis model, thereby improving the training efficiency of the audio synthesis model.

[0158] In some embodiments, after each compressed column data is written to its corresponding storage column, the model training method provided in this application embodiment can also be implemented in the following way: for the compressed column data corresponding to the voiceprint vector, a delayed reading flag is configured for the compressed column data, wherein the delayed reading flag is used to indicate that the reading operation of the compressed column data corresponding to the voiceprint vector is skipped during the model training stage.

[0159] Here, the delayed read flag refers to the control flag information configured in the metadata area of ​​the target storage file, used to indicate whether loading specific feature data should be delayed or skipped when the audio synthesis model performs model parameter updates. Skipping the reading operation of the compressed column data corresponding to the voiceprint vector during the model training phase means actively shielding the disk addressing and memory loading operations for the storage column storing the voiceprint vector when acquiring structured data to construct training samples.

[0160] In some embodiments, configuring a delayed read identifier for compressed column data corresponding to a voiceprint vector can be achieved in the following way: after the written operation of the compressed column data corresponding to the voiceprint vector is completed, obtain the column identifier information of the target storage column where the compressed column data is located; based on the column identifier information, create a configuration field corresponding to the voiceprint vector in the attribute description area of ​​the target storage file; generate a delayed read identifier to characterize the asynchronous read rule; write the delayed read identifier into the configuration field in the attribute description area to establish the correspondence between the delayed read identifier and the target storage column.

[0161] In some embodiments, a conditional triggering mechanism is used to dynamically respond to the delayed read flag. During the model training phase, the metadata of the target storage file is parsed. When the delayed read flag is detected in the storage column corresponding to the voiceprint vector, the loading status of the voiceprint vector is set to "pending read" by default, and the initial data reading operation is skipped. When the forward propagation calculation process of the audio synthesis model reaches a specific network layer that requires the fusion of voiceprint information, that specific network layer issues a call instruction for the voiceprint vector. Triggered by this call instruction, the compressed column data corresponding to the voiceprint vector is read from the target storage file based on the column flag information. This on-demand asynchronous reading mode can effectively reduce the instantaneous reading load when the model training starts.

[0162] For example, in the training scenario of a non-player character voice synthesis model during game development, compressed column data corresponding to the voiceprint vector is obtained and written to the storage column numbered "Col_5" in the target storage file. After writing, the column identifier information "Col_5" of the storage column containing the compressed column data is obtained. In the metadata attribute description area of ​​the target storage file, a configuration field corresponding to the voiceprint vector is created. Subsequently, a boolean value "True" is generated as the delayed read identifier. The delayed read identifier "True" is written into the configuration field in the attribute description area, establishing a correspondence between the delayed read identifier and "Col_5". When a training instruction is received and the structured data is read subsequently, the metadata attribute description area is parsed, and the delayed read identifier "True" is identified in "Col_5". Therefore, when constructing the initial training batch containing text data and audio feature sequences, the reading operation of the compressed column data corresponding to the voiceprint vector in "Col_5" is skipped. When the audio synthesis model reaches the stage where it needs to use voiceprint information and generates a request, the compressed column data corresponding to the voiceprint vector is extracted from "Col_5". By performing this step, delayed loading of voiceprint vectors with low request frequency is achieved.

[0163] For example, Figure 8This is a block diagram illustrating the principle of writing structured data to a target storage file and performing on-demand reading during the training phase, as provided in an embodiment of this application. See also... Figure 8 The left side shows structured data, which includes a sample identifier (Aria_01), text data ("Warrior, Welcome to the Magic World"), audio feature sequences, third phoneme feature sequences, and voiceprint vectors. The audio feature sequences, third phoneme feature sequences, and voiceprint vectors in the structured data form the corresponding columns of data to be stored: the audio feature sequence, the third phoneme feature sequence, and the voiceprint vector, respectively. These columns of data are compressed to form corresponding compressed columns. Subsequently, each compressed column is written to a target storage file. The target storage file contains the sample identifier, text data, audio feature sequences, third phoneme feature sequences, and voiceprint vectors stored in columns. It also includes a multi-level index and a delayed read flag. The multi-level index is used for locating and reading the audio feature sequences and third phoneme feature sequences, while the delayed read flag controls the reading of the voiceprint vector. The training module is connected to the target storage file and, using the multi-level index and delayed read flag, reads the corresponding data from the target storage file for training purposes.

[0164] In this embodiment, a delayed reading flag is configured for the compressed column data corresponding to the voiceprint vector. This delayed reading flag indicates that the reading operation of the compressed column data corresponding to the voiceprint vector is skipped during the model training stage. This simplifies the amount of data read at the beginning of the model training stage, reduces the instantaneous memory occupation, and ensures that the transmission efficiency of high-frequency feature sequences is prioritized without affecting the integrity of the audio synthesis model parameter update, thereby improving the training efficiency of the audio synthesis model.

[0165] In some embodiments, after storing the structured data in the target storage file, the model training method provided in this application embodiment can also be implemented in the following way: First, acquire the new audio data; then, perform feature extraction processing on the new audio data to obtain the new structured data; then, split the new structured data into multiple new column data; finally, append each new column data to the storage column corresponding to the new column data in the target storage file.

[0166] Here, "new audio data" refers to the raw sound waveform file newly added during the incremental update scenario of model training, which has not yet undergone feature extraction and structuring processing. Feature extraction processing refers to performing the same steps 101 to 104 on the new audio data as on the raw audio data. "New structured data" refers to data entries extracted from the new audio data, containing at least the sample identifier, audio feature sequence, and third phoneme feature sequence corresponding to the new audio data. "New column data" refers to the column data obtained after splitting the new structured data into single columns in memory according to different data dimensions, such as the column data to be stored corresponding to the new audio feature sequence, the column data to be stored corresponding to the new third phoneme feature sequence, etc.

[0167] In some embodiments, after storing structured data into a target storage file, new audio data and corresponding new text data are obtained; new audio feature sequences and new voiceprint vectors are extracted based on the new audio data, and new phoneme feature sequences are extracted based on the new audio data and the new text data; the new audio feature sequences, the new phoneme feature sequences, and the new voiceprint vectors are structurally associated to obtain new structured data; the new audio feature sequences in the new structured data are split into first new column data, the new phoneme feature sequences are split into second new column data, and the new voiceprint vectors are split into third new column data; for each new column data, the current tail write pointer of each corresponding storage column in the target storage file is obtained; each new column data is continuously written into the corresponding storage column of the target storage file, starting from the position of the corresponding tail write pointer.

[0168] For example, in the training scenario of a non-player character voice synthesis model during game development, after 1000 pieces of existing structured data have been stored in a target storage file (e.g., a Parquet format file), a newly recorded audio data segment and corresponding new text data are obtained. Feature extraction processing is performed on this new audio data to obtain new structured data containing new audio feature sequences, new phoneme feature sequences, and new voiceprint vectors. This new structured data is then split in memory, such that the new audio feature sequences constitute the first new column of data, the new phoneme feature sequences constitute the second new column of data, and the new voiceprint vectors constitute the third new column of data. The current tail write pointer of the first storage column (specifically storing audio feature sequences), the second storage column (specifically storing phoneme feature sequences), and the third storage column (specifically storing voiceprint vectors) in the target storage file is determined to be row 1001. The data of the first newly added column is directly appended to the 1001st row of the 1st storage column, the data of the second newly added column is appended to the 1001st row of the 2nd storage column, and the data of the third newly added column is appended to the 1001st row of the 3rd storage column.

[0169] This application embodiment acquires new audio data and performs feature extraction processing on the new audio data, realizing the transformation of incremental samples from original audio to new structured data; the new structured data is split into multiple new column data, and each new column data is appended to the storage column corresponding to the new column data in the target storage file. The dataset expansion is completed on the premise of ensuring that the existing structured data is not overwritten or reconstructed, thereby reducing the maintenance overhead and iteration time of the audio synthesis model training data.

[0170] In step 105, upon receiving a training instruction for the audio synthesis model to be trained, structured data is read from the target storage file, and the model parameters of the audio synthesis model to be trained are updated based on the data in the read structured data that are not configured with ignore flags, thus obtaining the trained audio synthesis model.

[0171] Here, the training instruction refers to the control signal used to initiate the parameter iterative update process of the audio synthesis model. The training instruction can be issued by the user through a human-computer interaction interface, or automatically generated by the model training scheduler when specific task triggering conditions are met. Reading structured data from the target storage file means performing on-demand extraction operations on the corresponding storage columns in the target storage file according to the data dimensions required by the audio synthesis model within a single forward and backward propagation computation cycle (i.e., the current training batch), skipping the loading process of the original audio data. Data without configured ignore flags refers to the set of discrete symbols in the feature label sequence contained in the structured data that have not been replaced with specific invalid flags. Updating the model parameters of the audio synthesis model to be trained refers to the data processing process of correcting the neural network weight matrix within the audio synthesis model using optimization algorithms based on the difference measure between the model's prediction results and the true feature label sequence contained in the structured data.

[0172] In some embodiments, a training instruction for the audio synthesis model to be trained is received from a user device or a task scheduler; in response to the training instruction, audio feature sequences and third phoneme feature sequences under specific data dimensions are read from multiple independent storage columns contained in the target storage file to obtain the extracted structured data; each position in the extracted structured data is scanned and identified to determine the target position configured with the ignore label; in the extracted structured data, data at the target position is excluded from the parameter update calculation range, and data not at the target position is extracted as data without the ignore label; the data without the ignore label is input into the audio synthesis model to be trained for forward propagation and model loss calculation; based on the loss calculation result, the model parameters of the audio synthesis model to be trained are corrected, and the trained audio synthesis model is output after the preset termination condition is met.

[0173] It should be noted that, in response to this training instruction, the structured data read from multiple independent storage columns contained in the target storage file can be customized according to actual needs. For example, text data, voiceprint vectors, etc. can also be read.

[0174] For example, in a non-player character voice synthesis model training scenario during game development, the target storage file stores different data dimensions, including text data, audio feature sequences, and third phoneme feature sequences, organized in a columnar format. A training instruction for the non-player character audio synthesis model to be trained is received, triggered by the game developer through the user device interface. In response to this training instruction, in the current first training batch, structured data belonging to one batch under each of the three data dimensions is directly extracted from the storage columns for text data, audio feature sequences, and third phoneme feature sequences in the target storage file. During this process, the loading and decoding operations of raw audio data (such as WAV files) not stored in the aforementioned storage columns are skipped. Each position in the extracted third phoneme feature sequence is scanned and identified, and positions with a value of "-100" are determined as target positions configured with an ignore flag (e.g., corresponding to the silent segment position of a comma in the text). The data at these target locations are removed from the parameter update calculation range. The remaining data representing the pronunciations of "warrior" and "welcome to the magical world" that are not at the target locations are extracted and treated as data without an ignore flag. This data without an ignore flag is input into the audio synthesis model to be trained. By calculating the error between the predicted result and the true feature sequence (i.e., the read audio feature sequence), the optimizer performs gradient descent and correction on the model parameters of the audio synthesis model. After 100,000 iterations of the current training batch, once the preset termination condition is met, the trained audio synthesis model is output.

[0175] In some embodiments, the audio synthesis model to be trained includes a large language model, an audio prediction network, and a phoneme prediction network. See also Figure 9 , Figure 9 The example shown in step 105 illustrates updating the model parameters of the audio synthesis model to be trained based on the data in the read structured data that does not have an ignore flag configured, to obtain the trained audio synthesis model. This may include the following steps 1051 to 1054.

[0176] In step 1051, the hidden state features are obtained by extracting features from the structured data without the ignore flag using a large language model.

[0177] Here, the audio prediction network refers to the first prediction head module connected to the output side of the large language model, used to generate audio label prediction results based on hidden state features. The phoneme prediction network refers to the second prediction head module connected to the output side of the large language model, used to generate phoneme label prediction results based on hidden state features. Hidden state features refer to the intermediate vector representation output by the large language model after feature extraction and nonlinear mapping of the input structured data, containing cross-modal association information.

[0178] In some embodiments, the hidden state features can be obtained by extracting features from structured data without configured ignore labels using a large language model in the following way: the large language model maps each feature label contained in the data without configured ignore labels to a preset vector space to obtain a feature vector sequence; the large language model performs multi-layer attention and feedforward neural network calculations on the feature vector sequence to obtain the hidden state features.

[0179] In some embodiments, the large language model performs feature extraction by performing cross-modal attention mechanism computation to obtain hidden state features.

[0180] For example, in the training scenario of a non-player character voice synthesis model during game development, the audio synthesis model to be trained includes a large language model with 7 billion parameters, an audio prediction network, and a phoneme prediction network. After obtaining structured data containing the text data "warrior" and the corresponding audio feature sequence and third phoneme feature sequence, firstly, the data at the position of the value "-100" (i.e., the ignore label) in the structured data is removed, and the remaining data representing valid features is identified as the data without the configured ignore label. The data without the configured ignore label is mapped to a feature vector sequence of dimension 1024. The large language model, through its internal attention mechanism layer, fuses the text feature "warrior" with the various feature labels in the audio feature sequence and the phoneme feature sequence to obtain a high-dimensional tensor of dimension 1024, and this high-dimensional tensor is identified as the hidden state feature.

[0181] In step 1052, the hidden state features are predicted using an audio prediction network to obtain the audio prediction result.

[0182] Here, audio prediction refers to the process of calculating the probability distribution or specific values ​​of audio feature labels corresponding to each time step by linearly or nonlinearly mapping the input hidden state features through fully connected or convolutional layers within the audio prediction network. The audio prediction result refers to the sequence data, composed of audio feature labels or representing the probabilities of audio feature labels, ultimately output by the audio prediction network based on the hidden state features.

[0183] In some embodiments, audio prediction of hidden state features using an audio prediction network can be achieved by the following steps: inputting the hidden state features extracted from a large language model into the audio prediction network along the forward propagation path; performing a linear transformation on the hidden state features in the feature dimension through the audio prediction network to obtain intermediate audio features; inputting the intermediate audio features into a preset activation function for nonlinear processing; calculating the prediction probability distribution on each candidate audio feature label based on the feature data obtained after nonlinear processing; and determining the sequence data containing the prediction probability distribution as the audio prediction result.

[0184] For example, in the training scenario of a non-player character voice synthesis model during game development, the audio synthesis model to be trained includes a large language model with 7 billion parameters, an audio prediction network, and a phoneme prediction network. After obtaining the hidden state features output by the large language model, these hidden state features are represented as a tensor data containing 100 time steps, with each time step having a feature dimension of 1024. This feature dimension... The hidden state features are input into this audio prediction network. This audio prediction network utilizes an internal dimension of... The feature mapping matrix (assuming the vocabulary size of the audio feature tags is 8192) is calculated by performing linear transformations such as matrix multiplication on the hidden state features to obtain a dimension of... The audio intermediate features are then input into the Softmax activation function for processing, yielding an output with a dimension of... The probability distribution tensor is used, where each row of the tensor represents the probability value predicted as each of the 8192 candidate audio feature labels at the current time step, and this probability distribution tensor is used as the audio prediction result.

[0185] In step 1053, the hidden state features are predicted using a phoneme prediction network to obtain the phoneme prediction results.

[0186] Here, phoneme prediction refers to the calculation operation of using the phoneme prediction network to spatially map and probabilistically classify the hidden state features, thereby predicting the phoneme labels corresponding to the hidden state features at each time step. The phoneme prediction result refers to the sequence data finally output by the phoneme prediction network based on the hidden state features, which consists of the predicted probabilities of each candidate phoneme feature label.

[0187] In some embodiments, phoneme prediction of hidden state features using a phoneme prediction network can be achieved by the following steps: obtaining the weight parameter matrix of the phoneme prediction network; inputting the hidden state features output by the large language model into the phoneme prediction network; transforming the feature dimension of the hidden state features using the weight parameter matrix to obtain intermediate phoneme features; performing normalized exponential function processing (i.e., Softmax processing) on ​​the intermediate phoneme features to calculate the predicted probability distribution for each candidate phoneme feature label at each time step; and determining the generated sequence data containing the predicted probability distribution as the phoneme prediction result.

[0188] For example, in the training scenario of a non-player character voice synthesis model during game development, the audio synthesis model to be trained includes a large language model, an audio prediction network, and a phoneme prediction network. After obtaining the hidden state feature output by the large language model, which has a sequence length of 100 and a feature dimension of 1024, the hidden state feature is input into the phoneme prediction network. The phoneme prediction network calls an internal weight parameter matrix with a dimension of 1024 multiplied by 256 (assuming a total of 256 candidate phoneme feature tags) to perform matrix multiplication calculation on the hidden state feature, obtaining an intermediate phoneme feature with a sequence length of 100 and a feature dimension of 256. Numerical normalization is performed on the intermediate phoneme feature to calculate the predicted probability value of the 256 phoneme feature tags at each time step. The tensor data containing the predicted probability value is determined as the phoneme prediction result.

[0189] In step 1054, based on the audio prediction results and phoneme prediction results, the model parameters of the large language model, the audio prediction network, and the phoneme prediction network are updated to obtain the trained audio synthesis model.

[0190] Here, model parameter update refers to using the target loss function to measure the difference between the audio prediction results and phoneme prediction results and the true label sequence, and then using an optimization algorithm to correct the calculation process of the internal weight matrix of the neural network based on this difference. The trained audio synthesis model refers to a neural network model that has completed parameter iterative optimization and can generate audio features and phoneme features with high timbre fidelity and pronunciation accuracy based on the input text. The true feature sequence refers to the audio feature sequence and the third phoneme feature sequence read from the target storage file.

[0191] This application embodiment extracts features from structured data without ignore labels using a large language model to obtain hidden state features, which can transform discrete sequence data into intermediate high-dimensional representations containing rich contextual information. Audio prediction is performed on the hidden state features using an audio prediction network to obtain audio prediction results, and phoneme prediction is performed on the hidden state features using a phoneme prediction network to obtain phoneme prediction results. This achieves multi-task joint feature generation based on shared hidden states, ensuring a high degree of alignment between the generated audio features and phoneme features in the semantic space. Based on the audio prediction results and phoneme prediction results, the model parameters of the large language model, audio prediction network, and phoneme prediction network are updated to obtain a trained audio synthesis model. This achieves efficient training based on discrete structured features, eliminates the influence of invalid data on model convergence, and improves the accuracy of parameter updates, thereby enhancing the training efficiency and representation accuracy of the audio synthesis model.

[0192] In some embodiments, in step 1054, the model parameters of the large language model, the audio prediction network, and the phoneme prediction network are updated based on the audio prediction results and the phoneme prediction results to obtain the trained audio synthesis model. This can be achieved in the following way: First, a first loss value is determined based on the audio prediction results; then, a second loss value is determined based on the phoneme prediction results; then, a target loss value is determined based on the first and second loss values; finally, the model parameters of the large language model, the audio prediction network, and the phoneme prediction network are updated based on the target loss value to obtain the trained audio synthesis model.

[0193] Here, the first loss value is a scalar value used to quantify the prediction deviation between the predicted audio result and the true audio feature sequence. The second loss value is a scalar value used to quantify the prediction deviation between the predicted phoneme result and the true third phoneme feature sequence. The target loss value is the error value derived from the first and second loss values, used to guide the audio synthesis model during the backpropagation process of the current batch.

[0194] In some embodiments, the first loss value is determined based on the audio prediction result, which can be achieved by: obtaining the audio feature sequence stored in the target storage file; calculating the prediction error distribution between the audio prediction result and the audio feature sequence at each position; and accumulating the prediction error distribution to obtain the first loss value.

[0195] In some embodiments, determining the second loss value based on the phoneme prediction result can be achieved by: obtaining the third phoneme feature sequence stored in the target storage file; extracting the valid position index of the third phoneme feature sequence that is not configured with the ignore flag; calculating the prediction error distribution of the phoneme prediction result and the third phoneme feature sequence at the valid position index; and accumulating the prediction error distribution to obtain the second loss value.

[0196] It should be noted that the specific loss function for the first loss value and the second loss value is not limited in the embodiments of this application. For example, it can be the cross-entropy loss function, etc.

[0197] In some embodiments, the target loss value is determined based on the first loss value and the second loss value, which can be achieved by weighting and summing the first loss value and the second loss value according to a preset weighting coefficient to obtain the target loss value.

[0198] In some embodiments, the model parameters of the large language model, audio prediction network, and phoneme prediction network are updated based on the target loss value to obtain the trained audio synthesis model. This can be achieved by: calculating the network gradient based on the target loss value, using the network gradient to update the internal parameter matrices of the large language model, audio prediction network, and phoneme prediction network, and outputting the trained audio synthesis model when the convergence condition is met.

[0199] For example, in the training scenario of a non-player character voice synthesis model during game development, a parameter update operation is performed on a feature sequence containing 100 time steps. First, the audio feature sequence (length 100) in the structured data is obtained, and the audio prediction result output by the audio synthesis model is obtained. The cross-entropy error between the audio prediction result and the audio feature sequence over 100 time steps is calculated, and the average is obtained to obtain the first loss value of 1.5. The third phoneme feature sequence (length 100) in the structured data is obtained, and the ignore flag with a value of "-100" is configured for positions 80 to 119 (40 positions in total, adjusted to positions 80 to 99 here due to exceeding the limit). Based on the remaining 80 valid position indices, the cross-entropy error between the phoneme prediction result and the third phoneme feature sequence is calculated, and the average is obtained to obtain the second loss value of 0.8. Subsequently, the first loss value (1.5) and the second loss value (0.8) are added together with a 1:1 weight ratio to determine the target loss value of 2.3. Finally, backpropagation is performed based on the target loss value of 2.3 to generate gradients for each model parameter. The weight matrices of the large language model, the audio prediction network, and the phoneme prediction network are updated according to the gradients using an optimization algorithm. After tens of thousands of iterations, training ends, resulting in the trained audio synthesis model.

[0200] This application's embodiments determine a first loss value based on audio prediction results and a second loss value based on phoneme prediction results, achieving independent quantitative evaluation of the accuracy of prediction tasks for different modalities; a target loss value is determined based on the first and second loss values, guiding the model to simultaneously improve acoustic and pronunciation performance during joint optimization; model parameters are updated based on the target loss value for the large language model, audio prediction network, and phoneme prediction network, enabling the various network components of the audio synthesis model to achieve parameter co-convergence through joint gradient descent while sharing the underlying representation, thereby improving the training efficiency and generation quality of the audio synthesis model under multi-task conditions.

[0201] In some embodiments, the target loss value is determined based on the first loss value and the second loss value, which can be achieved by: first, obtaining the current training stage of the audio synthesis model to be trained; then, if the training stage is a phoneme-independent training stage, determining the second loss value as the target loss value; and if the training stage is an audio-independent training stage, determining the first loss value as the target loss value.

[0202] Here, the training phase refers to a time interval with independent parameter update rules during model training, defined to achieve specific parameter optimization goals. The phoneme-independent training phase refers to freezing the model parameters of the large language model and the audio prediction network, and updating only the phoneme prediction network's model parameters. The audio-independent training phase refers to freezing the phoneme prediction network's model parameters, and updating both the large language model and the audio prediction network's model parameters. The training phase of the audio synthesis model to be trained can also include a joint training phase, which involves updating the model parameters of the large language model, the audio prediction network, and the phoneme prediction network.

[0203] In some embodiments, when the training phase is a phoneme-independent training phase, the model parameters of the large language model, the audio prediction network, and the phoneme prediction network are updated based on the target loss value to obtain the trained audio synthesis model. This can be achieved by updating the model parameters of the phoneme prediction network based on the second loss value.

[0204] In some embodiments, when the training phase is an audio-independent training phase, the model parameters of the large language model, the audio prediction network, and the phoneme prediction network are updated based on the target loss value to obtain the trained audio synthesis model. This can be achieved by updating the model parameters of the large language model and the audio prediction network based on the first loss value.

[0205] In some embodiments, when the training phase is a joint training phase, the first loss value and the second loss value are weighted and summed to obtain the target loss value. Based on the target loss value, the model parameters of the large language model, the model parameters of the audio prediction network, and the model parameters of the phoneme prediction network are updated to obtain the trained audio synthesis model.

[0206] In some embodiments, multiple training phases are performed in the order of phoneme-independent training phase, audio-independent training phase, and joint training phase.

[0207] For example, in the training scenario of a non-player character speech synthesis model during game development, the audio synthesis model to be trained includes a large language model, an audio prediction network, and a phoneme prediction network. In the current parameter update cycle, the current training stage of the audio synthesis model to be trained is first obtained. For example, it may currently be in the phoneme-independent training stage. The first loss value (e.g., 1.5) and the second loss value (e.g., 0.8) calculated by forward propagation are obtained. Since the current training stage is the phoneme-independent training stage, a low-level operator operation (e.g., a detach operation) is performed to truncate the gradient backpropagation path of the first loss value (1.5) during the network's backpropagation process, so that the first loss value does not generate any gradient information for updating model parameters. Subsequently, the gradient backpropagation path of the second loss value (0.8) is retained, and the second loss value (0.8) is determined as the target loss value. Based on the gradient information generated by the target loss value (0.8), only the model parameters of the phoneme prediction network are corrected, while the model parameters of the large language model and the audio prediction network remain unchanged.

[0208] This application embodiment obtains the current training stage of the audio synthesis model to be trained. In the phoneme-independent training stage, the second loss value is determined as the target loss value, or in the audio-independent training stage, the first loss value is determined as the target loss value. By dynamically determining the target loss value based on the training stage, gradient conflicts between different modality prediction tasks in the early stage of training are avoided, and the convergence stability of the model under the de-audioized training condition is guaranteed, thereby improving the training efficiency of the audio synthesis model.

[0209] In some embodiments, the audio synthesis model to be trained is deployed on a distributed computing cluster, which includes multiple computing nodes. The structured data also includes text data and voiceprint vectors extracted from the original audio data. Before reading the structured data from the target storage file, the model training method provided in this application embodiment can also be implemented in the following way: First, obtain the text length of the text data, the first feature distribution state of the audio feature sequence, and the second feature distribution state of the third phoneme feature sequence in the structured data; then, based on the voiceprint vector, text length, first feature distribution state, and second feature distribution state, perform block processing on the multiple structured data in the target storage file to obtain multiple data blocks; finally, allocate the multiple data blocks to multiple computing nodes.

[0210] Here, a distributed computing cluster refers to a hardware system resource pool consisting of multiple interconnected physical servers or virtual machine instances that collaboratively complete the same large-scale computing task. A computing node refers to an independent computing device within this distributed computing cluster that undertakes the training and parameter updating of the audio synthesis model. The first feature distribution state refers to the statistical distribution information of the audio feature sequence across various length intervals or the number of feature markers. The second feature distribution state refers to the statistical distribution information of the third phoneme feature sequence across various length intervals or the number of feature markers. Data partitioning refers to dividing all structured data in the target storage file according to specific partitioning rules and allocating the resulting subsets to each computing node.

[0211] In some embodiments, the text length of text data in each structured data is read from the target storage file. For each structured data, the number of discrete symbols contained in the audio feature sequence of the structured data is determined, and the number of discrete symbols is determined as the first feature distribution state corresponding to the structured data; the number of discrete symbols contained in the third phoneme feature sequence of the structured data is determined, and the number of discrete symbols is determined as the second feature distribution state corresponding to the structured data.

[0212] In some embodiments, multiple structured data in a target storage file are divided into blocks based on voiceprint vectors, text length, a first feature distribution state, and a second feature distribution state to obtain multiple data blocks. This can be achieved by: extracting voiceprint vectors contained in each structured data; clustering each structured data based on voiceprint vectors to obtain multiple clusters; sorting the multiple structured data within each cluster based on text length, the first feature distribution state, and the second feature distribution state; and dividing the sorted multiple structured data according to the number of computing nodes in the distributed computing cluster to obtain multiple data blocks.

[0213] In some embodiments, for each cluster, sorting multiple structured data within the cluster based on text length, first feature distribution state, and second feature distribution state can be achieved as follows: summing the text length, first feature distribution state, and second feature distribution state corresponding to the structured data to obtain the comprehensive length value of the structured data; and arranging the structured data within the cluster in descending order according to the comprehensive length value.

[0214] In some embodiments, the sorted structured data is divided into multiple data blocks according to the number of computing nodes in the distributed computing cluster. This can be achieved by: constructing a target data container in memory with the same number of computing nodes as the distributed computing cluster; obtaining the total length of the structured data already loaded into each target data container; selecting the container with the smallest total length from the constructed target data containers; for each cluster, loading the structured data that is first in descending order into the container, until all the structured data in the target storage file has been allocated; and determining each target data container after loading the structured data as a multiple data block.

[0215] After obtaining multiple data blocks, each data block is assigned to the corresponding computing node.

[0216] For example, in the training scenario of a non-player character voice synthesis model during game development, the audio synthesis model to be trained is deployed in a distributed computing cluster consisting of computing nodes with 8 accelerator cards. The target storage file contains 1 million pieces of structured data. First, the voiceprint vectors of these 1 million pieces of structured data are extracted, and a clustering algorithm is used to group structured data with the same game NPC voice into multiple clusters. Within a cluster containing 1000 pieces of structured data, the text length (e.g., 50), the first feature distribution state (e.g., 100 feature labels), and the second feature distribution state (e.g., 100 feature labels) corresponding to each piece of structured data are added together to obtain a comprehensive length value of 250. Then, these 1000 pieces of structured data are sorted in descending order according to the comprehensive length values ​​from 250 to 50. Subsequently, 8 target data containers are constructed in memory, with the sum of the comprehensive length values ​​of each container initially being 0. The first structured data entry, sorted in descending order, with a total length of 250, is extracted. Since the sum of all eight containers is currently 0, it is placed into the first current container, whose sum is then updated to 250. The second structured data entry is extracted; since the sums of the second to eighth containers are all 0 (less than 250), it is placed into the second current container. This process continues, and when a structured data entry with a smaller total length is encountered, the sums of the eight containers are automatically compared, and the entry with the smallest sum is placed in the container with the smallest sum. After all 1 million structured data entries in all clusters have been distributed, the total computational load within these eight target data containers is approximately equal. Finally, these eight target data containers filled with structured data are each designated as a data block.

[0217] For example, Figure 10 This is a block diagram illustrating the principle of structured data block allocation in a distributed computing cluster provided in this application embodiment. See also... Figure 10The target storage file includes multiple structured data sets. One of these structured data sets includes a sample identifier (Aria_01), text data ("Warrior, Welcome to the Magic World"), an audio feature sequence, a third phoneme feature sequence, and a voiceprint vector. The distributed computing cluster includes computing nodes 1, 2, 3, 4, 5, 6, 7, and 8. A global data cache area is configured above the distributed computing cluster, connected to each of these nodes. The global data cache area is a shared data storage area configured among the multiple computing nodes in the distributed computing cluster, used to centrally store frequently reused feature data. The data in the global data cache area can include at least one of the following: audio feature sequence, third phoneme feature sequence, and voiceprint vector. By sharing the feature data stored in the global data cache area among multiple computing nodes, each computing node can reduce the repeated reading and transmission of the same feature data when performing model training.

[0218] During operation, multiple structured data sets are first extracted from the target storage file. From these sets, text length, first feature distribution state, second feature distribution state, and voiceprint vector are extracted. Then, the structured data sets are clustered based on the voiceprint vectors to obtain clusters. Within each cluster, the structured data sets are sorted based on text length, first feature distribution state, and second feature distribution state, resulting in sorted structured data sets. These sorted data sets are then partitioned into multiple data blocks. These data blocks are then distributed to computing nodes 1, 2, 3, 4, 5, 6, 7, and 8 in the distributed computing cluster. Simultaneously, a data sharing relationship is established between the global data cache area and these computing nodes, providing unified data caching support for all nodes.

[0219] This application embodiment obtains the text length of the text data in the structured data, the first feature distribution state of the audio feature sequence, and the second feature distribution state of the third phoneme feature sequence. By combining the voiceprint vector, text length, first feature distribution state, and second feature distribution state corresponding to the structured data, multiple structured data are processed into blocks. This can ensure that samples with the same speaker features are clustered as much as possible, while uniformly dividing long sequence data and short sequence data into multiple data blocks. Distributing multiple data blocks to multiple computing nodes ensures that the memory usage and computing power consumption of each computing node tend to be balanced when loading feature sequences and performing matrix calculations in a distributed training scenario. This avoids single-node memory overflow or cluster computing bottleneck effect caused by the clustering of long sequence samples, thereby improving the overall training efficiency of the audio synthesis model in a distributed environment.

[0220] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0221] The training process for audio synthesis models typically includes the following steps: real-time loading of raw audio data, such as waveform audio file (WAV); decoding, resampling, and feature extraction of the raw audio data; and inputting the extracted acoustic features or intermediate representations into the acoustic model or large language model for training. This type of technique has been widely used in audio synthesis models based on large language models in recent years. In distributed training or large-scale speech data training scenarios, related techniques often involve frequent random reading and decoding of audio files.

[0222] The related technologies have the following technical problems: large audio files need to be loaded frequently during training, and in a distributed environment, data reading and processing have a significant impact on the overall processing performance; audio decoding and feature extraction involve a lot of computational processing, which affects the efficiency of model iteration; the training process involves the coordinated use of multiple resources such as storage, central processing unit (CPU) and memory, which is not conducive to the stability of large-scale model training; the audio processing process and the model training process are highly coupled logically, making the training process difficult to maintain and expand.

[0223] Therefore, in audio synthesis model training scenarios, it is necessary to reduce the dependence on processing raw audio data during the training phase, improve training efficiency, and optimize resource utilization.

[0224] Based on this, this application provides a model training method, which is a multi-granularity collaborative speech synthesis model training scheme based on structured feature decoupling. This application moves the steps that heavily rely on the original audio data during model training from the training stage to the offline preprocessing stage. By constructing a mechanism for the generation, storage, and transfer of multi-dimensional structured features, the model training process is decoupled from the original audio data during the training stage. Furthermore, this application does not simply "remove audio loading," but rather aims to solve the balance problem between model training efficiency, representation accuracy, and scalability under conditions where no original audio is involved in training, through multi-dimensional feature collaborative preprocessing, a dynamically adapted columnar storage mechanism, and a multi-granularity feature token collaborative training strategy.

[0225] The above objectives are achieved through the following processing methods in the embodiments of this application:

[0226] During the data preparation phase, a "collaborative preprocessing + incremental update" mechanism is introduced to perform one-time offline processing on the original audio data, simultaneously extracting multi-dimensional features such as audio feature sequences (voice tokens), phoneme feature sequences (phoneme tokens), and voiceprint vectors (speaker embedding vectors), and correcting temporal offsets through alignment processing; in the case of new data, feature extraction and structured writing are only performed on incremental data to reduce data iteration overhead.

[0227] In the data organization and storage stage, a columnar storage strategy based on dynamic indexing and differential compression is adopted to store multidimensional structured features in a unified training data file. The field index priority is dynamically adjusted based on the model type and training stage, which improves the reading efficiency of high-frequency features while balancing storage space usage.

[0228] During the model training phase, a "cross-modal attention alignment + loss feedback-driven token granularity adaptive mechanism" is used to achieve joint processing of multi-granularity features such as audio feature sequences and phoneme feature sequences without reading the original audio data, thereby improving the model's accuracy in representing speech temporal structure and pronunciation details.

[0229] In a distributed training scenario, a feature-aware block and shared caching mechanism is used to block the training samples (corresponding to the structured data in the above embodiments) by utilizing features such as voiceprint vectors, feature distribution states, and text length, so as to reduce the processing overhead caused by cross-node data transmission and random data access.

[0230] At the functional expansion level, a universal model adaptation interface and feature expansion channel are configured to enable different types of audio synthesis models to adapt to the training method provided in this application embodiment, while supporting the expansion and access of personalized features (such as emotion, prosodic features, etc.). In summary, this application embodiment proposes a three-in-one speech synthesis model training paradigm reconstruction scheme of "data-driven - model collaboration - scene adaptation". It differs from the technical route of processing the original audio data in real time during the training phase, and can avoid the problems of model training effect degradation and limited expansion caused by removing audio loading without reading the original audio data.

[0231] This application provides an efficient audio synthesis model training system, which includes: a data preprocessing module for offline processing of raw audio data to generate speech tokens, phoneme tokens, and related features; a structured data storage module for storing the preprocessing results in structured data format; a model training module for directly training the audio synthesis model based on the structured data; and a training scheduling and management module for efficiently scheduling training tasks in a distributed environment.

[0232] The data preprocessing module processes the raw audio data offline and outputs structured data; the structured data storage module writes the structured data into a structured training data file (corresponding to the target storage file in the above embodiment); during the training phase, the model training module reads only speech tokens (which can constitute the audio feature sequence in the above embodiment), phoneme tokens (which can constitute the third phoneme feature sequence in the above embodiment), and embedded features (corresponding to the voiceprint vector in the above embodiment) through the training data reading module to construct training samples and perform model training; during the training phase, the audio synthesis model training system is configured not to load or decode the raw audio data, thereby avoiding the input / output (I / O) bottleneck caused by random audio reading.

[0233] Figure 11 This is a schematic diagram of the audio synthesis model training process provided in an embodiment of this application. See also... Figure 11 Compared with the traditional speech synthesis model training process, the audio synthesis model training process in this application embodiment is a multi-granularity collaborative training process with structured feature decoupling, and the original audio data is no longer read during the training phase.

[0234] In the offline phase, namely the offline collaborative feature construction layer, step 1101 is executed first: obtain the original audio data and text data.

[0235] The original audio data can be in Waveform Audio File Format (WAV) or Free Lossless Audio Codec (FLAC) format. In this step, for incremental update scenarios, newly added data is only built offline.

[0236] Step 1102: Perform unified sampling and standardization processing.

[0237] Among them, preprocessing operations such as resampling and audio normalization are performed on the acquired raw audio data to provide standard media data for feature extraction.

[0238] Subsequently, feature extraction operations are performed in parallel, specifically including:

[0239] Step 1103: Extract speech feature sequences.

[0240] Specifically, the original audio data is acoustically encoded using a speech discretization model to generate a corresponding speech feature sequence (which may include multiple speech tokens, corresponding to the audio feature sequence in the above embodiment).

[0241] Step 1104: Extract phoneme feature sequences.

[0242] Specifically, a phoneme alignment model is used to extract corresponding phoneme feature sequences (which may include multiple phoneme tokens, corresponding to the first phoneme feature sequence in the above embodiment) based on the original audio data and text data.

[0243] Step 1105: Extract the embedding vector.

[0244] In this process, speaker modeling is used to extract the corresponding embedding vector (corresponding to the voiceprint vector in the above embodiment) from the original audio data.

[0245] Step 1106: Perform timing constraints and semantic correction.

[0246] Among them, temporal constraints and semantic corrections are performed through the cross-feature alignment optimization module.

[0247] Step 1107: Form structured feature samples.

[0248] In this process, the data features after alignment are logically correlated at the sample level to complete the preprocessing.

[0249] In the "structured storage" stage, step 1108, a structured training data file (corresponding to the target storage file in the above embodiment) is generated.

[0250] The files are organized using a columnar storage format (such as Parquet format), and each file contains physically independent data columns such as sample identifiers, text information (corresponding to the text data in the above embodiments), speech feature sequences, phoneme feature sequences, and embedding vectors.

[0251] Step 1109: Configure dynamic indexing and differential compression.

[0252] Specifically, multi-level index mapping is configured for high-frequency fields, and the LZ4 (Lempel–Ziv4) compression algorithm with low decompression latency is adopted; lazy loading rules are configured for low-frequency fields, and the ZSTD (Zstandard) compression algorithm with high compression ratio is adopted.

[0253] During the training phase, specifically in the training layers of the model based on structured features, it is strictly ensured that the original audio is not loaded, decoded, or processed. The specific steps are as follows:

[0254] Step 1110: Read the core features by field.

[0255] The model training module only reads the tokens and embedding vectors from the structured feature samples to construct training data.

[0256] Step 1111, joint modeling.

[0257] Among them, joint modeling is performed through a cross-granularity attention alignment module, that is, the input features of different granularities are predicted synchronously within the audio synthesis model by using a shared hidden state sequence.

[0258] Step 1112: Execute the loss feedback-driven feature token granularity adaptation mechanism.

[0259] Specifically, by truncating the gradient backpropagation path of a specific prediction task during the backpropagation process, the weight ratio of different losses is dynamically adjusted.

[0260] Step 1113: Forward propagation, loss calculation, and backpropagation.

[0261] The training process involves updating the model parameters of the audio synthesis model to be trained through forward propagation, loss calculation, and backpropagation, thereby completing the model training.

[0262] In distributed optimization and expansion, the underlying scheduling support for the above process is provided, and the following steps are specifically executed:

[0263] Step 1114, Feature-aware segmentation.

[0264] In this process, data is segmented using attributes such as text length and feature distribution status to avoid data skew.

[0265] Step 1115: Construct a global feature cache pool.

[0266] Among these methods, high-frequency multiplexed feature data is shared among multiple computing nodes to reduce the overhead of cross-node transmission.

[0267] Step 1116: Perform model and data adaptation.

[0268] Among them, the mainstream text-to-speech (TTS) migration is realized through the model-data adaptation layer, and standard calling interfaces are provided to adapt to audio synthesis models with different architectures.

[0269] Step 1117: Integrate custom features.

[0270] Among them, the pluggable feature interface supports the seamless integration of personalized extended features into the overall training process.

[0271] The method provided in this application includes at least an offline collaborative feature construction step. During the data preparation stage, a one-time offline collaborative preprocessing step is performed on the raw audio data to generate structured feature samples.

[0272] Figure 12 This is a schematic diagram illustrating offline collaborative feature construction and incremental update provided in an embodiment of this application. See also... Figure 12 First, obtain the raw audio data (such as raw audio files in WAV or FLAC format) and the corresponding text data.

[0273] Step 1201: Perform unified sampling and standardization processing.

[0274] The process involves resampling, loudness normalization, and silence segment processing of the original audio data to achieve basic normalization. Then, based on the normalized original audio data and text data, the feature extraction branch proceeds in parallel.

[0275] Step 1202: Extract speech feature sequences.

[0276] Specifically, a speech feature sequence (corresponding to the audio feature sequence in the above embodiment) is extracted from the original audio data through a speech discretization model.

[0277] Step 1203: Extract phoneme feature sequences.

[0278] Specifically, the phoneme alignment model outputs a phoneme feature sequence (corresponding to the first phoneme feature sequence in the above embodiment).

[0279] Step 1204: Extract speaker embedding vector.

[0280] Specifically, a speaker modeling network is used to output a speaker embedding vector associated with the text data (corresponding to the voiceprint vector in the above embodiment). Optionally, emotional features or prosodic control features can be extracted and output. Specifically, the complex rhythm, speech rate, stress, and intonation fluctuations in the original audio data are converted into a set of discrete symbol sequences using clustering or quantization methods.

[0281] Step 1205, cross-feature alignment optimization.

[0282] Specifically, based on temporal constraints and textual semantic information, the alignment relationship between the extracted speech feature sequence, phoneme feature sequence and speaker embedding vector is corrected to eliminate the offset error introduced by different feature extraction processes.

[0283] For example, to address the length inconsistency between the phoneme feature sequence (i.e., phone_id, extracted using the Montreal ForcedAligner (MFA) alignment tool with a frame shift of 10ms) and the speech feature sequence (i.e., speech_Token, extracted using the speech tokenizer model with a frame shift of 20ms), this application embodiment employs a resampling alignment method: The phoneme feature sequence is resampled to the same length as the speech feature sequence using a nearest neighbor interpolation algorithm, resulting in an aligned phoneme feature sequence (corresponding to the second phoneme feature sequence in the above embodiment), thus ensuring that each speech feature marker position has a corresponding phoneme feature marker. In this process, temporal constraints are based on temporal alignment relationships for resampling; while textual semantic information refers to configuring the text segments corresponding to non-pronunciation regions as ignored markers (i.e., marked as IGNORE_ID) during sequence construction, resulting in the third phoneme feature sequence, ensuring that only pronunciation regions (i.e., speech segments) participate in alignment correction.

[0284] Step 1206, sample-level association.

[0285] The multidimensional features obtained from the above processing are associated with the corresponding text data to generate structured feature samples (corresponding to the structured data in the above embodiments). The specific data structure included in the structured feature samples includes: sample identifier, text data, speech feature sequence, phoneme feature sequence, speaker embedding vector, optional emotional prosodic features, and optional retained original audio data, etc.

[0286] During the incremental update (new data access) phase, when new audio data is accessed, the above-mentioned offline collaborative feature construction (i.e., steps 1201-1206) is performed only on the new audio data to generate the corresponding incremental features.

[0287] Step 1207: Obtain the newly added audio data.

[0288] Here, when new audio data is obtained, return to step 1201.

[0289] Step 1208, field-level fusion.

[0290] Field-level fusion refers to performing the above feature construction process only once on the newly added raw audio data to obtain a new sample (or batch of samples) that maintains the same set of fields as the structured data. During the writing process, instead of performing coarse-grained overwriting on the entire sample, the incremental features after the above split are filled into the corresponding feature columns according to each field.

[0291] Step 1209, Index Update.

[0292] Specifically, an index update mechanism is implemented to append incremental features to existing structured training data files. After appending these incremental features to the structured training data files, the corresponding data addressing directories are updated synchronously (e.g., updating the index of the columnar storage format file (Parquet) containing sample identifiers, the index of the speaker embedding vector (Speaker, SPK) to the file list, or the data list file, etc.), thereby updating the structured training data files and index information. This mechanism allows newly added samples to be immediately located and found precisely by the training or retrieval logic through the index, without requiring a time-consuming full reconstruction operation of the entire dataset.

[0293] The model training method provided in this application involves a structured storage step and a model training step based on structured features. Figure 13 This is a schematic diagram of the structured storage and retrieval training mechanism provided in the embodiments of this application.

[0294] See Figure 13 The multidimensional feature results are stored in a columnar data organization in the structured training data file 1301. The structured training data file 1301 preferably uses a columnar storage format (such as Parquet format), supporting on-demand column reading and allowing direct reading of all fields from Parquet. The structured training data file 1301 is organized at the sample granularity, with each training sample corresponding to at least one set of core fields, which may include: a sample identifier field (Sample ID); a text data field (Text); a speech feature sequence field (high-frequency field); a phoneme feature sequence field (high-frequency field); a speaker embedding vector field (Speaker); and (optionally) extended feature fields. These fields are physically independent but logically maintain sample-level association (i.e., alignment based on the same Sample ID).

[0295] In the field access strategy (core decision layer) 1302, the access frequency of different fields is analyzed based on the training model type and the current training stage, thereby dynamically adjusting the field index priority. For core fields that are frequently accessed during the training stage (i.e., high-frequency fields), a multi-level index structure is established to reduce random access latency and achieve field-level positioning, thereby accelerating data location and field-by-field reading. For low-frequency access or optional extended fields (i.e., low-frequency fields), a lazy loading mechanism is triggered, and a lazy loading strategy is adopted to reduce invalid I / O overhead without affecting the main model training process.

[0296] Configure differential compression algorithms to optimize storage: First, form structured training data by column (Parquet columnar storage), then compress these columns. For different feature fields with varying data density, length distribution, and access patterns, use differential compression algorithms (i.e., algorithm binding) for storage optimization. This can be achieved through a mapping table between field access frequency and index / compression / read paths. For high-frequency fields (such as frequently accessed fields sensitive to response and decompression latency, including speech token sequence fields and phoneme token sequence fields), binding uses LZ4 or Snappy compression algorithms to reduce decompression overhead, combined with a multi-level index structure to achieve fast field-level reading, reducing disk reads while ensuring low decompression latency; for low-frequency fields (such as relatively sparse or less frequently accessed fields, including speaker embedding vector fields or (optionally) extended feature fields), binding uses ZSTD compression algorithms to improve the overall compression ratio, and employs a lazy loading path (reading on demand based on the index).

[0297] Through the aforementioned field-level index configuration and differentiated compression strategy, a balance is achieved between the storage space usage and I / O throughput of the structured training data file 1301. Based on independent physical storage, dynamic indexing optimizes I / O overhead. Without loading the original audio data, the training data reading module can complete the on-demand reading of multi-granularity features with low I / O overhead, significantly improving the overall throughput and stability during model training.

[0298] During the training phase of the structured feature-based model, the training process no longer relies on the original audio data. Instead, it executes the following steps: reading core feature data by field. Specifically, the training data reading module 1303 (field-level reading executor) performs data reading based on sample localization / text constraints. High-frequency fields are read directly column-by-column, while low-frequency fields are loaded lazily (read on demand based on the index). This allows the core feature data, such as multi-granularity token sequences and embedding vectors, to be read field-by-field from the structured training data file. This data is then used to construct training samples (batches) and input into the audio synthesis model for training.

[0299] Then, through a cross-granularity attention alignment module, speech tokens and phoneme tokens are jointly modeled to construct a unified speech representation. Joint modeling specifically refers to: first, sharing the hidden state: the output hidden state (lm_output) of the large language model is simultaneously input into two prediction heads; the first prediction head is used for audio prediction (speech token prediction), and the second prediction head is used for phoneme prediction (phoneme token prediction); then, audio and phonemes are modeled simultaneously through the hidden state of the same large language network (LLM); in the joint model optimization mode (Normal mode), they are jointly trained, with the target model loss equal to the audio prediction loss plus the phoneme prediction loss (loss=llm_loss+phone_loss), optimizing both tasks together; finally, cross-granularity attention alignment is performed, implicitly aligning the relationship between text, speech tokens, and the third phoneme token using the attention mechanism of the large language model (LLM).

[0300] A phased loss weight adjustment mechanism based on the training mode is implemented. Based on feedback information from the training loss, a token-granularity adaptive adjustment mechanism dynamically adjusts the weights or participation methods of features at different granularities during the training process. This mechanism dynamically adjusts the participation methods of different losses according to the model optimization stage (e.g., phone-only independent training stage, LLM-only independent training stage, or Normal joint training stage). Implicit weight adjustment is achieved by truncating the gradient flow (i.e., by using instruction control operators) to cut off the gradient backpropagation path during the network's backpropagation process.

[0301] The forward propagation, loss calculation, and back propagation processes of the model are executed to complete the parameter update.

[0302] At the level of distributed training optimization and model adaptation, the above training process also provides the following processing mechanisms:

[0303] 1) In a distributed computing scenario, the training scheduling module obtains the speaker identifier, text length, and token distribution features corresponding to the structured data. Based on the above feature information, it performs data-aware block processing (feature-aware slicing) on ​​the structured training data file containing training samples to avoid data skew problems in the distributed training scenario.

[0304] 2) Configure a global data cache area (establish a global feature cache pool) among multiple computing nodes in the distributed computing cluster. Store frequently reused features such as speaker embedding vectors and speech token sequences in this cache area to share the above data among multiple computing nodes, thereby reducing the network communication overhead caused by repeated transmission across nodes.

[0305] 3) By configuring the model data adaptation intermediate layer, various mainstream audio synthesis models can be smoothly adapted to the above training architecture based on structured data.

[0306] 4) By configuring the feature extension interface, it supports the direct access of custom feature types (such as sentiment, rhythm and other features) without adjusting the core model training process.

[0307] Through the above processing method, the original audio data processing logic is completely decoupled during the model parameter update stage.

[0308] In one embodiment, regarding the structured data fields and reading strategy, the structured training data file is organized and stored at the sample granularity. Each training sample contains at least the following independent feature data columns: sample identifier, text data, speech token sequence, phoneme token sequence, and speaker embedding vector. During model training, the training data reading module uses a data reading method based on feature fields or columns, reading only the feature columns related to model parameter updates (e.g., speech token sequence, phoneme token sequence, and speaker embedding vector), and skipping the reading of data columns corresponding to the original audio data. When constructing training data batches, a sequence padding and IGNORE_ID configuration mechanism is used to address inconsistent sequence lengths. This allows the model to mask the padded portions and non-speech regions during parameter update operations, avoiding interference with model gradient calculations.

[0309] In one embodiment, for data reading in a distributed training scenario, the training scheduling module performs sample-level or file-level sharding on the structured training data file in the distributed training environment, so that different training processes read different subsets of data; by using a columnar on-demand reading method, random access to small files in the shared storage system is reduced, thereby improving the overall training throughput and reducing the system load.

[0310] As an optional extension, the storage format of the structured training data file is not limited to a specific format (other columnar storage formats besides Parquet are also applicable); the generation method of voice tokens is not limited to a specific voice discretization model; data prefetching or caching acceleration mechanisms can be further introduced in the data reading stage; and, on the premise of ensuring that the main process of model training skips the original audio loading operation, the data columns corresponding to the original audio data can be retained in the structured training data file for backward compatibility or development and debugging operations.

[0311] The embodiments of this application have at least the following beneficial effects: Significantly reduced training I / O overhead: No original audio data is loaded during the training phase. Combined with a multi-level indexing and feature cache sharing mechanism, training I / O efficiency is improved by more than 50% in distributed scenarios; Improved model training efficiency: The audio decoding and real-time feature extraction processes are eliminated. Combined with incremental preprocessing and differentiated storage strategies, the overall training iteration speed is improved by approximately 30%-40%; Reduced system resource consumption: Through differentiated compression, delayed loading, and cache optimization mechanisms, CPU, memory, and storage resource consumption is reduced by approximately 40%-60%, improving the stability of large-scale training; Enhanced maintainability and scalability of the training process: The training process is completely decoupled from the audio processing process. Combined with a general model adaptation interface and pluggable feature extension channels, it facilitates model migration and functional upgrades; Guaranteed training results without audio dependency: Through multi-granularity token collaborative modeling and cross-modal alignment mechanisms, synthesized speech maintains consistency with traditional audio loading training schemes in terms of naturalness and pronunciation accuracy; Adaptable to flexible data iteration needs: The incremental preprocessing mechanism supports the rapid integration of new data into the training system, significantly reducing the time and computational costs required for data updates.

[0312] The following description continues to illustrate the exemplary structure of the model training device 455 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the model training device 455 in the memory 450 may include:

[0313] The acquisition module 4551 is used to acquire raw audio data and text data corresponding to the raw audio data, extract audio feature sequences from the raw audio data, and determine a first phoneme feature sequence based on the raw audio data and the text data.

[0314] Alignment module 4552 is used to align the first phoneme feature sequence based on the audio feature sequence to obtain the second phoneme feature sequence;

[0315] Configuration module 4553 is used to configure an ignore flag in the second phoneme feature sequence based on text data to obtain the third phoneme feature sequence;

[0316] Storage module 4554 is used to perform structured association between audio feature sequences and third phoneme feature sequences to obtain structured data, and store the structured data in the target storage file;

[0317] The training module 4555 is used to read structured data from the target storage file when it receives a training instruction for the audio synthesis model to be trained, and update the model parameters of the audio synthesis model to be trained based on the data in the read structured data that are not configured with ignore flags, so as to obtain the trained audio synthesis model.

[0318] In some embodiments, the alignment module 4552 is further configured to obtain a first time resolution of the first phoneme feature sequence and a second time resolution of the audio feature sequence; and to resample the first phoneme feature sequence based on the first time resolution and the second time resolution to obtain a second phoneme feature sequence, wherein the sequence length of the second phoneme feature sequence is the same as the sequence length of the audio feature sequence.

[0319] In some embodiments, the alignment module 4552 is further configured to determine a target scaling ratio based on a first time resolution and a second time resolution; and to perform interpolation processing on the first phoneme feature sequence according to the target scaling ratio to obtain a second phoneme feature sequence.

[0320] In some embodiments, the alignment module 4552 is further configured to construct an initial resampling sequence, wherein the sequence length of the initial resampling sequence is the same as the sequence length of the audio feature sequence; determine the mapping position of the target position index contained in the initial resampling sequence corresponding to the first phoneme feature sequence based on the target scaling ratio; determine the original position index that is closest to the mapping position from the first phoneme feature sequence, and obtain the original label value corresponding to the original position index in the first phoneme feature sequence; fill the original label value into the target position index of the initial resampling sequence to obtain the second phoneme feature sequence.

[0321] In some embodiments, the configuration module 4553 is further configured to determine, based on text data, non-pronunciation regions contained in the second phoneme feature sequence; configure an ignore flag in the non-pronunciation regions of the second phoneme feature sequence to obtain a third phoneme feature sequence.

[0322] In some embodiments, the storage module 4554 is further configured to obtain the sample identifier corresponding to the original audio data; based on the sample identifier, logically bind the audio feature sequence and the third phoneme feature sequence to obtain structured data containing the sample identifier.

[0323] In some embodiments, the storage module 4554 is further configured to divide the target storage file into multiple independent storage columns; and to write the audio feature sequence and the third phoneme feature sequence in the structured data into the corresponding storage columns respectively.

[0324] In some embodiments, the structured data further includes a voiceprint vector extracted from the original audio data; the storage module 4554 is further configured to construct the audio feature sequence, the third phoneme feature sequence, and the voiceprint vector in the structured data into mutually independent column data to be stored; determine the access frequency of each column data to be stored; compress each column data to be stored based on the access frequency to obtain compressed column data; and write each compressed column data into the corresponding storage column.

[0325] In some embodiments, the storage module 4554 is further configured to perform a first compression process on the column data to be stored corresponding to the audio feature sequence or the third phoneme feature sequence, when the access frequency of the column data to be stored is greater than or equal to a first preset threshold, to obtain compressed column data; and to perform a second compression process on the column data to be stored corresponding to the voiceprint vector, when the access frequency of the column data to be stored is less than the first preset threshold, to obtain compressed column data, wherein the data compression ratio of the second compression process is higher than the data compression ratio of the first compression process.

[0326] In some embodiments, the model training device 455 further includes an index building module, which, after writing each compressed column data into its corresponding storage column, obtains the physical storage location of the compressed column data in the corresponding storage column for the compressed column data corresponding to the audio feature sequence or the third phoneme feature sequence; determines a multi-level index of the compressed column data based on the physical storage location; and writes the multi-level index into the target storage file.

[0327] In some embodiments, the model training apparatus 455 further includes a delay flag module, which is used to configure a delay read flag for the compressed column data corresponding to the voiceprint vector after each compressed column data is written to the corresponding storage column. The delay read flag is used to indicate that the reading operation of the compressed column data corresponding to the voiceprint vector is skipped during the model training phase.

[0328] In some embodiments, the model training device 455 further includes a new module for acquiring new audio data; performing feature extraction processing on the new audio data to obtain new structured data; splitting the new structured data into multiple new column data; and appending each new column data to the storage column corresponding to the new column data in the target storage file.

[0329] In some embodiments, the audio synthesis model to be trained includes a large language model, an audio prediction network, and a phoneme prediction network; the training module 4555 is further configured to extract features from the structured data without configured ignore labels using the large language model to obtain hidden state features; perform audio prediction on the hidden state features using the audio prediction network to obtain audio prediction results; perform phoneme prediction on the hidden state features using the phoneme prediction network to obtain phoneme prediction results; and update the model parameters of the large language model, the audio prediction network, and the phoneme prediction network based on the audio prediction results and the phoneme prediction results to obtain the trained audio synthesis model.

[0330] In some embodiments, the training module 4555 is further configured to determine a first loss value based on the audio prediction result; determine a second loss value based on the phoneme prediction result; determine a target loss value based on the first loss value and the second loss value; and update the model parameters of the large language model, the audio prediction network and the phoneme prediction network based on the target loss value to obtain the trained audio synthesis model.

[0331] In some embodiments, the training module 4555 is further configured to obtain the current training stage of the audio synthesis model to be trained; if the training stage is a phoneme-independent training stage, determine the second loss value as the target loss value; if the training stage is an audio-independent training stage, determine the first loss value as the target loss value.

[0332] In some embodiments, the audio synthesis model to be trained is deployed on a distributed computing cluster, which includes multiple computing nodes. The structured data also includes text data and voiceprint vectors extracted from the original audio data. The model training device 455 further includes an allocation module for obtaining the text length of the text data in the structured data, the first feature distribution state of the audio feature sequence, and the second feature distribution state of the third phoneme feature sequence. Based on the voiceprint vector, text length, first feature distribution state, and second feature distribution state, the multiple structured data in the target storage file are divided into blocks to obtain multiple data blocks. The multiple data blocks are then allocated to multiple computing nodes.

[0333] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform the model training method described above in this application.

[0334] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the model training method provided in this application. For example, ... Figure 3 The model training method is shown.

[0335] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0336] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0337] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0338] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0339] In summary, by moving the processing of raw audio data to an offline stage through the embodiments of this application, and directly updating model parameters based on structured data during the training stage, the overhead of data reading, decoding, and feature extraction during the training stage can be reduced, thereby improving the model training throughput and optimizing the utilization of computing resources. On the other hand, by aligning audio feature sequences with phoneme feature sequences, configuring ignore markers at specific positions, and jointly modeling multi-granularity features, the accuracy of feature representation and the stability of model training can be improved. Furthermore, by combining columnar storage, differential compression, incremental data updates, distributed sharding, cache sharing, model adaptation, and feature expansion, the maintainability and scalability of the training process, as well as its adaptability to large-scale data and distributed scenarios, can also be improved.

[0340] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method includes: Obtain raw audio data and corresponding text data; extract audio feature sequences from the raw audio data; and determine a first phoneme feature sequence based on the raw audio data and the text data. Based on the audio feature sequence, the first phoneme feature sequence is aligned to obtain the second phoneme feature sequence; Based on the text data, determine the non-pronounceable regions contained in the second phoneme feature sequence; An ignore flag is configured in the non-pronounceable region of the second phoneme feature sequence to obtain the third phoneme feature sequence; The audio feature sequence and the third phoneme feature sequence are structurally correlated to obtain structured data, wherein the structured data also includes the voiceprint vector extracted from the original audio data; The target storage file is divided into multiple independent storage columns, and the audio feature sequence, the third phoneme feature sequence, and the voiceprint vector in the structured data are written into the corresponding storage columns respectively. Upon receiving a training instruction for the audio synthesis model to be trained, the structured data is read from the target storage file, and the model parameters of the audio synthesis model to be trained are updated based on the data in the read structured data that does not have the ignore flag configured, to obtain the trained audio synthesis model. During the model training phase, the metadata of the target storage file is parsed. When it is identified that the storage column corresponding to the voiceprint vector is configured with a delayed read flag, the loading status of the voiceprint vector is set to pending read by default, and the reading operation of the storage column data corresponding to the voiceprint vector is skipped.

2. The method of claim 1, wherein, The step of aligning the first phoneme feature sequence based on the audio feature sequence to obtain the second phoneme feature sequence includes: Obtain the first temporal resolution of the first phoneme feature sequence and the second temporal resolution of the audio feature sequence; Based on the first time resolution and the second time resolution, the first phoneme feature sequence is resampled to obtain a second phoneme feature sequence, wherein the sequence length of the second phoneme feature sequence is the same as the sequence length of the audio feature sequence.

3. The method according to claim 2, characterized in that, The step of resampling the first phoneme feature sequence based on the first time resolution and the second time resolution to obtain the second phoneme feature sequence includes: The target scaling ratio is determined based on the first time resolution and the second time resolution; The first phoneme feature sequence is interpolated according to the target scaling ratio to obtain the second phoneme feature sequence.

4. The method according to claim 3, characterized in that, The step of interpolating the first phoneme feature sequence according to the target scaling ratio to obtain the second phoneme feature sequence includes: Construct an initial resampling sequence, wherein the sequence length of the initial resampling sequence is the same as the sequence length of the audio feature sequence; Based on the target scaling ratio, determine the mapping position of the target position index contained in the initial resampled sequence in the first phoneme feature sequence; From the first phoneme feature sequence, determine the original position index that is closest to the mapped position, and obtain the original label value corresponding to the original position index in the first phoneme feature sequence; The original label values ​​are filled into the target position index of the initial resampled sequence to obtain the second phoneme feature sequence.

5. The method according to claim 1, characterized in that, The step of structurally associating the audio feature sequence and the third phoneme feature sequence to obtain structured data includes: Obtain the sample identifier corresponding to the original audio data; Based on the sample identifier, the audio feature sequence and the third phoneme feature sequence are logically bound to obtain the structured data containing the sample identifier.

6. The method according to claim 1, characterized in that, The step of writing the audio feature sequence, the third phoneme feature sequence, and the voiceprint vector from the structured data into the corresponding storage columns includes: The audio feature sequence, the third phoneme feature sequence, and the voiceprint vector in the structured data are respectively constructed into mutually independent column data to be stored; Determine the access frequency of each column of data to be stored; Based on the access frequency, each column of data to be stored is compressed to obtain compressed column data; Each compressed column data is written into its corresponding storage column.

7. The method according to claim 6, characterized in that, The step of compressing each column of data to be stored based on the access frequency to obtain compressed column data includes: For the column data to be stored corresponding to the audio feature sequence or the third phoneme feature sequence, if the access frequency of the column data to be stored is greater than or equal to a first preset threshold, the column data to be stored is subjected to a first compression process to obtain the compressed column data. For the column data to be stored corresponding to the voiceprint vector, if the access frequency of the column data to be stored is less than the first preset threshold, the column data to be stored is subjected to a second compression process to obtain the compressed column data, wherein the data compression ratio of the second compression process is higher than the data compression ratio of the first compression process.

8. The method according to claim 6, characterized in that, After writing each of the compressed column data into its corresponding storage column, the method further includes: For the compressed column data corresponding to the audio feature sequence or the third phoneme feature sequence, obtain the physical storage location of the compressed column data in the corresponding storage column; A multi-level index is determined based on the physical storage location of the compressed column data; Write the multi-level index into the target storage file.

9. The method according to claim 6, characterized in that, After writing each of the compressed column data into its corresponding storage column, the method further includes: For the compressed column data corresponding to the voiceprint vector, a delayed reading flag is configured for the compressed column data, wherein the delayed reading flag is used to indicate that the reading operation of the compressed column data corresponding to the voiceprint vector is skipped during the model training phase.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: Retrieve newly added audio data; The newly added audio data is subjected to feature extraction processing to obtain newly added structured data; The newly added structured data is split into multiple new column data; Each newly added column data is appended to the storage column corresponding to the newly added column data in the target storage file.

11. The method according to any one of claims 1-9, characterized in that, The audio synthesis model to be trained includes a large language model, an audio prediction network, and a phoneme prediction network; The step of updating the model parameters of the audio synthesis model to be trained based on the data in the read structured data that does not have the ignore flag configured, to obtain the trained audio synthesis model, includes: Using the large language model, feature extraction is performed on the data in the structured data that does not have the ignore flag configured, to obtain hidden state features; The hidden state features are used to perform audio prediction through the audio prediction network to obtain the audio prediction result; The hidden state features are predicted using the phoneme prediction network to obtain the phoneme prediction results. Based on the audio prediction results and the phoneme prediction results, the model parameters of the large language model, the audio prediction network, and the phoneme prediction network are updated to obtain the trained audio synthesis model.

12. The method according to claim 11, characterized in that, The step of updating the model parameters of the large language model, the audio prediction network, and the phoneme prediction network based on the audio prediction results and the phoneme prediction results to obtain the trained audio synthesis model includes: Based on the audio prediction results, a first loss value is determined; Based on the phoneme prediction results, a second loss value is determined; Based on the first loss value and the second loss value, determine the target loss value; Based on the target loss value, the model parameters of the large language model, the audio prediction network, and the phoneme prediction network are updated to obtain the trained audio synthesis model.

13. The method according to claim 12, characterized in that, Determining the target loss value based on the first loss value and the second loss value includes: Obtain the current training stage of the audio synthesis model to be trained; When the training phase is a phoneme-independent training phase, the second loss value is determined as the target loss value; In the case that the training phase is an audio-independent training phase, the first loss value is determined as the target loss value.

14. The method according to any one of claims 1-9, characterized in that, The audio synthesis model to be trained is deployed on a distributed computing cluster, which contains multiple computing nodes. The structured data also includes the text data and the voiceprint vector extracted from the original audio data. Before reading the structured data from the target storage file, the method further includes: Obtain the text length of the text data in the structured data, the first feature distribution state of the audio feature sequence, and the second feature distribution state of the third phoneme feature sequence; Based on the voiceprint vector, the text length, the first feature distribution state, and the second feature distribution state, the multiple structured data in the target storage file are divided into blocks to obtain multiple data blocks; The multiple data blocks are distributed to the multiple computing nodes.

15. A model training device, characterized in that, The device includes: The acquisition module is used to acquire raw audio data and text data corresponding to the raw audio data, extract audio feature sequences from the raw audio data, and determine a first phoneme feature sequence based on the raw audio data and the text data; An alignment module is used to align the first phoneme feature sequence based on the audio feature sequence to obtain a second phoneme feature sequence. A configuration module is used to determine, based on the text data, the non-pronunciation regions contained in the second phoneme feature sequence; and to configure an ignore flag in the non-pronunciation regions of the second phoneme feature sequence to obtain a third phoneme feature sequence. The storage module is used to perform a structured association between the audio feature sequence and the third phoneme feature sequence to obtain structured data, wherein the structured data also includes a voiceprint vector extracted from the original audio data; it is also used to divide the target storage file into multiple independent storage columns, and write the audio feature sequence, the third phoneme feature sequence, and the voiceprint vector in the structured data into the corresponding storage columns respectively; The training module is used to read the structured data from the target storage file upon receiving a training instruction for the audio synthesis model to be trained, and update the model parameters of the audio synthesis model to be trained based on the data in the read structured data that does not have the ignore flag configured, thereby obtaining the trained audio synthesis model. During the model training phase, the metadata of the target storage file is parsed. When a delayed read flag is detected in the storage column corresponding to the voiceprint vector, the loading status of the voiceprint vector is set to pending read by default, and the reading operation of the storage column data corresponding to the voiceprint vector is skipped.

16. The apparatus according to claim 15, characterized in that, The alignment module is further configured to obtain a first time resolution of the first phoneme feature sequence and a second time resolution of the audio feature sequence; based on the first time resolution and the second time resolution, the first phoneme feature sequence is resampled to obtain a second phoneme feature sequence, wherein the sequence length of the second phoneme feature sequence is the same as the sequence length of the audio feature sequence.

17. The apparatus according to claim 16, characterized in that, The alignment module is further configured to determine the target scaling ratio based on the first time resolution and the second time resolution; The first phoneme feature sequence is interpolated according to the target scaling ratio to obtain the second phoneme feature sequence.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the model training method according to any one of claims 1 to 14.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the model training method according to any one of claims 1 to 14.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the model training method according to any one of claims 1 to 14.

Images