Model training methods, speech processing methods, devices, electronic devices, computer-readable storage media, and computer program products

Abstract

This application provides a model training method, a speech processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The method includes: acquiring a first phoneme sequence sample, a first word sequence sample, and a first alignment relationship between the first phoneme sequence sample and the first word sequence sample; in a first training phase, training an initial prediction model based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weight, and a joint loss function of the initial prediction model; determining a state evaluation index for the initial prediction model; and triggering entry into a second training phase when the state evaluation index meets the phase switching conditions. This application can improve the accuracy of the model's prediction of alignment relationships.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, a speech processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In acoustic modeling fields such as speech synthesis and speech recognition, it is often necessary to interact and correlate phoneme sequences representing textual linguistic information with speech token sequences representing underlying acoustic features. However, due to the inherent frame rate differences between the two in terms of temporal resolution and information dimension, the lack of a precise temporal mapping mechanism can easily lead to deviations in model duration prediction and acoustic feature reconstruction. Therefore, the technical significance of research on aligning phoneme sequences with speech token sequences lies in constructing a structured cross-modal temporal alignment mechanism. By calculating and establishing a precise mapping between phoneme sequences and speech token sequences, accurate localization of acoustic boundaries and precise control of pronunciation duration can be achieved.

[0003] In related technologies, the alignment prediction of phoneme sequences and speech word sequences is performed on a single scale using a prediction model, which results in low accuracy in predicting the alignment relationship. Summary of the Invention

[0004] This application provides a model training method, a speech processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of model prediction of alignment relationships.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a model training method, the method comprising:

[0007] Obtain the first phoneme sequence sample, the first word sequence sample, and the first alignment relationship between the first phoneme sequence sample and the first word sequence sample;

[0008] The initial prediction model is trained iteratively until the training termination condition is met, resulting in the target prediction model:

[0009] In the first training phase, the initial prediction model is trained based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weight, and the joint loss function of the initial prediction model. The joint loss function includes the prediction task loss value and the phoneme loss value adjusted by the phoneme loss weight.

[0010] Determine the state evaluation index of the initial prediction model;

[0011] When the state evaluation index meets the stage switching conditions, it triggers the entry into the second training stage;

[0012] In the second training phase, the initial prediction model is trained again based on the first phoneme sequence sample, the first word sequence sample, the first alignment relation, the second alignment window, the second phoneme loss weight, and the joint loss function. The window length of the second alignment window is less than the window length of the first alignment window, and the first phoneme loss weight is less than the second phoneme loss weight.

[0013] This application provides a speech processing method, the method comprising:

[0014] Obtain the input feature sequence to be processed, wherein the input feature sequence includes at least one of the target phoneme sequence and the target word sequence;

[0015] The target prediction model is invoked to perform forward prediction processing on the input feature sequence to obtain the target prediction result; the target prediction model is trained using the model training method provided in the embodiments of this application.

[0016] Based on the target prediction results, generate task processing results for the prediction task;

[0017] Output the task processing results.

[0018] This application provides a model training apparatus, including:

[0019] The sample acquisition module is used to acquire a first phoneme sequence sample, a first word sequence sample, and a first alignment relationship between the first phoneme sequence sample and the first word sequence sample;

[0020] The first training module is used to perform the following iterative training on the initial prediction model until the training termination condition is met to obtain the target prediction model: In the first training stage, the initial prediction model is trained based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weight and the joint loss function of the initial prediction model. The joint loss function includes the prediction task loss value and the phoneme loss value adjusted by the phoneme loss weight.

[0021] The indicator determination module is used to determine the state evaluation indicators of the initial prediction model;

[0022] The second training module is used to trigger the entry into the second training phase when the state evaluation index meets the phase switching condition. In the second training phase, the initial prediction model is trained again based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the second alignment window, the second phoneme loss weight, and the joint loss function. The window length of the second alignment window is less than the window length of the first alignment window, and the first phoneme loss weight is less than the second phoneme loss weight.

[0023] This application provides a voice processing device, including:

[0024] A sequence acquisition module is used to acquire an input feature sequence to be processed, wherein the input feature sequence includes at least one of a target phoneme sequence and a target word sequence;

[0025] The prediction processing module is used to call the target prediction model to perform forward prediction processing on the input feature sequence to obtain the target prediction result. The target prediction model is trained using the model training method provided in the embodiments of this application.

[0026] The result generation module is used to generate task processing results for the prediction task based on the target prediction results; and output the task processing results.

[0027] This application provides an electronic device, the electronic device comprising:

[0028] Memory is used to store executable instructions or computer programs.

[0029] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the model training method or speech processing method provided in the embodiments of this application.

[0030] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the model training method or speech processing method provided in this application.

[0031] This application provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the model training method or speech processing method provided in this application.

[0032] The embodiments of this application have the following beneficial effects:

[0033] Applying the embodiments of this application, a first phoneme sequence sample, a first word sequence sample, and a first alignment relationship between the first phoneme sequence sample and the first word sequence sample are obtained. Then, the initial prediction model is iteratively trained until the training termination condition is met to obtain the target prediction model: In the first training phase, the initial prediction model is trained based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weight, and the joint loss function of the initial prediction model. The joint loss function includes the prediction task loss value and the phoneme loss value adjusted by the phoneme loss weight. This allows for the mapping of phoneme sequences and word sequences using a larger alignment window in the early stages of model training. A relaxed local search space is constructed to establish coarse-grained temporal correspondences, effectively reducing the temporal alignment constraint strength in the early training stage and preventing excessive interference of early local alignment errors on model parameter updates. Then, a state evaluation index for the initial prediction model is determined. When the state evaluation index meets the stage switching condition, the second training stage is triggered. In the second training stage, the initial prediction model is further trained based on the first phoneme sequence samples, the first word sequence samples, the first alignment relationship, the second alignment window, the second phoneme loss weight, and the joint loss function. The window length of the second alignment window is smaller than that of the first alignment window, and the first phoneme loss weight is smaller than that of the second phoneme loss weight. Thus, by periodically calculating the model's state evaluation index and automatically triggering stage switching when a threshold is met, the alignment window is simultaneously reduced and the phoneme loss weight is increased in the second training stage. This achieves a multi-scale alignment strategy from coarse to fine, effectively improving the accuracy of the alignment relationship between the model's predicted phoneme sequence and the speech word sequence. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the network architecture of the model training method provided in the embodiments of this application;

[0035] Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application;

[0036] Figure 3 This is a schematic diagram of the first process of the model training method provided in the embodiments of this application;

[0037] Figure 4 This is a schematic diagram of the second process of the model training method provided in the embodiments of this application;

[0038] Figure 5 This is a schematic diagram of the third process of the model training method provided in the embodiments of this application;

[0039] Figure 6 This is a schematic diagram of the fourth process of the model training method provided in the embodiments of this application;

[0040] Figure 7 This is a schematic diagram of the fifth process of the model training method provided in the embodiments of this application;

[0041] Figure 8 This is a schematic diagram of the sixth process of the model training method provided in the embodiments of this application;

[0042] Figure 9 This is a schematic diagram of the seventh process of the model training method provided in the embodiments of this application;

[0043] Figure 10 This is a schematic flowchart of the speech processing method provided in the embodiments of this application;

[0044] Figure 11 This is a schematic diagram of the resampling and alignment process provided in an embodiment of this application;

[0045] Figure 12 This is a schematic diagram of the mapping relationship provided in the embodiments of this application;

[0046] Figure 13 This is a flowchart illustrating the anomaly detection and dynamic filling process provided in an embodiment of this application;

[0047] Figure 14 This is a schematic diagram of the structure generated based on model prediction of alignment relationships provided in an embodiment of this application;

[0048] Figure 15 This is a schematic diagram of the multi-scale hierarchical alignment and bidirectional calibration mechanism provided in the embodiments of this application;

[0049] Figure 16 This is a schematic diagram of the alignment strategy dynamic adjustment mechanism based on three-dimensional state evaluation index provided in the embodiments of this application;

[0050] Figure 17 This is a schematic diagram of the model output results provided in the embodiments of this application;

[0051] Figure 18 This is a schematic diagram of the unsupervised adaptive calibration mechanism provided in the embodiments of this application.

[0052] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0055] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0056] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0057] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0058] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0059] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0060] 1) Phone Sequence: It is a sequence of phoneme identifiers arranged in chronological order. Each phoneme in the phoneme sequence represents the smallest unit of pronunciation.

[0061] 2) Speech Token Sequence: This is a discrete token sequence extracted from a speech signal through a speech discretization model. It is a discretized sequence used to represent the structural information of speech in the time dimension. Each token corresponds to a preset acoustic or linguistic unit in the time dimension.

[0062] 3) Ignore flag: This is a special flag used to mark invalid or non-participating positions in the loss calculation process during data alignment or sequence resampling. It is a dedicated placeholder flag (such as IGNORE_ID) that indicates the system to skip feature calculation.

[0063] This application provides a model training method, a speech processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of model prediction of alignment relationships.

[0064] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or as servers. The following will describe exemplary applications when the device is implemented as a server.

[0065] See Figure 1 , Figure 1 This is a schematic diagram of the network architecture of the model training method provided in the embodiments of this application, exemplified. Figure 1 The system involves server 200, network 300, and terminal 400. Terminal 400 connects to server 200 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0066] During model training, server 200 acquires the first phoneme sequence sample, the first word sequence sample, and the first alignment relationship between the first phoneme sequence sample and the first word sequence sample. The initial prediction model is iteratively trained until the training termination condition is met, resulting in the target prediction model: In the first training phase, the initial prediction model is trained based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weight, and the joint loss function of the initial prediction model. The joint loss function includes the prediction task loss value and the phoneme loss value adjusted by the phoneme loss weight. A state evaluation index for the initial prediction model is determined. When the state evaluation index meets the phase switching condition, the second training phase is triggered. In the second training phase, the initial prediction model is further trained based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the second alignment window, the second phoneme loss weight, and the joint loss function. The window length of the second alignment window is less than the window length of the first alignment window, and the first phoneme loss weight is less than the second phoneme loss weight.

[0067] Terminal 400 sends the input feature sequence to be processed to server 200. Server 200 obtains the input feature sequence to be processed, which includes at least one of the target phoneme sequence and the target word sequence. Server 200 calls the target prediction model to perform forward prediction processing on the input feature sequence to obtain the target prediction result. The target prediction model is trained using the model training method described above. Based on the target prediction result, a task processing result for the prediction task is generated and returned to terminal 400.

[0068] The model training method provided in this application can be applied to scenarios such as speech generation model training, phoneme prediction decoder training, and game character lip-sync animation generation model training. Examples are given below.

[0069] 1) In the speech generation model training scenario, the first phoneme sequence sample and the first word sequence sample extracted from the audio data are obtained, along with the first alignment relationship between them established in the initial state. The joint loss function of the initial prediction model is configured, where the prediction task loss value is configured as the mean squared error loss of speech acoustic feature prediction, and the phoneme loss value is configured as the cross-entropy loss of alignment prediction. In the first training phase, the initial prediction model is trained based on the search space defined by the first alignment window and the joint loss function to update the model parameters. The state evaluation index of the initial prediction model is periodically calculated. When the state evaluation index meets the stage switching condition, a trigger signal is generated to control the training process to enter the second training phase. In the second training phase, the alignment window is switched to the second alignment window, and the first phoneme loss weight is updated to the second phoneme loss weight, so that the initial prediction model continues to iterate under stricter local alignment constraints and higher alignment loss weights until the training termination condition is met, resulting in the target prediction model. The target prediction model is used to optimize the main speech generation task and the phoneme-word alignment prediction auxiliary task, solving the problems of temporal breaks or incoherent pronunciation in the speech generation process.

[0070] 2) In the phoneme prediction decoder training scenario, the initial prediction model is configured as a phoneme prediction decoder, and the prediction task loss value in its joint loss function is configured as the phoneme sequence recognition loss. In the first training phase, the initial prediction model is trained based on the search space defined by the first alignment window and the joint loss function to reduce the interference of early misalignment relationships on the phoneme prediction decoder parameter updates. During training, a state evaluation index for the initial prediction model is determined. When the state evaluation index meets the phase switching condition, the training phase transition is triggered. In the second training phase, the initial prediction model continues to be trained based on the search space defined by the second alignment window and the joint loss function, enabling the phoneme prediction decoder to focus on local effective articulation features, completing the convergence of the phoneme prediction decoder parameters, and obtaining the target prediction model. The target prediction model can be applied to phoneme recognition and prediction systems in dialect or noisy environments.

[0071] 3) In the training scenario of the game character lip-sync animation generation model, first phoneme sequence samples are obtained from the game dubbing script, and first word sequence samples are obtained from the corresponding audio data of the game dubbing script. Using an alignment tool or a rule-based resampling algorithm, preliminary temporal matching is performed on the first phoneme sequence samples and the first word sequence samples to obtain the first alignment relationship between them. The initial prediction model is configured as a multimodal lip-sync generation network, where the prediction task loss value is configured as the prediction loss of the lip-sync animation parameters, and the phoneme loss value is configured as the cross-entropy loss between the predicted alignment distribution output by the model's internal alignment module and the first alignment relationship. In the first training phase, the initial prediction model is trained based on the search space defined by the first alignment window and the joint loss function to learn the coarse-grained temporal correspondence between articulation units and mouth opening and closing movements. During training, a state evaluation index for the initial prediction model is determined. When the state evaluation index meets the stage switching conditions, the second training phase is triggered. In the second training phase, the initial prediction model is trained again using the same input data and joint loss function. The alignment window is switched to a second alignment window, and the first phoneme loss weights are updated to the second phoneme loss weights to optimize high-precision synchronization between phonemes and the instantaneous closure state of their corresponding lip movements. When the training termination condition is met, a target prediction model is obtained. When this target prediction model is invoked in the game engine, it can generate game character lip-sync animations that are highly synchronized with the input speech at the phoneme level and have a natural and smooth visual appearance.

[0072] In some embodiments, the server (e.g., server 200) can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0073] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2 The server 200 shown includes at least one processor 210, memory 230, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0074] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0075] The memory 230 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 230 may optionally include one or more storage devices physically located away from the processor 210.

[0076] The memory 230 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 230 described in this application embodiment is intended to include any suitable type of memory.

[0077] In some embodiments, memory 230 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0078] Operating system 231 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0079] The network communication module 232 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0080] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A model training device 233 stored in memory 230 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a sample acquisition module 2331, a first training module 2332, an index determination module 2333, and a second training module 2334. These modules are logically connected and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0081] The model training method provided in this application will be described in conjunction with exemplary applications and implementations of the server devices provided in the embodiments of this application.

[0082] The following describes the model training method provided in the embodiments of this application. For the purpose of understanding the model training method provided in the embodiments of this application, this application uses the application of a speech generation model training scenario as an example for explanation. It can also be applied to scenarios such as phoneme prediction decoder training and intelligent speech evaluation.

[0083] As mentioned above, the electronic device implementing the model training method of this application embodiment can be a terminal, a server, or a combination of both. The following explanation uses an electronic device as a server as an example to illustrate the model training method provided in this application embodiment. See also... Figure 3 , Figure 3 This is a schematic diagram of the first process of the model training method provided in the embodiments of this application, which will be combined with Figure 3 The steps shown are explained.

[0084] In step 301, the first phoneme sequence sample, the first word sequence sample, and the first alignment relationship between the first phoneme sequence sample and the first word sequence sample are obtained.

[0085] Here, the first phoneme sequence sample is the phoneme sequence used to train the initial prediction model. The phoneme sequence is represented as a sequence of phoneme identifiers arranged in chronological order, where each phoneme represents the smallest unit of articulation. The initial prediction model can be a speech generation model, a phoneme prediction model, etc. The first word sequence sample is the speech token sequence used to train the initial prediction model. The speech token sequence is a discrete token sequence extracted from the speech signal through a speech discretization model, used to represent the structural information of speech in the time dimension. Both the first phoneme sequence sample and the first word sequence sample originate from the first speech sample. The first alignment relationship is the annotation alignment relationship between the first phoneme sequence sample and the first word sequence sample, representing the one-to-one correspondence between each phoneme in the first phoneme sequence sample and each word in the first word sequence sample.

[0086] In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the second process of the model training method provided in the embodiments of this application. Figure 3 Step 301 shown can be achieved through... Figure 4 Steps 3011 to 3013 are implemented, and will be explained in detail below.

[0087] In step 3011, the initial alignment relationship between the first phoneme sequence sample and the first word sequence sample is determined.

[0088] Here, the initial alignment relationship is obtained by performing sample comparison learning based on the first phoneme sequence sample and the first word sequence sample, or the initial alignment relationship is obtained by constructing the alignment relationship based on the first phoneme sequence sample and the first word sequence sample.

[0089] In some embodiments, determining the initial alignment relationship between the first phoneme sequence sample and the first word sequence sample can be achieved by performing the following steps: constructing a positive sample pair from the first phoneme sequence sample and the first word sequence sample, wherein the first phoneme sequence sample and the first word sequence sample are derived from a first speech sample; obtaining a second phoneme sequence sample and constructing a negative sample pair from the second phoneme sequence sample and the first word sequence sample, wherein the second phoneme sequence sample is derived from a second speech sample, and the first speech sample and the second speech sample are different; performing temporal matching degree prediction on the positive sample pair and the negative sample pair respectively, thereby obtaining the first temporal matching degree of the positive sample pair and the second temporal matching degree of the negative sample pair; based on the first temporal matching degree and the second temporal matching degree, determining a target sample pair from the positive sample pair and the negative sample pair that meets the preset matching degree condition, wherein the target sample pair includes a target phoneme sequence sample and a target word sequence sample; and constructing an alignment relationship between the target phoneme sequence sample and the target word sequence sample to obtain the initial alignment relationship.

[0090] Here, the first phoneme sequence sample and the first word sequence sample originate from the first speech sample, meaning they come from the same speech sample. Therefore, they correspond in terms of semantic content, pronunciation timing, and acoustic origin. The first phoneme sequence sample and the first word sequence sample are combined into a positive sample pair. Based on the text content and audio information corresponding to the second speech sample, a second phoneme sequence sample is generated. Since the second phoneme sequence sample and the first word sequence sample come from different speech samples, they do not correspond to the same speech content. Therefore, the second phoneme sequence sample and the first word sequence sample are combined into a negative sample pair.

[0091] Temporal matching degree is a numerical feature used to quantify the similarity between phoneme sequence samples and word sequence samples in terms of temporal alignment and cross-modal feature distribution. The value of temporal matching degree is configured to range from 0 to 1. Temporal matching degree prediction is performed on positive and negative sample pairs separately. This involves inputting the positive and negative sample pairs into a pre-defined temporal correlation model (such as a Siamese dual-branch network structure). For the positive sample pair, the temporal correlation model encodes the features of the first phoneme sequence sample and the first word sequence sample through two parallel network branches, obtaining their respective latent space feature representations. The inner product or cosine distance between the two latent space feature representations is then calculated to obtain the result. The result is normalized, and the temporal matching degree of the positive sample pair, i.e., the first temporal matching degree, is output. Similarly, the output of the temporal correlation model for the negative sample pair is the second temporal matching degree.

[0092] Meeting the preset matching criteria, also known as reaching the matching threshold, involves comparing the first temporal matching score of positive sample pairs and the second temporal matching score of negative sample pairs with the matching threshold. Sample pairs that do not meet the preset matching criteria are filtered out, and those that do are identified as target sample pairs. This process eliminates sample pairs with temporal misalignment or extremely low relevance, retaining phoneme sequence samples and word sequence samples with high-confidence correspondence. For example, with a matching threshold of 0.7, the first temporal matching score of positive sample pairs is 0.8, and the second temporal matching score of negative sample pairs is 0.4. Therefore, sample pairs that meet the preset matching criteria are considered positive sample pairs and are identified as target sample pairs.

[0093] For the selected target sample pairs, phoneme node features from the target phoneme sequence samples and word time slot features from the target word sequence samples are extracted. Based on the similarity distribution matrix of phoneme node features and word time slot features in the latent space, a specific mapping path from phoneme index to word index is established using maximum likelihood path search (such as dynamic programming). This mapping path is recorded as a structured mapping dictionary of "phoneme index-word index" or "phoneme index-word time interval", which is the initial alignment relationship.

[0094] In this embodiment, a self-supervised correspondence establishment method based on contrastive learning mechanism is realized by extracting and constructing positive and negative sample pairs from the same and different speech samples respectively, and using the temporal correlation model to output the quantified temporal matching degree for sample pair screening. This enables the establishment of a high-confidence initial alignment relationship between phoneme sequence samples and word sequence samples under the objective condition of completely lacking manually labeled alignment data.

[0095] In some embodiments, see Figure 5 , Figure 5 This is a schematic diagram of the third process of the model training method provided in the embodiments of this application. Figure 4 Step 3011 shown can be achieved through... Figure 5 Steps 30111 to 30114 are implemented, and will be explained in detail below.

[0096] In step 30111, the original phoneme sequence corresponding to the first speech sample is obtained, and the original phoneme sequence is preprocessed to obtain the processed sequence sample.

[0097] Here, a first speech sample is selected from the speech dataset. The first speech sample can be a single sentence, a speech segment, or a complete speech entry. Using a phoneme alignment tool, a corresponding original phoneme sequence is generated based on the text content and audio data corresponding to the first speech sample. Each element in the original phoneme sequence is traversed, and non-phoneme identifiers representing silence, pauses (such as sil, sp), and noise (such as noise, breath), as well as invalid phoneme identifiers that exceed the range of the preset acoustic model vocabulary, are removed. This results in a processed sequence sample containing only valid phoneme identifiers representing the actual pronunciation content, and the number of elements in the processed sequence sample is determined as the first length.

[0098] In step 30112, the first word sequence sample corresponding to the first speech sample is obtained.

[0099] Here, a pre-defined discrete acoustic model for speech (such as the HuBERT model) is invoked to extract features and perform time-dimension discrete quantization on the underlying continuous acoustic waveform or spectral features of the first speech sample, outputting a one-dimensional discrete token set, i.e., the first token sequence sample. The length of the first token sequence sample, i.e., the second length, is determined based on the number of elements in the first token sequence sample. Since the modeling granularity and frame rate of the processed sequence sample and the first token sequence sample differ, the first length is usually different from the second length.

[0100] In step 30113, a target mapping function is constructed based on the first length and the second length, and the processed sequence sample is resampled based on the target mapping function to obtain the first phoneme sequence sample.

[0101] Here, the target mapping function is used to represent the index mapping relationship between the processed sequence sample and the first word sequence sample. The processed sequence sample is resampled based on the target mapping function, that is, it is converted into a first phoneme sequence sample using the target mapping function. The length of the first phoneme sequence sample is the second length, so that the first phoneme sequence sample is aligned with the first word sequence sample in the time dimension.

[0102] In some embodiments, constructing a target mapping function based on a first length and a second length can be achieved by performing the following steps: performing articulation stage boundary detection on the processed sequence sample to obtain a second target position in the processed sequence sample, wherein the second target position includes at least one of an articulation transition frame position and a syllable boundary position; determining a temporal offset coefficient based on the second target position and determining the length ratio between the second length and the first length; and constructing a target mapping function based on the length ratio and the temporal offset coefficient.

[0103] Here, the processed sequence samples undergo articulation stage boundary detection, which involves performing a temporal scan based on acoustic rules or pre-trained acoustic feature sequences. Specifically, by analyzing the linguistic properties between adjacent phoneme identifiers in the processed sequence samples (e.g., consonant-to-vowel transition nodes) or combining the energy / spectral abrupt change features of the underlying audio signal, key articulation nodes are identified from the processed sequence samples. Key articulation nodes include, but are not limited to, initial-to-final switching nodes indicating the position of articulation transition frames, and syllable pause / switching nodes indicating the position of syllable boundaries. The node positions corresponding to the key articulation nodes in the processed sequence samples are determined as the second target positions.

[0104] Based on the preset temporal alignment compensation rule base, a numerical offset, also known as a temporal offset coefficient, is dynamically calculated or allocated for the second target position. The temporal offset coefficient is a floating-point number within the range [0, 0.2]. The length ratio between the second length and the first length is calculated through division. Combining the length ratio and the temporal offset coefficient, a target mapping function for resampling the processed sequence samples is constructed. For example, the target mapping function is determined by the following formula (1):

[0105] (1)

[0106] in, This represents the source phoneme index in the processed sequence sample. This indicates the target word index corresponding to the source phoneme index. This indicates the first length of the processed sequence sample. This indicates the second length of the first word sequence sample. This represents the length ratio between the second length and the first length. Represents the timing offset coefficient. This represents the rounding operator, used to convert floating-point results into integers.

[0107] In this embodiment, by introducing a boundary detection mechanism based on phonetic physical features and constructing a target mapping function, the technical defects of traditional static interpolation methods that rely on fixed ratios are prone to temporal breaks or obvious alignment shifts when encountering different speech rates and different pronunciation structures (such as the transition period between initials and finals) are overcome. Based on the macroscopic alignment of the global time axis, the sub-frame-level adaptive temporal correction of key pronunciation nodes is realized through the temporal offset coefficient, thereby significantly reducing the micro-alignment error caused by the inherent frame shift difference of the heterogeneous acoustic discretization model, and ensuring the temporal continuity and acoustic semantic consistency of the phoneme sequence during the resampling process.

[0108] In some embodiments, see Figure 6 , Figure 6This is a schematic diagram of the fourth process of the model training method provided in the embodiments of this application. Figure 5 In step 30113 shown, "based on the target mapping function, the processed sequence samples are resampled to obtain the first phoneme sequence sample," which can be achieved through... Figure 6 Steps 301131 to 301133 are implemented, and will be explained in detail below.

[0109] In step 301131, for each source phoneme index in the processed sequence sample, the target word index corresponding to the source phoneme index is determined based on the target mapping function.

[0110] Here, for each source phoneme index in the processed sequence sample, the source phoneme index is substituted into the target mapping function to obtain the target word index corresponding to the source phoneme index.

[0111] In some embodiments, determining the target word index corresponding to the source phoneme index based on the target mapping function can be achieved by performing the following steps: scaling the source phoneme index based on the length ratio to obtain a scaled source phoneme index; superimposing the temporal offset coefficient and the scaled source phoneme index to obtain a superposition result; and determining the target word index corresponding to the source phoneme index based on the superposition result.

[0112] Here, the index mapping process based on the source phoneme index, as indicated by the target mapping function, involves multiplying the length ratio and the source phoneme index to scale the source phoneme index, resulting in a scaled source phoneme index. For example, the scaled source phoneme index is represented as follows: ,in Indicates the source phoneme index. This indicates the length ratio.

[0113] An addition operation is performed on the timing offset coefficients and the scaled source phoneme indices to achieve superposition, yielding the superposition result. For example, the superposition result is represented as follows: ,in This represents the temporal offset coefficient. Following rounding or nearest-neighbor rounding logic, the superposition result is rounded to obtain an integer value, which is then used as the target word index corresponding to the source phoneme index.

[0114] For example, the source phoneme index is 2, the length ratio is 1.2, and the temporal offset coefficient is 0.1. The source phoneme index is scaled based on the length ratio, resulting in a scaled source phoneme index of 2.4. The temporal offset coefficient and the scaled source phoneme index are then superimposed, resulting in a superimposed result of 2.5. The superimposed result is then rounded to obtain a target word index of 3 corresponding to the source phoneme index.

[0115] In this embodiment, by decomposing the complex mapping process of phoneme index and lexical index into steps such as "multiplication scaling, addition translation, and rounding", the control dimensions of macroscopic temporal ratio mapping (scaling based on length ratio) and microscopic local phase correction (superposition based on offset coefficient) are accurately separated. Finally, through the rounding operation, the phoneme time step is accurately and stably mapped to the lexical time axis with extremely low computational overhead, avoiding temporal cumulative drift in the resampling process and improving the computational efficiency and reliability of the alignment mapping relationship.

[0116] Continue to refer to Figure 6 In step 301132, when the target word index exceeds the length range of the processed sequence sample, the phoneme identifier corresponding to the last phoneme in the processed sequence sample is used as the phoneme identifier corresponding to the target word index.

[0117] Here, when the length of the first word sequence sample is greater than the length of the processed sequence sample (i.e.) When calculating the target term index, the calculated index may exceed the length range of the processed sequence sample (i.e., ...). At this point, the most recent index correction mechanism is triggered, which will change the phoneme identifier (i.e., the source phoneme index) corresponding to the last phoneme in the processed sequence sample. The corresponding phoneme identifier is determined as the phoneme identifier corresponding to the target word index.

[0118] For example, the first word sequence sample is represented as { , , , The processed sequence samples are represented as { , , The length of the processed sequence sample is 3, and the source phoneme index is... The corresponding target lexical index is 4, at which point the source phoneme index... If the corresponding target word index exceeds the length range of the processed sequence sample, then the phoneme identifier corresponding to the last phoneme in the processed sequence sample will be... , serving as the phoneme identifier corresponding to target word index 4.

[0119] In step 301133, the positions corresponding to the word indexes that do not match the source phonemes are filled with preset ignore flags to obtain the first phoneme sequence sample.

[0120] Here, for a lexical index that does not match a source phoneme, the position corresponding to that lexical index is filled with an ignore flag IGNORE_ID. The ignore flag is used to indicate that the position is not involved in the subsequent model loss calculation process.

[0121] In this embodiment, by constructing a closed-loop resampling data processing structure that includes "index mapping calculation, out-of-bounds truncation correction, and ignore flag filling", during the length alignment process of phoneme sequences and heterogeneous word sequences, on the one hand, the reuse mechanism of the last phoneme flag effectively blocks the index out-of-bounds anomaly caused by temporal scaling, and maintains the continuity of the pronunciation state at the temporal end of the phoneme sequence; on the other hand, by using a preset ignore flag to explicitly fill the word index positions that have not matched the source phoneme, the length alignment of phoneme sequences and heterogeneous word sequences is achieved without introducing false interpolation data, avoiding invalid alignment positions from participating in the model loss calculation, and significantly improving the convergence stability and alignment robustness of the model training process.

[0122] Continue to refer to Figure 5 In step 30114, the alignment relationship between the first phoneme sequence sample and the first word sequence sample is constructed to obtain the initial alignment relationship.

[0123] Here, the length of the first phoneme sequence sample is equal to the length of the first word sequence sample, both being the second length. The correspondence between the elements of the first phoneme sequence sample and the first word sequence sample under the same sequence index is determined as the initial alignment relationship. For example, the first word sequence sample is represented as { , , The first phoneme sequence sample is represented as { , , The sequence indices of the first word sequence sample include 0, 1, and 2, and the sequence indices of the first phoneme sequence sample also include 0, 1, and 2. The element at sequence index 0 of the first word sequence sample is... The elements at sequence index 0 of the first phoneme sequence sample are Then element With elements There is a correspondence; similarly, elements With elements There is a correspondence, elements With elements If a correspondence exists, the initial alignment relationship is represented as: → , → , → .

[0124] In this embodiment, preprocessing first removes non-pronunciation attributes and redundant phoneme identifiers that cross boundaries from the original phoneme sequence, ensuring the purity of the phoneme data in the processed sequence samples. Secondly, by using a target mapping function composed of the global length ratio and the local temporal offset coefficient, the temporal misalignment defect caused by the difference in temporal resolution (frame shift) of heterogeneous acoustic extraction tools is adaptively compensated. Under the premise of low computational overhead, the processed sequence samples can be accurately resampled to a data structure of the same length as the first word sequence sample, thereby constructing a strictly corresponding structured mapping framework in the time dimension. This directly overcomes the technical problem of inconsistent sequence lengths across modalities and provides temporally continuous training data for the joint optimization and loss calculation of the subsequent end-to-end prediction model.

[0125] Continue to refer to Figure 4 In step 3012, the trained alignment prediction model is invoked to perform alignment prediction on the first phoneme sequence sample and the first word sequence sample to obtain the predicted alignment relationship.

[0126] Here, the trained alignment prediction model can be a lightweight model containing a two-layer Transformer network structure. The first phoneme sequence sample and the first word sequence sample are encoded separately to obtain phoneme sequence embedding representations and word sequence embedding representations, respectively. These phoneme sequence embedding representations and word sequence embedding representations are then concatenated to obtain cross-modal context features. The phoneme sequence embedding representation, word sequence embedding representation, and cross-modal context features are input into the trained alignment prediction model. The trained alignment prediction model performs forward computation and outputs a probability distribution matrix, which represents the probability that each phoneme in the first phoneme sequence sample maps to each word. For each phoneme in the first phoneme sequence sample, maximum likelihood extraction or dynamic programming path search is performed on the probability distribution matrix to establish a mapping relationship between the word with the highest probability and the corresponding phoneme, thereby constructing and outputting the predicted alignment relationship.

[0127] In step 3013, a first confidence level of the predicted alignment relationship is determined. When the first confidence level is greater than or equal to the first confidence level threshold, the initial alignment relationship is updated based on the predicted alignment relationship to obtain the first alignment relationship.

[0128] Here, from the probability distribution matrix mentioned above, the edge probabilities or maximum activation probabilities along the paths where the predicted alignment relationship is located are extracted and used as the confidence level to quantify the reliability of the predicted alignment relationship, i.e., the first confidence level. Subsequently, the first confidence level is compared with a preset first confidence level threshold (e.g., configured as 0.5). When the first confidence level is greater than or equal to the first confidence level threshold, the original relationship data in the initial alignment relationship is updated using the predicted alignment relationship to obtain the first alignment relationship.

[0129] In this embodiment of the application, when processing the temporal mapping of phoneme sequence samples and word sequence samples, the trained alignment prediction model is used to predict the alignment of phoneme sequence samples and word sequence samples, which can obtain a refined predicted alignment relationship. A filtering mechanism based on a first confidence threshold is introduced, which only allows the predicted alignment relationship with high confidence to correct the initial alignment relationship. Thus, under the limitation of lacking manually labeled data, the alignment relationship with high robustness and high accuracy is adaptively output.

[0130] In some embodiments, see Figure 7 , Figure 7 This is a schematic diagram of the fifth step of the model training method provided in this application embodiment, which obtains the trained alignment prediction model through... Figure 7 Steps 310 to 314 are implemented, and will be explained in detail below.

[0131] In step 310, for each phoneme in the first phoneme sequence sample, based on the initial alignment relationship, the first word segment corresponding to the phoneme is determined in the first word segment sequence sample.

[0132] Here, for each phoneme in the first phoneme sequence sample, based on the initial alignment relationship, the target word corresponding to the phoneme is determined in the first word sequence sample, and the word interval corresponding to the phoneme is determined using the target word, that is, the first word interval.

[0133] In some embodiments, determining the first word segment interval corresponding to the phoneme in the first word segment sequence sample based on the initial alignment relationship can be achieved by performing the following steps: obtaining the target word segment mapped by the phoneme from the initial alignment relationship; and extracting multiple consecutive word segments centered on the target word segment from the first word segment sequence sample according to the preset window length to obtain the first word segment interval corresponding to the phoneme.

[0134] Here, for each phoneme in the first phoneme sequence sample, the index of the phoneme is used as the retrieval key. A query operation is performed in the initial alignment relationship to extract the word index that corresponds to the index of the phoneme. The word corresponding to the word index is determined as the word mapped by the phoneme, that is, the target word.

[0135] Set the preset window length to Each token determines the center position of the window as the location of the target word, and the location of the target word is marked as... Then the starting position of the window is The end position of this window is Under the spatial constraints of this window, multiple consecutive word units centered on the target word unit are located within the interval [ , Within ], the interval in the first word sequence sample [ , [Identified as the first word segment corresponding to the phoneme]

[0136] In this embodiment of the application, when processing the alignment prediction task of phoneme sequence samples and word sequence samples, the candidate search space of the word sequence sample time axis is shrunk from the full long sequence to a local interval (the first word interval). This not only significantly reduces the memory overhead and floating-point operation complexity required by the subsequent alignment prediction model, but also prevents long-span temporal drift or cross-pronunciation stage misalignment that may occur in the alignment prediction model during complex unsupervised iteration. This ensures that the mapping relationship from phoneme to word is strictly constrained within the local spatiotemporal range that conforms to the laws of pronunciation continuity and monotonicity, and significantly improves the structural stability of the model alignment prediction.

[0137] Continue to refer to Figure 7 In step 311, the index position of the target word corresponding to the phoneme in the initial alignment relationship in the first word sequence sample is determined as the first target position.

[0138] Here, the target word with a preset mapping association with the phoneme is determined from the initial alignment relationship, and the coordinate data of the target word on the time axis of the first word sequence sample is extracted, that is, the index position of the target word in the first word sequence sample, and it is determined as the first target position.

[0139] In step 312, within the first word interval, a target probability distribution is constructed in which the probability of the first target position is greater than the probability of other positions.

[0140] Here, within the first word interval corresponding to the phoneme, the probability value of the first target position is set to be greater than the probability value of other positions, and a target probability distribution is constructed based on the probability values ​​corresponding to the first target position and other positions within the first word interval.

[0141] In some embodiments, within a first word interval, constructing a target probability distribution where the probability of the first target position is greater than the probabilities of other positions can be achieved by performing the following steps: setting the probability value corresponding to the first target position within the first word interval as a first probability value; setting the probability values ​​corresponding to other positions within the first word interval besides the first target position as second probability values, wherein the first probability value is greater than the second probability value; and forming a target probability distribution corresponding to a phoneme by using the first probability value and the second probability value within the first word interval.

[0142] Here, a one-dimensional floating-point initial tensor with a length equal to the length of the first word interval is initialized. The probability value corresponding to the first target position within the first word interval is set as the first probability value, and the first probability value is mapped to the corresponding position in the initial tensor data to obtain the first tensor data. For example, the length of the first word interval is 5, the initial tensor data is represented as [0.0, 0.0, 0.0, 0.0, 0.0], the first probability value is 0.9, and the first probability value 0.9 is mapped to the corresponding position in the initial tensor data. At this time, the first tensor data is represented as [0.0, 0.0, 0.9, 0.0, 0.0].

[0143] The probability values ​​corresponding to positions other than the first target position within the first word interval are set as the second probability values. The first probability value is greater than the second probability value. The second probability value is then mapped to the corresponding position in the first tensor data to obtain the second tensor data. Continuing the example above, the first tensor data is [0.0, 0.0, 0.9, 0.0, 0.0], and the second probability value is 0.1. The second probability value 0.1 is mapped to the corresponding position in the first tensor data, and the second tensor data is then represented as [0.1, 0.1, 0.9, 0.1, 0.1].

[0144] After the above processing, the first probability value and the second probability value within the first word interval are both mapped to the initial tensor data, and the resulting second tensor data is determined as the target probability distribution corresponding to the phoneme. Continuing with the above example, the second tensor data is represented as [0.1, 0.1, 0.9, 0.1, 0.1], and the target probability distribution is also represented as [0.1, 0.1, 0.9, 0.1, 0.1].

[0145] In this embodiment, during the pre-training stage of the alignment prediction model, the initial alignment relationship is transformed into a continuous, structured probability distribution representation that can be processed by the neural network. This constructs a well-defined and low-interference supervision target for the alignment prediction model to be trained within a limited local feature space. This forces the alignment prediction model to learn and fit the initial alignment relationship with extremely high computational efficiency, avoiding gradient dissipation or temporal error convergence in the global large-span search, and ensuring the numerical stability and fitting accuracy of the iterative optimization of the alignment prediction model.

[0146] Continue to refer to Figure 7 In step 313, the first phoneme sequence sample and the first word sequence sample are input into the alignment prediction model to be trained, and the predicted probability distribution of the phoneme in the first word interval is output.

[0147] Here, firstly, the first phoneme sequence sample and the first word sequence sample are transformed into high-dimensional real vector representations through the embedding layer of the alignment prediction model to be trained, respectively, to obtain phoneme embedding vectors and word embedding vectors. The word embedding vector located within the first word interval is determined as the target embedding vector. Then, the phoneme embedding vector and the target embedding vector are concatenated, and positional encoding or cross-modal relative displacement features can be selectively superimposed to form a multimodal input feature matrix of the phoneme within the first word interval. Finally, after the multimodal input feature matrix is ​​processed by the model's self-attention mechanism and feedforward neural network, the top-level alignment prediction head (such as a fully connected layer combined with the Softmax activation function) outputs a floating-point array. The length of this array is the same as the length of the first word interval, and each floating-point value in the array represents the probability value of the phoneme being mapped to each word position within the first word interval under the current training parameters. This array is also the predicted probability distribution of the phoneme within the first word interval.

[0148] In step 314, the alignment loss value is determined based on the predicted probability distribution and the target probability distribution, and the model parameters of the alignment prediction model are updated based on the alignment loss value to obtain the trained alignment prediction model.

[0149] Here, the alignment loss value includes the initial loss value and the auxiliary loss value. The alignment loss value is used as an error feedback signal to trigger the backpropagation calculation process of the alignment prediction model, thereby updating the model parameters of the alignment prediction model and obtaining the trained alignment prediction model.

[0150] In some embodiments, determining the alignment loss value based on the predicted probability distribution and the target probability distribution can be achieved by performing the following steps: using a preset alignment loss function, determining an initial loss value based on the predicted probability distribution and the target probability distribution; obtaining the center positions of the first word intervals corresponding to two adjacent phonemes in the first phoneme sequence sample; when the indices of the center positions of two adjacent phonemes in the first word sequence sample do not satisfy the monotonically non-decreasing condition, determining an auxiliary loss value that violates the monotonically non-decreasing condition based on a preset penalty function; and fusing the auxiliary loss value and the initial loss value to obtain the alignment loss value.

[0151] Here, the preset alignment loss function can be the cross-entropy loss function. Using the preset alignment loss function, the predicted probability distribution and the target probability distribution are subjected to element-wise logarithmic operation and multiplication accumulation operation to calculate the information divergence between the two distributions, thereby obtaining the initial loss value.

[0152] For two logically adjacent phonemes on the time axis in the first phoneme sequence sample (e.g., with indices respectively), and Based on the initial alignment relationship, the first word segment intervals corresponding to the two adjacent phonemes are determined in the first word segment sequence sample, and the center position of the two first word segment intervals is determined based on the center anchor point coordinates of the two first word segment intervals on the first word segment sequence sample.

[0153] According to the objective temporal laws of phonetics, index The index of the center position corresponding to the phoneme must be less than or equal to the index. The index of the center position corresponding to the phoneme, i.e., the monotonically non-decreasing condition. The indices of two center positions in the first word sequence sample are compared. When the indices of two center positions in the first word sequence sample do not satisfy the monotonically non-decreasing condition, i.e., the indexes are... The index of the center position corresponding to the phoneme is greater than the index. The index of the center position corresponding to the phoneme triggers the calculation process of the auxiliary loss value.

[0154] The auxiliary loss value that violates the monotonically non-decreasing condition is determined based on a preset penalty function. The preset penalty function can be the absolute value of the difference between the two center position indices (L1 norm), the square of the difference between the two center position indices (L2 norm), or a fixed constant penalty term. The auxiliary loss value and the initial loss value are arithmetically summed and fused (the auxiliary loss value can also be pre-multiplied by a weight hyperparameter) to obtain the alignment loss value.

[0155] In this embodiment, when the alignment prediction model fits the initial alignment relationship through the local word interval (first word interval), it not only ensures the alignment accuracy of the model within the local window through the initial loss value (cross entropy), but also introduces an auxiliary loss value based on the comparison of adjacent center position indices to apply strict monotonicity regularization constraints on the backpropagation gradient. This prevents the model from generating problems such as temporal reversal and cross-stage disorder when fitting complex pronunciation data, and significantly improves the structural stability of the alignment relationship predicted by the model on the global time axis.

[0156] In this embodiment, the search space for alignment matching of the alignment prediction model is narrowed from the global long sequence to the first word interval, which effectively prevents long-span temporal drift or pronunciation stage misalignment that is prone to occur during the model iteration process. By determining the alignment loss value based on the predicted probability distribution and the target probability distribution, a local supervision constraint signal with clear numerical value and clear gradient direction is provided for the alignment prediction model, thereby guiding the alignment prediction model to fit the initial alignment relationship in the limited local feature space, overcoming the gradient oscillation problem that is easily caused by direct alignment of global sequences in unlabeled scenarios.

[0157] In some embodiments, the first word sequence sample is labeled with a temporal resolution, and the temporal resolution type includes at least a first resolution, a second resolution, and a third resolution, divided from low to high temporal resolution. See [link to documentation]. Figure 8 , Figure 8 This is a schematic diagram of the sixth step of the model training method provided in this application embodiment, which determines the first alignment relationship between the first phoneme sequence sample and the first word sequence sample, and can also be achieved through... Figure 8 Steps 320 to 323 are implemented, and will be explained in detail below.

[0158] In step 320, based on the first resolution, initial alignment is performed using a first-level search window to obtain the global temporal constraint relationship between the first phoneme sequence sample and the first word sequence sample at the pronunciation stage level.

[0159] Here, temporal resolution refers to the duration of audio represented by a single token on the timeline. The first resolution, also known as the coarse-grained resolution, can be set to 50 milliseconds per token. In the first token sequence sample at the first resolution, each token corresponds to a pronunciation stage unit; for example, a combination of "initial consonant + final vowel" is considered as one pronunciation stage unit. The first-level search window is the token interval used to perform initial alignment, and its size can be set to 5 tokens.

[0160] Using a first-level search window, initial alignment processing is performed on the first word sequence sample and the first phoneme sequence sample at the first resolution to achieve coarse alignment matching with the pronunciation stage unit as the basic unit, and to obtain the global temporal constraint relationship at the pronunciation stage level.

[0161] In some embodiments, based on a first resolution, initial alignment is performed using a first-level search window to obtain the global temporal constraint relationship between the first phoneme sequence sample and the first word sequence sample at the pronunciation stage level. This can be achieved by performing the following steps: extracting the first acoustic feature of the first phoneme sequence sample and the first text feature corresponding to the first word sequence sample; determining the first similarity matrix between the first acoustic feature and the first text feature under the constraint of the first-level search window; performing optimal path search based on the first similarity matrix to determine the start boundary and termination boundary between the pronunciation segment and the word text block, and using the start boundary and termination boundary as the global temporal constraint relationship.

[0162] Here, a pre-trained acoustic encoder network (such as a convolutional neural network) is used to perform feature mapping on the first phoneme sequence samples representing the pronunciation structure, transforming their discrete phoneme identifiers into continuous high-dimensional real-number vectors to obtain the first acoustic features. Simultaneously, a pre-trained text encoder is used to transform the first word sequence samples constituting the target time axis into corresponding high-dimensional real-number vectors to obtain the first text features.

[0163] The target acoustic features located within the first-level search window are determined from the first acoustic features, and the target text features located within the first-level search window are determined from the first text features. The dot product or cosine similarity is calculated on the target acoustic features and the target text features to obtain a two-dimensional numerical matrix representing the matching probability of the first acoustic features and the first text features under the first-level search window, and this matrix is ​​determined as the first similarity matrix.

[0164] The optimal path is determined by searching for the connected path with the highest cumulative matching probability from the top left corner to the bottom right corner of the first similarity matrix. This optimal path physically treats consecutive phoneme combinations (e.g., "initial consonant + final vowel") in the first phoneme sequence sample as an indivisible pronunciation segment and maps them to a consecutive segment of word text in the first word sequence sample. The starting and ending index positions of this optimal path connecting each pronunciation segment to its corresponding word text block in the first similarity matrix are defined as the starting and ending boundaries, respectively. These starting and ending boundaries constitute a global temporal constraint relationship.

[0165] In this embodiment of the application, when processing the temporal correspondence between complex phoneme sequence samples and word sequence samples, the global unconstrained search, whose computational complexity increases quadratically, is reconstructed into a constrained search within the first-level search window area. With extremely low memory overhead and computational cost, the temporal correspondence between phoneme sequence samples and word sequence samples at the pronunciation stage level (coarse-grained) is quickly established. At the same time, the obtained global temporal constraint relationship lays a reliable data foundation for the coarse-to-fine adaptive multi-scale hierarchical alignment mechanism.

[0166] Continue to refer to Figure 8 In step 321, based on the second resolution, within the scope defined by the global temporal constraint relationship, syllable-level alignment is performed using a second-level search window to obtain the first alignment result.

[0167] Here, the second resolution, also known as the medium-granularity resolution, can be set to 20 milliseconds per token. In the first token sequence sample at the second resolution, each token corresponds to a syllable unit; for example, a single syllable in pinyin is considered a syllable unit. The second-level search window is the token interval used to perform syllable-level alignment, and its size can be set to 3–5 tokens.

[0168] Within the scope of global temporal constraints, a second-level search window is used to perform syllable-level alignment processing on the first word sequence sample and the first phoneme sequence sample of the second resolution, so as to achieve secondary alignment matching with syllable units as the basic units and obtain the syllable-level alignment result, which is the first alignment result.

[0169] In some embodiments, based on the second resolution, within the scope defined by the global temporal constraints, syllable-level alignment is performed using a second-level search window to obtain a first alignment result. This can be achieved by performing the following steps: extracting the second acoustic features of the first phoneme sequence sample and obtaining the syllable-level lexical features of the first lexical sequence sample; mapping the global temporal constraints to an alignment search mask; determining the attention alignment probability between the second acoustic features and the syllable-level lexical features under the joint constraints of the alignment search mask and the second-level search window; and determining the first alignment result representing the syllable-level matching path based on the attention alignment probability.

[0170] Here, a pre-trained recurrent neural network is used to extract high-frequency or local dependency information from the first phoneme sequence samples, encoding it into continuous high-dimensional real vectors to obtain the second acoustic features. Simultaneously, a word embedding neural network is invoked to extract high-dimensional real vectors from the first word sequence samples that reflect the acoustic properties of complete syllables (such as single Chinese pinyin or English syllables), and these vectors are identified as syllable-level word features.

[0171] Generate a Boolean or floating-point two-dimensional matrix (such as a matrix with all negative infinity) that matches the second acoustic feature dimension and the syllable-level word feature dimension. Iterate through each start boundary and each end boundary in the global temporal constraint relationship, and set the element values ​​corresponding to each start boundary and each end boundary in the two-dimensional matrix to the flags that allow computation, and set the remaining element values ​​in the two-dimensional matrix to the flags that do not allow computation, to obtain a two-dimensional mask matrix, and determine it as the alignment search mask.

[0172] A local sparse matrix is ​​generated based on the alignment search mask and the second-level search window. This local sparse matrix is ​​then used to normalize the second acoustic features and syllable-level word features using a softmax operation, resulting in a highly sparse two-dimensional probability distribution matrix, i.e., the attention alignment probability. Subsequently, the maximum a posteriori path search algorithm is used to search for the syllable-word mapping path with the highest cumulative probability within the non-zero connected regions of the two-dimensional probability distribution matrix indicated by the attention alignment probability. This path data structure and its corresponding index mapping table represent the first alignment result of the syllable-level matching path.

[0173] In this embodiment, when performing syllable-level alignment on phoneme sequence samples and word sequence samples, the global temporal constraint relationship is mapped to an alignment search mask. A joint constraint mechanism of global and local constraints is constructed by combining the alignment search mask and the second-level search window. This shrinks the calculation of the global attention matrix to be performed within a local sparse matrix, effectively avoiding the long-term temporal drift and cross-pronunciation stage alignment defects that are easily caused in complex alignment calculations. This ensures that the output attention alignment probability has a high degree of concentration and determinism in numerical distribution, thereby providing data support for generating a temporally monotonic syllable-level matching path (the first alignment result).

[0174] Continue to refer to Figure 8 In step 322, based on the third resolution, within the range defined by the first alignment result, phoneme-level alignment is performed using a third-level search window to obtain the second alignment result.

[0175] Here, the third resolution, also known as the fine-grained resolution, can be set to 10 milliseconds per token. In the first token sequence sample at the third resolution, each token corresponds to a frame-level speech unit. The third-level search window is the token interval used to perform phoneme-level alignment, and its size can be set to 1–2 tokens. The window lengths of the first-level search window, the second-level search window, and the third-level search window decrease progressively.

[0176] Within the scope defined by the first alignment result, a third-level search window is used to perform phoneme-level alignment processing on the first word sequence sample and the first phoneme sequence sample at the third resolution, so as to achieve alignment matching with frame-level speech units as the basic units and obtain the phoneme-level alignment result, which is the second alignment result.

[0177] In some embodiments, based on a third resolution, within the range defined by the first alignment result, phoneme-level alignment is performed using a third-level search window to obtain a second alignment result. This can be achieved by performing the following steps: extracting frame-level acoustic features of the first phoneme sequence sample and obtaining phoneme-level lexical features of the first lexical sequence sample; generating a mask matrix based on the matching path indicated by the first alignment result and the window length of the third-level search window; and using the mask matrix to determine the alignment frame boundary between the frame-level acoustic features and the phoneme-level lexical features to obtain a second alignment result representing the frame-level matching correspondence.

[0178] Here, a pre-configured acoustic feature extraction network is used to extract high-temporal-resolution real-valued vector representations from the original audio data corresponding to the first phoneme sequence samples, and these representations are identified as frame-level acoustic features. Simultaneously, a fine-grained lexical embedding network is invoked to extract high-dimensional real-valued vector representations of discrete lexical units from the first lexical sequence samples, and these representations are used as phoneme-level lexical features.

[0179] For each phoneme in the first phoneme sequence sample, based on the matching path indicated by the first alignment result, the first target word corresponding to that phoneme is determined in the first word sequence sample. The first target word is used as the center of the third-level search window, and the target index boundary corresponding to that phoneme is calculated in combination with the window length of the third-level search window. A floating-point or Boolean two-dimensional matrix matching the frame-level acoustic feature dimension and the phoneme-level word feature dimension is created. The elements in this two-dimensional matrix corresponding to each target index boundary are set to activation values ​​(such as 1.0 or True), and the remaining elements in this two-dimensional matrix are set to inactive values ​​(such as negative infinity or False), thus obtaining the mask matrix.

[0180] The correlation score between frame-level acoustic features and phoneme-level word features is calculated using a mask matrix. Based on the correlation score, multiple phoneme index coordinates representing changes in phoneme structure (such as vowel start point and consonant release point) are extracted from the first phoneme sequence sample. These multiple phoneme index coordinates are then determined as the alignment frame boundary between the frame-level acoustic features and the phoneme-level word features. Based on the alignment frame boundary, the first phoneme sequence sample and the first word sequence sample are aligned to obtain the frame-level matching correspondence between the first phoneme sequence sample and the first word sequence sample, which is the second alignment result.

[0181] In this embodiment, a mask matrix is ​​generated based on the matching path indicated by the first alignment result and the window length of the third-level search window. Under the constraint of the mask matrix, the alignment frame boundary between the frame-level acoustic features and the phoneme-level lexical features is determined. Since the search space has been compressed, this process eliminates most of the interference from energy and spectral background noise in the alignment domain, thereby outputting a second alignment result with high confidence that can accurately characterize the frame-level matching correspondence. The second alignment result can be used to support downstream application scenarios with real-time requirements, such as pronunciation quality assessment.

[0182] In some embodiments, calibrating the second alignment result based on global temporal constraints can be achieved by performing the following steps: determining the pronunciation stage boundaries based on global temporal constraints, and using the pronunciation stage boundaries as constraints to divide the first phoneme sequence sample into multiple consecutive pronunciation stages; determining the corresponding alignment interval for each pronunciation stage on the first word sequence sample; when performing phoneme-level alignment, shrinking and restricting the effective mask region in the mask matrix to the alignment interval corresponding to the pronunciation stage to which each phoneme belongs; updating the mask matrix based on the temporal arrangement order of each phoneme in the first phoneme sequence sample to obtain the updated mask matrix; and calibrating the second alignment result using the updated mask matrix to obtain the calibrated second alignment result.

[0183] Here, based on global temporal constraints, multiple discrete index coordinates representing acoustic feature transition nodes (such as initial and final switching nodes, and phrase pause nodes) are extracted from the first phoneme sequence sample, and these multiple discrete index coordinates are determined as the boundaries of the pronunciation stages. Under the constraints of the pronunciation stage boundaries, a slicing operation is performed on multiple consecutive phonemes in the first phoneme sequence sample to obtain multiple non-overlapping sub-phoneme intervals arranged in their original order. Each sub-phoneme interval is an independent pronunciation stage, and multiple sub-phoneme intervals constitute multiple consecutive pronunciation stages.

[0184] For each pronunciation stage, based on the pronunciation stage boundary, the start target index and end target index corresponding to the pronunciation stage are extracted from the first word sequence sample, and the interval constructed by the start target index and end target index is determined as the alignment interval corresponding to the pronunciation stage.

[0185] When aligning the first phoneme sequence sample and the first word sequence sample at the phoneme level, for each valid mask region in the mask matrix, the intersection region between the valid mask region and the alignment interval corresponding to the pronunciation stage of each phoneme is determined, and this intersection region is redefined as the valid mask region to achieve the shrinkage restriction processing of the valid mask region. The valid mask region satisfies the monotonicity condition, which is that the alignment position of the next phoneme in the first phoneme sequence sample on the first word sequence sample is greater than or equal to the alignment position of the previous phoneme in the first phoneme sequence sample on the first word sequence sample.

[0186] Based on the temporal arrangement order of each phoneme in the first phoneme sequence sample, the mask matrix is ​​updated. This involves identifying target mask regions in the mask matrix that do not satisfy the temporal arrangement order and setting the elements within these target mask regions as masking markers, thus obtaining the updated mask matrix. The updated mask matrix is ​​then used to generate phoneme-level alignment paths. Based on these alignment paths, mapping relationships in the second alignment result that have experienced temporal drift or cross-stage misalignment are removed, thereby calibrating the second alignment result and obtaining the calibrated second alignment result.

[0187] In this embodiment, when calibrating the second alignment result based on global temporal constraints, the global temporal constraints are used to determine the corresponding alignment interval for each pronunciation stage in the phoneme sequence sample, and the effective mask area in the mask matrix is ​​shrunk and restricted to the alignment interval corresponding to the pronunciation stage of each phoneme. This avoids the long-distance drift error across stages that acoustic features are prone to cause under complex background noise. At the same time, the updated mask matrix is ​​used to calibrate the second alignment result, realizing bidirectional adaptive calibration of alignment from coarse-grained to fine-grained, which significantly improves the alignment accuracy of the calibrated second alignment result on the global time axis.

[0188] In some embodiments, calibrating the global temporal constraint relationship based on the second alignment result can be achieved by performing the following steps: calculating the phoneme consistency index based on the second alignment result; when there is a target mapping relationship in the global temporal constraint relationship that is inconsistent with the phoneme consistency index, deleting or adjusting the target mapping relationship in the global temporal constraint relationship to obtain the calibrated global temporal constraint relationship.

[0189] Here, for each phoneme in the first phoneme sequence sample, the first target word interval corresponding to that phoneme is determined in the first word sequence sample based on the second alignment result, and the matching confidence of that phoneme within the first target word interval is obtained. The matching confidence of each phoneme within its corresponding first target word interval is summed and averaged to obtain the phoneme consistency index corresponding to the second alignment result.

[0190] For each phoneme in the first phoneme sequence sample, the target word interval corresponding to that phoneme is determined based on the global temporal constraint relationship in the first word sequence sample, and the matching confidence of that phoneme within the target word interval is obtained. The matching confidence of each phoneme within its corresponding target word interval is summed and averaged to obtain the phoneme consistency index corresponding to the global temporal constraint relationship.

[0191] The phoneme consistency index corresponding to the second alignment result is compared with the phoneme consistency index corresponding to the global temporal constraint relationship. When the phoneme consistency indices corresponding to the two are inconsistent, the target mapping relationship that is inconsistent with the second alignment result is determined from the global temporal constraint relationship. The target mapping relationship can be deleted from the global temporal constraint relationship, or the target mapping relationship can be adjusted using the second alignment result to obtain the calibrated global temporal constraint relationship.

[0192] In this embodiment, when calibrating the global temporal constraint relationship based on the second alignment result, a phoneme consistency index is calculated based on the second alignment result. When there is a target mapping relationship in the global temporal constraint relationship that is inconsistent with the phoneme consistency index, the target mapping relationship in the global temporal constraint relationship is deleted or adjusted. This enables the comparison of the local consistency parameters corresponding to the fine-grained alignment result (second alignment result) and the coarse-grained alignment result (global temporal constraint relationship), automatically identifies unreasonable long-span stage offsets in the coarse-grained alignment result, and effectively eliminates the cumulative deviation problem that cannot be overcome by single-scale alignment, thus realizing bidirectional adaptive calibration from fine-grained to coarse-grained alignment.

[0193] Continue to refer to Figure 8 In step 323, based on the second alignment result, the first alignment relationship between the first phoneme sequence sample and the first word sequence sample is determined.

[0194] Here, based on the second alignment result, the duration of pronunciation corresponding to each word in the first word sequence sample is determined, and based on the duration of pronunciation corresponding to each word, the first alignment relationship between the first phoneme sequence sample and the first word sequence sample is determined.

[0195] In some embodiments, determining the first alignment relationship between the first phoneme sequence sample and the first word sequence sample based on the second alignment result can be achieved by performing the following steps: extracting the start acoustic frame index and end acoustic frame index corresponding to each word in the first word sequence sample from the second alignment result; determining the pronunciation duration corresponding to each word based on the start acoustic frame index and end acoustic frame index; and determining the mapping sequence composed of word and corresponding pronunciation duration as the first alignment relationship.

[0196] Here, for each word in the first word sequence sample, the target phoneme interval corresponding to the word is determined based on the second alignment result in the first phoneme sequence sample, and the starting index corresponding to the target phoneme interval is determined as the starting acoustic frame index corresponding to the word, and the ending index corresponding to the target phoneme interval is determined as the ending acoustic frame index corresponding to the word.

[0197] For each lexical unit in the first lexical unit sequence sample, the difference between the starting acoustic frame index and the ending acoustic frame index corresponding to the lexical unit is determined as the total number of acoustic frames. The total number of acoustic frames is multiplied by a preset single-frame temporal resolution to obtain the pronunciation duration corresponding to the lexical unit. Following the temporal arrangement of the first lexical unit sequence sample, each lexical unit and its corresponding pronunciation duration are mapped into a sequence to obtain the first alignment relationship.

[0198] In this embodiment, the second alignment result is transformed into a well-structured temporal mapping sequence of lexical-pronunciation duration. This not only reduces the storage overhead of alignment relationships in memory and the transmission bandwidth of inter-module communication, but also directly adapts to the alignment label format required in the speech generation model. This enables the second alignment result after multi-scale calibration to be transformed into control instructions that directly drive the generation of downstream acoustic features, significantly improving the temporal and physical continuity of the generated speech.

[0199] In this embodiment, a coarse-grained first resolution and a first-level search window are first used to establish a global temporal constraint relationship between phoneme sequence samples and word sequence samples at the pronunciation stage level. Then, within the range limited by the global temporal constraint relationship, the system sequentially switches to a medium-grained second resolution and a fine-grained third resolution, while simultaneously shrinking the search window to a second-level search window and a third-level search window. This realizes a coarse-to-fine cascaded alignment mechanism, which can use the alignment results of the previous coarse-grained level to limit the alignment process of the next fine-grained level, avoiding long-term temporal drift and cross-pronunciation stage misalignment that acoustic features are prone to cause under complex background noise.

[0200] In some embodiments, anomaly detection of the first phoneme sequence sample can be achieved by performing the following steps: when the first phoneme sequence sample is detected to be an empty phoneme sequence, a preset identifier is filled into the first phoneme sequence sample to obtain a third phoneme sequence sample, the sequence length of the third phoneme sequence sample being equal to the sequence length of the first word sequence sample, so as to use the third phoneme sequence sample to iteratively train the initial prediction model; or, when the proportion of valid phoneme identifiers in the first phoneme sequence sample is detected to be less than a first preset threshold, a preset identifier is filled into the position corresponding to the invalid phoneme identifier in the first phoneme sequence sample to obtain a fourth phoneme sequence sample, so as to use the fourth phoneme sequence sample to iteratively train the initial prediction model.

[0201] Here, an empty phoneme sequence is a phoneme sequence without any valid pronunciation entity identifier and with an empty payload. The length attribute value of the first phoneme sequence sample is read; if the length attribute value is zero, it is determined that the first phoneme sequence sample is an empty phoneme sequence. The first phoneme sequence sample is then fully populated with a preset identifier, which can be an ignore identifier (IGNORE_ID). Simultaneously, the sequence length of the first phoneme sequence sample is updated based on the sequence length of the first word sequence sample to obtain the third phoneme sequence sample. This ensures that the sequence length of the third phoneme sequence sample is the same as that of the first word sequence sample, allowing subsequent iterative training of the initial prediction model using the third phoneme sequence sample.

[0202] Valid phoneme identifiers are identifiers representing the actual pronunciation content in a phoneme sequence (e.g., vowel identifiers, consonant identifiers). Invalid phoneme identifiers are structured feature identifiers that do not have a mapping meaning to the actual pronunciation content, including but not limited to filler identifiers for silence / pause, boundary structure symbols, and identifiers that do not match the preset vocabulary. The first phoneme sequence sample is traversed, and the total number of phonemes belonging to valid phoneme identifiers is determined by table lookup or conditional matching. The ratio between this total number of phonemes and the length of the first phoneme sequence sample is determined as the proportion of valid phoneme identifiers. A first preset threshold is a preset lower limit for the proportion of valid phoneme identifiers. When the proportion of valid phoneme identifiers in the first phoneme sequence sample is detected to be less than the first preset threshold, the positions corresponding to each invalid phoneme identifier in the first phoneme sequence sample are determined, and preset identifiers are filled into the positions corresponding to each invalid phoneme identifier to update the first phoneme sequence sample, resulting in a fourth phoneme sequence sample. This fourth phoneme sequence sample is then used to perform iterative training on the initial prediction model.

[0203] In this embodiment, when the phoneme sequence sample is an empty phoneme sequence, or when the proportion of valid phoneme identifiers in the phoneme sequence sample is less than a threshold, the phoneme sequence sample is filled with a preset identifier (IGNORE_ID), which ensures the dimensional alignment between the phoneme sequence sample and the word sequence sample. At the same time, the preset identifier blocks gradient backpropagation in the loss calculation, avoiding the model parameter oscillation caused by abnormal phonemes without phonetic significance in the phoneme sequence sample during model iterative training, and significantly enhancing the parameter convergence stability of the model when facing complex data distributions containing noise.

[0204] In some embodiments, anomaly detection of the first phoneme sequence sample can be achieved by performing the following steps: obtaining a target phoneme template corresponding to the first phoneme sequence sample, the target phoneme template including multiple target phonemes with standard pronunciation; for each phoneme in the first phoneme sequence sample, determining the edit distance between the phoneme and the corresponding target phoneme; when the edit distance is greater than a third preset threshold, determining the phoneme as an abnormal phoneme; determining the second position of the abnormal phoneme in the first phoneme sequence sample, filling the second position with a preset identifier to obtain a fifth phoneme sequence sample, so as to use the fifth phoneme sequence sample to iteratively train the initial prediction model.

[0205] Here, the target phoneme template is a standard pronunciation template that strictly follows linguistic standard pronunciation rules and includes multiple target phonemes with standard pronunciations. For each phoneme in the first phoneme sequence sample, the corresponding target phoneme is determined based on the linguistic standard pronunciation rules, and a target phoneme template is constructed based on the target phonemes corresponding to each phoneme.

[0206] Edit distance is an indicator used to quantify the degree of difference between a phoneme and its corresponding target phoneme. It represents the total number of operation steps required to convert a phoneme into its corresponding target phoneme using the fewest possible editing operations (including insertion, deletion, or replacement). For each phoneme in the first phoneme sequence sample, the edit distance between the phoneme and its corresponding target phoneme is determined based on the total number of operation steps required to convert the phoneme into its corresponding target phoneme. A third preset threshold is a preset upper limit for the edit distance. When the edit distance between a phoneme and its corresponding target phoneme exceeds the third preset threshold, the phoneme is determined to be an abnormal phoneme.

[0207] For each abnormal phoneme in the first phoneme sequence sample, the position of the abnormal phoneme in the first phoneme sequence sample, i.e. the second position, is determined, and a preset identifier (IGNORE_ID) is filled in the second position corresponding to the abnormal phoneme to update the first phoneme sequence sample, thereby obtaining the fifth phoneme sequence sample. The fifth phoneme sequence sample is then used to perform iterative training on the initial prediction model.

[0208] In this embodiment, when processing abnormal phonemes with non-standard pronunciation in phoneme sequence samples, the edit distance is used as a quantification index to accurately locate the position (second position) of abnormal phonemes whose pronunciation deviates from the standard pronunciation template in the phoneme sequence samples. The preset identifier (IGNORE_ID) is used to fill only the position of the abnormal phonemes. This not only avoids the generation of erroneous gradient feedback by non-standard pronunciation features in the backpropagation of the model, but also retains and utilizes the supervision signals of the other standard pronunciation phonemes in the phoneme sequence samples, thereby significantly improving the parameter convergence stability and generalization robustness of the model in complex, non-standard pronunciation speech environments.

[0209] In some embodiments, anomaly detection of the first word sequence sample can be achieved by performing the following steps: determining the energy variance of the first word sequence sample; when the energy variance is greater than a second preset threshold, determining that there are noise words in the first word sequence sample; determining the first position of the noise word in the first word sequence sample; filling the first position with a preset identifier to obtain the second word sequence sample; and using the second word sequence sample to iteratively train the initial prediction model.

[0210] Here, energy variance is an assessment of the dispersion of the energy distribution of the acoustic feature sequence (such as a Mel spectrum sequence or waveform amplitude sequence) corresponding to the first word sequence sample in the time dimension. The energy variance of the first word sequence sample is obtained by acquiring the acoustic feature sequence corresponding to the first word sequence sample and calculating the summation and mean square error of the acoustic feature sequences.

[0211] The second preset threshold is the upper limit of the preset energy variance. When the energy variance of the first word sequence sample is greater than the second preset threshold, it is determined that there are noisy words in the first word sequence sample. Noisy words are words in the first word sequence sample whose acoustic features are composed of non-speech information (such as sudden environmental noise, random signals such as current microphone impact sound, etc.) and do not contain linguistic meaning.

[0212] For each noise word in the first word sequence sample, determine the position of the noise word in the first word sequence sample, i.e., the first position, and fill the first position corresponding to the noise word with a preset identifier (IGNORE_ID) to update the first word sequence sample, thereby obtaining the second word sequence sample. Then, the second word sequence sample is used to perform iterative training on the initial prediction model.

[0213] In this embodiment, when performing anomaly detection on the processed word sequence samples, the energy variance of the word sequence samples is used as a quantification index to accurately detect the position (first position) of the noise word in the word sequence samples. The preset identifier (IGNORE_ID) is used to fill only the position of the noise word, avoiding the error feedback link of the model parameter update propagation caused by the noise word. At the same time, the word segments with stable energy distribution in the word sequence samples are fully preserved and efficiently utilized, thereby significantly enhancing the model's noise robustness and parameter convergence ability in complex and uncontrolled acoustic environments.

[0214] Continue to refer to Figure 3 The following will be an explanation following step 301.

[0215] In step 302, during the first training phase, the initial prediction model is trained based on the first phoneme sequence samples, the first word sequence samples, the first alignment relationship, the first alignment window, the first phoneme loss weight, and the joint loss function of the initial prediction model.

[0216] Here, the first training phase is the initial training phase (the first 20% of training rounds). In the first training phase, the initial prediction model is trained based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weight, and the joint loss function of the initial prediction model. The first alignment window is used to constrain the candidate alignment interval between the first phoneme sequence sample and the first word sequence sample. The size of the first alignment window can be set to 5 tokens, and the first phoneme loss weight can be set to 0.2 to adopt a relaxed alignment strategy to reduce the alignment constraint strength and avoid the alignment error in the early training phase from interfering with the overall training of the model. The joint loss function includes the prediction task loss value and the phoneme loss value adjusted by the phoneme loss weight. The prediction task loss value is the loss value of the initial prediction model for the acoustic feature prediction task or the phoneme prediction task. The phoneme loss value is determined based on the feature matching degree between the phoneme sequence sample and the word sequence sample. For example, the joint loss function is determined by the following formula (2):

[0217] (2)

[0218] in, Denotes the joint loss function. This represents the predicted task loss value. Indicates the phoneme loss value. This indicates the phoneme loss weight.

[0219] In some embodiments, see Figure 9 , Figure 9 This is a schematic diagram of the seventh process of the model training method provided in the embodiments of this application. Figure 3 Step 302 shown can be achieved through... Figure 9 Steps 3021 to 3025 are implemented, and will be explained in detail below.

[0220] In step 3021, for each phoneme in the first phoneme sequence sample, a reference anchor point for the phoneme on the first word sequence sample is determined based on the first alignment relationship.

[0221] Here, for each phoneme in the first phoneme sequence sample, the reference anchor point of the phoneme on the first word sequence sample is the time position of the target word corresponding to the phoneme in the first word sequence sample.

[0222] In some embodiments, determining the reference anchor point of a phoneme on a first word sequence sample based on a first alignment relationship can be achieved by performing the following steps: parsing the first alignment relationship to obtain the index mapping relationship between the phoneme and the first word sequence sample; determining the word index of the target word corresponding to the phoneme in the first word sequence sample based on the index mapping relationship; and determining the time position of the word index as the reference anchor point corresponding to the phoneme.

[0223] Here, for each phoneme in the first phoneme sequence sample, the phoneme index is used as the retrieval key to query the target word mapped by the phoneme from the structured data corresponding to the first alignment relationship, so as to realize the parsing of the first alignment relationship, establish the index mapping relationship between the phoneme and the corresponding target word, and determine it as the index mapping relationship between the phoneme and the first word sequence sample.

[0224] For each phoneme in the first phoneme sequence sample, based on the index mapping relationship between the phoneme and the first word sequence sample, the word index of the target word that corresponds to the phoneme index is extracted from the first word sequence sample, and the time position of the word index of the target word in the first word sequence sample is determined as the reference anchor point corresponding to the phoneme.

[0225] In this embodiment, based on the pre-constructed first alignment relationship, a reference anchor point is determined for each phoneme in the phoneme sequence sample on the first word sequence sample. This achieves a unique and deterministic allocation of a reference anchor point for each phoneme in the first phoneme sequence sample, providing data support for the subsequent extraction of candidate alignment intervals. This enables the complex model computation space to be anchored from a globally unconstrained sequence to a restricted local position, thereby significantly reducing the computational complexity of determining the phoneme loss value.

[0226] Continue to refer to Figure 9 In step 3022, with the reference anchor point as the center, a second word segment corresponding to the window length of the first alignment window is extracted from the first word segment sequence sample as a candidate alignment segment.

[0227] Here, the window length of the first aligned window is set to Each token uses the reference anchor point as the center position of the first alignment window, and the position of the reference anchor point is marked as... Then the starting position of the first alignment window is The ending position of the first alignment window is Under the spatial constraints of the first alignment window, the intervals in the first word sequence sample are [ , The second word segment interval, corresponding to the window length of the first alignment window, is determined and identified as the candidate alignment interval.

[0228] In step 3023, at least one of the first phoneme sequence sample and the first word sequence sample is input into the initial prediction model for prediction processing to obtain the prediction result.

[0229] Here, the initial prediction model is used to predict acoustic features or phonemes for at least one of the first phoneme sequence samples and the first word sequence samples, obtaining the corresponding prediction results. For example, the initial prediction model is invoked to predict acoustic features for the first phoneme sequence samples and the first word sequence samples. Through the feature encoding network of the initial prediction model, the first phoneme sequence samples and their corresponding first word sequence samples are simultaneously embedded and encoded to generate a bimodal tensor representation. Then, through the intermediate fusion network of the initial prediction model, the bimodal tensor representation is used as a conditional variable to perform feature fusion (e.g., using the first phoneme sequence samples as linguistic content constraints and the first word sequence samples as acoustic duration / timbre priors), resulting in fused multimodal features. These multimodal features are then passed through the decoder network of the initial prediction model to output the acoustic feature prediction results.

[0230] In step 3024, within the candidate alignment interval, the phoneme loss value is determined based on the first phoneme sequence sample and the first word sequence sample, and the prediction task loss value is determined based on the prediction result.

[0231] Here, the phoneme loss value is determined based on the first phoneme sequence sample and the first word sequence sample within the candidate alignment interval. The loss value between the prediction result and the corresponding ground truth label of the prediction task is determined as the prediction task loss value.

[0232] In some embodiments, within the candidate alignment interval, determining the phoneme loss value based on the first phoneme sequence sample and the first word sequence sample can be achieved by performing the following steps: obtaining the first hidden layer feature corresponding to the first phoneme sequence sample obtained by the initial prediction model; obtaining the second hidden layer feature corresponding to the first word sequence sample within the candidate alignment interval obtained by the initial prediction model; determining the feature matching degree between the first hidden layer feature and the second hidden layer feature; and determining the phoneme loss value based on the feature matching degree.

[0233] Here, the first phoneme sequence sample is input into the initial prediction model. Through the feature encoding network of the initial prediction model, the first phoneme sequence sample is subjected to a series of matrix multiplications and nonlinear activation function (ReLU) operations. A high-dimensional real feature vector is output in the middle layer of the feature encoding network and determined as the first hidden layer feature corresponding to the first phoneme sequence sample.

[0234] The first word sequence sample is input into the initial prediction model. Through the feature encoding network of the initial prediction model, the first word sequence sample undergoes a series of matrix multiplications and ReLU operations. A high-dimensional real-valued feature vector is output from the intermediate layer of the feature encoding network and determined as the global hidden feature corresponding to the first word sequence sample. Based on the candidate alignment interval, feature slicing is performed on the global hidden feature corresponding to the first word sequence sample to obtain the second hidden feature.

[0235] A matrix dot product operation is performed on the first and second hidden layer features, and the result is numerically normalized to output a one-dimensional probability distribution array with values ​​ranging from 0 to 1. This array is then used as the feature matching degree between the first and second hidden layer features. The target label probability, constructed based on a standard mapping relationship, is obtained. Using the cross-entropy loss function, the loss is calculated on the feature matching degree between the first and second hidden layer features and the target label probability to obtain the phoneme loss value.

[0236] In this embodiment, when determining the feature matching degree between the first phoneme sequence sample and the first word sequence sample, a feature slicing operation is performed on the global hidden layer features of the first word sequence sample using the candidate alignment interval, thereby extracting the second hidden layer features of the first word sequence sample. This reduces the computational complexity of the feature matching degree from a global quadratic complexity to a linear complexity only within a local window. Simultaneously, the phoneme loss value derived based on the feature matching degree ensures that the model network weights achieve high-precision and high-stability numerical convergence within a limited search space.

[0237] Continue to refer to Figure 9 In step 3025, a first loss value is determined based on the first phoneme loss weight, the prediction task loss value, the phoneme loss value, and the joint loss function, and the network parameters of the initial prediction model are updated based on the first loss value.

[0238] Here, the first phoneme loss weight, the prediction task loss value, and the phoneme loss value are substituted into the joint loss function to obtain the first loss value. The first loss value is used as an error feedback signal to trigger the backpropagation calculation process of the initial prediction model, thereby updating the network parameters of the initial prediction model.

[0239] In this embodiment, during the first training phase, the initial prediction model is trained based on the first phoneme sequence samples, the first word sequence samples, the first alignment relationship, the first alignment window, the first phoneme loss weight, and the joint loss function of the initial prediction model. This allows for the use of a relaxed alignment window in the early stages of model training, enabling the learning of coarse-grained alignment relationships within a larger search space. This provides a structured temporal constraint with high spatial tolerance for the model network parameters, thereby avoiding early alignment errors from interfering with the overall training of the model.

[0240] Continue to refer to Figure 3 The following explanation will continue from step 302 above.

[0241] In step 303, the state evaluation index of the initial prediction model is determined.

[0242] Here, the state evaluation metrics include alignment error evaluation value, loss convergence evaluation value, and phoneme prediction accuracy.

[0243] In some embodiments, determining the state evaluation metric of the initial prediction model can be achieved by performing the following steps: determining the alignment confidence of the initial prediction model and determining the alignment error evaluation value based on the alignment confidence; determining the current round loss value of the initial prediction model in the current training round and the initial round loss value of the initial prediction model in the initial training round, and determining the ratio between the current round loss value and the initial round loss value as the loss convergence evaluation value; obtaining the predicted phoneme sequence and the preset phoneme sequence output by the initial prediction model in the current training round, and determining the proportion of consistency between the predicted phoneme identifiers in the predicted phoneme sequence and the preset phoneme identifiers in the preset phoneme sequence as the phoneme prediction accuracy; combining the alignment error evaluation value, the loss convergence evaluation value, and the phoneme prediction accuracy into a state evaluation metric.

[0244] Here, during the training of the initial prediction model, the alignment probability distribution matrix of each sample is obtained in the alignment prediction branch of the initial prediction model, and the probability value on the optimal mapping path in the alignment probability distribution matrix is ​​used as the alignment confidence. The alignment confidence of each sample is averaged to obtain the mean alignment confidence, which is used as the alignment confidence of the initial prediction model. The difference between the constant 1 and the alignment confidence of the initial prediction model is determined as the alignment error evaluation value. For example, the alignment error evaluation value can be determined by the following formula (3):

[0245] (3)

[0246] Determine the current prediction task loss value and the current phoneme loss value of the initial prediction model in the current training round. Substitute the first phoneme loss weight, the current prediction task loss value, and the current phoneme loss value into the joint loss function to obtain the current round loss value. Determine the initial prediction task loss value and the initial phoneme loss value of the initial prediction model in the initial training round. Substitute the first phoneme loss weight, the initial prediction task loss value, and the initial phoneme loss value into the joint loss function to obtain the initial round loss value. The ratio between the current round loss value and the initial round loss value is determined as the loss convergence evaluation value. For example, the loss convergence evaluation value can be determined by the following formula (4):

[0247] (4)

[0248] Obtain the predicted phoneme sequence and the preset phoneme sequence output by the initial prediction model in the current training round. The preset phoneme sequence is a labeled phoneme sequence, and each preset phoneme in the preset phoneme sequence is labeled as a phoneme identifier. Determine the target predicted phonemes that are consistent with each preset phoneme identifier from the predicted phoneme sequence. Determine the number of target predicted phonemes as the phoneme consistency number. Divide the phoneme consistency number by the total number of predicted phonemes in the predicted phoneme sequence to obtain the phoneme prediction accuracy. For example, the phoneme prediction accuracy can be determined by the following formula (5):

[0249] (5)

[0250] By combining the alignment error evaluation value, the loss convergence evaluation value, and the phoneme prediction accuracy, a three-dimensional state evaluation index is obtained.

[0251] In this embodiment, a multi-dimensional model state evaluation system is constructed by independently extracting quantitative evaluation values ​​from three dimensions: alignment quality, convergence speed, and prediction task performance. This system not only detects alignment quality through alignment error evaluation values ​​and the convergence trend of overall network parameters through loss convergence evaluation values, but also evaluates the performance of the prediction task through phoneme prediction accuracy. This overcomes the structural defects of traditional single metrics, which cannot reveal the uneven convergence of different task branches within the model or the contradiction between performance and loss metrics due to overfitting. The final determined state evaluation metrics provide a stable and reliable decision-making basis for the automation of subsequent model training strategies (such as triggering stage switching).

[0252] Continue to refer to Figure 3 The following explanation will continue from step 303 above.

[0253] In step 304, when the state evaluation index meets the stage switching conditions, the second training stage is triggered.

[0254] Here, when the alignment error evaluation value, loss convergence evaluation value, and phoneme prediction accuracy in the state evaluation index all reach the corresponding thresholds, it is determined that the state evaluation index meets the stage switching condition, and at this time, the second training stage is performed on the initial prediction model.

[0255] For example, the corresponding threshold for the alignment error evaluation value is 0.3, the corresponding threshold for the loss convergence evaluation value is 0.2, and the corresponding threshold for the phoneme prediction accuracy is 0.9. When the alignment error evaluation value is 0.4, the loss convergence evaluation value is 0.4, and the phoneme prediction accuracy is 0.9, all three values ​​have reached their respective thresholds. This indicates that the model's prediction alignment quality has reached the expected level in the current training phase, the model training has entered a stable convergence phase, and the model's phoneme prediction task has reached a usable accuracy level. At this point, the initial prediction model training enters the second training phase.

[0256] In step 305, during the second training phase, the initial prediction model is trained again based on the first phoneme sequence samples, the first word sequence samples, the first alignment relation, the second alignment window, the second phoneme loss weights, and the joint loss function.

[0257] Here, the second training phase is the middle of the training process (20%–80% of training rounds). In this phase, the initial prediction model is trained again based on the first phoneme sequence samples, the first word sequence samples, the first alignment relation, the second alignment window, the second phoneme loss weight, and the joint loss function. The window length of the second alignment window is smaller than that of the first alignment window, and the first phoneme loss weight is smaller than that of the second phoneme loss weight. The size of the second alignment window can be set to 3 tokens, and the second phoneme loss weight can be set to 0.4. The alignment constraints are gradually strengthened based on the first training phase, allowing the initial prediction model to improve the alignment accuracy between the phoneme and word sequences while maintaining stable training. The training process in the second phase is the same as that in the first phase, and will not be repeated here.

[0258] Continue to refer to Figure 3 In step 306, it is determined whether the training termination condition is met.

[0259] Here, the training termination condition is determined to be met when the initial prediction model completes training in the third training phase. The third training phase is the later stage of training (training rounds after 80%). In the third training phase, the initial prediction model continues to be trained based on the first phoneme sequence samples, the first word sequence samples, the first alignment relationship, the third alignment window, the third phoneme loss weight, and the joint loss function. The window length of the third alignment window is smaller than that of the second alignment window, and the third phoneme loss weight is greater than that of the second phoneme loss weight. The size of the third alignment window can be set to 1–2 tokens, and the third phoneme loss weight can be set to 0.5. A refined alignment strategy is adopted based on the second training phase to improve the alignment accuracy at the phoneme boundary level and the final model performance. The training process of the third training phase is the same as that of the first training phase described above, and will not be repeated here.

[0260] If the training termination condition is not met, proceed to step 307 to continue training the initial prediction model; if the training termination condition is met, proceed to step 308 to obtain the target prediction model.

[0261] In step 307, the initial prediction model is trained again.

[0262] Here, when it is detected that the initial prediction model is in the second training phase and the state evaluation metric of the initial prediction model does not meet the phase switching condition, training of the initial prediction model continues based on the first phoneme sequence samples, the first word sequence samples, the first alignment relationship, the second alignment window, the second phoneme loss weight, and the joint loss function. When it is detected that the initial prediction model is in the second training phase and the state evaluation metric of the initial prediction model meets the phase switching condition, the third training phase is triggered, and training of the initial prediction model continues based on the first phoneme sequence samples, the first word sequence samples, the first alignment relationship, the third alignment window, the third phoneme loss weight, and the joint loss function.

[0263] The speech processing method provided in the embodiments of this application will be described below. See also Figure 10 , Figure 10 This is a flowchart illustrating the speech processing method provided in the embodiments of this application, which will be combined with... Figure 10 The steps shown are explained.

[0264] In step 330, the input feature sequence to be processed is obtained.

[0265] Here, the input feature sequence is the sequence data input to the target prediction model during the model inference or application stage. The input feature sequence includes at least one of the target phoneme sequence and the target word sequence. The target prediction model is trained using the model training method provided in the embodiments of this application.

[0266] The system receives request data packets from upstream modules (such as text front-end or speech recognition front-end), parses the request data packets, and extracts the input feature sequence from the request data packets. When using the target prediction model for text-to-speech prediction tasks, the input feature sequence is the target phoneme sequence generated by text conversion; when using the target prediction model for phoneme-to-word alignment or recognition prediction tasks, the input feature sequence is the target phoneme sequence and target word sequence extracted from the speech dataset.

[0267] In step 331, the target prediction model is invoked to perform forward prediction processing on the input feature sequence to obtain the target prediction result.

[0268] Here, the target prediction model is invoked to perform matrix multiplication and nonlinear activation function calculations on the input feature sequence layer by layer, thereby realizing forward prediction processing of the input feature sequence. Finally, the target prediction result is generated through one or more output heads of the target prediction model.

[0269] In step 332, a task processing result for the prediction task is generated based on the target prediction result, and the task processing result is output.

[0270] Here, the target prediction result output by the target prediction model is transformed to obtain the task processing result for the prediction task. For the speech generation prediction task, when the target prediction result is a Mel spectrogram matrix, an independent vocoder network is invoked to synthesize the spectrogram matrix into one-dimensional continuous audio waveform data, which is then determined as the task processing result for the speech generation prediction task, and the task processing result is output.

[0271] For the alignment prediction task, when the target prediction result is a two-dimensional alignment probability distribution matrix, the maximum a posteriori probability decoding algorithm is called to search for a temporal monotonic path with the highest cumulative probability in the alignment probability distribution matrix. The temporal monotonic path is then transformed into an alignment relationship representing the phoneme-lexical mapping. The alignment relationship, along with the corresponding alignment confidence and timestamp, is then encapsulated in a structured manner to obtain the task processing result of the alignment prediction task, and the task processing result is then output.

[0272] In this embodiment, the target prediction model is invoked to perform forward prediction processing on the input feature sequence to obtain the target prediction result. Based on the target prediction result, task processing results for different prediction tasks are generated. This realizes a multi-stream output data structure design, avoiding the need for downstream applications to perform independent, repetitive, and easily error-inducing alignment calculations to obtain high-precision timing synchronization information. This greatly improves the end-to-end data integration efficiency, timing consistency, and engineering reuse value of the entire multimodal collaborative system (such as virtual lip reading drive and pronunciation evaluation).

[0273] In some embodiments, the model training method provided in this application can be applied to scenarios such as speech generation model training and phoneme prediction decoder training. The method involves obtaining a first phoneme sequence sample, a first word sequence sample, and a first alignment relationship between the first phoneme sequence sample and the first word sequence sample. Then, the initial prediction model is iteratively trained until the training termination condition is met to obtain the target prediction model. In the first training phase, the initial prediction model is trained based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weights, and the joint loss function of the initial prediction model. The joint loss function includes the prediction task loss value and the phoneme loss value adjusted by the phoneme loss weights. This allows for the construction of a relaxed mapping between the phoneme sequence and the word sequence using a larger alignment window in the early stages of model training. A local search space is used to establish coarse-grained temporal correspondences, effectively reducing the temporal alignment constraint strength in the early training stage and preventing excessive interference of early local alignment errors on model parameter updates. Then, the state evaluation index of the initial prediction model is determined. When the state evaluation index meets the stage switching condition, the second training stage is triggered. In the second training stage, the initial prediction model is trained again based on the first phoneme sequence samples, the first word sequence samples, the first alignment relationship, the second alignment window, the second phoneme loss weight, and the joint loss function. The window length of the second alignment window is smaller than that of the first alignment window, and the first phoneme loss weight is smaller than that of the second phoneme loss weight. Thus, by periodically calculating the model's state evaluation index and automatically triggering stage switching when a threshold is met, the alignment window is simultaneously reduced and the phoneme loss weight is increased in the second training stage. This achieves a multi-scale alignment strategy from coarse to fine, effectively improving the accuracy of the alignment relationship between the model's predicted phoneme sequence and the speech word sequence.

[0274] The following will describe an exemplary application of the model training method provided in the embodiments of this application in an audio processing scenario.

[0275] In existing speech generation systems (such as speech synthesis models), phoneme sequences and speech token sequences are typically generated by different tools or models. Phoneme sequences are usually obtained from labeled text and speech data using forced alignment tools, while speech token sequences are extracted from audio signals by speech discretization models. These two types of tools differ significantly in temporal frame settings and modeling granularity. For example, forced alignment tools typically use a frame shift of 10ms, while speech discretization models typically use 20ms. Furthermore, the former models at the phoneme level, while the latter models at the frame or subframe level. Additionally, differences in speech characteristics between different datasets (such as Mandarin / English, clean speech / noisy speech) further lead to inconsistencies between the length of the generated phoneme sequence (Lp) and the length of the speech token sequence (Ls) (Lp ≠ Ls).

[0276] To address the issue of inconsistent lengths between the phoneme sequence and the speech token sequence, the relevant technologies mainly employ the following solutions:

[0277] 1) Simple solution: This involves pruning or padding the sequence to avoid length inconsistencies. For example, longer sequences can be truncated, or shorter sequences can be padded with zeros or fixed values. Some speech synthesis models only use phoneme annotation information during data preparation, ignoring the phoneme sequence during model training or inference to bypass alignment issues. While simple to implement, this method is prone to information loss and fails to fully utilize phoneme-level supervision information.

[0278] 2) Rule-driven approach: This approach uses manually preset ratios (such as Ls / Lp) for index mapping, and resampling of phoneme sequences into speech token sequences through fixed rules. Common methods include nearest neighbor interpolation or linear interpolation. Because this type of method relies on static ratio assumptions, it is difficult to adapt to different speech rates, different languages, or complex noise scenarios, and its alignment accuracy and generalization ability are limited.

[0279] 3) Separate approach: The alignment model and the main model (such as the speech generation model or phoneme prediction model) are designed and trained independently, outputting only a single-scale alignment result, and usually do not provide structured alignment relationships for downstream systems to reuse. This approach lacks end-to-end joint optimization capabilities, makes it difficult to support multi-task collaborative training, and has high system integration and maintenance costs.

[0280] Several solutions have been proposed in related technologies to address the alignment problem between phoneme sequences and speech token sequences, but the following technical problems still exist:

[0281] 1) Severe temporal misalignment between phoneme sequences and speech token sequences: Existing simple proportional mapping or rule-driven alignment methods do not fully consider the temporal continuity characteristics of phoneme sequences. For example, the transition frames between initials and finals are not effectively modeled, resulting in a significant offset in the alignment results on the time axis. The alignment error can exceed 50ms in some scenarios, which directly affects the naturalness and coherence of speech generation.

[0282] 2) Poor adaptability of alignment rules: Alignment rules usually rely on fixed rules or proportional parameters preset by humans, making it difficult to adapt to different speech discretization models and different dataset characteristics (such as clean speech and noisy speech). In cross-model or cross-dataset scenarios, alignment rules often need to be redesigned or optimized, resulting in high reuse costs and low engineering adaptation efficiency.

[0283] 3) Insufficient ability to handle outlier samples: Existing alignment methods mainly handle simple outliers such as empty sequences and fully padded sequences, and are unable to cope with complex scenarios such as noisy speech, non-standard pronunciation, or cross-dataset adaptation anomalies. When such outlier samples are present, alignment failure or label contamination can easily occur, leading to gradient oscillations or instability during model training.

[0284] 4) Does not support multi-task joint training: The alignment model is usually designed separately from the main model (such as the speech generation model or phoneme prediction model). There is a lack of end-to-end joint optimization mechanism between the two, which makes it impossible to improve the alignment accuracy and the performance of the main task at the same time during training, thus limiting the further improvement of the overall system performance.

[0285] 5) Lack of coordination mechanism for multi-scale alignment: Although some schemes introduce multi-scale alignment strategies, they only process different time granularities in parallel and lack hierarchical constraints and coordination calibration mechanisms between scales, making it difficult to ensure global temporal consistency while taking into account the local detail alignment accuracy.

[0286] 6) Poor reusability of alignment results: Existing alignment methods usually only output simple index mapping relationships and do not provide structured alignment information such as confidence and timestamps. This causes downstream multimodal systems (such as lip-sync systems) to still need to perform additional secondary processing when using them, increasing the complexity of system integration.

[0287] 7) Lack of adaptability to unsupervised scenarios: Existing technologies generally rely on alignment data generated by manual annotation or forced alignment tools. In unsupervised scenarios where annotation data is lacking, it is difficult to achieve high-precision and stable phoneme and speech token alignment, which limits its promotion in large-scale, multilingual or cross-dataset applications.

[0288] The model training method and speech processing method provided in this application can specifically solve the problem of inconsistent lengths and difficulty in stable alignment between phoneme sequences and speech token sequences due to differences in temporal resolution, inconsistent modeling granularity, or differences in dataset characteristics in speech generation, phoneme prediction, or multimodal collaborative systems. This achieves the core objectives of cross-scene adaptation, end-to-end optimization, and structured reuse of alignment results. Compared to related technologies, the improvements include the following:

[0289] This application proposes a phoneme-speech token alignment construction mechanism based on "end-to-end joint training + adaptive multi-scale alignment + robust anomaly handling". This mechanism ensures temporal continuity and alignment stability while constructing scalable and transferable phoneme supervision targets and outputting reusable structured alignment relationships, thereby effectively solving technical problems such as low alignment accuracy, poor cross-model and cross-dataset adaptability, and inability to support multi-task joint training.

[0290] The key technical points of this application's embodiments mainly include:

[0291] 1) Unified time-axis driven adaptive resampling mechanism: Using the speech token sequence (supporting multi-scale hierarchical representation) as the target time axis, the phoneme sequence is automatically resampled through dynamic scaling to align the phoneme sequence length with the speech token sequence length, maintaining temporal continuity and semantic consistency of the phonemes during alignment. Resampling is achieved using rule-driven nearest neighbor interpolation, and an end-to-end alignment prediction model can be introduced to adaptively adjust the resampling process based on contextual information.

[0292] 2) Design of end-to-end joint training alignment prediction model: The speech modeling task and the alignment prediction task are jointly trained in an end-to-end manner. The alignment prediction model takes phoneme embedding, speech token embedding and cross-modal context features as input and directly predicts the correspondence between phonemes and speech tokens, thereby avoiding the failure problem of traditional static alignment rules in complex speech scenarios.

[0293] 3) Adaptive multi-scale hierarchical alignment mechanism: In order to balance the alignment accuracy and stability at different time granularities, a multi-scale hierarchical alignment strategy is introduced. The alignment relationship between phonemes and speech tokens is modeled at multiple time scales of coarse granularity, medium granularity and fine granularity. During the training phase, the alignment results of different scales are adaptively selected or fused according to the model convergence state and task requirements to improve the overall alignment robustness.

[0294] 4) Multi-scenario robust anomaly handling framework: To address common issues in practical applications such as empty sequence samples, noisy speech, non-standard pronunciation speech, and abnormal samples introduced by cross-dataset migration, a unified robust anomaly handling mechanism is constructed to automatically identify and filter invalid or low-confidence samples, avoiding interference from abnormal alignment relationships during model training and inference.

[0295] 5) Unsupervised adaptive calibration capability: Supports the automatic construction and optimization of alignment relationships in scenarios where there is no manually labeled phoneme-speech token alignment data, thereby adapting to different speech discretization models and different dataset distributions, significantly improving the system's generalization ability under cross-model and cross-scenario conditions.

[0296] 6) Structured alignment output and downstream reuse: The phoneme-speech token alignment relationship is output in a structured form of "index + confidence + timestamp". This alignment result can be used as an intermediate result within the model to participate in subsequent training or inference processes, or it can be directly called by downstream systems, including but not limited to lip-syncing, speech quality assessment and multimodal collaborative tasks, thereby improving the scalability and engineering reuse value of the overall system.

[0297] This application provides an adaptive multi-scale robust alignment system for automatic alignment of phonemes and speech tokens. This system can be integrated as a core functional module and applied to various voice-related products and system scenarios, including but not limited to the following aspects:

[0298] 1) Speech Generation Model Training System: The embodiments of this application can serve as target building modules in a speech generation model training system, used to adapt to multi-task joint training scenarios of text-to-speech models or speech synthesis models, such as achieving synchronous optimization of phoneme prediction tasks and speech generation tasks. This system supports both single-machine and distributed deployment methods and is suitable for large-scale speech model training environments.

[0299] 2) Phoneme Prediction Decoder Training System: The embodiments of this application can be used as an alignment module in a phoneme prediction decoder training system to construct a stable and robust phoneme supervision target. Especially under complex speech conditions such as dialect speech or noisy speech, it can effectively improve the accuracy of phoneme recognition and prediction.

[0300] 3) Multimodal cooperative system: Embodiments of this application can also be integrated into multimodal cooperative systems to provide high-precision alignment relationships for synchronization between speech, vision, or other modalities, specifically including:

[0301] I. Virtual Human Real-Time Voice-Driven System: This system can output a high-precision alignment relationship containing timestamp information, with an alignment accuracy of millimeter-level time resolution, thereby realizing real-time synchronous linkage between speech and lip movements;

[0302] II. Intelligent speech evaluation application system (e.g., English pronunciation evaluation system): By outputting fine-grained phoneme and speech token alignment results, it can accurately locate the specific time interval corresponding to the pronunciation error, which can be used to support refined pronunciation evaluation and feedback.

[0303] 4) Multilingual speech generation platform: In multilingual speech generation application scenarios, this application embodiment introduces an unsupervised adaptive calibration mechanism, which can quickly adapt to multilingual or cross-lingual speech generation scenarios such as English, Japanese and various dialects in the absence of manually labeled alignment data, and meet the deployment needs of globalized speech products.

[0304] 5) Speech Data Processing and Annotation Tools: Embodiments of this application can also be applied to the automatic alignment module of a speech data processing tool or speech annotation system, providing automated alignment services for tasks such as speech data annotation and dataset preprocessing, thereby replacing traditional manual alignment methods and improving overall processing efficiency by over 80%. Furthermore, the automatic alignment system provided in these embodiments can be seamlessly integrated with existing speech generation models and supports operation in Windows or Linux server environments and cloud-native deployment architectures, possessing excellent engineering adaptability and scalability.

[0305] The following describes the basic automatic resampling and alignment process provided in the embodiments of this application. This process uses the speech token sequence as the target time axis and achieves automatic resampling and alignment of the phoneme sequence through a dynamic scaling mechanism. For example, refer to... Figure 11 , Figure 11 This is a schematic diagram of the resampling and alignment process provided in an embodiment of this application. Combined with... Figure 11 The steps described herein will be explained to illustrate the basic automatic resampling and alignment process provided in the embodiments of this application.

[0306] In step 1101, the phoneme sequence is received and preprocessed.

[0307] Here, the phoneme sequence phone_id_seq output by the phoneme alignment tool is received, and its sequence length is marked as... Before entering the alignment process, the phoneme sequence is filtered for validity by the preprocessing module, removing non-phoneme identifiers (such as silence / pause: sil, sp, etc.). <sil>, pause, etc.; noise / breath: noise, breath, cough, etc.; boundary / structure: space, #, |, etc.) or invalid phoneme identifiers (phoneme identifiers are outside the model range) to obtain a normalized phoneme sequence.

[0308] In step 1102, the voice token sequence is received and the time resolution is marked.

[0309] Here, we receive the speech token sequence `speech_token_seq` output by the speech discretization model, whose sequence length is marked as... Simultaneously, the temporal resolution type of the speech token sequence is labeled to distinguish token representations at different granularity levels, including but not limited to coarse-grained, medium-grained, or fine-grained temporal resolutions.

[0310] In step 1103, a dynamic scaling mapping function is constructed.

[0311] Here, to improve the temporal consistency between phonemes and speech tokens, a dynamic proportional mapping function is constructed, specifically including: 1) identifying key temporal positions such as initial-final transition frames and syllable boundaries in the phoneme sequence through the articulation stage boundary detection module; 2) adaptively calculating the offset coefficient based on the identified key temporal positions. ,in [0, 0.2] is used to perform fine-grained time-series correction on the scaling results.

[0312] In step 1104, the index mapping relationship is calculated.

[0313] Here, based on the dynamic scaling mapping function, a correspondence is established between the phoneme sequence index and the speech token sequence index. The source phoneme index can be determined using the following formula (1). (satisfies 0≤) < ) and target voice token index (satisfies 0≤) < Mapping relationship between )

[0314] (1)

[0315] in, Indicates the target voice token index. Indicates the source phoneme index. Indicates the length of the phoneme sequence. Indicates the length of the voice token index. This represents the offset coefficient.

[0316] In step 1105, resampling is performed based on the mapping relationship.

[0317] Here, based on the above index mapping relationship, the corresponding phoneme identifier is selected from the phoneme sequence, and the phoneme sequence is resampled to generate a sequence of length [length missing]. The initial alignment of the phoneme sequence is made to keep it consistent with the speech token sequence on the timeline.

[0318] In step 1106, invalid location filling and supervision target construction are performed.

[0319] Here, for the portion of the mapping process that exceeds the phoneme sequence length... Target Index The predefined ignore flag IGNORE_ID is filled in at the corresponding position to construct the final phoneme supervision target sequence for subsequent model training or loss calculation stages.

[0320] Example, reference Figure 12 , Figure 12 This is a schematic diagram of the mapping relationship provided in the embodiments of this application. The phoneme identifier Phone4 in the phoneme sequence 1202 is mapped to the 9th slot on the voice token sequence 1201. At this time, the target index obtained by mapping exceeds the length of the phoneme sequence, and the ignore flag (IGNORE_ID) is filled at the index overflow position 1203 to indicate that it does not participate in the loss calculation.

[0321] In a preferred embodiment, the resampling process employs a nearest neighbor interpolation strategy, combined with a "mapping relationship + nearest index correction" mechanism. For example, when the calculated target index... When the length exceeds the phoneme sequence length range (i.e., the length of the speech token sequence is greater than the phoneme sequence length), use... The corresponding phoneme identifier is used as a substitute to avoid timing misalignment caused by boundary overflow. The initial alignment relationship between the phoneme sequence samples and the speech token sequence samples can be determined through the above-described basic automatic resampling and alignment process.

[0322] The following describes the multi-scenario anomaly detection and dynamic filling mechanism provided in the embodiments of this application. To improve the robustness of the phoneme sequence and speech token sequence alignment process in complex speech scenarios, a multi-scenario anomaly detection and dynamic filling mechanism is further introduced. This mechanism is used to uniformly process complex anomaly samples such as empty sequences, noisy speech, non-standard pronunciation, and cross-dataset adaptation anomalies, thereby avoiding the adverse effects of anomaly samples on the model training stability. For example, refer to... Figure 13 , Figure 13 This is a flowchart illustrating the anomaly detection and dynamic filling process provided in this application embodiment. The flowchart includes an input feature module 1301, a multi-scene anomaly detection module 1302, a local anomaly processing module 1303, a global anomaly processing module 1304, a phoneme supervision target construction module 1305, and a loss calculation and training stability assurance module 1306.

[0323] The input feature module 1301 includes phoneme sequences, speech token sequences, alignment prediction confidence, speech energy information, etc. The multi-scene anomaly detection module 1302 is used to perform empty sequence / fully filled sequence detection, noisy speech detection, non-standard pronunciation detection, cross-dataset adaptation anomaly detection, etc.

[0324] In the detection of empty / full-filled sequences, when the length of the detected phoneme sequence is 0, or the proportion of valid phoneme identifiers (identifiers representing the actual pronunciation content, usually excluding padding, IGNORE_ID, undefined identifiers, etc.) in the phoneme sequence is less than 10%, the phoneme sequence is determined to be an empty or full-filled sequence abnormality.

[0325] In noisy speech detection, energy statistical analysis is performed on the speech token sequence to calculate the energy variance of the speech token sequence (which is the variance statistic of the energy distribution of the continuous acoustic representation or speech signal corresponding to the speech token sequence in the time dimension). When the energy variance is greater than a preset threshold (e.g., 0.8), the speech token sequence is determined to be noisy speech abnormality.

[0326] In the detection of non-standard pronunciation, the phoneme sequence is compared with the corresponding standard phoneme template, and the edit distance between the two is calculated. When the edit distance is greater than 5 (for single-sentence speech scenarios), the phoneme sequence is judged to be non-standard pronunciation.

[0327] In cross-dataset adaptation anomaly detection, the mean alignment confidence score is calculated based on the alignment confidence score output by the alignment prediction model. When the mean alignment confidence score is less than 0.6, the sample is determined to be a cross-dataset adaptation anomaly.

[0328] After determining the anomaly type during the anomaly detection phase, a differentiated dynamic filling strategy is adopted based on the determination type and the scope of the anomaly's impact. The local anomaly processing module 1303, when an anomaly is detected occurring only within a portion of a single sentence's speech time interval, fills only the position corresponding to the anomaly interval with the ignore flag IGNORE_ID, while retaining the original phoneme alignment results for the remaining valid intervals, thereby maximizing the utilization of effective supervisory information. The global anomaly processing module 1304, when detecting a sample belonging to a global anomaly situation (including but not limited to whole-sequence noisy speech or empty phoneme sequences), generates a length of... The complete IGNORE_ID phoneme sequence is used as the final phoneme supervision target for this sample. The loss calculation and training stability assurance module 1306 is used during model training to calculate the effective phoneme positions normally when calculating phoneme-related losses, so as to update the gradient normally. The IGNORE_ID marker positions are automatically masked to prevent gradient backpropagation, ensuring they do not participate in the backpropagation of phoneme loss and only retaining the forward calculation process, thereby outputting a model update gradient that ensures stability. Before entering the training phase, the above-mentioned multi-scene anomaly detection and dynamic filling mechanism can be used to handle anomaly phoneme sequence samples or speech token sequence samples.

[0329] The alignment relationship generation mechanism based on model prediction provided in this application will be described below. To reduce the alignment error introduced by fixed-ratio or rule-based resampling methods in complex speech scenarios, a model prediction-based alignment relationship generation mechanism is further introduced. The alignment relationship is generated by the trained alignment prediction model, rather than relying entirely on preset rules. For example, refer to... Figure 14 , Figure 14 This is a schematic diagram of the structure generated based on model prediction of alignment relationships provided in the embodiments of this application.

[0330] The alignment prediction model 1404 takes phoneme sequence encoding representation 1401, speech token sequence encoding representation 1402, and their contextual features 1403 as input. The contextual features 1403 characterize the temporal dependencies and cross-modal associations between adjacent time steps (phonemes and speech tokens are encoded separately, fused by concatenation, and then modeled using a sequence model to obtain a hidden state representation containing cross-modal dependencies). The alignment prediction model 1404 models the correspondence between phonemes and speech tokens using a neural network structure. Its output can be an alignment probability distribution matrix 1405 from phonemes to speech tokens, or an index mapping or time interval mapping from phonemes to speech tokens. The Viterbi algorithm is then used to extract the optimal index path from the alignment probability distribution matrix 1405 to generate an initial alignment relationship. Subsequently, based on alignment confidence screening, if the alignment confidence of the corresponding position in the initial alignment relationship is greater than or equal to 0.5, the alignment result is retained, thus obtaining the final alignment relationship. Based on the aforementioned final alignment relationship, a phoneme-supervised target sequence is constructed and applied to the joint training process in phoneme prediction or speech generation tasks. The model-predictive alignment relationship generation mechanism described above can be used to perform preliminary alignment prediction on phoneme sequence samples and speech token sequence samples, thus obtaining the predicted alignment relationship.

[0331] The alignment prediction model 1404 and the main model 1406 (including the speech generation model or phoneme prediction model) are optimized through end-to-end joint training. The joint loss function consists of the alignment loss and the main task loss. The alignment loss uses cross-entropy loss to optimize the alignment accuracy of the alignment prediction model. The main task loss uses mean squared error loss (MSE loss) to optimize the performance of the main task of speech generation or phoneme prediction. The weight coefficients of the alignment loss and the main task loss are dynamically adjusted during the training process to balance alignment accuracy and main task performance at different training stages. The alignment prediction model can also be pre-trained as an independent module, and its parameters can be fixed after training for use in the subsequent main model training or inference stage for generating alignment relationships.

[0332] The following describes the multi-scale hierarchical alignment and bidirectional calibration mechanism provided in the embodiments of this application. To further improve the alignment accuracy and robustness of phoneme sequences and speech token sequences at different temporal resolutions, a multi-scale alignment mechanism supporting three levels of alignment—coarse-grained, medium-grained, and fine-grained—is provided. A bidirectional calibration strategy is introduced to jointly optimize the alignment results at each scale, balancing global temporal consistency with local boundary accuracy. For example, refer to... Figure 15 , Figure 15 This is a schematic diagram of the multi-scale hierarchical alignment and bidirectional calibration mechanism provided in the embodiments of this application.

[0333] Based on the requirements of speech modeling, the speech token sequence is divided into three scales according to temporal resolution: coarse-grained tokens, medium-grained tokens, and fine-grained tokens. Coarse-grained tokens have a temporal resolution of 50ms / token and are used to represent articulation stage-level information, with one token corresponding to a complete articulation stage unit, such as "initial consonant + final vowel" as a complete articulation stage. Medium-grained tokens have a temporal resolution of 20ms / token and are used to represent syllable-level information, with one token corresponding to a syllable unit, such as "ba" as a syllable unit. Fine-grained tokens have a temporal resolution of 10ms / token and are used to represent frame-level speech information, corresponding to the original speech feature frames, and can characterize fine-grained phoneme boundary changes.

[0334] The alignment of phonemes and speech tokens is performed in a coarse-to-fine hierarchical manner, specifically including three stages: coarse-grained alignment, medium-grained alignment, and fine-grained alignment. In the coarse-grained alignment stage, at the coarse-grained token scale, a loose alignment window of 5 tokens is used for initial alignment to obtain the overall temporal relationship at the pronunciation stage level, providing global constraints for subsequent fine alignment. In the medium-grained alignment stage, at the medium-grained token scale, syllables are used as the basic alignment unit, with an alignment window size of 3–5 tokens, improving local syllable-level alignment accuracy while maintaining global temporal consistency. In the fine-grained alignment stage, at the fine-grained token scale, a refined alignment strategy is adopted, reducing the alignment window size to 1–2 tokens to accurately obtain phoneme boundaries and frame-level temporal details.

[0335] In the aforementioned multi-scale alignment modeling, the alignment window essentially represents a restricted search interval on the speech token axis. That is, under monotonic alignment constraints (incrementing phoneme indices correspond to non-decreasing speech token indices), the current processing unit is aligned within a continuous interval of speech token indices. The entire process employs a coarse-to-fine multi-scale strategy: In the coarse-grained alignment stage, the entire sentence or large segment of phonemes is divided into blocks using a larger alignment window (approximately 5 tokens). The approximate position of each coarse unit on the token axis is determined through uniform stretching or initial anchor points, and restricted alignment is performed within this alignment window, thereby establishing a global temporal correspondence. In the medium-grained alignment stage, the units are refined to the syllable level. Building upon the global constraint band inherited from the coarse-grained alignment stage, the alignment is further reduced to a local alignment window of 3–5 tokens. Dynamic programming or local attention mechanisms are used to optimize the alignment position of syllables to speech tokens within this alignment window. In the fine-grained alignment stage, a tiny alignment window of only 1-2 tokens is retained near the reference position from the medium-grained alignment stage. This window finely adjusts the phoneme boundaries, determining the specific distribution of start and end tokens or across tokens, and can be further refined to the time interval by incorporating frame-level acoustic features. Overall, by progressively shrinking the alignment window, global coarse alignment and local fine alignment are achieved, ensuring both monotonicity and computational efficiency while achieving stable and high-precision phoneme-to-speech token alignment.

[0336] To avoid cumulative errors from alignment results at a single scale, this application further introduces a bidirectional calibration mechanism to perform cross-constraints and dynamic corrections on alignment results between different scales. A phoneme consistency index is calculated based on the fine-grained alignment results. When a significantly inconsistent mapping relationship is detected in the coarse-grained alignment results, consistency correction (fine-grained → coarse-grained calibration) is performed on the coarse-grained alignment results, such as deleting or adjusting inconsistent coarse-grained mappings, to improve the overall accuracy of coarse-grained alignment.

[0337] Using the pronunciation stage boundaries determined by coarse-grained alignment as constraints, the fine-grained alignment process is subjected to stage constraints (coarse-grained → fine-grained calibration) to avoid temporal drift or cross-stage misalignment in local areas, thereby improving the stability of fine-grained alignment.

[0338] Alignment results at different scales are applied to different downstream task scenarios. Coarse-grained alignment results are used to impose global temporal constraints on the main model, improving the overall temporal consistency of speech generation or phoneme prediction. Medium-grained alignment results are used for multimodal collaborative tasks, such as lip-sync control in virtual human speech-driven systems. Fine-grained alignment results are used for high-precision applications, such as pronunciation quality assessment and phoneme-level error localization. The above multi-scale hierarchical alignment and bidirectional calibration mechanism can be used to determine the annotation alignment relationship (the first alignment relationship in the above embodiments) between phoneme sequence samples and speech token sequence samples.

[0339] The following describes the dynamic adjustment mechanism for alignment strategies based on a three-dimensional state evaluation index provided in the embodiments of this application. To achieve adaptive optimization of the phoneme and speech token alignment process at different training stages, a dynamic adjustment mechanism for alignment strategies based on a three-dimensional state evaluation index is proposed. This mechanism periodically evaluates the model training state and automatically adjusts the alignment constraint strength and loss weight accordingly, thereby gradually improving alignment accuracy while ensuring training stability. For example, refer to... Figure 16 , Figure 16 This is a schematic diagram of the alignment strategy dynamic adjustment mechanism based on three-dimensional state evaluation index provided in the embodiments of this application.

[0340] During model training, the model training status is evaluated at fixed intervals. A three-dimensional state evaluation index is calculated every 10 training rounds. The three-dimensional state evaluation index includes the alignment error evaluation value, the loss convergence evaluation value, and the phoneme prediction accuracy index. For example, the alignment error evaluation value can be determined using the following formula (3):

[0341] (3)

[0342] When the alignment error evaluation value meets the target threshold At 0.3, it is determined that the alignment quality of the current stage has reached the expected level. For example, the loss convergence evaluation value can be determined by the following formula (4):

[0343] (4)

[0344] When the loss convergence evaluation value meets the target threshold At a value of 0.2, it is determined that the model training has entered a stable convergence phase. The phoneme prediction accuracy metric, also known as the phoneme prediction accuracy rate, is used as the evaluation indicator. When the phoneme prediction accuracy rate (the proportion of predicted phoneme labels that match the annotations at effective supervised locations) meets the target threshold... At 90%, it is determined that the phoneme prediction task has reached a usable level of accuracy.

[0345] Based on the aforementioned three-dimensional state evaluation metrics, the training process is divided into early training, mid-training, and late training phases (each phase corresponds to a set of optimization strategies (window + weight), with the same set of evaluation metrics used throughout the observation). Different alignment window sizes and phoneme loss weight configurations are adopted at different phases. In the early training phase (first 20% of epochs), the alignment window size is set to 5 tokens, and the phoneme loss weight is set to 0.2, employing a relaxed alignment strategy to reduce the alignment constraint strength and avoid early alignment errors interfering with the overall model training. In the mid-training phase (20%–80% of epochs), the alignment window size is adjusted to 3 tokens, and the phoneme loss weight is increased to 0.4, gradually strengthening the alignment constraint to improve the alignment accuracy between phonemes and speech tokens while maintaining stable training. In the late training phase (after 80% of epochs), the alignment window size is further reduced to 1–2 tokens, and the phoneme loss weight is increased to 0.5, employing a refined alignment strategy to improve the alignment accuracy at the phoneme boundary level and the final model performance.

[0346] In this multi-scale alignment and joint training framework, the alignment window and phoneme loss weights are coordinated and scheduled according to the training progress to achieve a coarse-to-fine alignment convergence process. Specifically, during training, the alignment window and phoneme loss weights are adjusted according to the current training progress ratio. (e.g., the current training progress represents a proportion of the total training progress) Dynamically determine the alignment window size. Phoneme loss weights. In the early stages of training (p≤0.2), a wider alignment window (e.g., 5 tokens) is used, and smaller phoneme loss weights are set (e.g., ...). (= 0.2), the model is made to learn the coarse-grained alignment relationship within a large search space while avoiding unstable training caused by overly strong alignment supervision. In the middle stage of training (0.2 < p ≤ 0.8), the alignment window is reduced to 3 Tokens, and at the same time, the phoneme loss weight is increased (such as 0.4) to gradually strengthen the local consistency between phonemes and speech Tokens; in the late stage of training (p > 0.8), the alignment window is further shrunk to 2 Tokens, and the phoneme loss weight is increased to a relatively high level (such as 0.5), making the model focus on the fine alignment of phoneme-level boundaries. The alignment window takes effect by restricting the alignment search space or constructing an attention / loss mask, that is, in the forward calculation, only candidate Tokens falling within the interval centered on the reference anchor point and with a width of are allowed to participate in the alignment modeling or loss calculation. The phoneme loss weight directly acts on the joint loss function and dynamically adjusts the contribution of phoneme supervision to the overall gradient at different training stages. The two work together to enable the model to have a strong global alignment fault tolerance ability in the early stage of training and gradually converge to a fine-grained alignment result with high precision and low drift in the later stage. For example, the joint loss function is determined by the following formula (2):

[0347] (2)

[0348] where, represents the joint loss function, represents the prediction task loss, represents the phoneme loss, represents the phoneme loss weight.

[0349] In the stage switching trigger mechanism 1601, the switching between each training stage does not depend on manual setting but is realized through an automatic trigger mechanism. When the alignment error evaluation value, the loss convergence evaluation value, and the phoneme prediction accuracy index simultaneously meet the corresponding target threshold conditions, it is automatically triggered to enter the next training stage, and the alignment window size and the phoneme loss weight configuration are synchronously adjusted to achieve the adaptive dynamic adjustment of the alignment strategy without manual intervention. Through the above alignment strategy dynamic adjustment mechanism based on three-dimensional state evaluation indicators, in each training stage, the main model (the initial prediction model in the above embodiment) can be trained in combination with the corresponding alignment window, phoneme sequence samples, and speech Token sequence samples.

[0350] In the inference stage of the embodiment of the present application, in addition to outputting the speech generation result, the alignment relationship between phonemes and speech Tokens is also output as an explicit model output, so that the alignment information can be directly reused by the downstream system without additional post-processing or secondary alignment operations. For example, referring to Figure 17 , Figure 17 This is a schematic diagram of the model output results provided in the embodiments of this application. The model inference output includes speech output 1701, phoneme sequence output 1702, and alignment relationship output 1703. The speech output 1701, phoneme sequence output 1702, and alignment relationship output 1703 are synchronous in time. The alignment relationship output 1703 can be applied to downstream application scenarios such as lip-syncing / pronunciation evaluation.

[0351] The unsupervised adaptive calibration mechanism provided in the embodiments of this application will be described below. In application scenarios where there is no manually labeled phoneme-speech token alignment data, to achieve adaptive alignment capabilities across datasets and speech discretization models, an unsupervised adaptive calibration mechanism is further proposed. Through self-supervised learning and iterative optimization, a stable and reliable alignment relationship is constructed in the absence of explicit alignment labels. For example, refer to... Figure 18 , Figure 18 This is a schematic diagram of the unsupervised adaptive calibration mechanism provided in an embodiment of this application. The diagram includes an input sequence module 1801, a self-supervised contrastive learning module 1802, an initial alignment generation module 1803, an iterative alignment optimization module 1804, and a cross-model adaptive module 1805. The input sequence module 1801 includes a phoneme sequence and a speech token sequence.

[0352] The self-supervised contrastive learning module 1802 is used in the self-supervised pre-training stage to construct a self-supervised contrastive learning task based on phoneme sequences and speech token sequences. This task learns the temporal correlation between the two sequences. Training samples are constructed by temporally matching the phoneme sequences and speech token sequences. Positive sample pairs are phoneme sequences and speech token sequences from the same speech sample, while negative sample pairs are random pairings of phoneme sequences and speech token sequences from different speech samples. A temporal correlation model is used to encode the phoneme sequences and speech token sequences separately, and the similarity between them is output. The similarity value ranges from [0, 1], representing the degree of temporal matching between the phonemes and speech tokens.

[0353] After completing self-supervised pre-training, the initial phoneme-to-speech token mapping relationship is constructed using the similarity output of the temporal correlation model. The initial alignment generation module 1803 is used to filter potential mapping relationships based on similarity, and phoneme-to-token pairs with a similarity greater than or equal to 0.7 are identified as valid mapping relationships. The initial alignment results are generated based on the valid mapping relationships for subsequent model optimization.

[0354] After obtaining the initial alignment results, the process enters an iterative optimization phase, continuously improving alignment accuracy through model prediction. The iterative alignment optimization module 1804 trains a lightweight alignment prediction model based on the initial alignment results. This model employs a two-layer Transformer network structure to further model the correspondence between phonemes and speech tokens. The prediction results output by the alignment prediction model are used to update the phoneme-speech token mapping, and the prediction results are filtered by confidence, retaining mappings with a confidence level greater than or equal to 0.5. This process of model training and iterative mapping updates is repeated 3–5 times until the average alignment confidence level reaches or exceeds 0.8, thus obtaining a stable and reliable final alignment.

[0355] The cross-model adaptive module 1805 is used for automatic adaptation under different speech discretization models or different dataset scenarios. When adapting to different speech discretization models, the system automatically adjusts the temperature parameter in the contrastive learning task. For example, when using the first speech discretization model (HuBERT) for speech token extraction, the temperature parameter is set to 0.1; when using the second speech discretization model (w2v-BERT) for speech token extraction, the temperature parameter is set to 0.05. In this way, a stable similarity distribution and alignment quality can be maintained under different model structures and data distribution conditions. The above unsupervised adaptive calibration mechanism can be used to determine the annotation alignment relationship between phoneme sequence samples and speech token sequence samples (the first alignment relationship in the above embodiment), thereby using the phoneme sequence samples, speech token sequence samples, and annotation alignment relationship to train the main model (the initial prediction model in the above embodiment).

[0356] In this embodiment, stable joint training of phoneme prediction and speech generation is supported, and robust processing capability is provided for abnormal samples. The alignment mechanism can be extended to model prediction, multi-scale or adaptive strategies. The alignment relationship can be used as a structured intermediate representation for reuse by downstream systems. It supports coarse / medium / fine multi-scale alignment and is adapted to multiple downstream scenarios such as speech generation, lip-syncing, and pronunciation evaluation. It solves the problem of inconsistent lengths between phoneme sequences and speech token sequences.

[0357] The following description continues to illustrate the exemplary structure of the model training device 233 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the model training device 233 of the memory 230 may include: a sample acquisition module 2331, used to acquire a first phoneme sequence sample, a first word sequence sample, and a first alignment relationship between the first phoneme sequence sample and the first word sequence sample; and a first training module 2332, used to perform the following iterative training on the initial prediction model until the training termination condition is met to obtain the target prediction model: in the first training stage, the initial prediction model is trained based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weight, and the joint loss function of the initial prediction model. The joint loss function includes the prediction task loss value and the phoneme loss value adjusted by the phoneme loss weight; the index determination module 2333 is used to determine the state evaluation index of the initial prediction model; the second training module 2334 is used to trigger the entry into the second training phase when the state evaluation index meets the phase switching condition; in the second training phase, the initial prediction model is trained again based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the second alignment window, the second phoneme loss weight and the joint loss function, the window length of the second alignment window is less than the window length of the first alignment window, and the first phoneme loss weight is less than the second phoneme loss weight.

[0358] In some embodiments, the sample acquisition module 2331 is further configured to determine the initial alignment relationship between the first phoneme sequence sample and the first word sequence sample; call the trained alignment prediction model to perform alignment prediction on the first phoneme sequence sample and the first word sequence sample to obtain the predicted alignment relationship; determine the first confidence level of the predicted alignment relationship; when the first confidence level is greater than or equal to the first confidence threshold, update the initial alignment relationship based on the predicted alignment relationship to obtain the first alignment relationship.

[0359] In some embodiments, the sample acquisition module 2331 is further configured to, for each phoneme in the first phoneme sequence sample, determine the first word interval corresponding to the phoneme in the first word interval sample based on the initial alignment relationship; determine the index position of the target word corresponding to the phoneme in the initial alignment relationship in the first word interval sample as the first target position; construct a target probability distribution in the first word interval where the probability of the first target position is greater than the probability of other positions; input the first phoneme sequence sample and the first word interval sample into the alignment prediction model to be trained, and output the predicted probability distribution of the phoneme in the first word interval; determine the alignment loss value based on the predicted probability distribution and the target probability distribution, and update the model parameters of the alignment prediction model based on the alignment loss value to obtain the trained alignment prediction model.

[0360] In some embodiments, the sample acquisition module 2331 is further configured to acquire the target word character mapped by the phoneme from the initial alignment relationship; and according to the preset window length, extract a series of consecutive word characters centered on the target word character from the first word character sequence sample to obtain the first word character interval corresponding to the phoneme.

[0361] In some embodiments, the sample acquisition module 2331 is further configured to set the probability value corresponding to the first target position within the first word interval as a first probability value; set the probability values ​​corresponding to other positions within the first word interval besides the first target position as second probability values, wherein the first probability value is greater than the second probability value; and form a target probability distribution corresponding to a phoneme by the first probability value and the second probability value within the first word interval.

[0362] In some embodiments, the sample acquisition module 2331 is further configured to: determine an initial loss value based on the predicted probability distribution and the target probability distribution using a preset alignment loss function; acquire the center positions of the first word intervals corresponding to two adjacent phonemes in the first phoneme sequence sample; when the index of the center positions of two adjacent phonemes in the first word sequence sample does not satisfy the monotonically non-decreasing condition, determine an auxiliary loss value that violates the monotonically non-decreasing condition based on a preset penalty function; and fuse the auxiliary loss value and the initial loss value to obtain the alignment loss value.

[0363] In some embodiments, the sample acquisition module 2331 is further configured to construct a positive sample pair from a first phoneme sequence sample and a first word sequence sample, wherein the first phoneme sequence sample and the first word sequence sample are derived from a first speech sample; acquire a second phoneme sequence sample and construct a negative sample pair from the second phoneme sequence sample and the first word sequence sample, wherein the second phoneme sequence sample is derived from a second speech sample, and the first speech sample and the second speech sample are different; perform temporal matching degree prediction on the positive sample pair and the negative sample pair respectively to obtain a first temporal matching degree of the positive sample pair and a second temporal matching degree of the negative sample pair; based on the first temporal matching degree and the second temporal matching degree, determine a target sample pair that meets a preset matching degree condition from the positive sample pair and the negative sample pair, wherein the target sample pair includes a target phoneme sequence sample and a target word sequence sample; and construct an alignment relationship between the target phoneme sequence sample and the target word sequence sample to obtain an initial alignment relationship.

[0364] In some embodiments, the sample acquisition module 2331 is further configured to acquire the original phoneme sequence corresponding to the first speech sample, and preprocess the original phoneme sequence to obtain a processed sequence sample, the length of which is a first length; acquire the first word sequence sample corresponding to the first speech sample, the length of which is a second length; construct a target mapping function based on the first length and the second length, and resample the processed sequence sample based on the target mapping function to obtain a first phoneme sequence sample, the length of which is a second length; and construct an alignment relationship between the first phoneme sequence sample and the first word sequence sample to obtain an initial alignment relationship.

[0365] In some embodiments, the sample acquisition module 2331 is further configured to perform articulation stage boundary detection on the processed sequence sample to obtain a second target position in the processed sequence sample, the second target position including at least one of the articulation transition frame position and the syllable boundary position; determine a temporal offset coefficient based on the second target position, and determine the length ratio between the second length and the first length; and construct a target mapping function based on the length ratio and the temporal offset coefficient.

[0366] In some embodiments, the sample acquisition module 2331 is further configured to determine the target word index corresponding to each source phoneme index in the processed sequence sample based on the target mapping function; when the target word index exceeds the length range of the processed sequence sample, the phoneme identifier corresponding to the last phoneme in the processed sequence sample is used as the phoneme identifier corresponding to the target word index; and the positions corresponding to the word indexes that do not match the source phonemes are filled with a preset ignore identifier to obtain the first phoneme sequence sample.

[0367] In some embodiments, the sample acquisition module 2331 is further configured to scale the source phoneme index based on the length ratio to obtain a scaled source phoneme index; superimpose the temporal offset coefficient and the scaled source phoneme index to obtain a superposition result; and determine the target word index corresponding to the source phoneme index based on the superposition result.

[0368] In some embodiments, the first word sequence sample is labeled with a temporal resolution. The temporal resolution type includes at least a first resolution, a second resolution, and a third resolution, which are divided from low to high temporal resolution. The sample acquisition module 2331 is further configured to perform initial alignment using a first-level search window based on the first resolution to obtain the global temporal constraint relationship between the first phoneme sequence sample and the first word sequence sample at the pronunciation stage level; perform syllable-level alignment using a second-level search window within the range defined by the global temporal constraint relationship based on the second resolution to obtain a first alignment result; perform phoneme-level alignment using a third-level search window within the range defined by the first alignment result based on the third resolution to obtain a second alignment result; and determine the first alignment relationship between the first phoneme sequence sample and the first word sequence sample based on the second alignment result; wherein the window length of the first-level search window, the window length of the second-level search window, and the window length of the third-level search window decrease progressively.

[0369] In some embodiments, the sample acquisition module 2331 is further configured to extract the first acoustic features of the first phoneme sequence sample and the first text features corresponding to the first word sequence sample; under the limitation of the first-level search window, determine the first similarity matrix between the first acoustic features and the first text features; perform optimal path search based on the first similarity matrix, determine the start boundary and the end boundary between the pronunciation segment and the word text block, and use the start boundary and the end boundary as a global temporal constraint relationship.

[0370] In some embodiments, the sample acquisition module 2331 is further configured to extract the second acoustic features of the first phoneme sequence sample and acquire the syllable-level lexical features of the first lexical sequence sample; map the global temporal constraint relationship to an alignment search mask; determine the attention alignment probability between the second acoustic features and the syllable-level lexical features under the joint constraints of the alignment search mask and the second-level search window; and determine the first alignment result representing the syllable-level matching path based on the attention alignment probability.

[0371] In some embodiments, the sample acquisition module 2331 is further configured to extract frame-level acoustic features of the first phoneme sequence sample and obtain phoneme-level lexical features of the first lexical sequence sample; generate a mask matrix based on the matching path indicated by the first alignment result and the window length of the third-level search window; and use the mask matrix to determine the alignment frame boundary between the frame-level acoustic features and the phoneme-level lexical features to obtain a second alignment result representing the frame-level matching correspondence.

[0372] In some embodiments, the sample acquisition module 2331 is further configured to extract the start acoustic frame index and end acoustic frame index corresponding to each word in the first word sequence sample from the second alignment result; determine the pronunciation duration corresponding to each word based on the start acoustic frame index and end acoustic frame index; and determine the mapping sequence composed of word and corresponding pronunciation duration as the first alignment relationship.

[0373] In some embodiments, the sample acquisition module 2331 is further configured to determine the pronunciation stage boundary based on the global temporal constraint relationship, and divide the first phoneme sequence sample into multiple consecutive pronunciation stages using the pronunciation stage boundary as a constraint condition; determine the corresponding alignment interval for each pronunciation stage on the first word sequence sample; when performing phoneme-level alignment, shrink and restrict the effective mask region in the mask matrix to the alignment interval corresponding to the pronunciation stage to which each phoneme belongs; update the mask matrix based on the temporal arrangement order of each phoneme in the first phoneme sequence sample to obtain the updated mask matrix; and calibrate the second alignment result using the updated mask matrix to obtain the calibrated second alignment result.

[0374] In some embodiments, the sample acquisition module 2331 is further configured to calculate a phoneme consistency index based on the second alignment result; when there is a target mapping relationship in the global temporal constraint relationship that is inconsistent with the phoneme consistency index, the target mapping relationship in the global temporal constraint relationship is deleted or adjusted to obtain a calibrated global temporal constraint relationship.

[0375] In some embodiments, the sample acquisition module 2331 is further configured to, when the first phoneme sequence sample is detected to be an empty phoneme sequence, fill the first phoneme sequence sample with a preset identifier to obtain a third phoneme sequence sample, wherein the sequence length of the third phoneme sequence sample is equal to the sequence length of the first word sequence sample, so as to use the third phoneme sequence sample to iteratively train the initial prediction model; or, when the proportion of valid phoneme identifiers in the first phoneme sequence sample is detected to be less than a first preset threshold, fill the position corresponding to the invalid phoneme identifier in the first phoneme sequence sample with a preset identifier to obtain a fourth phoneme sequence sample, so as to use the fourth phoneme sequence sample to iteratively train the initial prediction model.

[0376] In some embodiments, the sample acquisition module 2331 is further configured to determine the energy variance of the first word sequence sample, and when the energy variance is greater than a second preset threshold, determine that there are noise words in the first word sequence sample; determine the first position of the noise word in the first word sequence sample, fill the first position with a preset identifier, and obtain the second word sequence sample, so as to use the second word sequence sample to iteratively train the initial prediction model.

[0377] In some embodiments, the sample acquisition module 2331 is further configured to acquire a target phoneme template corresponding to the first phoneme sequence sample, the target phoneme template including multiple target phonemes with standard pronunciation; for each phoneme in the first phoneme sequence sample, determine the edit distance between the phoneme and the corresponding target phoneme; when the edit distance is greater than a third preset threshold, determine the phoneme as an abnormal phoneme; determine the second position of the abnormal phoneme in the first phoneme sequence sample, fill the second position with a preset identifier, and obtain a fifth phoneme sequence sample, so as to use the fifth phoneme sequence sample to iteratively train the initial prediction model.

[0378] In some embodiments, the first alignment window is used to constrain the candidate alignment interval between the first phoneme sequence sample and the first word sequence sample. The first training module 2332 is further used to determine, for each phoneme in the first phoneme sequence sample, a reference anchor point on the first word sequence sample based on the first alignment relationship; to extract a second word interval on the first word sequence sample centered on the reference anchor point, corresponding to the window length of the first alignment window, as a candidate alignment interval; to input at least one of the first phoneme sequence sample and the first word sequence sample into the initial prediction model for prediction processing to obtain a prediction result; to determine the phoneme loss value based on the first phoneme sequence sample and the first word sequence sample within the candidate alignment interval, and to determine the prediction task loss value based on the prediction result; to determine the first loss value based on the first phoneme loss weight, the prediction task loss value, the phoneme loss value, and the joint loss function, and to update the network parameters of the initial prediction model based on the first loss value.

[0379] In some embodiments, the first training module 2332 is further configured to parse the first alignment relationship, obtain the index mapping relationship between the phoneme and the first word sequence sample; determine the word index of the target word corresponding to the phoneme in the first word sequence sample based on the index mapping relationship; and determine the time position of the word index as the reference anchor point corresponding to the phoneme.

[0380] In some embodiments, the first training module 2332 is further configured to obtain the first hidden layer features corresponding to the first phoneme sequence samples processed by the initial prediction model; obtain the second hidden layer features corresponding to the first word sequence samples within the candidate alignment interval processed by the initial prediction model; determine the feature matching degree between the first hidden layer features and the second hidden layer features; and determine the phoneme loss value based on the feature matching degree.

[0381] In some embodiments, the metric determination module 2333 is further configured to determine the alignment confidence of the initial prediction model and determine the alignment error evaluation value based on the alignment confidence; determine the current round loss value of the initial prediction model in the current training round and the initial round loss value of the initial prediction model in the initial training round, and determine the ratio between the current round loss value and the initial round loss value as the loss convergence evaluation value; obtain the predicted phoneme sequence and the preset phoneme sequence output by the initial prediction model in the current training round, and determine the consistency ratio between the predicted phoneme identifiers in the predicted phoneme sequence and the preset phoneme identifiers in the preset phoneme sequence as the phoneme prediction accuracy; and combine the alignment error evaluation value, the loss convergence evaluation value, and the phoneme prediction accuracy into a state evaluation metric.

[0382] The following describes an exemplary structure of the speech processing device provided in the embodiments of this application as a software module. In some embodiments, the software module in the speech processing device may include: a sequence acquisition module, used to acquire an input feature sequence to be processed, the input feature sequence including at least one of a target phoneme sequence and a target word sequence; a prediction processing module, used to call a target prediction model to perform forward prediction processing on the input feature sequence to obtain a target prediction result, the target prediction model being trained using the model training method provided in the embodiments of this application; a result generation module, used to generate a task processing result for the prediction task based on the target prediction result; and output the task processing result.

[0383] This application provides a computer program product, which includes computer-executable instructions or a computer program. The processor of an electronic device reads the computer-executable instructions or computer program from a computer-readable storage medium, and executes the computer-executable instructions or computer program, causing the electronic device to perform the model training method or speech processing method described in this application.

[0384] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the model training method or speech processing method provided in this application. For example, ... Figure 3 The model training method is shown.

[0385] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0386] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0387] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0388] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0389] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.< / sil>

Claims

1. A model training method, characterized in that, The method includes: Obtain the first phoneme sequence sample, the first word sequence sample, and the first alignment relationship between the first phoneme sequence sample and the first word sequence sample; The initial prediction model is trained iteratively until the training termination condition is met, resulting in the target prediction model: In the first training phase, the initial prediction model is trained based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weight, and the joint loss function of the initial prediction model. The joint loss function includes a prediction task loss value and a phoneme loss value adjusted by the phoneme loss weight. The phoneme loss value is determined based on the feature matching degree between the first phoneme sequence sample and the first word sequence sample. Determine the state evaluation index of the initial prediction model; When the state evaluation index meets the stage switching conditions, it triggers the entry into the second training stage; In the second training phase, the initial prediction model is trained again based on the first phoneme sequence sample, the first word sequence sample, the first alignment relation, the second alignment window, the second phoneme loss weight, and the joint loss function. The window length of the second alignment window is less than the window length of the first alignment window, and the first phoneme loss weight is less than the second phoneme loss weight.

2. The method according to claim 1, characterized in that, The method further includes: Determine the initial alignment relationship between the first phoneme sequence sample and the first word sequence sample; The trained alignment prediction model is invoked to perform alignment prediction on the first phoneme sequence sample and the first word sequence sample to obtain the predicted alignment relationship. A first confidence level is determined for the predicted alignment relationship. When the first confidence level is greater than or equal to a first confidence threshold, the initial alignment relationship is updated based on the predicted alignment relationship to obtain the first alignment relationship.

3. The method according to claim 2, characterized in that, The method further includes: For each phoneme in the first phoneme sequence sample, based on the initial alignment relationship, a first word segment interval corresponding to the phoneme is determined in the first word segment sequence sample; The index position of the target word corresponding to the phoneme in the initial alignment relationship in the first word sequence sample is determined as the first target position; Within the first word segment interval, construct a target probability distribution in which the probability of the first target position is greater than the probability of other positions; Input the first phoneme sequence sample and the first word sequence sample into the alignment prediction model to be trained, and output the predicted probability distribution of the phoneme in the first word interval; The alignment loss value is determined based on the predicted probability distribution and the target probability distribution, and the model parameters of the alignment prediction model are updated based on the alignment loss value to obtain the trained alignment prediction model.

4. The method according to claim 3, characterized in that, The step of determining the first word interval corresponding to the phoneme in the first word sequence sample based on the initial alignment relationship includes: From the initial alignment relationship, obtain the target word character mapped by the phoneme; Based on the preset window length, a series of consecutive word units centered on the target word unit are extracted from the first word unit sequence sample to obtain the first word unit interval corresponding to the phoneme.

5. The method according to claim 3, characterized in that, The step of constructing a target probability distribution within the first word interval, where the probability of the first target position is greater than the probabilities of other positions, includes: Set the probability value corresponding to the first target position within the first word interval as the first probability value; Set the probability values ​​corresponding to other positions within the first word interval besides the first target position as the second probability value, wherein the first probability value is greater than the second probability value; The target probability distribution corresponding to the phoneme is formed by the first probability value and the second probability value within the first word interval.

6. The method according to claim 3, characterized in that, Determining the alignment loss value based on the predicted probability distribution and the target probability distribution includes: Using a preset alignment loss function, an initial loss value is determined based on the predicted probability distribution and the target probability distribution; Obtain the center positions of the first word intervals corresponding to two adjacent phonemes in the first phoneme sequence sample; When the index of the center position of the two adjacent phonemes in the first word sequence sample does not satisfy the monotonically non-decreasing condition, an auxiliary loss value for violating the monotonically non-decreasing condition is determined based on a preset penalty function. The auxiliary loss value and the initial loss value are fused together to obtain the alignment loss value.

7. The method according to claim 2, characterized in that, Determining the initial alignment relationship between the first phoneme sequence sample and the first word sequence sample includes: The first phoneme sequence sample and the first word sequence sample are constructed into a positive sample pair, and the first phoneme sequence sample and the first word sequence sample are derived from the first speech sample. Obtain a second phoneme sequence sample and construct a negative sample pair with the second phoneme sequence sample and the first word sequence sample. The second phoneme sequence sample comes from a second speech sample. The first speech sample and the second speech sample are different. The temporal matching degree prediction is performed on the positive sample pair and the negative sample pair respectively to obtain the first temporal matching degree of the positive sample pair and the second temporal matching degree of the negative sample pair. Based on the first temporal matching degree and the second temporal matching degree, target sample pairs that meet the preset matching degree conditions are determined from the positive sample pairs and the negative sample pairs. The target sample pairs include target phoneme sequence samples and target word sequence samples. An alignment relationship is constructed between the target phoneme sequence sample and the target word sequence sample to obtain the initial alignment relationship.

8. The method according to claim 1, characterized in that, The first alignment window is used to constrain the candidate alignment interval between the first phoneme sequence sample and the first word sequence sample. The step of training the initial prediction model based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weight, and the joint loss function of the initial prediction model includes: For each phoneme in the first phoneme sequence sample, a reference anchor point for the phoneme on the first word sequence sample is determined based on the first alignment relationship; Centered on the reference anchor point, a second word segment corresponding to the window length of the first alignment window is extracted from the first word segment sequence sample as the candidate alignment segment; Input at least one of the first phoneme sequence sample and the first word sequence sample into the initial prediction model for prediction processing to obtain the prediction result; Within the candidate alignment interval, a phoneme loss value is determined based on the first phoneme sequence sample and the first word sequence sample, and a prediction task loss value is determined based on the prediction result; A first loss value is determined based on the first phoneme loss weight, the prediction task loss value, the phoneme loss value, and the joint loss function, and the network parameters of the initial prediction model are updated based on the first loss value.

9. The method according to claim 8, characterized in that, The step of determining the reference anchor point of the phoneme on the first word sequence sample based on the first alignment relationship includes: Parse the first alignment relationship to obtain the index mapping relationship between the phoneme and the first word sequence sample; Based on the index mapping relationship, the lexical index of the target lexical corresponding to the phoneme in the first lexical sequence sample is determined; The time position of the word index is determined as the reference anchor point corresponding to the phoneme.

10. The method according to claim 8, characterized in that, Within the candidate alignment interval, determining the phoneme loss value based on the first phoneme sequence sample and the first word sequence sample includes: Obtain the first hidden layer features corresponding to the first phoneme sequence sample obtained by the initial prediction model; Obtain the second hidden layer features corresponding to the first word sequence sample within the candidate alignment interval, obtained from the initial prediction model; Determine the feature matching degree between the first hidden layer feature and the second hidden layer feature; The phoneme loss value is determined based on the feature matching degree.

11. The method according to claim 2, characterized in that, Determining the initial alignment relationship between the first phoneme sequence sample and the first word sequence sample includes: Obtain the original phoneme sequence corresponding to the first speech sample, and preprocess the original phoneme sequence to obtain a processed sequence sample, wherein the length of the processed sequence sample is a first length; Obtain the first word sequence sample corresponding to the first speech sample, wherein the length of the first word sequence sample is the second length; A target mapping function is constructed based on the first length and the second length, and the processed sequence sample is resampled based on the target mapping function to obtain the first phoneme sequence sample, wherein the length of the first phoneme sequence sample is the second length; An alignment relationship is constructed between the first phoneme sequence sample and the first word sequence sample to obtain the initial alignment relationship.

12. The method according to claim 11, characterized in that, The construction of the target mapping function based on the first length and the second length includes: The processed sequence sample is subjected to articulation stage boundary detection to obtain a second target position in the processed sequence sample. The second target position includes at least one of the articulation transition frame position and syllable boundary position. The timing offset coefficient is determined based on the second target position, and the length ratio between the second length and the first length is determined. The target mapping function is constructed based on the length ratio and the time offset coefficient.

13. The method according to claim 12, characterized in that, The step of resampling the processed sequence samples based on the target mapping function to obtain the first phoneme sequence sample includes: For each source phoneme index in the processed sequence sample, the target word index corresponding to the source phoneme index is determined based on the target mapping function; When the target word index exceeds the length range of the processed sequence sample, the phoneme identifier corresponding to the last phoneme in the processed sequence sample is used as the phoneme identifier corresponding to the target word index. The positions corresponding to the word indexes that do not match the source phoneme are filled with preset ignore flags to obtain the first phoneme sequence sample.

14. The method according to claim 13, characterized in that, The step of determining the target lexical index corresponding to the source phoneme index based on the target mapping function includes: Based on the length ratio, the source phoneme index is scaled to obtain a scaled source phoneme index; The timing offset coefficient and the scaled source phoneme index are superimposed to obtain the superposition result; The target word index corresponding to the source phoneme index is determined based on the superposition result.

15. The method according to claim 1, characterized in that, The first lexical sequence sample is labeled with a temporal resolution, wherein the temporal resolution type includes at least a first resolution, a second resolution, and a third resolution, which are divided according to temporal resolution from low to high. The method further includes: Based on the first resolution, an initial alignment is performed using a first-level search window to obtain the global temporal constraint relationship between the first phoneme sequence sample and the first word sequence sample at the pronunciation stage level. Based on the second resolution, within the scope defined by the global temporal constraint relationship, syllable-level alignment is performed using a second-level search window to obtain the first alignment result; Based on the third resolution, within the range defined by the first alignment result, phoneme-level alignment is performed using a third-level search window to obtain the second alignment result; Based on the second alignment result, the first alignment relationship between the first phoneme sequence sample and the first word sequence sample is determined; The window lengths of the first-level search window, the second-level search window, and the third-level search window decrease progressively.

16. The method according to claim 15, characterized in that, The step of performing initial alignment based on the first resolution using a first-level search window to obtain the global temporal constraint relationship between the first phoneme sequence sample and the first word sequence sample at the pronunciation stage includes: Extract the first acoustic features of the first phoneme sequence sample and the first text features corresponding to the first word sequence sample; Within the constraints of the first-level search window, a first similarity matrix is ​​determined between the first acoustic feature and the first text feature; Based on the first similarity matrix, perform optimal path search to determine the start boundary and termination boundary between the pronunciation segment and the word text block, and use the start boundary and termination boundary as the global temporal constraint relationship.

17. The method according to claim 15, characterized in that, Based on the second resolution, within the scope defined by the global temporal constraints, syllable-level alignment is performed using a second-level search window to obtain the first alignment result, including: Extract the second acoustic features of the first phoneme sequence sample and obtain the syllable-level word features of the first word sequence sample; Map the global temporal constraint relationship to an alignment search mask; Under the joint constraints of the alignment search mask and the second-level search window, the attention alignment probability between the second acoustic feature and the syllable-level word feature is determined, and the first alignment result representing the syllable-level matching path is determined based on the attention alignment probability.

18. The method according to claim 15, characterized in that, Based on the third resolution, within the range defined by the first alignment result, phoneme-level alignment is performed using a third-level search window to obtain a second alignment result, including: Extract the frame-level acoustic features of the first phoneme sequence sample and obtain the phoneme-level word features of the first word sequence sample; A mask matrix is ​​generated based on the matching path indicated by the first alignment result and the window length of the third-level search window. The alignment frame boundary between the frame-level acoustic features and the phoneme-level lexical features is determined using the mask matrix, resulting in the second alignment result representing the frame-level matching correspondence.

19. The method according to claim 15, characterized in that, Determining the first alignment relationship between the first phoneme sequence sample and the first word sequence sample based on the second alignment result includes: From the second alignment result, extract the start acoustic frame index and end acoustic frame index corresponding to each word in the first word sequence sample in the first phoneme sequence sample; Based on the starting acoustic frame index and the ending acoustic frame index, the pronunciation duration corresponding to each word is determined; The mapping sequence consisting of the lexical and the corresponding pronunciation duration is determined as the first alignment relationship.

20. The method according to claim 18, characterized in that, The method further includes: Based on the global temporal constraint relationship, the boundaries of the pronunciation stages are determined, and the first phoneme sequence sample is divided into multiple consecutive pronunciation stages using the boundaries of the pronunciation stages as constraints. On the first word sequence sample, a corresponding alignment interval is determined for each of the pronunciation stages; When performing phoneme-level alignment, the effective mask region in the mask matrix is ​​shrunk and restricted to the alignment interval corresponding to the pronunciation stage of each phoneme; Based on the temporal arrangement order of each phoneme in the first phoneme sequence sample, the mask matrix is ​​updated to obtain the updated mask matrix; The second alignment result is calibrated using the updated mask matrix to obtain the calibrated second alignment result.

21. The method according to any one of claims 15 to 19, characterized in that, The method further includes: Calculate the phoneme consistency index based on the second alignment result; When there is a target mapping relationship in the global temporal constraint relationship that is inconsistent with the phoneme consistency index, the target mapping relationship in the global temporal constraint relationship is deleted or adjusted to obtain the calibrated global temporal constraint relationship.

22. The method according to any one of claims 1 to 19, characterized in that, The determination of the state evaluation index of the initial prediction model includes: Determine the alignment confidence of the initial prediction model, and determine the alignment error evaluation value based on the alignment confidence; Determine the current loss value of the initial prediction model in the current training round, and the initial loss value of the initial prediction model in the initial training round, and determine the ratio between the current loss value and the initial loss value as the loss convergence evaluation value; The predicted phoneme sequence and the preset phoneme sequence output by the initial prediction model in the current training round are obtained, and the proportion of consistency between the predicted phoneme identifier in the predicted phoneme sequence and the preset phoneme identifier in the preset phoneme sequence is determined as the phoneme prediction accuracy. The alignment error evaluation value, the loss convergence evaluation value, and the phoneme prediction accuracy are combined to form the state evaluation index.

23. The method according to any one of claims 1 to 19, characterized in that, The method further includes: When the first phoneme sequence sample is detected to be an empty phoneme sequence, a preset identifier is filled into the first phoneme sequence sample to obtain a third phoneme sequence sample. The sequence length of the third phoneme sequence sample is equal to the sequence length of the first word sequence sample, so as to use the third phoneme sequence sample to iteratively train the initial prediction model; or, When the proportion of valid phoneme identifiers in the first phoneme sequence sample is less than a first preset threshold, the preset identifier is filled in the position corresponding to the invalid phoneme identifier in the first phoneme sequence sample to obtain a fourth phoneme sequence sample, so as to use the fourth phoneme sequence sample to iteratively train the initial prediction model.

24. The method according to any one of claims 1 to 19, characterized in that, The method further includes: Determine the energy variance of the first word sequence sample; when the energy variance is greater than a second preset threshold, determine that there are noise words in the first word sequence sample. The first position of the noise word in the first word sequence sample is determined, and a preset identifier is filled in the first position to obtain a second word sequence sample, so as to use the second word sequence sample to iteratively train the initial prediction model.

25. The method according to any one of claims 1 to 19, characterized in that, The method further includes: Obtain the target phoneme template corresponding to the first phoneme sequence sample, wherein the target phoneme template includes multiple target phonemes with standard pronunciations; For each phoneme in the first phoneme sequence sample, determine the edit distance between the phoneme and the corresponding target phoneme; When the edit distance is greater than a third preset threshold, the phoneme is determined to be an abnormal phoneme; The second position of the abnormal phoneme in the first phoneme sequence sample is determined, and a preset identifier is filled in the second position to obtain a fifth phoneme sequence sample, so as to use the fifth phoneme sequence sample to iteratively train the initial prediction model.

26. A speech processing method, characterized in that, The method includes: Obtain the input feature sequence to be processed, wherein the input feature sequence includes at least one of the target phoneme sequence and the target word sequence; The target prediction model is invoked to perform forward prediction processing on the input feature sequence to obtain the target prediction result. The target prediction model is trained using the model training method described in any one of claims 1 to 25. Based on the target prediction results, generate task processing results for the prediction task; Output the task processing results.

27. A model training device, characterized in that, The device includes: The sample acquisition module is used to acquire a first phoneme sequence sample, a first word sequence sample, and a first alignment relationship between the first phoneme sequence sample and the first word sequence sample; The first training module is used to iteratively train the initial prediction model until the training termination condition is met to obtain the target prediction model: In the first training stage, the initial prediction model is trained based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the first alignment window, the first phoneme loss weight, and the joint loss function of the initial prediction model. The joint loss function includes a prediction task loss value and a phoneme loss value adjusted by the phoneme loss weight. The phoneme loss value is determined based on the feature matching degree between the first phoneme sequence sample and the first word sequence sample. The indicator determination module is used to determine the state evaluation indicators of the initial prediction model; The second training module is used to trigger the entry into the second training phase when the state evaluation index meets the phase switching condition. In the second training phase, the initial prediction model is trained again based on the first phoneme sequence sample, the first word sequence sample, the first alignment relationship, the second alignment window, the second phoneme loss weight, and the joint loss function. The window length of the second alignment window is less than the window length of the first alignment window, and the first phoneme loss weight is less than the second phoneme loss weight.

28. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the model training method according to any one of claims 1 to 25, or the speech processing method according to claim 26.

29. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the model training method according to any one of claims 1 to 25, or the speech processing method according to claim 26.

30. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the model training method according to any one of claims 1 to 25, or the speech processing method according to claim 26.

Images