Deep learning-based chinese language audio phoneme segmentation method, device and medium

Abstract

The application discloses a deep learning-based Chinese language audio phoneme segmentation method and device and a medium, belongs to the technical field of speech analysis, and solves the problem of how to improve the precision and robustness of phoneme boundary detection; the application provides an audio phoneme segmentation model combining an audio signal and a mel spectrum feature, simultaneously receives an original audio signal and a mel spectrogram as input; the original audio signal is input into a HuBERT encoder, and the mel spectrogram is input into a Conformer encoder; two output features are firstly subjected to a residual attention mechanism to realize preliminary cross-modal fusion, and then subjected to multi-scale dilated convolution to extract context information in different receptive field ranges; the features subjected to the multi-scale dilated convolution are further input into a gating mechanism for feature selection; the selected features are subjected to a phoneme classification linear layer to obtain classification features, and the application has significant advantages in reducing manual labeling cost, improving segmentation precision and robustness.

Description

Technical Field

[0001] This invention belongs to the field of speech analysis technology, and relates to a method, device and medium for Chinese audio phoneme segmentation based on deep learning. Background Technology

[0002] In phonetics and linguistics research, phonemes, as the most basic and distinctive units of speech in language, are always fundamentally and critically defined and segmented. Phonemes not only constitute the smallest semantic unit of speech, but also serve as the foundation for constructing higher-level language structures (such as syllables, words, and sentences). Accurately obtaining the temporal boundaries of phonemes is of great significance for building high-performance speech models, optimizing acoustic feature representations, and even improving the overall performance of downstream speech processing tasks.

[0003] Phoneme segmentation is a technique for identifying precise temporal boundaries of phonemes in continuous speech. As a fundamental step in speech processing, it supports a wide range of applications such as pronunciation assessment, speech synthesis, and speech analysis. Unlike phoneme recognition, which focuses on assigning category labels to speech segments, phoneme segmentation requires accurately locating boundary transition points at a fine temporal resolution, thus posing a greater technical challenge.

[0004] Traditional phoneme segmentation methods typically rely on statistical acoustic models or forced alignment tools, such as the widely used Montreal Forced Aligner (MFA) in recent years. These tools are generally based on hybrid HMM-GMM or HMM-DNN models, achieving phoneme-level temporal alignment using existing transcribed text and dictionary information. Phoneme boundaries are derived from the state alignment results of a given phoneme sequence. While effective in controlled environments, these methods heavily depend on pronunciation dictionaries and strong frame-level assumptions, treating phoneme boundaries as implicit outcomes rather than explicit learning targets.

[0005] With the development of deep learning, neural network-based acoustic modeling methods have made breakthrough progress in the field of speech processing. In particular, the successful application of models such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Transformers in speech sequence modeling transforms phoneme segmentation into frame-level or segment-level classification tasks, predicting phoneme labels or boundary indicators through acoustic features. Existing technologies, such as the invention patent with publication number CN121214975A, disclose a multi-scale, multi-feature-based speech emotion recognition method. This method employs a multi-layer cross-attention mechanism for feature fusion, allowing MFCC features and spectrogram features to interact bidirectionally at multiple levels, dynamically calculating their correlation. This method fuses the temporal features F_M2 of MFCC and the spatial features F_S2 of the spectrogram, enabling interaction between two different perspectives of acoustic features to generate more powerful emotion classification features. Although existing technologies alleviate some of the limitations of traditional alignment methods, they still confuse boundary detection with phoneme classification, making it difficult for models to capture potential and context-dependent phoneme transitions.

[0006] From the perspective of representation learning, phoneme boundaries do not correspond to a single, uniform acoustic pattern. Depending on the speech environment and coarticulation effects, boundaries may manifest as waveform abrupt changes, spectral gradients, or context-dependent temporal transitions. However, most existing methods rely on single acoustic representations (such as the original waveform or time-frequency features), which limits their ability to encode diverse boundary cues. Therefore, improving the accuracy and robustness of phoneme boundary detection remains a pressing challenge. Summary of the Invention

[0007] The technical problem to be solved by this invention is how to improve the accuracy and robustness of phoneme boundary detection.

[0008] The present invention solves the above-mentioned technical problems through the following technical solutions:

[0009] A deep learning-based method for Chinese audio phoneme segmentation includes the following steps: acquiring the original waveform signal and corresponding spectrogram of the audio sample; constructing a Chinese phoneme segmentation model, using the audio sample and its corresponding Mel spectrogram as input, and training the model, including: The original waveform signal is input into the HuBERT encoder to extract waveform semantic features; Input the spectrogram into the Conformer encoder to extract the spectral acoustic features; Waveform semantic features and spectral acoustic features are input into the feature fusion module for cross-modal feature interaction to generate fused features; The fused features are input into the multi-scale temporal domain analysis module to extract multi-scale contextual features; Multi-scale contextual features are input into a gating network, and the multi-scale contextual features are adaptively selected by dynamically calculating the feature importance weights to generate boundary-sensitive features.

[0010] Furthermore, the method also includes: Boundary-sensitive features are input into a classifier to detect speech activity and determine whether each frame is a silent frame; based on the determination result, frame-level phoneme classification is assisted; wherein, the classifier is a linear layer.

[0011] Furthermore, the feature fusion module employs a residual cross-attention mechanism, specifically including the following: First, the waveform semantic features are mapped to a query matrix, and the spectral acoustic features are mapped to a key matrix and a value matrix; Next, the attention weights of the query matrix and the key matrix are calculated, and the value matrix is ​​weighted and aggregated according to the attention weights to obtain the attention fusion features; Then, the attention fusion features are residually connected with the waveform semantic features to generate fusion features.

[0012] Furthermore, the multi-scale temporal analysis module includes at least two parallel dilated convolutional layers, each with a different dilation rate.

[0013] Furthermore, the multi-scale contextual features are obtained through the following process: First, the fused features are processed through a one-dimensional convolutional layer. Perform downsampling; Next, the outputs of multiple parallel dilated convolutional layers are summed along the channel dimension; Then, the fused features are fused with the downsampled features through residual connections to generate multi-scale contextual features.

[0014] Furthermore, the gated network includes a linear transformation layer and an activation function layer, and the specific process for generating boundary-sensitive features is as follows: First, the multi-scale contextual features are input into the gating network, which outputs an importance weight vector, represented by the following logic:

[0015] in, Weight coefficients generated for different dimensions , For multi-scale contextual features, , For feature embedding dimension, Represents a linear transformation layer. Indicates the activation function layer; Subsequently, the multi-scale contextual features are input into another linear transformation layer to obtain the transformed features. This can be represented using the following logic:

[0016] Finally, the importance weight vector is compared with the transformed features. Perform element-wise multiplication to obtain boundary-sensitive features. :

[0017] in, It represents the Hadamardi (or Hadama) stack.

[0018] Furthermore, during model training, a combined loss function is used to optimize the model. This combined loss function consists of the loss function of the main classification task. Loss function for auxiliary classification tasks The weighted sum is obtained, represented as ,in To assist in task weighting.

[0019] Furthermore, the loss function of the main classification task is expressed as follows: ,in For weighted focus loss, The weighted focus loss is represented using the following logic, which is the Dice loss. :

[0020]

[0021] in, and These are the balance factor and modulation factor for focus loss, respectively. For the model, the t-th sample is in the true category The predicted probability, For the real category Category weights, Let be the true label of the t-th sample. Used to reduce the loss of easily classified samples and to emphasize difficult-to-classify samples. The class weights representing cross-entropy This is a mask used to exclude ignored tags. It is an indicator function used to filter out samples from which loss calculations are not required. To ignore the labels, the Dice loss is represented using the following logic. :

[0022] in, For feature embedding dimension, For the model to the first Each sample belongs to category The predicted probability, For the first One-Hot encoding of the true labels of each sample, when the true category is Time value Otherwise, it is 0; For category Corresponding weights A smoothing constant to prevent division by zero errors.

[0023] An electronic device includes a memory and a processor, the memory being used to store a program that supports the processor in executing the aforementioned deep learning-based Chinese audio phoneme segmentation method, and the processor being configured to execute the program stored in the memory.

[0024] A storage medium storing a computer program, wherein the computer program is executed by a processor to perform the steps of the above-described deep learning-based Chinese audio phoneme segmentation method.

[0025] The advantages of this invention are: This invention systematically explores the Chinese phoneme segmentation task from the perspective of deep learning, proposing an audio phoneme segmentation model that combines audio signals and Mel spectrogram features. It simultaneously receives the original audio signal and Mel spectrogram as input: the original audio signal serves as the input to the HuberT encoder, while the Mel spectrogram is input to the Conformer encoder. The two output features first undergo preliminary cross-modal fusion through a residual attention mechanism, and then extract contextual information within different receptive fields through multi-scale dilated convolution. The features processed by multi-scale dilated convolution are further input into a gating mechanism for feature selection. The selected features are processed through a phoneme classification linear layer to obtain classification features, which, along with phoneme labels, serve as the input to the Combin Loss function for the phoneme classification task. This invention also introduces a VAD (Voice over Activated Silence) silent classification auxiliary head. The previously acquired classification features are processed through a simple linear layer to generate auxiliary features, which are used in subsequent auxiliary tasks to determine whether a frame is silent.

[0026] Compared with traditional techniques, this invention uses the high-level representation of the self-supervised pre-trained model (HuBERT branch) as an "implicit text" or "content guide" to guide the fine-grained acoustic model (Conformer branch) for boundary localization through an attention mechanism. This reduces reliance on expensive transcribed text while achieving or approaching the segmentation accuracy of traditional text-dependent methods. It demonstrates significant advantages in reducing manual annotation costs, improving segmentation accuracy and robustness, and achieving stable phoneme segmentation without requiring precise word-by-word transcription. This provides a new technical approach to solving the transcription cost problem in large-scale corpus processing. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the Chinese phoneme segmentation model according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram showing the distribution of some phonemes in the AISHELL dataset of Embodiment 1 of the present invention; Figure 3 This is a visualization of the phoneme segmentation task performed based on the AISHELL dataset in Embodiment 1 of the present invention; Figure 4 This is a visualization of the phoneme segmentation task performed based on the ST-CMDS dataset in Embodiment 1 of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments: Example 1 like Figure 1 Specifically, a deep learning-based method for Chinese audio phoneme segmentation is disclosed, including the following steps: S1, Obtain the training dataset, which includes multiple Chinese audio samples.

[0030] In this embodiment, to evaluate the performance of the deep learning model, sample data from the AISHELL-1 dataset and the ST-CMDS dataset were selected as training data to construct the training dataset. The AISHELL-1 dataset belongs to the AISHELL-ASR0009 Mandarin speech database, with a total duration of 178 hours. It includes recordings from 400 speakers representing different Chinese dialect regions, transcribed and annotated by professional speech annotators, and underwent rigorous quality checks. The AISHELL-1 dataset maintains a text accuracy of no less than 95% and includes training, development, and test sets.

[0031] The ST-CMDS dataset is a publicly available Mandarin command speech dataset, totaling approximately 108 hours. Recordings were taken by about 855 speakers from different regions of China using mobile phones in relatively quiet indoor environments. Each recording contains a single-sentence command from a single speaker, with no overlap. These are primarily short instructions controlled by smart devices, with an average sentence length of 4-8 seconds. After manual proofreading and rigorous quality checks, the ST-CMDS dataset achieves a text transcription accuracy exceeding 98%. The ST-CMDS dataset is divided into a training set (approximately 102 hours) and a test set. The training set is used for model training, while the test set is reserved for internal evaluation. During training, uniform randomization control ensures repeatability of results. All operations involving randomness are executed under a fixed random seed of 42, thus avoiding fluctuations in results between different runs.

[0032] In this embodiment, the dataset processing employs a fixed-ratio random partitioning method: the union of all subsets of the dataset is divided into a training set and a validation set in a 4:1 ratio. Since the partitioning process is dominated by a uniform random control mechanism, the training set and validation set maintain consistency across different experimental runs. This strategy effectively ensures the stability of model evaluation and the reliability of experimental results, such as... Figure 2 This shows the proportional distribution of specific phonemes in the AISHELL dataset.

[0033] S2, preprocessing the audio samples in the training dataset, specifically includes the following: like Figure 1 This embodiment presents the overall architecture of a deep learning-based Chinese audio phoneme segmentation model. The model's input includes two types of acoustic features: the original speech waveform (WAV) and its corresponding Mel spectrum. This embodiment obtains the two types of acoustic features required by the model and the pseudo-labels needed for training by preprocessing Chinese audio samples, specifically including the following: S21 uses resampling technology to standardize the audio signal at a preset sampling rate while maintaining the mono format, and obtains the original speech waveform (WAV) as the input of the HuBERT encoder in the model to obtain high-level speech content encoding.

[0034] S22 generates a Mel-spectrum (Mel spectrum) of the audio signal, which is used as input to the Conformer encoder in the model to capture local acoustic features at the phoneme level.

[0035] In this embodiment, the phoneme segmentation task is first modeled as a frame-by-frame multi-classification task of audio. Specifically, according to the classification of Chinese phonemes disclosed in the existing literature "Mandarin (China) MFA dictionary v3.0.0" (McAuliffe, M., & Sonderegger, M. (2024). Montreal Forced Aligner.), Chinese phonemes are divided into categories including "a", "aj", and "aj". ","aj ","aj ","aj ","aj ","aj 144 phonemes, including "aw", were used. An MFA aligner was used to generate phoneme temporal boundaries for each audio signal, serving as pseudo-labels for the training task. These pseudo-labels included phoneme category and phoneme temporal boundaries. The pseudo-labels were then converted into frame-level labels using the frame-to-time conversion formula below, serving as labels for the supervised learning task. The following logical representation was used:

[0036] in, For audio sampling rate, The sampling step size, For frame index, The target time point is defined as follows. In this embodiment, to complete the auxiliary classification task, frames marked as pronunciation in the frame-level labels are assigned a value of 1, and frames marked as silent are assigned a value of 0, thereby obtaining the silence label.

[0037] Furthermore, for the Mel frequency cepstral (Mel-Spectrogram, or Mel spectrum for short), the sampling points are converted into a frame-level representation using the following logical representation:

[0038] in, This refers to the number of sampling points in the audio. This represents the total number of frames.

[0039] In a preferred embodiment, Set to 16kHz, The settings are set to 320, with a sampling frequency of 16kHz input to the deep learning model for the original audio; for the Mel spectrogram, a 512-point FFT with an 80-dimensional Mel filter bank is used, and the frame shift is... The sampling points are set to 320 (20 ms) and the window length is set to 512 (32 ms), resulting in a standard overlap of about 12 ms between adjacent frames.

[0040] In a preferred embodiment, an MFA aligner is used to generate phoneme time boundaries for each audio signal as pseudo-labels for the training task. The audio signals are ultimately mapped to... Space, pseudo-tags are mapped to The space, where 145 represents 144 phoneme classes and silence frames, B represents batch size, and T represents the number of time frames. It represents the space of real numbers.

[0041] S3, data augmentation of the audio samples in the training dataset, specifically includes the following: S31, noise is added to the original audio signal, the intensity of which is adjusted by random sampling signal-to-noise ratio.

[0042] In this embodiment, to enhance the model's robustness in diverse acoustic environments and prevent the HuBERT branch from overfitting to clean studio speech, additive white Gaussian noise is added to the original audio waveform. The noise intensity is dynamically adjusted by randomly sampling the signal-to-noise ratio (SNR) to simulate background noise interference in the real world. Specifically, let the original audio signal be... The enhanced signal is defined as:

[0043] in, For the enhanced audio signal, n represents the Gaussian noise vector, generated based on the target signal-to-noise ratio (SNR), specifically within the SNR range. Uniform sampling within the region is represented by the following logic:

[0044] in, Indicates a normal distribution. The standard deviation of the noise. The average power of the original audio signal. , These are the upper and lower limits of the signal-to-noise ratio, respectively; specifically, they are represented using the following logical expression. :

[0045] in, This represents the audio signal value corresponding to the t-th time step.

[0046] In a preferred embodiment, and Set the values ​​to 15 and 30 respectively, and add noise randomly with a 50% probability.

[0047] S32 uses the SpecAugment regularization strategy to augment the Mel spectrogram.

[0048] In this embodiment, for the Mel spectrogram input to the Conformer branch, a SpecAugment regularization strategy is employed for data augmentation. This strategy includes frequency masking and temporal masking. By randomly masking consecutive frequency bands or time frames in the spectrogram, signal loss or transient changes are simulated, thereby forcing the model to reconstruct missing boundary information using the semantic features of the HuBERT branch and the residual acoustic context, thus significantly improving generalization ability. Specifically, let the input logarithmic Mel spectrogram be... ,in Indicates the number of frequency bands.

[0049] For frequency masks, the interval All frequency channels within are set to 0, where For frequency masking width, from uniform distribution Mid-sampling, The preset maximum frequency masking width, and The starting position for frequency masking. From the interval Mid-sampling, represented by the following logic:

[0050] in, Let be the logarithmic Mel energy value on the i-th Mel frequency channel in the t-th time frame.

[0051] For time masks, the interval All time frames within the range are set to 0, where For the time masking width, from a uniform distribution sampling, The preset maximum time mask width, The starting position of the time masking. from Sampling is represented using the following logic:

[0052] in, Let be the logarithmic Mel energy value on the f-th Mel frequency channel of the j-th time frame.

[0053] In step S3, based on the above data augmentation scheme, waveform-level noise is applied to enhance the noise resistance of Hubert semantic features, while spectrogram-level masking prevents Conformer from relying solely on local acoustic features. These two data augmentation methods work synergistically through a residual cross-attention module, enabling the model to accurately predict phoneme boundaries even when noise renders speech inaudible or acoustic cues are obscured.

[0054] S4, construct a Chinese phoneme segmentation model, using audio samples and corresponding Mel spectrograms as input, and train the model. The model includes at least a HuberT encoder, a Conformer encoder, a feature fusion module, a multi-scale temporal analysis module, a gating network, and a classifier; specifically, S4 includes the following: S41, input the original audio waveform signal of the audio sample into the HuBERT encoder to extract waveform semantic features.

[0055] In this embodiment, the HuberT encoder is a feature extractor for the raw audio signal. It is a self-supervised pre-trained model for speech processing, and its core idea is to learn high-level structural information of speech by predicting the sequence of hidden units. The HuberT encoder consists of multiple Transformer modules, which can efficiently model long-term dependencies. Through the hierarchical structure of the encoder, acoustic features containing multi-dimensional information such as speech content, phoneme boundaries, and speaker style are extracted step by step.

[0056] Specifically, the Hubert encoder includes a convolutional neural network layer (CNN Block) and a Transformer layer (Transformer Block). The original audio waveform signal of the audio sample is first fed into the pre-trained Hubert encoder. The CNN Block converts the audio waveform into low-level acoustic features, and then the Transformer Block converts the low-level acoustic features into high-level semantic features.

[0057] In a preferred embodiment, the HuBERT encoder can employ various pre-trained model variants. This embodiment uses the chinese-hubert-base version and freezes the convolutional neural network layer parameters during training. The HuBERT encoder possesses strong context-aware capabilities, utilizing contextual information to mitigate the uncertainty caused by noise and ambiguous phonemes. Therefore, when used as an audio feature encoder, it not only generates semantically coherent representations but also improves the overall performance of the model. Thus, when used as an audio feature encoder, the HuBERT encoder not only provides semantically coherent representations but also enhances the overall robustness of the model in phoneme boundary detection tasks.

[0058] S42, input the Mel spectrum corresponding to the original audio waveform signal into the Conformer encoder to extract the spectral acoustic features.

[0059] In this embodiment, the Conformer encoder is used to extract features from the Mel spectrogram corresponding to the original audio waveform signal. As an acoustic model architecture, the Conformer encoder combines a convolutional neural network with a self-attention mechanism, which can simultaneously capture local and global dependencies in the speech signal.

[0060] Specifically, the Conformer encoder comprises multiple Conformer layers, each of which includes a first feedforward network, a multi-head self-attention module, a convolutional module, and a second feedforward network. After the Mel spectrogram is input into the pre-trained Conformer encoder, the encoder outputs a high-dimensional representation as an abstract feature of the Mel spectrogram mode.

[0061] In a preferred embodiment, the number of Conformer layers is set to 2, the kernel size of the convolutional module is 31, the multi-head self-attention module is set to 4 attention heads, and a dropout rate of 30% is applied to the first feedforward network, the convolutional module, and the second feedforward network. This embodiment effectively alleviates the model overfitting problem by retaining 4 attention heads while randomly discarding neurons in the corresponding sub-modules.

[0062] In this embodiment, the Conformer encoder is highly sensitive to local details, while phoneme boundaries often exhibit instantaneous spectral changes, requiring the capture of short-term abrupt changes. This embodiment leverages the Conformer encoder's temporal modeling capabilities and local receptive field advantages to achieve a richer and more stable acoustic feature representation of the overall model.

[0063] S43, waveform semantic features and spectral acoustic features are input into the feature fusion module for cross-modal feature interaction to generate fused features; wherein the feature fusion module adopts a residual cross-attention mechanism, specifically including the following: Because the HuberT branch in the model is pre-trained on large-scale data, it can generate semantically rich and highly robust features; while the Conformer branch uses random initialization and needs to learn from scratch. Therefore, in the early stages of training, the Conformer encoder will generate noise. If this noise is directly concatenated with the features output by HuberT and then input into subsequent structures, it will contaminate the high-quality features output by the HuberT encoder, leading to a decrease in model performance. Furthermore, the high sensitivity of the Conformer encoder to local details is crucial for capturing instantaneous spectral changes at phoneme boundaries. Based on this, the feature fusion module in this embodiment uses a residual cross-attention mechanism to achieve cross-modal feature interaction.

[0064] Specifically, the waveform semantic features output by the HuberT encoder are first mapped to a query matrix. The spectral acoustic features output by the Conformer encoder are mapped to a key matrix. Sum matrix This can be represented using the following logic:

[0065] in, For waveform semantic features, , The embedding dimension is obtained by processing the audio waveform signal through a Hubert encoder. For spectral acoustic characteristics, , The embedding dimension, obtained by processing the Mel spectrogram using a Conformer encoder, is mapped to the same feature space through the aforementioned logical waveform semantic features and spectral acoustic features. , The embedding dimension of the key vector. , , All of these are learnable parameter matrices.

[0066] Next, the attention weights A between the query matrix and the key matrix are calculated using a cross-attention mechanism, and the value matrix is ​​weighted and aggregated based on these attention weights to obtain the attention fusion features. This can be represented using the following logic:

[0067]

[0068] Then, before normalization, attention is used to fuse the features. With waveform semantic features Perform residual connections to generate fused features. This can be represented using the following logic:

[0069] In a preferred embodiment, the feature fusion module is configured with 8 attention heads, and 30% of the neurons are randomly discarded.

[0070] In this embodiment, the weights of the Conformer branch can be dynamically adjusted through a residual cross-attention mechanism, ensuring that its attention output approaches zero when the Conformer's learning performance is poor. As long as the Conformer branch learns useful information, it can add value to the output of the Hubert branch. Furthermore, the residual cross-attention mechanism can mitigate the impact of minor frame differences caused by different rounding strategies employed by the two encoders during subsampling.

[0071] S44, input the fused features into the multi-scale temporal analysis module to obtain multi-scale contextual features; wherein, the multi-scale temporal analysis module includes at least two parallel dilated convolutional layers, each of which has a different dilation rate.

[0072] In phoneme segmentation tasks, the model must simultaneously capture long-term temporal context information and fine acoustic features near the boundaries to accurately locate phoneme boundaries. This embodiment introduces multi-scale dilated convolution. Specifically, the multi-scale temporal analysis module includes three parallel dilated convolutional layers, each using a 3×3 convolutional kernel, with different dilation rates of 1, 3, and 5. When the dilation rate of the dilated convolutional layer is 1, the current dilated convolutional layer is used to capture high-frequency local details, such as the instantaneous start point of plosive sounds in audio; while when the dilation rate of the dilated convolutional layer is 3 and / or 5, the current dilated convolutional layer is used to capture mid- to long-term temporal context information of the audio.

[0073] Specifically, the fused features of the S34 output are first processed through a 1×1 one-dimensional convolutional layer. Downsampling is performed; then, the outputs of three parallel dilated convolutional layers are summed along the channel dimension to fuse local detail information with mid- to long-term temporal context information; next, the fused features are fused with the downsampled features through residual connections to obtain multi-scale contextual features; finally, normalized residual sums and features are applied to further stabilize the training process and improve representation quality. This multi-scale contextual feature generation process prevents gradient vanishing in deep networks while preserving the original features.

[0074] S45 is a multi-scale contextual feature input gating network that dynamically calculates feature importance weights to adaptively select multi-scale contextual features and generate boundary-sensitive features.

[0075] In this embodiment, the gating network includes a linear transformation layer and a sigmoid activation function layer. By inputting multi-scale contextual features into the gating network and outputting an importance weight vector, it dynamically determines how much information should be retained in each frame, using the following logical representation:

[0076] in, The weighting coefficients generated for different dimensions, also known as gating weights. , For feature embedding dimension, For multi-scale contextual features, , Represents a linear transformation layer. This represents the activation function layer.

[0077] Subsequently, the multi-scale contextual features will undergo another linear transformation layer to obtain the transformed features. :

[0078] Finally, the gating network output is multiplied element-wise with the linearly transformed features through gating weights to obtain boundary-sensitive features. :

[0079] in, It represents the Hadamardi (or Hadama) stack.

[0080] like Figure 1 As shown, the features output by HuberT emphasize semantic analysis, while the features extracted based on Mel spectrograms emphasize acoustic features. During the steady-state period between phonemes, HuberT branches should be given higher weights; conversely, during the transition period at phoneme boundaries, the acoustic features of the Mel spectrogram are more critical. The gating mechanism automatically selects the features to be processed first in each frame: during the steady-state period of a phoneme, it allows more contextual features to pass through (e.g., dilated convolutions with a dilation rate of 5); while during the phoneme transition period, it allows more short-term high-frequency features to be passed (e.g., dilated convolutions with a dilation rate of 1).

[0081] Furthermore, the gating network outputs boundary-sensitive features After performing a linear transformation on the main classification head, the representation of the main classification task is obtained. ,like Figure 1 As shown.

[0082] S46, input boundary-sensitive features into the classifier to perform speech activity detection and determine whether each frame is a silent frame; based on the determination result, assist frame-level phoneme classification to correct the phoneme classification result. The classifier is a linear layer. Mapping to auxiliary task representation through linear layers .

[0083] In this embodiment, considering the large number of simple silent frames in the audio samples, which leads the model to tend to predict silent frames while ignoring speech frame features, this is not the desired result. Therefore, this embodiment introduces a Voice Activity Detection (VAD) auxiliary head as a classifier to assist the model in identifying whether a frame is silent. This auxiliary task adds a simple linear layer after the phoneme classification head to achieve low-level feature sharing, and uses the generated auxiliary features to construct a simple binary classification problem. A direct cross-entropy loss function is used to determine whether a frame belongs to the silent class. This auxiliary task helps the feature layer better distinguish between speech and background noise, thereby indirectly assisting the main classifier.

[0084] Furthermore, during training, a combined loss function is used to optimize the model. This combined loss function consists of the loss function of the main classification task. Loss function for auxiliary classification tasks We get the weighted sum, such as Figure 1 As shown; wherein, the loss function for the main classification task includes Dice loss and weighted focus loss, and the loss function for the auxiliary classification task is... Specifically, it is cross-entropy loss, used to determine whether a frame is silent.

[0085] In this embodiment, although 144 phoneme categories and silence frames were divided in step S2, extreme class imbalance still exists in actual multi-class classification tasks: the number of frames for a single phoneme category is far less than that for silence frames, and there are also significant differences in the frame distribution between different phoneme categories, such as... Figure 2 As shown. To address this issue, this embodiment employs a loss function for the main classification task during model training. To optimize, the loss function for the main classification task... Combining Dice Loss and Weighted Focus Loss loss to Dice Weighted focus loss Further explanation: Weighted focus loss is based on weighted cross-entropy loss by introducing a moderating factor. This guides the model to focus on phoneme samples that are difficult to classify, while reducing the weight allocation for the silent category. The weighted focus loss can be represented using the following logic. :

[0086]

[0087] in, and These are the balance factor and modulation factor for focus loss, respectively, which are set to 0.5 and 2 in this embodiment. For the model, the t-th sample is in the true category The predicted probability, For the real category Category weights, Let be the true label of the t-th sample. Used to reduce the loss of easily classified samples and to emphasize difficult-to-classify samples. The category weights represent the cross-entropy. In this embodiment, the weight of the silent class is set to 0.05, and the weight of the phoneme class is set to 1. Label smoothing with a weight of 0.1 is used in the cross-entropy calculation. This is a mask used to exclude ignored tags. It is an indicator function used to filter out samples from which loss calculations are not required. To ignore tags, this embodiment adopts the following masking strategy: after identifying the maximum time step within the batch, frame-level tags are filled with -100 to indicate that the tags are ignored, and silent tags are filled with 0.

[0088] Dice loss Originating from the field of medical image segmentation, the loss function designed in this embodiment is used to optimize the model's focus on the overlap of class sets, ignoring the size of silent frames and only focusing on phoneme overlap; specifically, the Dice loss is represented by the following logic. :

[0089] in, The feature embedding dimension also represents the total number of categories. In this embodiment, it includes 144 phoneme categories and a silence category. The model is mapped by the Softmax function to the first... Each sample belongs to category The predicted probability, For the first One-hot encoding of the true labels of each sample, when the true category is Time value Otherwise, it is 0; For category The corresponding weights are consistent with those used in the cross-entropy loss calculation, and are used to further weaken the dominant effect of the silent category on the Dice loss. To prevent division by zero errors, the smoothing constant is set to [value missing] in this embodiment. ; The mask, also an indicator function, is used to filter out ignored tags.

[0090] Furthermore, the total loss function of the main classification task Represented as By combining Dice loss and weighted focus loss, the model can both use focus loss to grasp the overall direction of gradient optimization and discover hard-to-classify samples during training, and use Dice loss to fine-tune the classification boundary under extremely imbalanced data, thereby improving the overlap of prediction results.

[0091] In a preferred embodiment, the phoneme segmentation task is first modeled as a multi-class deep learning task containing 145 categories (including 144 phonemes and a silence class). After converting the audio into a frame-level representation, the phoneme category is predicted for each frame. Finally, the temporal boundary of each phoneme is obtained through the frame-time conversion formula provided by S2. Next, the preprocessed and enhanced original audio along with the Mel spectrogram are input into the audio phoneme segmentation model. After encoder processing, cross-modal feature interaction, multi-scale temporal analysis, and gating mechanism, frame-level features are obtained. After linear transformation by the main classification head, the representation of the main classification task is obtained. This indicates that the loss is calculated by combining the input with the frame-level label. At the same time, Mapping to auxiliary task representation via linear layers This indicates that the cross-entropy loss is calculated along with the silence label. Here, weights [0.5, 1.0] are used to emphasize speech categories; the final combined loss function... ,in To assist in task weighting, a masking strategy is introduced during model training to improve efficiency; Finally, for all samples in the batch, the maximum time step for each sample is first determined. And fill the remaining samples to that maximum length; by creating a zero-filled value The effective time step is copied into the matrix, while the other cells remain unchanged, thereby unifying the time step of all samples in the batch. For frame-level labels and silent labels, this embodiment adopts a similar masking strategy: after identifying the maximum time step in the batch, frame-level labels are filled with -100 to indicate that the label is ignored, and silent labels are filled with 0.

[0092] Furthermore, simulation experiments were conducted on the deep learning-based Chinese audio phoneme segmentation method proposed in this embodiment to verify the model performance, including the following: (1) Since the Hubert encoder has been trained on large-scale data and has powerful feature extraction capabilities, this embodiment aims to fine-tune the Hubert encoder based on the data rather than make significant changes. However, other modules are initialized randomly and require larger step sizes to learn quickly. Based on this, this embodiment adopts a differential learning rate and implements a staged training strategy.

[0093] In the first phase, the learning rate of the HuberT encoder was set to 1e-5, and that of other modules to 2e-4, to ensure they quickly kept pace with the HuberT branch progress. Simultaneously, the patience value of the learning rate scheduler was set to 4, giving it ample room to adapt. After the first phase, we reduced the learning rate to enter the second phase of fine-tuning.

[0094] In the second stage, the learning rate of the HuberT encoder is reduced to 5e-6, the base learning rate of other modules is reduced to 5e-5, and the patience value of the learning rate scheduler is reduced to 3 to enhance its sensitivity. For the specific training task, the batch size is set to 16, and the weight decay is set to 1e-4. In a preferred embodiment, AdamW is used as the optimizer, ReduceLROnPlateau as the learning rate scheduler, the learning rate scheduler mode is set to min, and the learning rate adjustment factor is 0.5. During training, the following metrics are used to evaluate model performance: phoneme accuracy, VAD accuracy, phoneme F1 score, and VAD F1 score, which are represented by the following logic:

[0095]

[0096]

[0097]

[0098] in, , , , These represent phoneme accuracy, VAD accuracy, phoneme F1 score, and VAD F1 score, respectively. , These represent the precision and recall of the model classification, respectively. All the code above is implemented based on the PyTorch deep learning framework and trained using an RTX 3090 graphics card.

[0099] (2) Comparative experiments were conducted on the ST-CMDS dataset and the AISHELL-1 dataset, examining both unimodal and bimodal configurations and adjusting the weight parameters in the auxiliary task. As shown in Tables 1 and 2 below: Table 1 Performance Comparison of Different Modal Models

[0100] Table 1 shows a comparison of metrics for the ST-CMDS and AISHELL-1 datasets under single-modal (HuBERT or Conformer only) and bimodal (HuBERT + Conformer) configurations. In Table 1, Phone Accuracy, Phone F1, VAD Accuracy, and VAD F1 represent the accuracy of phoneme classification, the F1 score of phoneme classification, the accuracy of VAD auxiliary head classification, and the F1 score of VAD auxiliary head classification, respectively.

[0101] Table 1 shows that using only the Mel spectrogram modality leads to suboptimal results, while using only the audio modality exhibits quite good performance. However, the dual-modal approach significantly outperforms the single-audio modality in both F1 score and accuracy. This is because HuBERT, after extensive pre-training on large amounts of data, possesses powerful phoneme recognition capabilities. However, the Conformer's sensitivity to local details cannot be ignored: phoneme boundaries often manifest as instantaneous spectral changes, requiring features capable of capturing short-term abrupt changes. By combining the Conformer's robust temporal modeling capabilities, local receptive field advantages, and residual attention weighting mechanism, the synergistic advantages of the audio and Mel spectrogram dual-modality approach are achieved. It is evident that audio serves as the primary modality, while the spectral features of the Mel spectrogram play a supporting but indispensable role.

[0102] Table 2 Weights of Different Auxiliary Tasks Performance Comparison Table

[0103] Table 2 shows the weight metrics of different auxiliary tasks on the ST-CMDS and AISHELL-1 datasets. The comparison results show that when... The model performs best when the value is set to 0.1. This is because... When the accuracy is 0.1, the model prioritizes the main classification task while maintaining high accuracy in the auxiliary task. Furthermore, this embodiment provides a VAD-based head localization method; the auxiliary task has the characteristics of low cost and high return, therefore... The setting of =0.1 is appropriate and reasonable, realizing the complementary advantages between the main classification task and the auxiliary task.

[0104] In practical experiments, the prediction accuracy and F1 score of the VAD-assisted head localization method approached 100%, indicating that the model can accurately distinguish between silent and non-silent areas. In inference tasks, VAD prediction can force frames misclassified as non-silent to be corrected to silent frames. This step significantly filters out erroneous phoneme predictions, thereby improving overall accuracy. Furthermore, with VAD assistance, not only can the start and end boundaries of phonemes in the entire audio segment be predicted with high reliability, but the start and end boundaries of phonemes at both ends of brief silent segments in the audio can also be accurately located.

[0105] (3) In the inference task, the frame index is converted into a time index using the frame-time conversion formula provided in step S2, and the phoneme time boundary is determined by observing the change in phoneme category between consecutive frames. Specifically, when consecutive frames suddenly transition from one phoneme category to another, the point is marked as the time boundary. In addition, the accuracy of phoneme segmentation is improved by using a VAD auxiliary head for silence detection. Specifically, when the main classifier identifies a frame as a phoneme category, but the VAD auxiliary head determines it as a silent frame, this embodiment will force it to be corrected to the silent category based on the high confidence of the VAD identification. This effectively reduces the occurrence of false positive samples.

[0106] like Figure 3 and Figure 4 The results of phoneme segmentation on the AISHELL-1S and ST-CMDS datasets are shown respectively. Figure 3 and Figure 4 The first image in the image represents the Mel spectrogram corresponding to the audio signal. It can be seen that the Mel spectrogram also changes significantly at the boundaries of phonemes, which is precisely due to the advantage of this mode being sensitive to boundary details. Figure 3 and Figure 4 The second image in the diagram represents the original phoneme boundaries, and the third image shows the phoneme boundaries segmented by the model. A comparison reveals that the phoneme segmentation model proposed in this invention achieves remarkable results in phoneme segmentation, almost perfectly matching the original boundaries, which sufficiently demonstrates the superiority of the method presented in this invention.

[0107] Through the above experiments, the audio phoneme segmentation model proposed in this invention has achieved significant results in phoneme segmentation tasks. This model exhibits almost zero error in recognizing silent and non-silent frames, which is a significant advantage for identifying the time boundaries of phonemes on both sides of brief silent intervals. While performing phoneme segmentation, this invention also achieves high-precision phoneme recognition. Although this invention primarily focuses on Chinese audio data, deep learning models have strong transfer capabilities; for different languages, high-precision phoneme segmentation models can be trained using their corresponding monolingual audio data. This invention mainly focuses on single-speaker recognition, but in real-world dialogue scenarios, multi-speaker interaction accompanied by environmental noise is very common. Therefore, by continuously optimizing the model, it can automatically separate the signals of each speaker and perform corresponding phoneme segmentation in complex environments, while simultaneously enhancing the model's robustness to environmental noise to improve recognition performance under adverse conditions.

[0108] Example 2 An electronic device includes a memory and a processor, the memory being used to store a program that supports the processor in executing the deep learning-based Chinese audio phoneme segmentation method of Embodiment 1, and the processor being configured to execute the program stored in the memory.

[0109] Example 3 A storage medium storing a computer program, which, when executed by a processor, performs the steps of the deep learning-based Chinese audio phoneme segmentation method in Embodiment 1.

[0110] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning-based method for Chinese audio phoneme segmentation, characterized in that, Includes the following steps: Obtain the original waveform signal and corresponding spectrogram of the audio sample; A Chinese phoneme segmentation model is constructed, and the model is trained using audio samples and corresponding Mel spectrograms as input, including: The original waveform signal is input into the HuBERT encoder to extract waveform semantic features; Input the spectrogram into the Conformer encoder to extract the spectral acoustic features; Waveform semantic features and spectral acoustic features are input into the feature fusion module for cross-modal feature interaction to generate fused features; The fused features are input into the multi-scale temporal domain analysis module to extract multi-scale contextual features; Multi-scale contextual features are input into a gating network, and the multi-scale contextual features are adaptively selected by dynamically calculating the feature importance weights to generate boundary-sensitive features.

2. The Chinese audio phoneme segmentation method based on deep learning according to claim 1, characterized in that, The method further includes: Boundary-sensitive features are input into a classifier to detect speech activity and determine whether each frame is a silent frame; based on the determination result, frame-level phoneme classification is assisted; wherein, the classifier is a linear layer.

3. The Chinese audio phoneme segmentation method based on deep learning according to claim 1, characterized in that, The feature fusion module employs a residual cross-attention mechanism, specifically including the following: First, the waveform semantic features are mapped to a query matrix, and the spectral acoustic features are mapped to a key matrix and a value matrix; Next, the attention weights of the query matrix and the key matrix are calculated, and the value matrix is ​​weighted and aggregated according to the attention weights to obtain the attention fusion features; Then, the attention fusion features are residually connected with the waveform semantic features to generate fusion features.

4. The Chinese audio phoneme segmentation method based on deep learning according to claim 1, characterized in that, The multi-scale temporal analysis module includes at least two parallel dilated convolutional layers, each with a different dilation rate.

5. The Chinese audio phoneme segmentation method based on deep learning according to claim 4, characterized in that, The multi-scale contextual features are obtained through the following process: First, the fused features are processed through a one-dimensional convolutional layer. Perform downsampling; Next, the outputs of multiple parallel dilated convolutional layers are summed along the channel dimension; Then, the fused features are fused with the downsampled features through residual connections to generate multi-scale contextual features.

6. The Chinese audio phoneme segmentation method based on deep learning according to claim 1, characterized in that, The gated network includes a linear transformation layer and an activation function layer. The specific process for generating boundary-sensitive features is as follows: First, the multi-scale contextual features are input into the gating network, which outputs an importance weight vector, represented by the following logic: in, Weight coefficients generated for different dimensions , For multi-scale contextual features, , For feature embedding dimension, Represents a linear transformation layer. Indicates the activation function layer; Subsequently, the multi-scale contextual features are input into another linear transformation layer to obtain the transformed features. This can be represented using the following logic: Finally, the importance weight vector is compared with the transformed features. Perform element-wise multiplication to obtain boundary-sensitive features. : in, It represents the Hadamardi (or Hadama) stack.

7. The Chinese audio phoneme segmentation method based on deep learning according to claim 1, characterized in that, During model training, a combined loss function is used to optimize the model. This combined loss function consists of the loss function of the main classification task. Loss function for auxiliary classification tasks The weighted sum is obtained, represented as ,in To assist in task weighting.

8. The Chinese audio phoneme segmentation method based on deep learning according to claim 7, characterized in that, The loss function of the main classification task is expressed as follows: ,in For weighted focus loss, The weighted focus loss is represented using the following logic, which is the Dice loss. : in, and These are the balance factor and modulation factor for focus loss, respectively. For the model, the t-th sample is in the true category The predicted probability, For the real category Category weights, Let be the true label of the t-th sample. Used to reduce the loss of easily classified samples and to emphasize difficult-to-classify samples. The class weights representing cross-entropy This is a mask used to exclude ignored tags. It is an indicator function used to filter out samples from which loss calculations are not required. To ignore the labels, the Dice loss is represented using the following logic. : in, For feature embedding dimension, For the model to the first Each sample belongs to category The predicted probability, For the first One-Hot encoding of the true labels of each sample, when the true category is Time value Otherwise, it is 0; For category Corresponding weights A smoothing constant to prevent division by zero errors.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the deep learning-based Chinese audio phoneme segmentation method according to any one of claims 1 to 8, and the processor is configured to execute the program stored in the memory.

10. A storage medium storing a computer program, characterized in that, When the computer program is run by the processor, it performs the steps of the Chinese audio phoneme segmentation method based on deep learning as described in any one of claims 1 to 8.

Images