Multidimensional assessment method for cleft palate speech, model training method, and computer program product

Abstract

The present application relates to the technical field of speech signal processing and medical auxiliary diagnosis, and particularly relates to a cleft palate speech multi-dimensional evaluation method, a model training method and a computer program product. The evaluation method provided by the present application comprises: obtaining a cleft palate speech signal to be evaluated; inputting the cleft palate speech signal into a shared encoder to obtain an acoustic feature sequence, wherein the shared encoder is constructed based on a deep learning architecture, and the deep learning architecture comprises at least one of conformer architecture, a multi-layer bidirectional long short-term memory network or a transformer encoder backbone network; inputting the acoustic feature sequence into a vowel evaluation module, a consonant evaluation module and a vocabulary evaluation module respectively; outputting a vowel hypernasality grade through the vowel evaluation module, outputting a consonant pronunciation state through the consonant evaluation module, and outputting a vocabulary pronunciation correctness rate through the vocabulary evaluation module. The present application significantly improves the objectivity, accuracy and consistency of evaluation.

Description

Technical Field

[0001] This invention relates to the fields of speech signal processing and medical auxiliary diagnosis technology, and more specifically, to a multi-dimensional assessment method for cleft palate speech, a training method for a cleft palate speech assessment model, and a computer program product. Background Technology

[0002] In the clinical diagnosis and rehabilitation of patients with cleft palate, speech assessment is a crucial basis for developing treatment plans and evaluating efficacy. Currently, clinical judgment mainly relies on the subjective auditory perception of speech therapists. This method is not only time-consuming, but the assessment results are also highly dependent on the assessor's personal experience, exhibiting significant subjective variability and instability, making it difficult to establish unified quantitative standards. To overcome the limitations of subjective assessment, existing technologies attempt to introduce computer-aided techniques, but most are limited to the identification of single pathological features, such as detecting only specific compensatory pronunciation errors like nasal air leakage, pharyngeal fricatives, or laryngeal plosives. These methods typically use manually designed acoustic features combined with traditional machine learning classifiers, resulting in a single assessment dimension that cannot simultaneously cover multiple core clinical indicators such as excessive nasality of vowels, consonant articulation status (e.g., weakening or dropping), and word pronunciation accuracy. Furthermore, existing general speech assessment schemes based on deep learning are mostly designed for normal speech or general voice disorders, and their model architectures often lack the ability to specifically extract complex pathological features unique to cleft palate speech, such as high-frequency transient loss and abnormal low-frequency nasal energy. Especially when processing continuous speech, the pronouncements of cleft palate patients deviate significantly from standard text, making commonly used forced alignment techniques prone to failure. This leads to an inability to accurately locate the speech segment to be evaluated, thus affecting the accuracy of the assessment. Therefore, there is an urgent need for an assessment method that can integrate multi-dimensional assessment tasks, adaptively extract pathological features of cleft palate, and provide objective quantitative results. Summary of the Invention

[0003] To solve, or at least partially solve, the above-mentioned technical problems, the present invention provides a method and computer program product for multi-dimensional assessment of speech in cleft palate.

[0004] This invention provides a multi-dimensional assessment method for speech in patients with cleft palate, comprising: Acquire the speech signal of the cleft palate to be evaluated; The cleft palate speech signal is input into a shared encoder to obtain an acoustic feature sequence. The shared encoder is built based on a deep learning architecture, which includes at least one of a conformal architecture, a multi-layer bidirectional long short-term memory network architecture, or a converter encoder backbone network architecture. The acoustic feature sequences are respectively input into the vowel evaluation module, consonant evaluation module, and vocabulary evaluation module; The vowel evaluation module outputs the vowel over-nasalization level, the consonant evaluation module outputs the consonant pronunciation status, and the vocabulary evaluation module outputs the vocabulary pronunciation accuracy.

[0005] This method constructs a shared encoder based on a conformal architecture to simultaneously extract global contextual dependencies and local spectral patterns from cleft palate speech signals, effectively capturing the unique pathological acoustic features of cleft palate speech. By inputting the extracted acoustic feature sequences in parallel into assessment modules designed for vowels, consonants, and words at three different granularities, multi-task collaborative processing is achieved, solving the problem that existing technologies have a single assessment dimension and cannot comprehensively reflect the patient's speech status. This method can provide comprehensive quantitative indicators for vowel over-nasalization level, consonant pronunciation status, and word pronunciation accuracy, significantly improving the objectivity, consistency, and clinical guidance value of the assessment.

[0006] Optionally, the acquired raw speech signal is preprocessed to obtain the Mel spectrum feature matrix; wherein, the preprocessing includes pre-emphasis processing, frame windowing processing, and Mel filter bank filtering processing performed sequentially.

[0007] By using a standardized preprocessing procedure, the original time-domain speech signal is converted into a frequency-domain representation that is adapted to the pathological features of cleft palate. This effectively improves the recognizability of high-frequency transient features and nasal formants in the signal, providing a high-quality input foundation for subsequent deep feature extraction.

[0008] As a preferred approach, to capture the high-frequency transient features and nasal formants of cleft palate speech, the pre-emphasis processing employs a transfer function. ,in This is the pre-emphasis coefficient. The delay operator is used; the frame length for the frame-by-frame windowing process is set to 25ms, the frame shift is set to 10ms, and the window function is a Hamming window.

[0009] By using a clearly defined mathematical model and specific parameter configuration, the weight of the high-frequency component is specifically enhanced to capture the transient features of consonants, while ensuring the stability of the intra-frame signal to facilitate the analysis of nasal formants. This significantly improves the sensitivity to capturing subtle pathological features such as consonant weakening and excessive nasality.

[0010] Optionally, speech data of cleft palate patients reading a pre-set corpus are collected. The pre-set corpus includes six basic vowel phonetic symbols for assessing the degree of excessive nasality, twenty-six Mandarin consonant phonetic symbols for assessing pronunciation state, and one hundred commonly used words for assessing pronunciation accuracy. In the Mel filter bank filtering process, the number of Mel filter banks is set to eighty to obtain a dimension of 80× The feature matrix of , where The number of frames.

[0011] It ensures a high degree of consistency between the data collection content and the multi-dimensional evaluation task, achieving full coverage of pathological features; at the same time, the specific feature matrix dimension configuration achieves a balance between frequency domain resolution and computational complexity, taking into account both the richness of pathological features and the efficiency of model processing.

[0012] As a preferred embodiment, the cleft palate speech signal is input into multiple sequentially stacked conformal blocks for processing. During the processing of each conformal block, the following operations are performed sequentially: a first feedforward neural network is used to perform a nonlinear transformation on the input data to obtain first intermediate data; a multi-head self-attention mechanism is used to capture global dependencies in the first intermediate data to obtain second intermediate data; a convolution operation is used to extract local features from the second intermediate data to obtain third intermediate data; a second feedforward neural network is used to perform another nonlinear transformation on the third intermediate data to obtain the output data of the current conformal block; the output data of the current conformal block is processed by layer normalization and used as the input data of the next level conformal block, until the output data of the last conformal block is used as the acoustic feature sequence.

[0013] By serially fusing global contextual information and local spectral patterns during data flow, a multi-layered feature representation is formed. Combined with layer normalization, the data distribution of deep networks is stabilized, significantly improving the extraction accuracy of complex temporal structures and spectral distortion features of cleft palate speech.

[0014] Optionally, the input data is processed sequentially through a first linear layer, a nonlinear activation function layer, and a second linear layer; the third intermediate data is processed sequentially through a first linear layer, a nonlinear activation function layer, and a second linear layer; and the second intermediate data is processed sequentially through pointwise convolution, depthwise separable convolution, and gated linear units.

[0015] The algorithm kernel for nonlinear mapping and local feature extraction has been refined. While reducing computational complexity, it dynamically adjusts and efficiently extracts local spectral distortion features, thereby enhancing the model's sensitivity to subtle acoustic differences.

[0016] As a preferred embodiment, the shared encoder is a multi-layer bidirectional long short-term memory network architecture, or the shared encoder is a transducer encoder backbone network architecture. When the shared encoder is a multi-layer bidirectional long short-term memory network, the cleft palate speech signal is input into the multi-layer bidirectional long short-term memory network for temporal feature extraction to obtain initial temporal features. The initial temporal features are then mapped using a feature adaptation layer to transform them into a unified feature space, resulting in the acoustic feature sequence. When the shared encoder is a transducer encoder backbone network, the cleft palate speech signal is input into the transducer encoder backbone network. The transducer encoder backbone network is used to process the cleft palate speech signal using a self-attention mechanism to extract deep acoustic features and directly output them as the acoustic feature sequence.

[0017] It provides alternative feature extraction paths, which can capture long temporal context through bidirectional recursion or directly establish long-range correlations between different parts of the input signal through global self-attention mechanism, thereby enhancing the applicability of the technical solution under different computing power environments or temporal modeling requirements.

[0018] Optionally, attention statistical pooling is performed on the acoustic feature sequence to obtain a vowel feature vector; the vowel feature vector is then input into a classifier to obtain the vowel over-nasalization level.

[0019] Adaptive weighted aggregation highlights speech frames relevant to nasal assessment, enabling specific feature extraction for the vowel dimension and enhancing the model's sensitivity to pathological features of excessive nasality in cleft palate speech.

[0020] As a preferred embodiment, the acoustic feature sequence is obtained by concatenating the output features of multiple conformal blocks of the shared encoder and then performing layer normalization processing; the attention statistical pooling processing of the acoustic feature sequence includes calculating the weight of each frame feature in the acoustic feature sequence through trainable parameters, and performing a weighted average calculation of the features of each frame based on the weights; the vowel over-nasalization level includes four levels: none, light, medium, and heavy, and the classifier outputs the probability distribution of the four levels.

[0021] It integrates multi-resolution features of different network depths and automatically focuses on speech frames that contribute more to nasal sound assessment through a learnable attention mechanism, clarifying the grading intervals of pathological severity and providing confidence output, thereby enhancing the clinical guidance significance of the assessment results.

[0022] Optionally, using a forced alignment technique, the acoustic feature sequence is matched with a standard text phoneme sequence to determine the start and end time boundaries of the consonant segment to be evaluated; a consonant segment feature sequence located between the start and end time boundaries is extracted from the acoustic feature sequence; a temporal average pooling operation is performed on the consonant segment feature sequence to obtain a fixed-dimensional consonant segment representation vector; the consonant segment representation vector is input into a three-classifier to output the consonant pronunciation state, which includes a normal state, a weakened state, and a dropped state.

[0023] By using forced alignment technology, the specific position of the consonant to be evaluated on the time axis was accurately located, eliminating the interference of irrelevant speech segments. The overall acoustic information of the consonant segment was aggregated through temporal average pooling, enabling refined automatic identification of cleft palate-specific pathological conditions such as consonant weakening and dropout.

[0024] As a preferred embodiment, when using the forced alignment technique for matching, a blank label is introduced to adjust the state transition probability. The blank label is used to tolerate the difference in phoneme sequence length caused by the absence of pronunciation in the cleft palate speech signal. The three-classifier is trained based on the cross-entropy loss function, which is calculated based on the difference between the true label of the consonant pronunciation state and the probability distribution output by the three-classifier.

[0025] By adding an empty state transition path to the state transition model, alignment deviations caused by consonant omissions are effectively tolerated, enhancing the algorithm's robustness to pathological speech. At the same time, a specific loss function configuration strengthens the classifier's ability to distinguish between the three pronunciation states, ensuring the reliability of the evaluation results.

[0026] Optionally, the start and end time boundaries of the words to be evaluated in the cleft palate speech signal are determined; based on the start and end time boundaries, a lexical segment feature sequence corresponding to the words to be evaluated is extracted from the acoustic feature sequence; the lexical segment feature sequence is aggregated using a multi-head attention mechanism to obtain a lexical-level feature vector; based on the lexical-level feature vector, a binary classifier is used to output the pronunciation correctness judgment result of the words to be evaluated; and the pronunciation correctness judgment results of all the words to be evaluated are statistically analyzed to obtain the pronunciation accuracy rate of the words.

[0027] By accurately locating word boundaries and adaptively focusing on key pathological feature frames within word segments using a multi-head attention mechanism, end-to-end refined evaluation at the word level is achieved, improving the sensitivity to word pronunciation errors and providing a quantitative indicator of word pronunciation accuracy.

[0028] As a preferred approach, the lexical segment feature sequence is mapped to a query matrix, a key matrix, and a value matrix, respectively; the attention weight distribution of the query matrix and the key matrix is ​​calculated; and the value matrix is ​​weighted and summed based on the attention weight distribution to obtain the lexical-level feature vector.

[0029] By constructing the linear transformation space required for the attention mechanism and dynamically allocating weights based on the correlation between features, key pathological acoustic features are highlighted while irrelevant background information is suppressed. This outperforms a fixed average pooling strategy and improves the accuracy of feature aggregation.

[0030] Optionally, the vocabulary-level feature vector is input into a logistic activation function to obtain the pronunciation correctness probability; the pronunciation correctness probability is compared with a preset threshold; if the pronunciation correctness probability is greater than or equal to the preset threshold, the pronunciation correctness judgment result is correct; if the pronunciation correctness probability is less than the preset threshold, the pronunciation correctness judgment result is incorrect.

[0031] It provides clear and explicit binary classification criteria for word pronunciation, transforming continuous probability values ​​into explicit correct or incorrect judgments, ensuring the consistency and interpretability of the assessment results, and making it easy for clinicians to understand and use.

[0032] As a preferred approach, a forced alignment technique is used to align the cleft palate speech signal with a preset 100-word list text at the phoneme level; based on the phoneme timestamps in the alignment results, the start and end timestamps of the words to be evaluated are aggregated and used as the start and end time boundaries.

[0033] Determining coarse-grained lexical boundaries based on fine-grained phoneme boundaries improves the robustness of lexical boundary localization under pathological speech conditions of cleft palate, provides a reliable time reference for subsequent feature extraction, and reduces evaluation bias caused by alignment errors.

[0034] Optionally, a labeled training dataset is obtained, which includes training speech signals and their corresponding vowel over-nasalization level labels, consonant pronunciation state labels, and vocabulary pronunciation accuracy labels; the training speech signals are input into the shared encoder, vowel evaluation module, consonant evaluation module, and vocabulary evaluation module to be trained to obtain prediction results; based on the prediction results and the corresponding labels, vowel task loss, consonant task loss, and vocabulary task loss are calculated respectively; based on the vowel task loss, consonant task loss, and vocabulary task loss, a total loss function is constructed; based on the total loss function, the parameters of the shared encoder, vowel evaluation module, consonant evaluation module, and vocabulary evaluation module are updated using a backpropagation algorithm.

[0035] By synchronously updating the parameters of the shared encoder and the three evaluation modules through a joint optimization mechanism, end-to-end multi-task collaborative training is achieved, ensuring that the shared encoder can learn a general acoustic feature representation compatible with the three evaluation tasks, thereby improving the overall generalization ability of the model.

[0036] As a preferred embodiment, the vowel task loss is calculated using a cross-entropy loss function based on the vowel over-nasalization level label and the probability distribution output by the vowel evaluation module; the consonant task loss is calculated using a cross-entropy loss function based on the consonant pronunciation state label and the probability distribution output by the consonant evaluation module; and the vocabulary task loss is calculated using a binary cross-entropy loss function based on the vocabulary pronunciation accuracy label and the binary classification probability output by the vocabulary evaluation module.

[0037] By matching specific loss calculation methods to the data distribution characteristics of different tasks, the prediction errors of each module are accurately quantified, making gradient backpropagation more accurate and accelerating the convergence speed of the model on specific pathological features.

[0038] Optionally, a preset task weight coefficient is obtained, which includes vowel weight, consonant weight, and vocabulary weight; the product of the vowel task loss and the vowel weight, the product of the consonant task loss and the consonant weight, and the product of the vocabulary task loss and the vocabulary weight are summed to obtain the total loss function; wherein, the task weight coefficient is configured to balance the gradient magnitude differences among the vowel task loss, the consonant task loss, and the vocabulary task loss.

[0039] By adjusting the weighting coefficients to adjust the contribution ratio of each task's loss to the total loss, gradient conflict and negative transfer problems in multi-task training are effectively alleviated, ensuring a balanced improvement in evaluation performance across the three dimensions of vowels, consonants, and vocabulary, and avoiding performance bottlenecks in a single task.

[0040] As a preferred embodiment, the steps of acquiring the cleft palate speech signal to be evaluated and subsequently inputting the cleft palate speech signal into a shared encoder to output the evaluation result are performed locally by the mobile terminal device; wherein, the shared encoder is a lightweight conformal encoder, which has fewer layers than the standard conformal architecture, and the lightweight conformal encoder is trained by knowledge distillation technology, wherein the teacher model of the knowledge distillation technology is a standard conformal architecture model with more parameters than the lightweight conformal encoder.

[0041] This enables a high-precision, multi-dimensional assessment method for cleft palate speech to run offline and in real time on resource-constrained mobile devices, significantly reducing reliance on cloud computing power and network latency. It improves diagnostic and treatment efficiency and patient privacy in primary healthcare settings, while maintaining high sensitivity to pathological features of cleft palate through knowledge distillation.

[0042] Optionally, the step of acquiring the cleft palate speech signal to be evaluated is performed by the user terminal, and the step of inputting the cleft palate speech signal to the shared encoder to output the evaluation result is performed by the cloud server; the method further includes: the user terminal uploading the acquired cleft palate speech signal to the cloud server; the cloud server generating a structured multi-dimensional evaluation report and distributing the structured multi-dimensional evaluation report to the user terminal.

[0043] It breaks through the limitations of geographical location and terminal hardware performance, enabling users with low-configuration devices to enjoy standardized cleft palate speech assessment services based on high-precision deep learning models, promoting the downward flow of high-quality medical resources, and supporting long-term home rehabilitation monitoring and data archiving.

[0044] In summary, this invention systematically solves the technical problems of existing technologies, such as the single dimension of cleft palate speech assessment, insufficient extraction of pathological features, and difficulty in aligning continuous speech, by constructing a technical solution that combines a shared encoder based on a conformal architecture with a multi-dimensional assessment module. This solution achieves end-to-end mapping from raw speech signals to quantitative assessment results across three dimensions: vowels, consonants, and vocabulary. This not only significantly improves the objectivity, accuracy, and consistency of the assessment but also adapts to diverse application scenarios, from clinical outpatient clinics to home rehabilitation, through flexible deployment, providing strong technical support for the accurate diagnosis and rehabilitation of cleft palate speech. Attached Figure Description

[0045] To more clearly illustrate the embodiments of the present invention, the relevant accompanying drawings will be briefly described below. It should be understood that the drawings described below are only for illustrating some embodiments of the present invention, and those skilled in the art can obtain many other technical features and connections not mentioned herein based on these drawings.

[0046] Figure 1 This is a flowchart of a multi-dimensional assessment method for speech in cleft palate according to an embodiment of the present invention; Figure 2 This is a flowchart of the evaluation process for the level of excessive nasality in vowels according to an embodiment of the present invention; Figure 3 This is a flowchart of the evaluation process for consonant articulation states according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the evaluation of vocabulary pronunciation accuracy according to an embodiment of the present invention; Figure 5 This is a flowchart of model training according to an embodiment of the present invention. Detailed Implementation

[0047] The present invention will now be described in detail with reference to the accompanying drawings.

[0048] In the clinical practice of speech therapy for cleft palate, accurate, objective, and comprehensive speech assessment is crucial for developing surgical plans, planning speech therapy pathways, and evaluating rehabilitation outcomes. Due to velopharyngeal closure dysfunction, patients with cleft palate exhibit highly complex and multidimensional speech disorders, typically characterized by excessive nasalization of vowel sounds, weakening or loss of consonant sounds, and a decline in overall vocabulary clarity. Currently, clinical assessment primarily relies on the subjective auditory perception of speech therapists. However, this traditional approach has significant limitations: firstly, assessment results are highly dependent on the assessor's personal experience and auditory sensitivity, leading to significant differences in judgments between different assessors and even within the same assessor at different times, lacking a unified quantitative standard; secondly, manual assessment is time-consuming, making it difficult to meet the needs of large-scale screening or high-frequency rehabilitation monitoring.

[0049] With the development of artificial intelligence technology, assisted assessment schemes based on automatic speech recognition (ASR) have gradually attracted attention. In this field, a typical implementation uses an architecture combining deep neural networks and hidden Markov models (DNN-HMM). Such schemes typically train acoustic models using large-scale normal speech data, perform phoneme-level matching between input speech and text using forced alignment techniques, and then extract manual or shallow acoustic features such as fundamental frequency jitter, amplitude jitter, and Mel-frequency cepstral coefficients. Finally, these features are input into a classifier to obtain a score for voice problems. In addition, some schemes attempt to use convolutional neural networks or recurrent neural networks to extract frame-level features of continuous speech, and then determine the severity of articulation disorders through statistical pooling.

[0050] However, the aforementioned existing technical solutions still face inherent technical contradictions when dealing with the specific pathological speech of cleft palate. First, existing general speech assessment models are mostly designed for normal speech or general pathological voices. Their feature extraction backbone networks often struggle to simultaneously capture the subtle high-frequency transient features (such as plosives before consonant dropout) and low-frequency energy anomalies (such as formant shifts caused by vowel nasalization) in cleft palate speech. General pooling strategies (such as global average pooling) tend to smooth out these crucial pathological details, resulting in insufficient sensitivity of the model to mild or specific types of cleft palate speech disorders. Second, cleft palate speech assessment is a typical multi-granularity, multi-task problem, requiring simultaneous processing of four levels of vowel severity, three states of consonants, and binary classification accuracy of words. Existing solutions typically employ serial processing or independent single-task models. During joint training, they often face negative transfer problems caused by differences in gradient magnitudes. The lack of a unified shared encoder to collaboratively learn common acoustic representations across different tasks not only increases computational redundancy but also makes model convergence difficult due to gradient conflicts between tasks, hindering the achievement of balanced, high-precision multi-dimensional evaluation within a single framework. Furthermore, cleft palate patients often have missing or severely deformed phonemes. Traditional forced alignment algorithms, based on the monotonicity assumption, are prone to alignment deviations when dealing with such atypical pronunciations, leading to subsequent feature extraction errors and severely impacting the reliability of evaluation results.

[0051] To address the aforementioned issues, those skilled in the art might consider adding feature dimensions or employing more complex independent models to handle each evaluation task separately. However, simply stacking features or models not only significantly increases computational complexity, making it difficult to deploy on mobile terminals or primary healthcare devices, but also fails to fundamentally solve the problems of feature sharing and gradient balancing in multi-task learning, and may even reduce generalization ability due to model overfitting. Therefore, how to construct an end-to-end evaluation method that can simultaneously cover the three dimensions of vowels, consonants, and vocabulary, accurately capture the pathological acoustic features unique to cleft palate, achieve multi-task collaborative optimization, and possess good generalization ability has become a pressing technical challenge in this field.

[0052] In view of this, the embodiments of the present invention aim to provide a multi-dimensional assessment method for speech in cleft palate, in order to solve or at least partially alleviate the above-mentioned technical problems.

[0053] First Implementation Method This embodiment provides a multi-dimensional assessment method for speech in cleft palate patients. This method can be applied to terminal devices such as smartphones, tablets, computers, or dedicated medical assessment instruments. The method aims to achieve automated, multi-dimensional pathological feature analysis of speech signals from cleft palate patients through a unified deep learning architecture.

[0054] like Figure 1As shown, firstly, the speech signal of the cleft palate to be evaluated is acquired. In a specific embodiment, the process of acquiring the speech signal of the cleft palate to be evaluated may include acquiring the patient's vocal audio through a microphone array. The acquired audio may be isolated vocalizations or continuous speech sequences. To adapt to subsequent neural network processing, the acquired raw speech signal may be preprocessed to obtain a Mel-spectral feature matrix adapted to the pathological characteristics of the cleft palate.

[0055] Specifically, the preprocessing process may include sequential pre-emphasis processing, frame-by-frame windowing processing, and Mel filter bank filtering. The purpose of pre-emphasis processing is to enhance the high-frequency components of the speech signal. This is because the transient features of consonants in cleft palate speech (such as plosives) are often concentrated in the high-frequency region, while the energy of nasal formants is mainly concentrated in the low-frequency region. Pre-emphasis can balance the spectrum, making the high-frequency features more prominent. In a specific embodiment, pre-emphasis processing can employ a transfer function... .in, For delay operators, it represents the unit of time delay of a signal; This is the pre-emphasis coefficient, and its value can be adjusted according to the specific application scenario. For example, in a preferred embodiment, The value of can be 0.97. This transfer function indicates that the output signal at the current time is equal to the input signal at the current time minus the input signal at the previous time. This multiplies the filtering speed, thus achieving the effect of high-pass filtering.

[0056] After pre-emphasis processing, frame-segmentation and windowing can be performed. Since speech signals are non-stationary, but can be approximated as stationary signals for a very short time (e.g., tens of milliseconds), it is necessary to segment the continuous speech signal into short frames. In a specific embodiment, the frame length for frame-segmentation and windowing can be set to 25ms, and the frame shift can be set to 10ms. The reason for setting the frame length to 25ms is that this time length is sufficient to capture the fundamental frequency period and formant structure of the speech signal, while being short enough to ensure the stationarity of the signal within the frame. Setting the frame shift to 10ms means that there is a 15ms overlap between adjacent frames. This overlap can effectively reduce information loss caused by frame segmentation and ensure the temporal continuity of the feature sequence. The window function can be a Hamming window, a commonly used window function with rapid sidelobe decay, which can effectively reduce spectral leakage and improve the accuracy of spectral analysis.

[0057] Subsequently, each frame of the windowed signal can be filtered using a Mel filter bank. The Mel filter bank simulates the non-linear frequency perception characteristics of the human ear, exhibiting high resolution in the low-frequency region and low resolution in the high-frequency region. In a specific embodiment, the number of Mel filter banks can be set to eighty. Through this processing, a dimension of 80× can be obtained. The feature matrix of , where 80 represents the number of frames and the number of feature points in the frequency dimension. Setting the number of Mel filter banks to 80 strikes a balance between frequency domain resolution and computational complexity. A larger number of filters can provide finer spectral details, helping to capture subtle pathological features in cleft palate speech, such as slight nasalization or consonant weakening, but it also increases the computational burden on subsequent models; while 80 filters are generally considered an optimal value that provides sufficient information in speech recognition tasks.

[0058] To support the aforementioned assessment task, it is necessary to collect speech data of cleft palate patients reading a pre-set corpus as input or training data. In one specific embodiment, the pre-set corpus may include six basic vowel phonetic symbols for assessing the degree of excessive nasality, twenty-six Mandarin consonant phonetic symbols for assessing articulation state, and one hundred commonly used words for assessing pronunciation accuracy. The six basic vowel phonetic symbols may be, for example,... These vowels cover the main formant space of oral articulation, effectively reflecting the degree of nasalization caused by velopharyngeal closure insufficiency. The twenty-six Mandarin consonant phonetic symbols cover multiple articulation points, including labial, lingual, and dental consonants, facilitating a comprehensive assessment of consonant articulation. One hundred commonly used words simulate everyday communication scenarios to assess the patient's pronunciation accuracy in continuous speech. This corpus design ensures a targeted correspondence between data collection and the three subsequent assessment tasks (vowels, consonants, and vocabulary), avoiding assessment biases that might arise from general corpora.

[0059] After obtaining the preprocessed Mel-spectral feature matrix, the cleft palate speech signal is input into a shared encoder to obtain an acoustic feature sequence. In one specific embodiment, the shared encoder can be constructed based on a conformal architecture. A conformal architecture is a deep learning model that combines convolutional neural networks and self-attention mechanisms. It can simultaneously capture local features and global dependencies of speech signals, making it highly suitable for processing cleft palate speech signals that contain both local transient anomalies and long-term prosodic anomalies.

[0060] Specifically, the process of inputting the cleft palate speech signal into a shared encoder to obtain an acoustic feature sequence may include: inputting the cleft palate speech signal into multiple sequentially stacked conformal blocks for processing. A conformal block is the basic building block of a conformal architecture; stacking multiple conformal blocks can form a deep network, thereby extracting higher-order abstract features. During the processing of each conformal block, the following operations may be performed sequentially: The first step involves using a first feedforward neural network to perform a nonlinear transformation on the input data, obtaining the first intermediate data. The feedforward neural network is used to perform point-by-point nonlinear mapping on the features, enhancing the expressive power of the model.

[0061] The second step involves using a multi-head self-attention mechanism to capture global dependencies in the first intermediate data, resulting in the second intermediate data. This mechanism allows the model to simultaneously focus on features from other points in the sequence while processing features at the current time step, thus capturing long-range contextual dependencies. This is particularly important for cleft palate speech evaluation, as the nasalization of vowels often requires integral evaluation over the entire vowel segment, while the articulation of consonants may be influenced by the co-articulation of preceding and following phonemes.

[0062] The third step involves using convolution operations to extract local features from the second intermediate data, resulting in the third intermediate data. Convolution operations are translation-invariant and excel at capturing local spectral patterns, such as spectral abrupt changes during consonant plosives or the specific shapes of nasal formants. The conformal architecture, by introducing convolution modules, compensates for the shortcomings of pure self-attention mechanisms in local feature extraction.

[0063] The fourth step involves using a second feedforward neural network to perform another nonlinear transformation on the third intermediate data to obtain the output data of the current conformal block.

[0064] Finally, the output data of the current conformal block is processed by layer normalization and used as the input data for the next level conformal block, until the output data of the last conformal block is used as the acoustic feature sequence. Layer normalization is used to stabilize the training process of deep networks, prevent gradient vanishing or exploding, and ensure that pathological features can be effectively accumulated without being submerged during network transmission.

[0065] To more clearly illustrate the internal data processing logic of the conformal block, a specific mathematical expression is provided below. Assume the... The input of each conformal block is The output is The calculation process can be expressed as: in, , , These represent intermediate data processed by different modules; 1 represents the first feedforward neural network; MHSA represents the multi-head self-attention mechanism; Conv represents the convolution operation; This represents the second feedforward neural network; LayerNorm represents layer normalization; 0.5 is a scaling factor used to stabilize gradient flow at residual connections. This formula illustrates the structure using residual connections within the conformal block, which facilitates the direct transfer of information in deep networks and avoids feature loss during multiple nonlinear transformations.

[0066] Furthermore, specific algorithmic implementations can be adopted for each sub-step in the aforementioned conformal block. In a specific embodiment, the nonlinear transformation of the input data using the first feedforward neural network may include processing the input data sequentially through a first linear layer, a nonlinear activation function layer, and a second linear layer. The linear layer is used to transform the dimensions, for example, by first enlarging and then compressing the dimensions. The nonlinear activation function layer is used to introduce nonlinear factors; commonly used activation functions can be the Swish function or the ReLU function. Similarly, the nonlinear transformation of the third intermediate data again using the second feedforward neural network may also include processing the third intermediate data sequentially through a first linear layer, a nonlinear activation function layer, and a second linear layer. This structure is commonly referred to as a feedforward network module, which is widely used in Transformer and its variant architectures to enhance the model's feature mapping capabilities.

[0067] In one specific embodiment, the local feature extraction of the second intermediate data using convolution operations may include processing the second intermediate data sequentially through pointwise convolution, depthwise separable convolution, and gated linear units. Pointwise convolution is used to adjust the number of channels, depthwise separable convolution is used to extract spatial features while reducing computational load, and gated linear units are used to control the flow of information, similar to the gating mechanism in long short-term memory networks, dynamically selecting which local features to retain or discard. This convolution module design improves the ability to recognize local abnormal features (such as transient noise and spectral distortion) in cleft palate speech while ensuring computational efficiency.

[0068] In addition to the shared encoder based on a conformal architecture, this implementation also provides alternative feature extraction schemes to adapt to different computing power environments or timing modeling requirements. In a specific embodiment, the shared encoder can be a multi-layer bidirectional long short-term memory network architecture, or the shared encoder can be a converter encoder backbone network architecture.

[0069] When the shared encoder is a multi-layer bidirectional long short-term memory (BSSM) network architecture, inputting the cleft palate speech signal into the shared encoder to obtain an acoustic feature sequence may include: inputting the cleft palate speech signal into the multi-layer bidirectional BSSM network for temporal feature extraction to obtain initial temporal features. The bidirectional BSSM network can simultaneously utilize past and future contextual information to process the current input, making it very suitable for processing temporal signals such as speech. Subsequently, a feature adaptation layer is used to map the initial temporal features, transforming them into a unified feature space to obtain the acoustic feature sequence. The function of the feature adaptation layer is to adjust the feature dimension output by the bidirectional BSSM network to a dimension compatible with subsequent evaluation modules, ensuring consistency of feature spaces under different encoder architectures.

[0070] When the shared encoder is a transducer encoder backbone network architecture, inputting the cleft palate speech signal into the shared encoder to obtain the acoustic feature sequence may include: inputting the cleft palate speech signal into the transducer encoder backbone network. The transducer encoder primarily relies on a self-attention mechanism and lacks convolutional inductive bias, but it often achieves excellent performance under large-scale data training. Subsequently, the transducer encoder backbone network is used to process the cleft palate speech signal using a self-attention mechanism to extract deep acoustic features and directly output them as the acoustic feature sequence. This approach is suitable for scenarios with sufficient computing power and large amounts of data, and can capture deeper semantic and acoustic associations.

[0071] This implementation method enables the system to generate high-quality acoustic feature sequences adapted to the pathological characteristics of cleft palate. Specifically, by using specific preprocessing parameters (such as a 25ms frame length and an 80-Vimel filter), the recognizability of high-frequency transient features and nasal formants in the signal is enhanced; by using a shared encoder based on a conformal architecture, the fusion extraction of global contextual information and local spectral patterns is achieved; and by providing multiple encoder alternatives, the applicability of the technical solution in different hardware environments is enhanced. These acoustic feature sequences provide a solid foundation for subsequent multi-dimensional evaluation tasks of vowels, consonants, and vocabulary, ensuring that the subsequent classification module can determine the pathological state based on rich and accurate feature inputs.

[0072] Second Implementation Method Building upon the first embodiment, this embodiment further improves the multi-dimensional assessment method for cleft palate speech by adding specific assessment modules for vowels, consonants, and vocabulary. Based on the acoustic feature sequence output by the shared encoder in the first embodiment, this embodiment executes three assessment tasks in parallel to achieve a comprehensive, automated, and quantitative assessment of the pathological features of cleft palate speech.

[0073] First, regarding the assessment of vowel hypernasalization levels. In a specific embodiment, such as... Figure 2 As shown, the process of outputting the vowel over-nasalization level through the vowel evaluation module may include performing attention statistical pooling on the acoustic feature sequence to obtain a vowel feature vector, and then inputting the vowel feature vector into a classifier to obtain the vowel over-nasalization level.

[0074] Specifically, to fully utilize the feature information at different levels in the shared encoder, the acoustic feature sequence can be obtained by concatenating the output features of multiple conformal blocks of the shared encoder and then performing layer normalization. For example, the output features of the conformal blocks at layers 1, 6, and 12 can be concatenated along the channel dimension. Shallow features typically contain more acoustic detail information, such as the fine structure of nasal formants, while deep features contain more semantic and contextual information. By concatenating multi-resolution features, the feature type requirements of different evaluation tasks can be simultaneously considered, improving evaluation accuracy. The concatenated features undergo layer normalization, which stabilizes the feature distribution and eliminates dimensional differences between features at different layers.

[0075] Subsequently, the acoustic feature sequence is subjected to attentional statistical pooling. In one specific embodiment, this process includes calculating the weight of each frame feature in the acoustic feature sequence using trainable parameters, and performing a weighted average of the frame features based on the weights. The attentional statistical pooling mechanism allows the model to adaptively focus on speech frames that contribute more to nasal sound evaluation, such as the core stable segments of vowels, while automatically suppressing interference from silent segments or transitional segments.

[0076] In a specific implementation example, the calculation process of attention statistical pooling can be represented by the following formula: in, Represents the first in the acoustic feature sequence The feature vector of the frame; This is the first trainable weight matrix used to perform a linear transformation on the input features; This is the first trainable bias vector; This is the transpose of the trainable second weight vector, used to map the result of the nonlinear transformation to a scalar fraction; is the trainable second bias scalar; tanh is the hyperbolic tangent activation function, used to introduce nonlinear factors; Indicates the first Unnormalized importance score of frame features; exp represents the exponential function; Indicates the first The normalized attention weights for frame features are such that the sum of the weights for all frames is 1. This represents the final vowel feature vector, which is the aggregated feature after weighted averaging. Using this formula, the model can learn which time frames are most critical for determining the degree of nasality, thus achieving soft feature selection.

[0077] After obtaining the vowel feature vectors, they are input into a classifier. In one specific embodiment, the level of excessive nasality in vowels includes four levels: none, mild, moderate, and severe. These four levels correspond to the typical clinical grading standards for the degree of nasalization in cleft palate speech. The classifier outputs the probability distribution of the four levels. For example, the classifier can be a fully connected layer followed by a Softmax activation function, outputting a four-dimensional vector, where each dimension represents the probability of belonging to the corresponding level. Finally, the level with the highest probability can be selected as the evaluation result, or the complete probability distribution can be output for the doctor's reference. Compared with simple binary classification (with / without nasality), this four-level classification design can more delicately reflect the severity of the patient's pathology, providing richer evidence for surgical planning or evaluation of rehabilitation effects.

[0078] Secondly, regarding the assessment of consonant articulation states. In a specific embodiment, such as... Figure 3 As shown, the process of outputting the consonant pronunciation state through the consonant evaluation module may include using a forced alignment technique to match the acoustic feature sequence with a standard text phoneme sequence to determine the start and end time boundaries of the consonant segment to be evaluated. The forced alignment technique can establish a precise temporal correspondence between the speech signal and the text phonemes, thereby accurately locating the position of a specific consonant in a continuous acoustic feature sequence.

[0079] Subsequently, consonant segment feature sequences located between the start and end time boundaries are extracted from the acoustic feature sequence. This step eliminates interference from irrelevant speech segments (such as vowel segments or silence segments), ensuring that subsequent analysis focuses solely on the target consonant. Next, temporal average pooling is performed on the consonant segment feature sequence to obtain a fixed-dimensional consonant segment representation vector. Temporal average pooling compresses the variable-length consonant segment feature sequence into a fixed-dimensional vector, aggregating the overall acoustic information within that time period. Compared to attention pooling, average pooling is often more stable when processing short consonant segments, avoiding feature bias caused by abnormal fluctuations in individual frames.

[0080] Finally, the consonant segment representation vector is input into a three-classifier, which outputs the consonant pronunciation state. In a specific embodiment, the consonant pronunciation state includes a normal state, a weakened state, and a dropped state. The normal state indicates that the consonant is pronounced clearly and completely; the weakened state indicates that the consonant is present but lacks energy or the articulation point is incorrect; the dropped state indicates that the consonant is completely absent. This three-classification design covers the main manifestations of consonant pronunciation abnormalities in patients with cleft palate.

[0081] To further improve alignment robustness under pathological speech conditions, in one specific embodiment, when using forced alignment techniques for matching, blank labels can be introduced to adjust the state transition probability. Patients with cleft palate often suffer from missing or severely deformed phonemes due to abnormalities in the structure of their speech organs. Traditional forced alignment algorithms, which assume monotonically ordered phoneme sequences, may not be able to handle such cases. The blank labels are used to tolerate differences in phoneme sequence lengths caused by missing pronunciation in the cleft palate speech signal. For example, adding empty state jump paths to the state transition model allows the acoustic sequence to maintain the continuity of the alignment path even when corresponding phoneme features are lacking, thereby avoiding alignment failures or boundary misalignments.

[0082] Furthermore, to ensure the effectiveness of model training and the reliability of classification results, the three-classifier can be trained based on the cross-entropy loss function. The cross-entropy loss function is calculated based on the difference between the true label of the consonant pronunciation state and the probability distribution output by the three-classifier. By minimizing the cross-entropy between the predicted probability distribution and the true label, the three-classifier parameters are updated, strengthening the ability to distinguish between normal, weakened, and dropped states. This ensures that the model converges to the optimal state during training, enabling the evaluation system to stably output high-confidence consonant pronunciation state judgments.

[0083] Secondly, regarding the assessment of vocabulary pronunciation accuracy. In a specific embodiment, such as... Figure 4 As shown, the process of outputting the pronunciation accuracy of words through the vocabulary evaluation module may include determining the start and end time boundaries of the word to be evaluated in the cleft palate speech signal. Subsequently, based on the start and end time boundaries, a vocabulary segment feature sequence corresponding to the word to be evaluated is extracted from the acoustic feature sequence.

[0084] In one specific implementation, determining the start and end time boundaries of the words to be evaluated in the cleft palate speech signal may include using a forced alignment technique to align the cleft palate speech signal with a preset 100-word corpus at the phoneme level. The preset corpus includes one hundred commonly used words for evaluating pronunciation accuracy. Based on the phoneme timestamps in the alignment results, the start and end timestamps of the words to be evaluated are aggregated and used as the start and end time boundaries. Determining coarse-grained word boundaries based on fine-grained phoneme boundaries can improve the robustness of boundary localization and reduce word segment truncation deviations caused by single phoneme alignment errors.

[0085] After obtaining the lexical segment feature sequence, a multi-head attention mechanism is used to aggregate the features of the lexical segment feature sequence to obtain a lexical-level feature vector. Traditional average pooling methods cannot distinguish the importance of different frames within a lexical segment and are easily affected by non-pathological noise or pauses in cleft palate speech. The multi-head attention mechanism can adaptively focus on key frames in the lexical segment that contribute significantly to pathological features, generating robust lexical-level representations.

[0086] In one specific embodiment, the feature aggregation of the vocabulary segment feature sequence using a multi-head attention mechanism may include mapping the vocabulary segment feature sequence into a query matrix, a key matrix, and a value matrix, respectively. Subsequently, the attention weight distribution of the query matrix and the key matrix is ​​calculated. Finally, the value matrix is ​​weighted and summed based on the attention weight distribution to obtain the vocabulary-level feature vector. Specifically, the query matrix, key matrix, and value matrix are obtained through linear transformation, and the attention weight distribution reflects the correlation between features in each frame within the vocabulary segment. Through weighted summation, the model can highlight key pathological acoustic features, suppress irrelevant background information, and enhance the flexibility and accuracy of feature aggregation.

[0087] Based on the lexical-level feature vector, a binary classifier outputs the pronunciation correctness judgment result of the word to be evaluated. In a specific embodiment, the step of outputting the pronunciation correctness judgment result of the word to be evaluated through a binary classifier based on the lexical-level feature vector may include inputting the lexical-level feature vector into a logistic activation function to obtain the pronunciation correctness probability. The logistic activation function maps the feature vector to a probability value between zero and one, quantifying the possibility of correct pronunciation. Subsequently, the pronunciation correctness probability is compared with a preset threshold. If the pronunciation correctness probability is greater than or equal to the preset threshold, the pronunciation correctness judgment result is correct pronunciation; if the pronunciation correctness probability is less than the preset threshold, the pronunciation correctness judgment result is incorrect pronunciation. The preset threshold can be adjusted according to specific application scenarios. For example, in a preferred embodiment, the preset threshold can be set to 0.5. This explicit binary classification decision logic ensures the consistency and interpretability of the evaluation results, making it easy for clinicians to understand and use.

[0088] Finally, the pronunciation accuracy of all the evaluated words is statistically analyzed to obtain the pronunciation accuracy rate. For example, if 80 out of 100 commonly used words are judged to be pronounced correctly, the pronunciation accuracy rate is 80%. This indicator quantifies the patient's overall pronunciation level at the vocabulary level, providing a macro-level indicator for clinical reference.

[0089] This implementation method enables a comprehensive and automated assessment of cleft palate speech. Specifically, the vowel assessment module accurately captures the severity of excessive nasality through attention-based statistical pooling and four-level classification; the consonant assessment module effectively identifies consonant weakening and omission by introducing forced alignment with blank labels and a three-classifier; and the vocabulary assessment module quantifies overall articulation clarity through multi-head attention mechanisms and binary classification statistics. These three modules share underlying acoustic features, reducing computational redundancy and enhancing overall generalization ability through multi-task collaborative learning. This approach overcomes the limitations of existing technologies with their single assessment dimension, providing clinicians with a more comprehensive tool for cleft palate speech assessment, significantly improving assessment efficiency and consistency, and offering multi-dimensional quantitative evidence for clinical diagnosis and rehabilitation.

[0090] Third Implementation Method Based on the first and second embodiments, this embodiment further provides a model training and optimization process for the above-mentioned multi-dimensional assessment method for cleft palate speech. To ensure that the shared encoder can simultaneously adapt to the extraction requirements of pathological features at three different granularities—vowels, consonants, and words—and to guarantee a balanced improvement in the performance of the three assessment modules, an end-to-end multi-task joint fine-tuning mechanism can be adopted.

[0091] like Figure 5As shown, firstly, a labeled training dataset is obtained. In a specific embodiment, the labeled training dataset may include training speech signals and their corresponding vowel over-nasalization level labels, consonant pronunciation state labels, and vocabulary pronunciation accuracy labels. The training speech signals may be original recordings collected from cleft palate patients, and the resulting Mel-spectral feature matrix is ​​obtained after preprocessing. The corresponding labels are manually labeled by professional speech therapists or digitized based on existing clinical assessment results. For example, the vowel over-nasalization level label may be one of four levels: none, light, medium, and heavy, usually represented using one-hot encoding; the consonant pronunciation state label may be one of three states: normal, weakened, and dropped, also using one-hot encoding; the vocabulary pronunciation accuracy label may be a binary classification label of correct or incorrect, usually represented by a value of 0 or 1. To enhance the generalization ability of the model, data augmentation processing can also be performed on the training speech signals, such as adding background noise, changing the speech rate, adjusting the pitch, or performing spectral masking, thereby expanding the diversity of the training dataset.

[0092] Subsequently, the training speech signal is input to the shared encoder, vowel evaluation module, consonant evaluation module, and vocabulary evaluation module to obtain prediction results. This process is consistent with the evaluation procedures described in the first and second embodiments. Specifically, the training speech signal first passes through the shared encoder to extract acoustic feature sequences, which are then fed into the three evaluation modules respectively. The vowel evaluation module outputs the probability distribution of vowel over-nasalization levels, the consonant evaluation module outputs the probability distribution of consonant pronunciation states, and the vocabulary evaluation module outputs the binary classification probability of word pronunciation correctness. These output results are the model's prediction results, which will be compared with the true labels in subsequent steps to calculate the error.

[0093] Based on the prediction results and corresponding labels, the loss for the vowel task, consonant task, and vocabulary task are calculated respectively. Since the label types for different evaluation tasks are different, a loss function matching the specific data distribution characteristics can be used to accurately measure the prediction error of each task.

[0094] In a specific embodiment, for the vowel task, the vowel task loss is calculated using the cross-entropy loss function based on the vowel hypernasalization level label and the probability distribution output by the vowel evaluation module. The vowel hypernasalization level is a multi-class classification task (four classes), and the cross-entropy loss function can effectively measure the difference between the predicted probability distribution and the true one-hot encoded label. The formula for calculating the vowel task loss can be expressed as: in, Indicates loss of vowel task; The true value representing the level label of vowel over-nasalization is usually a four-dimensional one-hot encoded vector. For example, if the true level is "light", the value of the corresponding dimension is 1, and the rest are 0. The probability distribution output by the vowel evaluation module is represented by a four-dimensional vector, where each element represents the predicted probability of belonging to the corresponding level, and the sum of all elements is 1. Represents the natural logarithm function; The summation symbol represents the summation of the losses across the four categories. This formula drives the model to more accurately predict nasal tone levels by minimizing the cross-entropy between the predicted probabilities and the true labels.

[0095] Similarly, for the consonant task, the cross-entropy loss function is used to calculate the consonant task loss based on the consonant pronunciation state labels and the probability distribution output by the consonant evaluation module. Consonant pronunciation state is also a multi-class task (three classes), and its loss function calculation method is similar to that of the vowel task. The formula for calculating the consonant task loss can be expressed as: in, Indicates consonant task loss; The true value representing the consonant pronunciation state label is usually a three-dimensional one-hot encoded vector, corresponding to the three states of normal, weakened, and dropped. The probability distribution output by the consonant evaluation module is represented by a three-dimensional vector, where each element represents the predicted probability of belonging to the corresponding state. Represents the natural logarithm function; The summation symbol is used. This loss function enhances the model's ability to distinguish between normal, weakened, and dropped states, ensuring the accuracy of consonant evaluation.

[0096] For vocabulary tasks, the loss is calculated using a binary cross-entropy loss function based on the vocabulary pronunciation accuracy labels and the binary classification probabilities output by the vocabulary evaluation module. Vocabulary pronunciation accuracy is a binary classification task (correct / incorrect), and the binary cross-entropy loss function is suitable for this type of probability output. The formula for calculating the vocabulary task loss can be expressed as: in, Indicates the loss in the lexical task; This represents the actual value of the vocabulary pronunciation accuracy label, with a value of 1 indicating correct pronunciation and a value of 0 indicating incorrect pronunciation. This represents the probability of correct pronunciation output by the vocabulary evaluation module, with a value ranging from 0 to 1. This represents the natural logarithm function. This formula penalizes the deviation between the predicted probability and the true label, enabling the model to more accurately distinguish between correctly pronounced and incorrectly pronounced words.

[0097] Different loss functions are used to calculate the loss for each task, improving the matching degree between the loss function and the label type of each evaluation task. For example, cross-entropy loss is suitable for optimizing multi-class probability distributions, while binary cross-entropy loss is suitable for optimizing binary probability distributions. This targeted design makes gradient backpropagation more accurate, accelerates the convergence speed of the model on specific pathological features, and avoids low training efficiency or performance bottlenecks caused by loss function mismatch.

[0098] After calculating the losses for the three tasks separately, a total loss function is constructed based on the vowel task loss, the consonant task loss, and the vocabulary task loss. In multi-task learning, the magnitudes of the loss functions and gradients for different tasks often differ. Direct summation may lead to gradient dominance, where tasks with larger gradient magnitudes dominate the training process, causing the model to favor optimizing easier tasks while neglecting more difficult ones. To address this issue, a weighted summation method can be used to construct the total loss function.

[0099] In a specific embodiment, constructing the total loss function based on the vowel task loss, the consonant task loss, and the lexical task loss may include obtaining preset task weight coefficients, whereby the task weight coefficients include vowel weights, consonant weights, and lexical weights. Subsequently, the products of the vowel task loss and the vowel weights, the consonant task loss and the consonant weights, and the lexical task loss and the lexical weights are summed to obtain the total loss function. The formula for calculating the total loss function can be expressed as: in, Represents the total loss function; These represent the loss for vowel task, consonant task, and lexical task, respectively. Indicates vowel weight. Indicates consonant weight. These three factors—word weights, task weight coefficients, and lexical weights—are collectively referred to as task weight coefficients.

[0100] The task weight coefficients are configured to balance the differences in gradient magnitudes among the vowel task loss, the consonant task loss, and the lexical task loss. For example, if the loss values ​​for the vowel task are generally large, the vowel weights can be appropriately reduced. If the vocabulary task is difficult to converge, the vocabulary weight can be appropriately increased. In a specific example, vowel weights, consonant weights, and lexical weights can all be set to 1.0, indicating that the three tasks are equally important. In another example, the weights can be dynamically adjusted based on performance on the validation set; for example, the vowel weight could be set to 0.8, the consonant weight to 1.0, and the lexical weight to 1.2, to balance the contributions of each task. By adjusting the weight coefficients to regulate the contribution ratio of each task's loss to the total loss, it is possible to effectively suppress tasks with larger gradient magnitudes from dominating the training process, thus mitigating gradient conflicts and negative transfer problems in multi-task training.

[0101] Finally, based on the total loss function, the parameters of the shared encoder, the vowel evaluation module, the consonant evaluation module, and the vocabulary evaluation module are updated using the backpropagation algorithm. Backpropagation is a commonly used neural network training method. It minimizes the total loss by calculating the gradient of the total loss function with respect to the parameters of each module and using an optimizer (e.g., stochastic gradient descent (SGD), Adam optimizer, etc.) to adjust the parameter values ​​in the reverse direction of the gradient. In a specific embodiment, an end-to-end fine-tuning strategy can be adopted, that is, updating all parameters of the shared encoder and the three evaluation modules simultaneously. This allows the shared encoder to learn a general acoustic feature representation compatible with three evaluation tasks, rather than optimizing only for a single task. For example, the shared encoder may learn feature representations that include both nasal formant information and consonant transient information, thereby improving the overall generalization ability of the model.

[0102] This implementation method achieves end-to-end multi-task joint fine-tuning. Specifically, by constructing a total loss function and backpropagating to update parameters, it ensures that the shared encoder can simultaneously adapt to the extraction needs of pathological features at three different granularities: vowels, consonants, and vocabulary. Specific loss functions (cross-entropy or binary cross-entropy) are used for different tasks, improving the accuracy of loss calculation and training convergence speed. By constructing the total loss function through weighted summation, gradient conflicts in multi-task training are effectively mitigated, ensuring balanced improvement in evaluation performance across the three dimensions of vowels, consonants, and vocabulary, avoiding performance limitations in a single task. This training strategy enables the model to achieve high-precision multi-dimensional evaluation within a single framework, providing reliable technical support for clinical practice.

[0103] Fourth Implementation Method Building upon the multi-dimensional assessment method for cleft palate speech and its model training process described in the foregoing embodiments, this embodiment further provides a system deployment architecture and execution scheme for practical applications. Considering the varying needs of different medical scenarios regarding computing power, network environment, and data privacy, this embodiment offers two main deployment modes: a local execution mode on mobile terminal devices and a collaborative execution mode on cloud servers. These two modes are based on the same core algorithm architecture, but adaptive adjustments have been made to the model structure, data flow, and interaction methods to meet the clinical application needs of different scenarios, from primary healthcare to home rehabilitation.

[0104] First, regarding the local execution mode on the mobile terminal device. In one specific embodiment, the steps of acquiring the cleft palate speech signal to be evaluated and subsequently inputting the cleft palate speech signal to a shared encoder to output the evaluation result are performed locally by the mobile terminal device. The mobile terminal device can be a smartphone, tablet, laptop, or a dedicated handheld medical evaluation terminal. The core advantage of this local execution mode is that it can complete the closed-loop evaluation without a network connection, significantly reducing reliance on cloud computing power and network latency, and improving diagnostic and treatment efficiency and patient privacy security in primary healthcare settings.

[0105] To achieve low-latency real-time inference on computationally limited mobile devices, the shared encoder is a lightweight conformal encoder. The lightweight conformal encoder has fewer layers than the standard conformal architecture. In one specific embodiment, a standard conformal architecture model may contain twelve conformal blocks, while the lightweight conformal encoder may contain four to six conformal blocks. By physically reducing the number of network layers, the computational load and memory consumption during forward inference are directly reduced. For example, if the standard model has approximately 200 million parameters, the lightweight model can compress the number of parameters to less than 50 million, thereby enabling a high-precision, multi-dimensional assessment method for cleft palate speech to run offline and in real-time on mobile devices.

[0106] However, simply reducing the number of network layers may lead to a decrease in model accuracy, making it difficult to accurately extract pathological features of cleft palate. To compensate for the accuracy loss caused by the reduction in the number of layers, the lightweight conformal encoder is trained using knowledge distillation technology. The teacher model in the knowledge distillation technique is a standard conformal architecture model with a larger number of parameters than the lightweight conformal encoder. Knowledge distillation is a model compression technique whose basic principle is to allow a small student model to learn the output behavior or intermediate feature representation of a large teacher model, thereby retaining most of the generalization ability of the teacher model in the student model.

[0107] In a specific training embodiment, the knowledge distillation process may include the following steps. First, a pre-trained standard conformal architecture model is prepared as the teacher model, which has achieved high evaluation accuracy on a large-scale cleft palate speech dataset. Second, a lightweight conformal architecture model with fewer layers is constructed as the student model. Then, the same training speech signal is simultaneously input into both the teacher and student models. The teacher model outputs a teacher probability distribution, and the student model outputs a student probability distribution. To enable the student model to better mimic the teacher model, a temperature parameter can be introduced to soften the probability distribution. The formula for calculating the softened probability distribution can be expressed as: in, This represents the softening probability of the i-th category; The output layer of the model is represented by the first... The original value of each logits unit; For temperature parameters, they are usually set to values ​​greater than 1, for example... It can be version 2.0 or 3.0; Represents an exponential function; Indicates all categories Perform summation. When When the value is greater than 1, the probability distribution becomes smoother, contains more hidden knowledge, and helps the student model learn the relative relationships between categories.

[0108] Subsequently, based on the softening probability distributions output by the teacher model and the student model, the distillation loss is calculated. The distillation loss can be calculated using KL divergence or mean squared error. In a specific embodiment, the formula for calculating the distillation loss can be expressed as: in, Indicates distillation loss; The output of the teacher model represents the first... The softening probability of each category; The first output of the student model represents the... Softening probability of each category Represents the natural logarithm function; Indicates all categories Summation is performed. This formula measures the difference between the probability distributions of the student model and the teacher model.

[0109] The final total training loss can be a weighted sum of the task loss and the distillation loss. The formula for calculating the total training loss can be expressed as: in, This represents the total loss during distillation training; This represents the task loss calculated based on the real labels, such as cross-entropy loss; This is a distillation loss weighting coefficient used to balance the contributions of task loss and distillation loss, for example... It can be set to 0.5 or 1.0; This represents the distillation loss described above. It is minimized using the backpropagation algorithm. It can update the parameters of the student model, so that the lightweight model can still accurately extract the pathological features of cleft palate while maintaining low computational complexity and high sensitivity to the pathological features of cleft palate.

[0110] In local execution mode, the user's voice is captured via the mobile terminal's microphone, the model performs inference locally, and the evaluation results are displayed directly on the terminal screen. The entire process involves data remaining within the device, effectively protecting the patient's voice privacy. This is particularly suitable for privacy-sensitive home rehabilitation scenarios or remote areas with poor network conditions.

[0111] Secondly, regarding the cloud server collaborative execution mode. In one specific embodiment, the step of acquiring the cleft palate speech signal to be evaluated is performed by the user terminal, while the step of inputting the cleft palate speech signal to the shared encoder and outputting the evaluation result is performed by the cloud server. This mode is suitable for scenarios where the user terminal hardware performance is low, or where the powerful computing power of the cloud is required for high-precision evaluation. The user terminal can be a low-configuration mobile phone, an older tablet computer, or a dedicated simple recording device.

[0112] Specifically, the method further includes: the user terminal uploading the collected cleft palate speech signal to the cloud server. The upload process can be performed via a wireless network, such as Wi-Fi, 4G, or 5G. To save bandwidth, the user terminal can perform preliminary compression encoding of the speech signal locally, for example, using OPUS or AAC encoding formats, before uploading. After receiving the speech signal, the cloud server uses a high-performance computing cluster deployed in the cloud to execute the inference steps of the shared encoder and evaluation module. The cloud server can run standard models with a larger number of parameters, or even ensemble learning schemes integrating multiple models, thereby achieving higher evaluation accuracy than local lightweight models.

[0113] Subsequently, the cloud server generates a structured multi-dimensional assessment report and distributes it to the user terminal. This structured multi-dimensional assessment report contains more than just simple scores; it includes rich clinical information. In one specific embodiment, the structured multi-dimensional assessment report may include the following: a patient basic information area, recording patient ID, age, assessment date, etc.; a vowel assessment details area, listing the nasal level and probability distribution of each basic vowel phoneme, for example, the nasal level of / a / is moderate, with a probability of 0.8; a consonant assessment details area, listing the pronunciation state of each consonant phoneme, for example, the state of / b / is weakened, and marking phonemes requiring focused training; a vocabulary assessment details area, displaying the pronunciation accuracy rate of a 100-word list, for example, an accuracy rate of 75%, and listing a list of incorrectly pronounced words; and a comprehensive suggestion area, generating personalized rehabilitation suggestions based on the assessment results, such as suggesting focused practice of labial plosive strength or soft palate elevation training.

[0114] The report can be in portable document format, web page format, or structured data format. After the cloud server generates the report, it is distributed to the user's terminal via the network. Upon receiving the report, the user's terminal can visualize it through an application, such as using a bar chart to show the distribution of nasal tone levels or a line graph to show the trend of accuracy changes across assessments. This model overcomes the limitations of geographical location and terminal hardware performance, enabling users with low-configuration devices to enjoy standardized cleft palate speech assessment services based on high-precision deep learning models. This promotes the downward flow of high-quality medical resources and supports long-term home rehabilitation monitoring and data archiving. Doctors or therapists can view patients' historical assessment reports through the cloud backend, remotely monitor rehabilitation progress, and adjust treatment plans in a timely manner.

[0115] This implementation provides a flexible deployment architecture. The mobile terminal local execution solution, through a lightweight model and knowledge distillation technology, achieves offline, real-time, and privacy-secure assessments, solving application problems in scenarios with limited computing power. The cloud server collaborative execution solution, through end-to-cloud division of labor, achieves high-precision, remote, and archiveable assessment services, addressing the needs of low-configuration terminals and remote assessments. These two solutions complement each other, together forming a complete intelligent judgment and assessment grading system for cleft palate speech, meeting the clinical application needs of different scenarios from primary healthcare to home rehabilitation, and significantly improving the accessibility and convenience of assessments.

[0116] Fifth Implementation Method Based on the multi-dimensional assessment method, model training process, and system deployment architecture for cleft palate speech described in the first to fourth embodiments above, this embodiment further provides a computer program product that can solidify the technical solutions in any of the aforementioned embodiments into a software entity that runs on a general-purpose or special-purpose data processing device.

[0117] In one specific embodiment, the computer program product may include a computer-readable storage medium storing a computer program. The computer-readable storage medium can be any tangible medium capable of containing, storing, communicating, propagating, or transmitting a program for use by an instruction execution system, apparatus, or device. For example, the computer-readable storage medium may be an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices (e.g., punch cards or recessed protrusions storing instructions thereon), and any suitable combination thereof. It should be noted that the computer-readable storage medium is defined herein as a non-transitory tangible medium, excluding transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0118] When the computer program is loaded onto a computer, other programmable data processing device, or other device and executed, it can implement the multi-dimensional assessment method for cleft palate speech in any of the foregoing embodiments. Specifically, when the computer program runs on a processor, it can execute the following series of operation sequences: First, the program commands control the audio acquisition interface to acquire the speech signal of the cleft palate to be evaluated. In a specific embodiment, this step may include calling the microphone driver of a mobile terminal or computer to acquire the raw audio data of the user reading a preset corpus (such as six basic vowels, twenty-six Chinese consonants, or one hundred commonly used words), and converting it into a digital signal sequence.

[0119] Secondly, the program instructions preprocess the acquired raw speech signal. This step may include sequentially performing pre-emphasis processing, frame-by-frame windowing processing, and Mel filter bank filtering. For example, the program may apply a transfer function. .

[0120] Pre-emphasis is applied, where 'a' is the pre-emphasis coefficient; the signal is divided into short frames with a frame length of 25ms and a frame shift of 10ms, and a Hamming window is applied; subsequently, a Mel filter bank containing eighty filters is used to generate a signal with a dimension of 80×. The Mel-spectral feature matrix. These specific parameter configurations and processing logic are encoded in program instructions to ensure that the extracted features are adapted to the pathological characteristics of cleft palate.

[0121] Subsequently, the program instructions input the preprocessed feature matrix into the shared encoder model. This shared encoder model can be a neural network model built based on a conformal architecture, a multi-layer bidirectional long short-term memory network, or a transformer encoder backbone network. The program instructions control the processor to load the pre-trained model weight parameters and perform forward inference calculations. In the specific implementation of the conformal architecture, the program instructions can control the processor to sequentially execute steps such as the first feedforward neural network transformation, multi-head self-attention mechanism for global dependency capture, convolutional operation for local feature extraction, the second feedforward neural network transformation, and layer normalization processing, ultimately outputting an acoustic feature sequence.

[0122] Next, the program instructions input the acoustic feature sequences into the vowel evaluation module, consonant evaluation module, and vocabulary evaluation module, respectively.

[0123] For vowel evaluation, the program executes attention statistical pooling to calculate the weights of features in each frame and perform a weighted average to obtain a vowel feature vector. Then, the classifier outputs the probability distribution of vowel over-nasalization levels (none, light, medium, heavy).

[0124] For consonant evaluation, the program uses forced alignment to determine the temporal boundaries of consonant segments, extracts feature sequences, performs temporal average pooling, and outputs the consonant pronunciation state (normal, weakened, or dropped) through a three-classifier. During this process, the program can also execute alignment logic that introduces blank labels to tolerate missing phonemes.

[0125] For vocabulary evaluation, the program instructions determine the vocabulary boundaries, use a multi-head attention mechanism to aggregate vocabulary segment features, output the pronunciation correctness judgment through a binary classifier and threshold comparison, and statistically obtain the vocabulary pronunciation accuracy rate.

[0126] If the program corresponds to the training flow of the third implementation, the program instructions can also control the processor to execute model training steps. This includes loading the labeled training dataset and constructing the total loss function. The network parameters are updated using the backpropagation algorithm. The calculation logic of the loss function (such as cross-entropy loss and binary cross-entropy loss) and the weight balancing strategy are all encoded in the program instructions.

[0127] If the program corresponds to the deployment scenario of the fourth implementation, the program instructions can also control the processor to execute specific deployment logic. In local execution mode, the program loads a lightweight conformal encoder model (trained using knowledge distillation technology), completes the entire inference process locally, and generates an evaluation report. In cloud-collaborative mode, the program instructions control the communication module to upload the collected voice signals to the cloud server, receive the structured multi-dimensional evaluation report generated in the cloud, and visualize it on the user interface.

[0128] The computer program product can also be presented as a transient data signal carrying the aforementioned computer program code, transmitted to the target device via wired or wireless network. For example, the program product can be distributed from a server to a user's mobile terminal or a medical institution's assessment device via internet download, local area network transmission, or Bluetooth transmission. Once the receiving device receives the data signal and stores it in its local memory, it can execute the multi-dimensional assessment method for cleft palate speech according to the above process.

[0129] Furthermore, the computer program product can be combined with hardware devices to form an integrated hardware and software solution. For example, the computer program can be embedded in a dedicated speech evaluation chip or embedded system. In this embodiment, program instructions directly drive hardware circuits to perform operations such as feature extraction, encoding reasoning, and classification judgment, thereby achieving low-power, high-real-time cleft palate speech evaluation functionality.

[0130] This implementation provides a flexible and portable computer program product. This product transforms the abstract multi-dimensional assessment method for cleft palate speech into concrete executable code, enabling the method to be widely applied across various computing platforms, from high-performance cloud servers to resource-constrained mobile terminals. This not only promotes the popularization and application of the technology but also ensures the standardization and automation of the assessment process, allowing non-professionals to easily use the system for preliminary screening and rehabilitation monitoring of cleft palate speech, greatly improving the accessibility and efficiency of medical services.

[0131] Finally, it should be noted that those skilled in the art will understand that many technical details have been presented in the embodiments of the present invention to facilitate a better understanding of the invention. However, even without these technical details and various variations and modifications based on the above embodiments, the technical solutions claimed in the claims of the present invention can be substantially achieved. Therefore, in practical applications, various changes in form and detail can be made to the above embodiments without departing from the spirit and scope of the present invention.

Claims

1. A multi-dimensional assessment method for speech in cleft palate, characterized in that, include: Acquire the speech signal of the cleft palate to be evaluated; The cleft palate speech signal is input into a shared encoder to obtain an acoustic feature sequence. The shared encoder is built based on a deep learning architecture, which includes at least one of a conformal architecture, a multi-layer bidirectional long short-term memory network architecture, or a converter encoder backbone network architecture. The acoustic feature sequences are respectively input into the vowel evaluation module, consonant evaluation module, and vocabulary evaluation module; The vowel evaluation module outputs the vowel over-nasalization level, the consonant evaluation module outputs the consonant pronunciation status, and the vocabulary evaluation module outputs the vocabulary pronunciation accuracy.

2. The method according to claim 1, characterized in that, The acquisition of the speech signal of the cleft palate to be evaluated includes: The collected raw speech signal is preprocessed to obtain the Mel spectrum feature matrix; The preprocessing includes pre-emphasis processing, frame-by-frame windowing processing, and Mel filter bank filtering processing performed sequentially, with the number of Mel filter banks configured to be 80 to 120.

3. The method according to claim 2, characterized in that, To capture the high-frequency transient features and nasal formants of cleft palate speech, the pre-emphasis processing employs a transfer function. Where a is the pre-weighting coefficient, For delay operators; The frame length for the frame-segmentation and windowing process is set to 25ms, the frame shift is set to 10ms, and the window function is a Hamming window. The process of acquiring the speech signal of the cleft palate to be evaluated also includes: Speech data of patients with cleft palate reading a pre-set corpus were collected. The pre-set corpus includes six basic vowel phonetic symbols for assessing the degree of excessive nasality, twenty-six Chinese consonant phonetic symbols for assessing the pronunciation state, and one hundred commonly used words for assessing the pronunciation accuracy. In the Mel filter bank filtering process, the number of Mel filter banks is set to 80 to obtain a dimension of 80× The feature matrix of , where The number of frames.

4. The method according to claim 1, characterized in that, The step of inputting the cleft palate speech signal into a shared encoder to obtain an acoustic feature sequence includes: The cleft palate speech signal is input into multiple sequentially stacked conformal blocks for processing; During the processing of each conformal block, the following operations are performed sequentially: The first intermediate data is obtained by performing a nonlinear transformation on the input data using a first feedforward neural network. The second intermediate data is obtained by using a multi-head self-attention mechanism to capture global dependencies on the first intermediate data; The third intermediate data is obtained by extracting local features from the second intermediate data using convolution operations. The third intermediate data is then subjected to another nonlinear transformation using a second feedforward neural network to obtain the output data of the current conformal block. The output data of the current conformal block is processed by layer normalization and then used as the input data of the next level conformal block, until the output data of the last conformal block is used as the acoustic feature sequence.

5. The method according to claim 4, characterized in that, The nonlinear transformation of the input data using a first feedforward neural network includes: The input data is processed sequentially through a first linear layer, a nonlinear activation function layer, and a second linear layer. The step of using a second feedforward neural network to perform another nonlinear transformation on the third intermediate data includes: The third intermediate data is processed sequentially through a first linear layer, a nonlinear activation function layer, and a second linear layer. The step of extracting local features from the second intermediate data using convolution operations includes: The second intermediate data is processed sequentially through pointwise convolution, depthwise separable convolution, and gated linear units.

6. The method according to claim 1, characterized in that, The shared encoder is a multi-layer bidirectional long short-term memory network architecture. The process of inputting the cleft palate speech signal into the shared encoder to obtain an acoustic feature sequence includes: The cleft palate speech signal is input into the multi-layer bidirectional long short-term memory network for temporal feature extraction to obtain initial temporal features; The initial temporal features are mapped using a feature adaptation layer and transformed into a unified feature space to obtain the acoustic feature sequence. or, The shared encoder is a converter encoder backbone network architecture. The process of inputting the cleft palate speech signal into the shared encoder to obtain an acoustic feature sequence includes: The cleft palate speech signal is input into the converter encoder backbone network; The transducer encoder backbone network is used to process the cleft palate speech signal using a self-attention mechanism, extracting deep acoustic features and directly outputting them as the acoustic feature sequence.

7. The method according to claim 1, characterized in that, The output of the vowel over-nasalization level through the vowel evaluation module includes: Attention statistical pooling is performed on the acoustic feature sequence to obtain vowel feature vectors; The vowel feature vector is input into the classifier to obtain the vowel nasalization level.

8. The method according to claim 7, characterized in that, The acoustic feature sequence is obtained by concatenating the output features of multiple conformal blocks of the shared encoder and then performing layer normalization. The attention statistical pooling process for the acoustic feature sequence includes calculating the weight of each frame feature in the acoustic feature sequence using trainable parameters, and performing a weighted average calculation on the features of each frame based on the weights. The vowel nasalization level includes four levels: none, light, medium, and heavy. The classifier outputs the probability distribution of the four levels.

9. The method according to claim 1, characterized in that, The step of outputting the consonant pronunciation state through the consonant evaluation module includes: By using forced alignment technology, the acoustic feature sequence is matched with the standard text phoneme sequence to determine the start and end time boundaries of the consonant segment to be evaluated; Extract the consonant segment feature sequence located between the start time boundary and the end time boundary from the acoustic feature sequence; Perform temporal average pooling on the consonant segment feature sequence to obtain a consonant segment representation vector of fixed dimension; The consonant segment representation vector is input into a three-classifier, which outputs the consonant pronunciation state, which includes a normal state, a weakened state, and a dropped state.

10. The method according to claim 9, characterized in that: When matching using the forced alignment technique, a blank label is introduced to adjust the state transition probability. The blank label is used to tolerate the difference in phoneme sequence length caused by the absence of pronunciation in the cleft palate speech signal. The three-classifier is trained based on the cross-entropy loss function, which is calculated based on the difference between the true label of the consonant pronunciation state and the probability distribution output by the three-classifier.

11. The method according to claim 1, characterized in that, The output of vocabulary pronunciation accuracy through the vocabulary evaluation module includes: Determine the start and end time boundaries of the words to be evaluated in the cleft palate speech signal; Based on the start and end time boundaries, extract the lexical segment feature sequence corresponding to the word to be evaluated from the acoustic feature sequence; The feature sequence of the vocabulary segment is aggregated using a multi-head attention mechanism to obtain a vocabulary-level feature vector; Based on the vocabulary-level feature vector, a binary classifier is used to output the pronunciation correctness judgment result of the vocabulary to be evaluated; The pronunciation accuracy of all the words to be evaluated is statistically analyzed to obtain the pronunciation accuracy rate of the words.

12. The method according to claim 11, characterized in that, The feature aggregation of the vocabulary segment feature sequence through a multi-head attention mechanism includes: The word segment feature sequences are mapped into query matrix, key matrix, and value matrix, respectively. Calculate the attention weight distribution of the query matrix and the key matrix; The value matrix is ​​weighted and summed based on the attention weight distribution to obtain the vocabulary-level feature vector.

13. The method according to claim 11, characterized in that, The step of outputting the pronunciation correctness judgment result of the word to be evaluated through a binary classifier based on the vocabulary-level feature vector includes: The vocabulary-level feature vectors are input into the logistic activation function to obtain the probability of correct pronunciation; The probability of correct pronunciation is compared with a preset threshold. If the probability of correct pronunciation is greater than or equal to the preset threshold, then the pronunciation correctness judgment result is correct. If the probability of correct pronunciation is less than the preset threshold, then the pronunciation correctness judgment result is a pronunciation error.

14. The method according to claim 11, characterized in that, Determining the start and end time boundaries of the words to be evaluated in the cleft palate speech signal includes: Using forced alignment technology, the cleft palate speech signal is aligned with a preset 100-word list text at the phoneme level. Based on the phoneme timestamps in the alignment results, the start and end timestamps of the words to be evaluated are aggregated and used as the start and end time boundaries.

15. A training method for a cleft palate speech assessment model, characterized in that, include: Obtain a labeled training dataset, which includes training speech signals and their corresponding vowel over-nasalization level labels, consonant pronunciation state labels, and word pronunciation accuracy labels; The training speech signal is input into the shared encoder, vowel evaluation module, consonant evaluation module and vocabulary evaluation module to be trained, and the prediction result is obtained; Based on the prediction results and the corresponding labels, the vowel task loss, consonant task loss and vocabulary task loss are calculated respectively. Based on the vowel task loss, the consonant task loss, and the vocabulary task loss, a total loss function is constructed; Based on the total loss function, the parameters of the shared encoder, the vowel evaluation module, the consonant evaluation module, and the vocabulary evaluation module are updated using the backpropagation algorithm.

16. The method according to claim 15, characterized in that, The step of calculating vowel task loss, consonant task loss, and vocabulary task loss based on the prediction results and corresponding labels includes: Based on the vowel over-nasalization level label and the probability distribution output by the vowel evaluation module, the vowel task loss is calculated using the cross-entropy loss function; Based on the consonant pronunciation state label and the probability distribution output by the consonant evaluation module, the consonant task loss is calculated using the cross-entropy loss function; Based on the vocabulary pronunciation accuracy labels and the binary classification probabilities output by the vocabulary evaluation module, the vocabulary task loss is calculated using the binary cross-entropy loss function.

17. The method according to claim 15, characterized in that, The construction of the total loss function based on the vowel task loss, the consonant task loss, and the vocabulary task loss includes: Obtain preset task weight coefficients, which include vowel weights, consonant weights, and vocabulary weights; The total loss function is obtained by summing the products of the vowel task loss and the vowel weight, the consonant task loss and the consonant weight, and the lexical task loss and the lexical weight. The task weight coefficients are configured to balance the gradient magnitude differences between the vowel task loss, the consonant task loss, and the vocabulary task loss.

18. A computer program product having a computer program that, when executed, enables the implementation of the method as described in any one of claims 1-17.

Images