A state evaluation system based on a lightweight Chinese speech rehabilitation large model

Abstract

This invention discloses a state assessment system based on a lightweight Chinese speech rehabilitation large-scale model, belonging to the technical field of artificial intelligence. The technical problems it addresses are: poor domain adaptability, an imbalance between lightweight design and accuracy, and weak cross-modal generalization in existing Chinese speech rehabilitation large-scale models. The system includes: a teacher model construction module for building a teacher model, which performs speech defect mask reconstruction and cross-modal assessment based on cross-modal fusion features, enhanced acoustic features, and text features; a teacher model training module for training the teacher model; a student model construction and training module for tailoring the trained teacher model into a student model, performing intra-modal and inter-modal self-supervised training on the student model, and performing knowledge distillation; and a model deployment and application module for performing low-dimensional embedding compression on the trained student model and introducing quantized perceptual training to obtain a lightweight model comprising multiple independent modules.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically a state assessment system based on a lightweight Chinese speech rehabilitation model. Background Technology

[0002] As AI technology penetrates the medical field, large-scale models are gradually being applied to speech rehabilitation diagnosis and treatment (such as pronunciation defect identification and rehabilitation effect prediction). However, current technology has three major shortcomings that make it difficult to meet actual clinical needs:

[0003] (1) Insufficient domain adaptability: Existing general Chinese models (such as general speech recognition models and text understanding models) are not optimized for the data characteristics of the speech rehabilitation field. Chinese speech rehabilitation data contains a large number of "non-standard speech" (such as the pronunciation deviation of patients with articulation disorders, speech samples under the influence of dialects / accents) and "diagnosis and treatment related data" (such as pathological diagnosis records and rehabilitation training logs). General models are prone to recognition bias, resulting in low accuracy of diagnosis and treatment assessment (such as misjudging "pronunciation defects" as "speech noise").

[0004] (2) Imbalance between lightweight model and accuracy: Large speech rehabilitation models need to balance "diagnosis and assessment accuracy" and "device compatibility" - primary healthcare institutions mostly use low-computing-power terminals (such as ordinary PCs and embedded devices), while existing large models (such as speech models with hundreds of billions of parameters) occupy more than 10GB of storage and take more than 500ms for a single inference, which cannot meet the needs of real-time diagnosis and treatment; if traditional lightweight methods (such as simply pruning network layers) are used, the model will lose key features in the field of speech rehabilitation (such as subtle pronunciation differences), and the assessment accuracy will decrease by more than 20%;

[0005] (3) Weak cross-modal knowledge transfer capability: Speech rehabilitation diagnosis and treatment relies on the fusion of cross-modal information such as “speech-text-pathological labels” (e.g., judging the condition by combining patient speech samples, text descriptions, and doctor's diagnostic labels). Existing knowledge distillation technology focuses on a single modality (e.g., distilling only speech features) and lacks a self-supervised cross-modal learning mechanism, resulting in poor generalization ability of student models (e.g., when faced with unseen dialect pronunciation defect samples, the evaluation accuracy drops sharply).

[0006] The existing large-scale Chinese speech rehabilitation models suffer from poor adaptability, an imbalance between lightweight design and accuracy, and weak cross-modal generalization, which are technical problems that need to be solved. Summary of the Invention

[0007] The technical objective of this invention is to address the above-mentioned shortcomings by providing a state assessment system based on a lightweight Chinese speech rehabilitation model, thereby resolving the technical problems of poor adaptability, imbalance between lightweight design and accuracy, and weak cross-modal generalization in existing Chinese speech rehabilitation models.

[0008] This invention discloses a state assessment system based on a lightweight Chinese speech rehabilitation model, comprising:

[0009] The data acquisition module is used to collect and preprocess speech data, text data, and cross-modal assessment labels related to Chinese speech rehabilitation to form a sample set;

[0010] The teacher model construction module is used to construct a large-scale Chinese speech rehabilitation model as a teacher model. The teacher model extracts features from the input speech and text data, performs cross-modal feature alignment on the extracted acoustic and text features to obtain cross-modal fusion features, and performs speech defect mask reconstruction and cross-modal evaluation based on the cross-modal fusion features and the enhanced acoustic and text features. The speech defect mask reconstruction outputs the completed speech data, pronunciation defect labels and probability distributions, and the cross-modal evaluation outputs the pronunciation defect types and probability distributions, state type labels and probability distributions, and the evaluation results of the training phase.

[0011] The teacher model training module is used to construct a loss function based on cross-modal alignment training, speech defect mask reconstruction training, and state type training. The teacher model is trained based on the sample set by minimizing the loss function.

[0012] The student model construction and training module is used to trim the trained teacher model into a student model, perform intra-modal and inter-modal self-supervised training on the student model, and perform knowledge distillation on the student model based on the cross-modal fusion features and state type label probability distribution output by the teacher model.

[0013] The model deployment application module is used to perform low-dimensional embedding compression on the trained student model and introduce quantization-aware training during the training phase of the student model to obtain a lightweight model that includes multiple independent modules. After evaluating the lightweight model based on predefined evaluation metrics, Chinese speech state evaluation is performed based on the lightweight model.

[0014] Preferably, the teacher model includes a base model layer, a cross-modal feature alignment layer, and a feature enhancement evaluation layer;

[0015] The base model layer is used to perform semantic encoding on the input text data to obtain text features, and to extract acoustic features from the input speech data to obtain acoustic features including pronunciation details and defect features.

[0016] The cross-modal feature alignment layer is used to fuse and align textual and acoustic features based on a dual cross-attention mechanism and a cross-loss function, and output cross-modal fused features.

[0017] The feature enhancement and evaluation layer is used to enhance acoustic features through convolutional structures, capture defect features including alveolar consonant defects and articulation shifts, and obtain defect acoustic features as enhanced acoustic features. It is used to enhance text features through dictionary matching, obtain domain-enhanced text features as enhanced text features. It is used to reconstruct speech defect masks based on the masked speech data based on defect acoustic features and cross-modal fusion features, and outputs the speech defect type and the completed speech data. It is used to perform cross-modal evaluation based on the speech defect mask reconstruction generated as a prior, based on defect acoustic features, domain-enhanced text features and cross-modal fusion features, and outputs speech defect type and probability distribution, state type label and probability distribution, and evaluation results of the training phase.

[0018] As a preferred embodiment, the basic model layer includes a text coding layer and a speech coding layer;

[0019] The text encoding layer is used to extract semantic features from speech data using the Transformer encoder of ERNIE-3.0, while retaining general semantic understanding capabilities, and outputting text features;

[0020] The speech coding layer is used to extract speech features from the input speech data based on the convolutional and Transformer structures in Wav2Vec2.0, and outputs acoustic features including pronunciation details and defect features.

[0021] Preferably, the cross-modal feature alignment layer takes text features and acoustic features as input, and maps the text features and acoustic features to a unified shared feature space. It constructs a speech-text mutual attention matrix based on a bidirectional cross-attention mechanism, fuses text features and acoustic features through the speech-text mutual attention matrix, constructs positive and negative samples, introduces contrast loss to constrain the feature distance between text features and acoustic features, and obtains cross-modal fused features, where the matched speech feature and text feature pairs are used as positive samples, and the mismatched speech feature and text feature pairs are used as negative samples.

[0022] The formula for calculating the contrast loss is as follows:

[0023] ,

[0024] in, Represents cosine similarity. This represents a positive sample pair constructed from the same user's voice and text features. ( )) represents negative sample pairs constructed by different users based on speech and text features. This indicates the number of negative samples in each batch. This represents the temperature coefficient.

[0025] As a preferred embodiment, the feature enhancement evaluation layer includes a defect feature extraction layer, a knowledge integration layer, and a mask reconstruction and prediction layer;

[0026] The defect feature extraction layer is connected in series with the output of the speech coding layer. It takes acoustic features as input, captures specific acoustic features including alveolar consonant defects and articulation shifts, and outputs defect acoustic features.

[0027] The knowledge integration layer is used to weight and enhance text features through domain dictionary matching, and outputs domain-enhanced text features.

[0028] The mask reconstruction and prediction layer includes a speech defect mask prediction sublayer and a cross-modal label prediction sublayer. The speech defect mask prediction sublayer is used to perform cross-attention reconstruction based on enhanced defect acoustic features and cross-modal fusion features, reconstruct artificially masked segments in speech data, and output the speech defect type, probability distribution, and completed speech data. The cross-modal label prediction sublayer uses the speech defect type output by feature reconstruction as a prior, and performs evaluation analysis based on enhanced defect acoustic features, domain enhanced text features, and cross-modal fusion features. The output includes the speech defect type and probability distribution, state type label and probability distribution, and evaluation results from the training phase.

[0029] As a preferred option, the loss function corresponding to the teacher model... Represented as:

[0030] ,

[0031] in, This represents the cross-entropy loss constructed based on state type training. express The weight, The calculation formula is expressed as follows:

[0032] ,

[0033] Indicates the number of samples in the sample set. This indicates the number of categories in the status type label. Indicates sample In category One-hot encoding on, Indicates sample In category The probability of the teacher model predicting the output, in scenarios with multiple diagnosis and treatment labels, Multi-label binary encoding;

[0034] in, This represents the loss constructed based on cross-modal alignment training. express The weight, The calculation formula is:

[0035] ,

[0036] Represents the acoustic eigenvector. Represents the text feature vector. Represents the L2 norm;

[0037] in, This represents the CTC loss constructed based on speech defect mask reconstruction training. express The weight, The calculation formula is expressed as follows:

[0038] ,

[0039] This represents the pronunciation defect labels corresponding to the original speech data. This represents the probability distribution of pronunciation defect labels reconstructed by the teacher model through a speech defect mask.

[0040] As a preferred approach, student model construction and training includes the following operations:

[0041] Student model initialization: Based on the trained teacher model, retain the network layers used for feature extraction, cross-modal alignment, feature enhancement, speech defect mask reconstruction, and cross-modal evaluation in the teacher model, and prune redundant network layers to obtain the initialized student model;

[0042] Intramodal self-supervision: Random segment masking is performed on speech data. The masked speech data is used as input, and a student model is used to reconstruct the masked regions. A loss function is constructed based on the probability distribution of pronunciation defect labels output by the masked region reconstruction and the probability value of the real pronunciation defects. The model is trained by minimizing the loss function. Similarly, random segment masking is performed on text data. The masked text data is used as input, and a student model is used to reconstruct the masked regions. A loss function is constructed based on the probability distribution of text data output by the masked region reconstruction and the probability value of the real text data. The model is trained by minimizing the loss function.

[0043] Intermodal self-supervision: Voice and text features from the same user are used as positive samples, while cross-modal fused features from different users are used as negative samples. The cross-modal feature alignment capability of the student model is optimized through contrastive loss. The formula for calculating contrastive loss is as follows:

[0044] ,

[0045] in, Indicates the size of the learning batch. Indicates the first Acoustic features corresponding to each speech data point Indicates the first Text features corresponding to each piece of text data Indicates the temperature coefficient;

[0046] Feature distillation: Based on the cross-modal fusion features output by the teacher model, the student model learns the feature distribution of the cross-modal fusion features through L2 loss, and the corresponding loss function is... The calculation formula is:

[0047] ,

[0048] in, This represents the cross-modal fusion features output by the teacher model. This represents the cross-modal fusion features of the student model. The dimension representing the cross-modal fusion feature;

[0049] Probabilistic distillation: Based on the state type label probability distribution output by the teacher model, the student model learns the diagnosis label probability distribution through KL divergence loss, and the corresponding loss function... The calculation formula is:

[0050] ,

[0051] in, , , This represents the probability distribution of the state type labels output by the teacher model. This represents the probability distribution of the state type labels output by the student model. Indicates the number of state types. This represents an empirical value used to balance the smoothness and discriminative power of a probability distribution.

[0052] Preferably, the model deployment application module is used to perform the following operations:

[0053] Low-dimensional embedding compression: For the cross-modal fusion features output by the student model, the encoder in the student model compresses the high-dimensional cross-modal fusion features to a low dimension. The decoder in the student model reconstructs the compressed cross-modal fusion features. A reconstruction loss is constructed based on the deviation between the high-dimensional and low-dimensional cross-modal fusion features, and state type label constraints are introduced. A label constraint loss is constructed based on the probability distribution of state type labels output by the student model and the true state type label probability values. The weighted sum of the reconstruction loss and the label constraint loss is taken as the total low-dimensional embedding compression loss. The variance explained rate of the compressed cross-modal fusion features is analyzed by PCA. The student model is trained based on the total low-dimensional embedding compression loss.

[0054] Among them, the total loss of low-dimensional embedding compression The calculation formula is:

[0055] ,

[0056] in, Indicates the reconstruction loss. Represents the label constraint loss. Indicates the weighting coefficient. The calculation formula is:

[0057] ,

[0058] Indicates high-dimensional cross-modal fusion characteristics. This represents the cross-modal fusion features of the reconstruction. The original dimension representing the cross-modal fusion feature;

[0059] Among them, the variance explained rate The calculation formula is as follows:

[0060] ,

[0061] In representing the high-dimensional cross-modal fusion features, the first... The variance of each principal component;

[0062] Quantization-aware training: The model weights of the student model are quantized using INT8, and the activation values ​​in the student model are also quantized using INT8. Quantization error compensation is inserted during model training, and the accuracy loss caused by quantization is corrected through backpropagation. The calculation formula is:

[0063] ,

[0064] The quantization error loss is represented by the following formula:

[0065] ,

[0066] This represents the weights of the student model before quantization. This represents the weights of the quantized student model. Indicates the total weight. Indicates the weighting coefficient. The loss function representing the teacher model;

[0067] Among them, the key network layers preset in the student model are quantized using INT8, and the non-key layers preset in the student model are quantized using INT4. During quantization-aware training, the weight range and quantization step size are set, and progressive quantization is performed. FP32 is used for training in the first training rounds, and quantization is gradually introduced in the back rounds.

[0068] Modular output: The trained student model is divided into three independent modules: pronunciation assessment module, state type classification module, and training phase suggestion module. The pronunciation assessment module is used to output the pronunciation defect type and probability distribution, the state type classification module is used to output the state type and probability distribution, and the training phase suggestion module is used to output the training phase.

[0069] As a preferred option, when evaluating the lightweight model based on predefined evaluation metrics, the evaluation metrics include diagnostic and treatment assessment accuracy, F1 score for pronunciation defect identification, and model stability.

[0070] Accuracy of diagnosis and treatment assessment The calculation formula is expressed as follows:

[0071] ,

[0072] The formula for calculating the F1 score for pronunciation defect identification is as follows:

[0073] ,

[0074] in, , ;

[0075] When evaluating model stability, determine whether the accuracy fluctuation of consecutive predetermined number of inferences is less than a threshold.

[0076] The state assessment system based on a lightweight Chinese speech rehabilitation model of the present invention has the following advantages:

[0077] 1. Significantly improved domain adaptability and high assessment accuracy: Through domain-adaptive pre-training and fine-tuning, the model specifically learns "non-standard speech features" and "state-related features" in the field of Chinese speech rehabilitation. Compared with general large models, the accuracy of Chinese speech defect recognition is improved by more than 35%, and the accuracy of type assessment is stable at ≥90%, meeting the accuracy requirements of business (such as the accuracy of distinguishing between "dialect pronunciation differences" and "pathological pronunciation defects" reaches 92%).

[0078] 2. Strong cross-modal generalization ability and excellent robustness: Cross-modal self-supervised knowledge distillation combined with intra-modal / inter-modal self-supervised tasks enables the student model to integrate multi-source information of "speech-text-label". When faced with unseen dialect samples (such as Cantonese and Sichuanese speech defect samples), the assessment accuracy decreases by ≤5%, which is far better than the single-modal distillation model (decreases by ≥15%), and is suitable for Chinese speech rehabilitation scenarios in different regions.

[0079] 3. Outstanding lightweight effect, adapted to low computing power terminals: Multi-dimensional lightweight technology reduces model storage space from 10GB+ of traditional large models to less than 500MB (compression ratio 20:1), and inference time from 500ms+ to less than 100ms. It can run smoothly on ordinary PCs (4GB memory) and home tablets (2GB memory) in grassroots institutions, solving the problem of "large models are unaffordable and unusable".

[0080] 4. Highly practical and promotes widespread adoption: The modular output structure supports on-demand access to assessment functions (e.g., primary care facilities only need "pronunciation assessment + type classification", while home terminals only need "training phase"), reducing operational complexity; the model can directly connect to existing speech rehabilitation equipment (e.g., voice acquisition microphones, rehabilitation training apps), without the need to reconstruct the hardware system, significantly reducing implementation costs and helping Chinese speech rehabilitation technology penetrate into primary care and home settings. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0082] The invention will be further described below with reference to the accompanying drawings.

[0083] Figure 1 This is a structural block diagram of a state assessment system based on a lightweight Chinese speech rehabilitation big data model, as an example. Detailed Implementation

[0084] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0085] This invention provides a state assessment system based on a lightweight Chinese speech rehabilitation model to address the technical problems of poor adaptability, imbalance between lightweight design and accuracy, and weak cross-modal generalization in existing Chinese speech rehabilitation models.

[0086] Example: The present invention provides a state assessment system based on a lightweight Chinese speech rehabilitation model, comprising a data acquisition module, a teacher model construction module, a teacher model training module, a student model construction and training module, and a model deployment and application module.

[0087] The data acquisition module is used to collect and preprocess speech data, text data, and cross-modal assessment labels related to Chinese speech rehabilitation to form a sample set.

[0088] As a specific implementation of the data acquisition module, this module is used to collect specialized datasets in the field of Chinese speech rehabilitation, including:

[0089] ① Voice data (100,000+ records, including pronunciation samples from patients with articulation disorders, aphasia, etc., covering Mandarin and 7 major dialect regions);

[0090] ②Text data (50,000+ records, including patient condition descriptions, doctor's treatment records, and rehabilitation training plans);

[0091] ③ Cross-modal labels (each data entry is labeled with cross-modal labels such as "pronunciation defect type", "state type", "training stage" etc.)

[0092] The data was then cleaned (silent segments were removed, and annotation errors were corrected) and standardized (the speech sampling rate was uniformly set to 16kHz, and the text was uniformly encoded in UTF-8).

[0093] The teacher model construction module is used to construct a large-scale Chinese speech rehabilitation model as a teacher model. The teacher model extracts features from the input speech and text data, performs cross-modal feature alignment on the extracted acoustic and text features to obtain cross-modal fusion features, and performs speech defect mask reconstruction and cross-modal evaluation based on the cross-modal fusion features and the enhanced acoustic and text features. The speech defect mask reconstruction outputs the completed speech data, pronunciation defect labels and probability distributions, and the cross-modal evaluation outputs the pronunciation defect types and probability distributions, state type labels and probability distributions, and the evaluation results of the training phase.

[0094] As a specific implementation of the teacher model, the model includes a base model layer, a cross-modal feature alignment layer, and a feature enhancement evaluation layer.

[0095] The base model layer is used to semantically encode the input text data to obtain text features, and to extract acoustic features from the input speech data to obtain acoustic features including pronunciation details and defect features.

[0096] As a specific implementation of the basic model layer, it includes a text encoding layer and a speech encoding layer.

[0097] The text encoding layer is used to pass the Transformer encoder of ERNIE-3.0. It contains 6-12 stacked Transformer blocks and is responsible for extracting the semantic features of text data (disease description, medical records, etc.) as text feature output. While retaining general semantic understanding capabilities, it reserves optimization space for subsequent domain adaptation.

[0098] The speech coding layer is based on the convolutional and Transformer structures in Wav2Vec2.0. It contains 3 Conv1d feature extraction layers (capturing acoustic details) + 12 Transformer coding layers (modeling speech temporal features). It processes 16kHz standardized speech data (samples of articulation disorders, dialect pronunciations, etc.), extracts speech features from the input speech data, and outputs acoustic features including pronunciation details and defect features.

[0099] The cross-modal feature alignment layer is used to fuse and align textual and acoustic features based on a dual cross-attention mechanism and a cross-loss function, and output cross-modal fused features.

[0100] As a specific implementation of the cross-modal feature alignment layer, this network layer takes text features and acoustic features as input, mapping 768-dimensional text features and 768-dimensional acoustic features to a unified shared feature space to eliminate modal dimensional differences. A speech-text mutual attention matrix is ​​constructed based on a bidirectional cross-attention mechanism. This matrix fuses text features and acoustic features (e.g., using the speech and acoustic features of "apple" to correct semantic bias in the text "pingguo"). Positive and negative samples are constructed, and a contrastive loss is introduced to constrain the feature distance between text features and acoustic features, resulting in cross-modal fused features. Matched speech and text feature pairs are considered positive samples, and mismatched pairs are considered negative samples. The contrastive loss is calculated as follows:

[0101] ,

[0102] in, Represents cosine similarity. This represents a positive sample pair constructed from the same user's voice and text features. ( )) represents negative sample pairs constructed by different users based on speech and text features. This indicates the number of negative samples in each batch. This represents the temperature coefficient.

[0103] The feature enhancement and evaluation layer is used to enhance acoustic features through convolutional structures, capture defect features including alveolar consonant defects and articulation shifts, and obtain defect acoustic features as enhanced acoustic features. It is used to enhance text features through dictionary matching, obtain domain-enhanced text features as enhanced text features. It is used to reconstruct speech defect masks based on the masked speech data based on defect acoustic features and cross-modal fusion features, and outputs the speech defect type and the completed speech data. It is used to perform cross-modal evaluation based on the speech defect mask reconstruction generated as a prior, based on defect acoustic features, domain-enhanced text features and cross-modal fusion features, and outputs speech defect type and probability distribution, state type label and probability distribution, and evaluation results of the training phase.

[0104] As a specific implementation of the feature enhancement evaluation layer, it includes a defect feature extraction layer, a knowledge integration layer, and a mask reconstruction and prediction layer.

[0105] The defect feature extraction layer consists of three lightweight Conv1d convolutional layers (kernel size 3×3), which are concatenated at the output of the speech coding layer. This replaces the irrelevant noise filtering layer in the general model. It takes acoustic features as input, captures specific acoustic features including alveolar consonant defects and articulation shifts, and outputs defect acoustic features.

[0106] The knowledge integration layer is used to enhance the weights of text features through domain dictionary matching (including terms such as "diarrhea" and "motor aphasia"), and output domain-enhanced text features.

[0107] The mask reconstruction and prediction layer includes a speech defect mask prediction sublayer and a cross-modal label prediction sublayer. The speech defect mask prediction sublayer is used to perform cross-attention reconstruction based on enhanced defect acoustic features and cross-modal fusion features, reconstruct artificially masked segments in speech data, and output the speech defect type (such as "stuttering" or "unclear pronunciation"), probability distribution, and completed speech data. The cross-modal label prediction sublayer uses the speech defect type output by feature reconstruction as a prior, and performs evaluation analysis based on enhanced defect acoustic features, domain-enhanced text features, and cross-modal fusion features. The output includes the speech defect type and probability distribution, state type label and probability distribution, and evaluation results from the training phase.

[0108] The teacher model training module is used to construct a loss function based on cross-modal alignment training, speech defect mask reconstruction training, and state type training. The teacher model is trained based on the sample set by minimizing the loss function.

[0109] As a specific implementation of the teacher model training module, this module is used to build cross-modal alignment pre-training and domain feature pre-training.

[0110] The cross-modal alignment pre-training process matches speech samples with text descriptions and optimizes inter-modal feature mapping through contrastive learning (e.g., inputting "speech of user pronouncing 'apple' + incorrect text 'pingguo'", the model learns to correct the text and align speech features). The goal is to optimize the modal consistency between speech and text features, employing a dual loss function of "cosine similarity loss + contrastive loss".

[0111] The speech-text feature cosine similarity loss, as a loss constructed based on cross-modal alignment training, is calculated using the following formula:

[0112] ,

[0113] The acoustic feature vector, with 768 dimensions, is obtained by concatenating features from 16kHz speech frames extracted by Wav2Vec2.0. This represents the text feature vector, which has 768 dimensions and is extracted from text features by ERNIE-3.0. This represents the L2 norm, with a loss value ranging from [0,2]. A smaller value indicates better modal alignment.

[0114] Domain feature pre-training – Design the “pronunciation defect mask reconstruction” task (masking defect segments in speech to allow the model to predict the defect type and the original pronunciation) and the “state type evaluation” task (predicting state type labels based on speech / text features) to enable the model to learn the specific features of the Chinese speech rehabilitation domain (such as the acoustic features of “apical consonant pronunciation defect”).

[0115] In the pronunciation defect mask reconstruction task, for 10%-20% of the random frame masks in the speech samples (mask interval length of 100ms-300ms, corresponding to 1600-4800 sampling points), the CTC loss is used to calculate the reconstruction error. Correspondingly, the CTC loss is constructed based on the speech defect mask reconstruction training. The calculation formula is expressed as follows:

[0116] ,

[0117] The label represents the pronunciation defect label corresponding to the original speech data (N is the number of speech frames, and the label value is {0: normal, 1: alveolar consonant defect, 2: velar consonant defect, 3: tone defect}). This indicates that the teacher model reconstructs the output pronunciation defect label probability distribution through a speech defect mask, ensuring that the model learns the acoustic feature patterns of defect frames.

[0118] State type label prediction task: For multi-class labels such as "state type" (mild / moderate / severe, labels are {0,1,2}), cross-entropy loss is used. The cross-entropy loss constructed based on state type label prediction is as follows. The calculation formula is expressed as follows:

[0119] ,

[0120] Indicates the number of samples in the sample set. This indicates the number of categories in the status type label. Indicates sample In category One-hot encoding on, Indicates sample In category The probability of the teacher model predicting the output, in multi-state type label scenarios (such as when a user has both "dialectics" and "tone deficiency"), For multi-label binary encoding, the loss function remains unchanged.

[0121] Based on application scenario requirements, the teacher model was fine-tuned using fully annotated "speech-text-cross-modal label" data (20,000+ records). The total loss during the fine-tuning phase was a weighted sum of the losses from each task. Adapting to the priorities of the actual application scenario, the loss function of the teacher model was then determined. Represented as:

[0122] ,

[0123] in, express The weight, express The weight, express The weight. (The state type evaluation loss has the highest weight, so priority is given to ensuring the accuracy of state type classification). (Cross-modal consistency weighting is secondary, ensuring voice-text information matching.) (Defect reconstruction has the lowest weight, assisting in optimizing feature extraction), and The training batch size is set to 32, and the learning rate uses a cosine annealing strategy (initial...). , minimum ).

[0124] The student model construction and training module is used to trim the trained teacher model into a student model, perform intra-modal and inter-modal self-supervised training on the student model, and perform knowledge distillation on the student model based on the cross-modal fusion features and state type label probability distribution output by the teacher model.

[0125] As a specific implementation of the student model construction and training module, this module designs a cross-modal self-supervised knowledge distillation framework to transfer cross-modal knowledge from the teacher model to the student model, thereby improving the robustness and generalization ability of the student model. This module supports the following operations:

[0126] (1) Initialization of student model: Based on the trained teacher model, retain the network layers in the teacher model used for feature extraction, cross-modal alignment, feature enhancement, speech defect mask reconstruction and cross-modal evaluation prediction, and prune redundant network layers (such as the redundant Transformer block in ERNIE-3.0 for processing "general news text" and the unnecessary convolutional layer in Wav2Vec 2.0 for filtering "general environmental noise") to obtain the initialized student model, with the initial parameter size being 1 / 4 of the teacher model;

[0127] (2) Intramodal self-supervision: Random segment masking (masking 10%-20% of speech frames) is performed on speech data. The masked speech data is used as input, and the masked region is reconstructed through the student model. The loss function is constructed based on the probability distribution of pronunciation defect labels output by the masked region reconstruction and the probability value of the real pronunciation defect. The model is trained by minimizing the loss function. Random segment masking (masking terms such as "dialectics" and "motor aphasia") is performed on text data. The masked text data is used as input, and the masked region is reconstructed through the student model. The loss function is constructed based on the probability distribution of text data output by the masked region reconstruction and the probability value of the real text data. The model is trained by minimizing the loss function, thereby enhancing the ability to capture domain features.

[0128] (3) Intermodal self-supervision: Construct a "cross-modal contrastive learning task", using the speech features and text features of the same user as positive samples and the cross-modal fusion features of different users as negative samples. Optimize the cross-modal feature alignment ability of the student model through contrastive loss. The contrastive loss calculation formula is as follows:

[0129] ,

[0130] in, Indicates the size of the learning batch. Indicates the first Acoustic features (feature dimensions) corresponding to each speech data point (simplified speech feature dimensions of the student model). Indicates the first Text features (feature dimensions) corresponding to each text data point ), (Smaller temperature coefficient enhances the distinction between positive and negative samples) Each sample in each batch corresponds to 3 negative samples (text features of other samples in the same batch are randomly selected).

[0131] (4) Feature distillation: Based on the cross-modal fusion features output by the teacher model (such as the concatenated vector of speech acoustic features and text semantic features), the student model learns the feature distribution of the cross-modal fusion features through L2 loss to ensure that key domain features are not lost. The corresponding loss function is... The calculation formula is:

[0132] ,

[0133] in, This represents the cross-modal fusion features output by the teacher model. This represents the cross-modal fusion features of the student model. The dimension of the cross-modal fusion features is represented by D=2048, obtained by concatenating speech and text features through a single fully connected layer. A smaller loss value indicates a closer similarity in feature distributions. During training, it is required that... ;

[0134] (5) Probability Distillation: Based on the probability distribution of state type labels output by the teacher model (e.g., "probability of dysarthria 80%, probability of aphasia 15%)", the student model learns the probability distribution of diagnosis labels through KL divergence loss to improve the consistency of diagnosis and treatment assessment. During the distillation process, "temperature coefficient adjustment" is used to soften the probability distribution, improve the distillation effect, and avoid the student model from getting trapped in local optima. The corresponding loss function is... The calculation formula is:

[0135] ,

[0136] in, , , This represents the probability distribution of the state type labels output by the teacher model. This represents the probability distribution of the state type labels output by the student model. Indicates the total number of state types. This represents an empirical value used to balance the smoothness and discriminative power of a probability distribution. It degenerates into ordinary cross-entropy loss.

[0137] The model deployment application module is used to perform low-dimensional embedding compression on the trained student model and introduce quantization-aware training during the training phase of the student model to obtain a lightweight model that includes multiple independent modules. After evaluating the lightweight model based on predefined evaluation metrics, Chinese speech state evaluation is performed based on the lightweight model.

[0138] As a specific implementation of model evaluation deployment, the module reduces model overhead while maintaining evaluation accuracy through a triple technique of "low-dimensional embedding compression + quantization-aware training + modular output". The module performs the following operations:

[0139] (1) Low-dimensional embedding compression: For the cross-modal fusion features output by the student model (usually 2048 dimensions), a "domain-adapted low-dimensional embedder" is designed. The encoder in the student model compresses the high-dimensional cross-modal fusion features to a low dimension (256 dimensions, compression ratio 8:1). The decoder in the student model reconstructs the compressed cross-modal fusion features. Based on the deviation between the high-dimensional cross-modal fusion features and the low-dimensional cross-modal fusion features, a reconstruction loss (MSE) is constructed, and a state type label constraint is introduced (the embedded features must be strongly correlated with the state type labels). Based on the probability distribution of the state type labels output by the student model and the true state type label probability values, a label constraint loss (cross-entropy) is constructed. The reconstruction loss and the label constraint loss are weighted and summed as the total loss of low-dimensional embedding compression. The variance explained rate of the compressed cross-modal fusion features is analyzed by PCA (≥92%) to ensure the retention of domain features. The student model is trained based on the total loss of low-dimensional embedding compression. The calculation formula is:

[0140] ,

[0141] in, Indicates the reconstruction loss. This represents the label constraint loss, ensuring that the embedded features are strongly correlated with the state type labels for accurate prediction. The weighting coefficients are used to prioritize reconstruction accuracy and avoid feature distortion. After embedding, the feature dimension d=256, and the compression ratio is... , The calculation formula is:

[0142] ,

[0143] Indicates high-dimensional cross-modal fusion characteristics. This represents the cross-modal fusion features of the reconstruction. The original dimension representing the cross-modal fusion feature;

[0144] Among them, the variance explained rate The calculation formula is as follows:

[0145] ,

[0146] In representing the high-dimensional cross-modal fusion features, the first... The variance of each principal component;

[0147] (2) Quantization-aware training: Quantization logic is integrated into the model training stage to achieve a balance between accuracy and lightweighting. The model weights of the student model are quantized using INT8, and the activation values ​​in the student model are also quantized using INT8. Quantization error compensation is inserted during model training, and the accuracy loss caused by quantization is corrected through backpropagation. The calculation formula is:

[0148] ,

[0149] The quantization error loss is represented by the following formula:

[0150] ,

[0151] This represents the weights of the student model before quantization. This represents the weights of the quantized student model. Indicates the total weight. This represents the weighting coefficient (which controls the impact of quantization error on the total loss). The loss function representing the teacher model;

[0152] In this study, the key network layers in the student model are quantized using INT8, while the non-key layers are quantized using INT4. During quantization-aware training, a weight range and quantization step size are set, and progressive quantization is implemented. FP32 is used in the early training rounds, and quantization is gradually introduced in the later rounds. For INT8 quantization, the weight range is mapped to [-127, 127], and the quantization step size is:

[0153] ;

[0154] INT4 quantization is mapped to [-7, 7] with a step size of:

[0155] ,

[0156] Non-critical layers (such as dropout layers for text features) use INT4, while critical layers (such as cross-modal fusion layers) use INT8.

[0157] (3) Modular output: Based on the actual application scenario requirements, the trained student model is divided into three independent modules that can be called on demand: the pronunciation evaluation module, the state type classification module, and the training phase suggestion module.

[0158] The pronunciation assessment module is used to output the types of pronunciation defects and their probability distributions, such as "pronunciation defect type" (e.g., apical consonant defect, velar consonant defect) and "pronunciation accuracy score" (0-100 points).

[0159] The state type classification module is used to output state type labels and probability distributions, such as outputting "pathological type of speech disorder" (e.g., dysarthria, sensory aphasia) and "severity rating" (mild / moderate / severe).

[0160] The training phase suggestion module is used to output the training phase, such as outputting "personalized training" (e.g., "tongue tip against palate exercise" for deficiencies in alveolar consonants).

[0161] Deploy lightweight models to target terminals (such as basic PCs and home tablets), optimize the inference process through "model inference acceleration engines" (such as TensorRT and ONNXRuntime), and ensure that the time for a single evaluation is ≤100ms and the storage usage is ≤500MB.

[0162] (4) Model evaluation: The deployed lightweight model is evaluated based on predefined evaluation indicators. As a specific implementation of model evaluation, the evaluation indicators are the accuracy of diagnosis and treatment assessment, the F1 score of pronunciation defect recognition, and the model stability.

[0163] Accuracy of diagnosis and treatment assessment The calculation formula is expressed as follows:

[0164] ,

[0165] The formula for calculating the F1 score for pronunciation defect identification is as follows:

[0166] ,

[0167] in, , ;

[0168] When evaluating model stability, determine whether the accuracy fluctuation of consecutive predetermined number of inferences is less than a threshold.

[0169] (5) Model application: The speech and text data to be analyzed are collected by the data acquisition module as input data. The Chinese speech state is evaluated based on the lightweight model. The output includes pronunciation defect type and probability distribution, state type label and probability distribution, and evaluation results of the training phase.

[0170] Based on the system disclosed in this embodiment, a specific case is given. This embodiment addresses the assessment needs of children with articulation disorders in a community health service center. The goal is to build a lightweight Chinese speech rehabilitation model, which is deployed on a regular PC terminal (not a professional AI server) used by community doctors. This model enables rapid assessment of "apical / velar consonant defects" in children aged 3-6 (single assessment time ≤100ms), classification of condition types (mild / moderate / severe), and generation of personalized training suggestions, thus solving the pain points of "limited computing power and insufficient professional assessment personnel" in community institutions.

[0171] Step 1: Data Preparation and Preprocessing

[0172] 1. Dataset Construction: Collect dedicated data on Chinese children's articulation disorders, including: ① Speech data (30,000 entries, including 20 core words of alveolar / velar consonants such as "bàbà" and "gēge" pronounced by children aged 3-6, of which 10,000 are normal pronunciations, 12,000 are alveolar consonant defects, and 8,000 are velar consonant defects, with a sampling rate of 16kHz and a single entry duration of 1-3 seconds); ② Text data (15,000 entries, including parents' descriptions of "child's speech is lisping" and doctors' initial consultation records of "weakness in raising the tip of the tongue"); ③ Status type labels (each data entry is labeled with "defect type" and "severity", such as "alveolar consonant defect - mild").

[0173] 2. Data cleaning: Environmental noise in the speech samples was removed using the Audacity tool (samples with a signal-to-noise ratio <20dB were removed, and a total of 28,000 valid speech samples were selected); the text data was preprocessed using Jieba word segmentation to filter out meaningless interjections such as "um" and "ah", and retain key descriptions such as "pronunciation is lisping" and "tongue is not flexible".

[0174] Step 2: Domain-Adaptive Teacher Model Construction (Executed on Data Center Server)

[0175] 1. Domain-adaptive pre-training continues

[0176] (1) Cross-modal alignment pre-training:

[0177] Speech features s (D_s=768) were extracted using Wav2Vec2.0 (Chinese fine-tuned version), and text features t (D_s=768) were extracted using ERNIE-3.0. Each batch inputs 32 sets of "voice-text" samples (including 24 positive samples and 8 negative samples, K=10), with a temperature coefficient. .

[0178] (2) Reconstruction of pronunciation defect mask:

[0179] Speech samples are randomly masked at a ratio of 15% (masking interval 200ms, corresponding to 3200 sampling points), and CTC loss is applied. optimization.

[0180] (3) State type label prediction:

[0181] For the "severity level" (C=3), cross-entropy loss is used. .

[0182] 2. Targeted fine-tuning and optimization

[0183] According to the formula The total loss was calculated using cosine annealing (initial lr=1e-5, reduced to 1e-7 in round 30) and a batch size of 32.

[0184] Step 3: Cross-modal self-supervised knowledge distillation (student model construction)

[0185] 1. Student model initialization

[0186] Based on the teacher model, the "general semantic understanding layer" (accounting for 35% of the teacher model parameters) was trimmed, while the "cross-modal feature fusion layer" and "diagnosis and evaluation layer" were retained, reducing the initial parameter size from 2.4GB (FP32) of the teacher model to 1.56GB.

[0187] 2. Two-stage knowledge distillation

[0188] (1) Characteristic distillation (L2 loss):

[0189] According to the formula Calculation, teacher model fusion features Student model fusion features .

[0190] (2) Probabilistic distillation (KL divergence):

[0191] Temperature coefficient T=1.5, number of pathological types C=4 (normal / apical consonant defect / velar consonant defect / mixed defect), according to the formula calculate.

[0192] 3. Cross-modal self-supervised contrastive learning

[0193] According to the formula Calculations: Batch size B=16, temperature coefficient Each sample corresponds to 3 negative samples.

[0194] Step 4: Multi-dimensional lightweight optimization (deployment-side adaptation)

[0195] 1. Low-dimensional embedding compression

[0196] Autoencoder training: according to the formula The original feature D=2048 was calculated, and the embedded feature d=256 (compression ratio 8:1).

[0197] 2. Quantitative Awareness Training (QAT)

[0198] Quantization configuration: The critical layer (cross-modal fusion layer) uses INT8 quantization, and the non-critical layer (text dropout layer) uses INT4 quantization. The quantization step size is calculated according to the formula:

[0199] (1) INT8 quantization: The FP32 weight range of the teacher model is [-3.2, 2.8], and the quantization step size is... ;

[0200] (2) INT4 quantization: FP32 weight range for text layer [-1.5, 1.3], quantization step size ;

[0201] (3) Quantization effect: The model parameters were reduced from 1.56GB (FP32) to 210MB (INT8+INT4 hybrid quantization), and the storage usage was compressed by 7.4 times, meeting the requirement of "≤500MB".

[0202] 3. Inference latency optimization

[0203] On the community PC (Intel i5-12400, integrated graphics), follow the formula. calculate.

[0204] Step 5: Application Deployment (Practical Testing by Community Staff)

[0205] 1. Accuracy Verification: The model evaluation results were compared with the expert manual evaluation results in 280 children's speech samples in the test set;

[0206] 2. Stability verification: Perform 1000 consecutive inferences, calculate the accuracy fluctuation range and fluctuation amplitude, and determine whether there are any abnormal crashes.

[0207] The system in this embodiment first constructs a teacher model by continuing pre-training or fine-tuning based on the data characteristics of the medical and health care field, and then designs a cross-modal self-supervised knowledge distillation method to construct a more robust student model with stronger generalization ability. Finally, it significantly reduces the output expression dimension, computational overhead and storage footprint of the model by designing low-dimensional embedding compression technology, implementing quantitative perception training and adopting modular output structure.

[0208] The above provides a detailed description of the state assessment system based on a lightweight Chinese speech rehabilitation model provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A state assessment system based on a lightweight Chinese speech rehabilitation model, characterized in that, include: The data acquisition module is used to collect and preprocess speech data, text data, and cross-modal assessment labels related to Chinese speech rehabilitation to form a sample set; The teacher model construction module is used to construct a large-scale Chinese speech rehabilitation model as a teacher model. The teacher model extracts features from the input speech and text data, performs cross-modal feature alignment on the extracted acoustic and text features to obtain cross-modal fusion features, and performs speech defect mask reconstruction and cross-modal evaluation based on the cross-modal fusion features and the enhanced acoustic and text features. The speech defect mask reconstruction outputs the completed speech data, pronunciation defect labels and probability distributions, and the cross-modal evaluation outputs the pronunciation defect types and probability distributions, state type labels and probability distributions, and the evaluation results of the training phase. The teacher model training module is used to construct a loss function based on cross-modal alignment training, speech defect mask reconstruction training, and state type training. The teacher model is trained based on the sample set by minimizing the loss function. The student model construction and training module is used to trim the trained teacher model into a student model, perform intra-modal and inter-modal self-supervised training on the student model, and perform knowledge distillation on the student model based on the cross-modal fusion features and state type label probability distribution output by the teacher model. The model deployment application module is used to perform low-dimensional embedding compression on the trained student model and introduce quantization-aware training during the training phase of the student model to obtain a lightweight model that includes multiple independent modules. After evaluating the lightweight model based on predefined evaluation metrics, Chinese speech state evaluation is performed based on the lightweight model. The teacher model includes a base model layer, a cross-modal feature alignment layer, and a feature enhancement evaluation layer. The base model layer is used to perform semantic encoding on the input text data to obtain text features, and to extract acoustic features from the input speech data to obtain acoustic features including pronunciation details and defect features. The cross-modal feature alignment layer is used to fuse and align textual and acoustic features based on a dual cross-attention mechanism and a cross-loss function, and output cross-modal fused features. The feature enhancement and evaluation layer is used to enhance acoustic features through convolutional structures, capture defect features including alveolar consonant defects and articulation shifts, and obtain defect acoustic features as enhanced acoustic features. It is used to enhance text features through dictionary matching, obtain domain-enhanced text features as enhanced text features. It is used to reconstruct speech defect masks based on speech data masked by defect acoustic features and cross-modal fusion features, and outputs speech defect types and completed speech data. It is used to perform cross-modal evaluation based on speech defect mask reconstruction and defect acoustic features, domain-enhanced text features and cross-modal fusion features, using the speech defect types generated by speech defect mask reconstruction as priors, and outputs speech defect types and probability distributions, state type labels and probability distributions, and evaluation results of the training phase. The feature enhancement evaluation layer includes a defect feature extraction layer, a knowledge integration layer, and a mask reconstruction and prediction layer. The defect feature extraction layer is connected in series with the output of the speech coding layer. It takes acoustic features as input, captures specific acoustic features including alveolar consonant defects and articulation shifts, and outputs defect acoustic features. The knowledge integration layer is used to weight and enhance text features through domain dictionary matching, and outputs domain-enhanced text features. The mask reconstruction and prediction layer includes a speech defect mask prediction sublayer and a cross-modal label prediction sublayer. The speech defect mask prediction sublayer is used to perform cross-attention reconstruction based on enhanced defect acoustic features and cross-modal fusion features, and to reconstruct artificially masked segments in speech data, outputting pronunciation defect types, probability distributions, and completed speech data. The cross-modal label prediction sublayer uses the pronunciation defect types output by feature reconstruction as priors, and performs evaluation and analysis based on enhanced defect acoustic features, domain-enhanced text features, and cross-modal fusion features, outputting pronunciation defect types and probability distributions, state type labels and probability distributions, and evaluation results from the training phase. Among them, the loss function corresponding to the teacher model Represented as: ， in, This represents the cross-entropy loss constructed based on state type training. express The weight, The calculation formula is expressed as follows: ， Indicates the number of samples in the sample set. This indicates the number of categories in the status type label. Indicates sample In category One-hot encoding on, Indicates sample In category The teacher model predicts the probability of output in a multi-state type label scenario. Multi-label binary encoding; in, This represents the loss constructed based on cross-modal alignment training. express The weight, The calculation formula is: ， Represents the acoustic eigenvector. Represents the text feature vector. Represents the L2 norm; in, This represents the CTC loss constructed based on speech defect mask reconstruction training. express The weight, The calculation formula is expressed as follows: ， This represents the pronunciation defect labels corresponding to the original speech data. This represents the probability distribution of pronunciation defect labels reconstructed by the teacher model through a speech defect mask. The student model construction and training includes the following operations: Student model initialization: Based on the trained teacher model, retain the network layers used for feature extraction, cross-modal alignment, feature enhancement, speech defect mask reconstruction, and cross-modal evaluation in the teacher model, and prune redundant network layers to obtain the initialized student model; Intramodal self-supervision: Random segment masking is performed on speech data. The masked speech data is used as input, and a student model is used to reconstruct the masked regions. A loss function is constructed based on the probability distribution of pronunciation defect labels output by the masked region reconstruction and the probability value of the real pronunciation defects. The model is trained by minimizing the loss function. Similarly, random segment masking is performed on text data. The masked text data is used as input, and a student model is used to reconstruct the masked regions. A loss function is constructed based on the probability distribution of text data output by the masked region reconstruction and the probability value of the real text data. The model is trained by minimizing the loss function. Intermodal self-supervision: Voice and text features from the same user are used as positive samples, while cross-modal fused features from different users are used as negative samples. The cross-modal feature alignment capability of the student model is optimized through contrastive loss. The formula for calculating contrastive loss is as follows: ， in, Indicates the size of the learning batch. Indicates the first Acoustic features corresponding to each speech data point Indicates the first Text features corresponding to each piece of text data Indicates the temperature coefficient; Feature distillation: Based on the cross-modal fusion features output by the teacher model, the student model learns the feature distribution of the cross-modal fusion features through L2 loss, and the corresponding loss function is... The calculation formula is: ， in, This represents the cross-modal fusion features output by the teacher model. This represents the cross-modal fusion features of the student model. The dimension representing the cross-modal fusion feature; Probabilistic distillation: Based on the state type label probability distribution output by the teacher model, the student model learns the diagnosis label probability distribution through KL divergence loss, and the corresponding loss function... The calculation formula is: ， in, , , This represents the probability distribution of the state type labels output by the teacher model. This represents the probability distribution of the state type labels output by the student model. Indicates the number of state types. This represents an empirical value used to balance the smoothness and discriminative power of a probability distribution. The model deployment application module is used to perform the following operations: Low-dimensional embedding compression: For the cross-modal fusion features output by the student model, the encoder in the student model compresses the high-dimensional cross-modal fusion features to a low dimension. The decoder in the student model reconstructs the compressed cross-modal fusion features. A reconstruction loss is constructed based on the deviation between the high-dimensional and low-dimensional cross-modal fusion features, and state type label constraints are introduced. A label constraint loss is constructed based on the state type label probability distribution output by the student model and the true state type label probability values. The weighted sum of the reconstruction loss and the label constraint loss is taken as the total low-dimensional embedding compression loss. The variance explained rate of the compressed cross-modal fusion features is analyzed by PCA. The student model is trained based on the total low-dimensional embedding compression loss. The calculation formula is: ， in, Indicates the reconstruction loss. Represents the label constraint loss. Indicates the weighting coefficient. The calculation formula is: ， Indicates high-dimensional cross-modal fusion characteristics. This represents the cross-modal fusion features of the reconstruction. The original dimension representing the cross-modal fusion feature; Among them, the variance explained rate The calculation formula is as follows: ， In representing the high-dimensional cross-modal fusion features, the first... The variance of each principal component; Quantization-aware training: The model weights of the student model are quantized using INT8, and the activation values ​​in the student model are also quantized using INT8. Quantization error compensation is inserted during model training, and the accuracy loss caused by quantization is corrected through backpropagation. The calculation formula is: ， The quantization error loss is represented by the following formula: ， This represents the weights of the student model before quantization. This represents the weights of the quantized student model. Indicates the total weight. Indicates the weighting coefficient. The loss function representing the teacher model; Among them, the key network layers preset in the student model are quantized using INT8, and the non-key layers preset in the student model are quantized using INT4. During quantization-aware training, the weight range and quantization step size are set, and progressive quantization is performed. FP32 is used for training in the first training rounds, and quantization is gradually introduced in the back rounds. Modular output: The trained student model is divided into three independent modules: pronunciation assessment module, state type classification module, and training phase suggestion module. The pronunciation assessment module is used to output the pronunciation defect type and probability distribution, the state type classification module is used to output the state type and probability distribution, and the training phase suggestion module is used to output the training phase.

2. The state assessment system based on a lightweight Chinese speech rehabilitation model according to claim 1, characterized in that, The basic model layer includes a text encoding layer and a speech encoding layer; The text encoding layer is used to extract semantic features from speech data using the Transformer encoder of ERNIE-3.0, while retaining general semantic understanding capabilities, and outputting text features; The speech coding layer is used to extract speech features from the input speech data based on the convolutional and Transformer structures in Wav2Vec2.0, and outputs acoustic features including pronunciation details and defect features.

3. The state assessment system based on a lightweight Chinese speech rehabilitation model according to claim 1, characterized in that, The cross-modal feature alignment layer takes text features and acoustic features as input and maps them to a unified shared feature space. It constructs a speech-text mutual attention matrix based on a bidirectional cross-attention mechanism, fuses text features and acoustic features through the speech-text mutual attention matrix, constructs positive and negative samples, and introduces contrast loss to constrain the feature distance between text features and acoustic features to obtain cross-modal fused features. Matched speech and text feature pairs are used as positive samples, and mismatched speech and text feature pairs are used as negative samples. The formula for calculating the contrast loss is as follows: ， in, Represents cosine similarity. This represents a positive sample pair constructed from the same user's voice and text features. ( () represents negative sample pairs constructed by different users based on speech and text features. This indicates the number of negative samples in each batch. This represents the temperature coefficient.

4. The state assessment system based on a lightweight Chinese speech rehabilitation model according to claim 1, characterized in that, When evaluating the lightweight model based on predefined evaluation metrics, the evaluation metrics are diagnostic assessment accuracy, F1 score for pronunciation defect identification, and model stability. Accuracy of diagnosis and treatment assessment The calculation formula is expressed as follows: ， The formula for calculating the F1 score for pronunciation defect identification is as follows: ， in, , ; When evaluating model stability, determine whether the accuracy fluctuation of consecutive predetermined number of inferences is less than a threshold.

Images