Spoken language expression ability AI evaluation method, device, equipment and medium

Abstract

The invention relates to a spoken language expression ability AI evaluation method and device, equipment and a medium. The method comprises the following steps: performing multi-granularity feature decoupling processing on an original voice signal to obtain acoustic, linguistic and sub-linguistic feature sets; generating a specific feature representation of each scoring task based on a shared underlying encoder and a task specific expert network; constructing a differential scoring rule graph to quantify a relationship between tasks; identifying a gradient conflict task pair through forward gradient prediction and conflict detection; implementing a dynamic parameter freezing strategy according to the conflict index to generate an optimized gradient direction; model parameters are updated in combination with a multi-target loss function of dynamic weight calibration; and finally, generating a scoring result through multi-task score fusion. By adopting the method, the problems of gradient conflicts, inconsistent scoring standard semantics and calibration failure in multi-task learning can be effectively solved through task relation modeling and dynamic gradient reconciliation, and the evaluation accuracy and stability are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to an oral expression ability AI evaluation method, device, equipment and medium. BACKGROUND

[0002] With the in-depth application of artificial intelligence technology in the field of language education, oral expression ability AI evaluation systems based on multi-task learning have gradually become popular. Such systems usually adopt a shared encoder architecture to simultaneously evaluate the pronunciation accuracy, language fluency, vocabulary diversity and logical coherence of learners in multiple dimensions, aiming to provide comprehensive oral ability analysis.

[0003] In traditional technology, the system processes the speech signal through a unified feature extraction network, and generates multiple task-specific prediction heads in parallel based on shared representations. Each scoring task uses an independent loss function for joint optimization, relying on gradient weighted averaging or simple loss weighting to balance the learning process of different tasks.

[0004] However, the existing multi-task learning architecture has obvious technical limitations. First, the optimization objectives of different scoring tasks are inherently contradictory, for example, fluency evaluation requires punishing all long pauses, while logical coherence evaluation needs to reward reasonable thinking pauses. This conflict of objectives leads to the gradient direction of the model being mutually opposed during training, significantly reducing the final performance of each task. Second, traditional methods cannot perceive the semantic context of the task, and the same speech feature is given opposite scoring significance in different scenarios, causing inconsistency in the scoring standard. In addition, the bias cancellation phenomenon that occurs during multi-score calibration, as well as the calibration failure problem when facing unknown speech content, all seriously affect the accuracy and reliability of the evaluation results. These defects limit the effective application of existing systems in actual educational scenarios. SUMMARY

[0005] Therefore, it is necessary to provide an oral expression ability AI evaluation method, device, equipment and medium to solve the above technical problems.

[0006] In a first aspect, the present application provides an oral expression ability AI evaluation method, comprising:

[0007] S1, performing multi-granularity feature decoupling processing on the collected original speech signal, extracting multi-dimensional feature data in the acoustic feature space, the linguistic feature space and the paralinguistic feature space, and generating a decoupled feature set;

[0008] S2, performing task-aware coding processing on the decoupled feature set through a hierarchical structure of a shared bottom-layer encoder and a task-specific top-layer expert network, to generate specific feature representations for pronunciation accuracy, fluency, vocabulary diversity and logical coherence scoring tasks;

[0009] S3, constructing a differentiable scoring rule graph according to the semantic description of each scoring task and the label co-occurrence relationship in the training data based on the specific feature representation, and generating a task relationship graph based on the differentiable scoring rule graph;

[0010] S4, based on the task relationship graph and the model parameters of the shared bottom layer encoder and the task-specific top layer expert network, calculating the predicted gradient direction of each task on the shared layer through forward gradient prediction to generate an initial gradient direction set;

[0011] S5, based on the initial gradient direction set and the task relationship graph, calculating the conflict index of each task through a conflict detection mechanism, and implementing a dynamic parameter freezing strategy according to the conflict index to generate an optimized gradient update direction;

[0012] S6, based on the optimized gradient update direction, combining the multi-objective loss function of each scoring task after dynamic weight calibration, updating the model parameters of the shared bottom layer encoder and the task-specific top layer expert network, and generating an updated oral assessment model;

[0013] S7, inputting the voice to be evaluated into the updated oral assessment model, weighting and fusing the predicted scores of each scoring task to generate an oral expression ability score result.

[0014] In one of the embodiments, the collected original voice signal is subjected to multi-granularity feature decoupling processing, and multi-dimensional feature data in the acoustic feature space, the linguistic feature space and the paralinguistic feature space are extracted to generate a decoupled feature set, including:

[0015] S11, performing volume calibration processing on the original voice signal collected by the dual-channel synchronization to obtain standardized voice waveform data; the standardized voice waveform data is used to represent that the voice signal quality meets the preset standard;

[0016] S12, extracting low-level acoustic features of mel-frequency cepstrum coefficient, fundamental frequency trajectory and energy distribution from the standardized voice waveform data, and generating acoustic feature space data by calculating vowel format deviation and consonant voicing contrast; the calculation formula of the vowel format deviation is:

[0017]

[0018] wherein, is the vowel format deviation, represents the th format frequency, represents the original vowel, represents the standard vowel;

[0019] S13, convert the voice into text through an end-to-end voice recognition engine, extract lexical diversity, syntactic complexity and semantic coherence features based on the converted text, calculate semantic density in combination with lexical embedding vectors in a pre-trained semantic space, and generate a linguistic feature space data; the calculation formula of the semantic density is:

[0020]

[0021] wherein, is the semantic density, is the embedding vector of the lexical , and is the total number of words;

[0022] S14, extract pause frequency, pause duration distribution, speech rate change and prosody features from the original voice signal, distinguish hesitation pauses from thinking pauses through a context-aware pause classification model, and generate paralinguistic feature space data;

[0023] S15, splice the acoustic feature space data, the linguistic feature space data and the paralinguistic feature space data according to the feature dimensions, and generate a decoupled feature set.

[0024] In one of the embodiments, based on the model parameters of the task relationship graph and the shared bottom layer encoder and the task-specific top layer expert network, the predicted gradient directions of each task on the shared layer are calculated through forward gradient prediction to generate an initial gradient direction set, including:

[0025] S21, based on the current parameters of the shared bottom layer encoder and the decoupled feature set, the intermediate activation values of each scoring task are calculated through forward propagation to obtain a forward propagation result;

[0026] S22, based on the forward propagation result and the current parameters of the task-specific top layer expert network, the base loss values of each scoring task corresponding to its true label are calculated to obtain a task loss set;

[0027] S23, according to the task loss set, the predicted gradient of the shared bottom layer encoder parameters for each task is approximately calculated through the gradient direction prediction network to generate an initial gradient direction set.

[0028] In one of the embodiments, based on the initial gradient direction set and the task relationship graph, the conflict index of each task is calculated through a conflict detection mechanism, and a dynamic parameter freezing strategy is implemented according to the conflict index to generate an optimized gradient update direction, including:

[0029] S31, based on the initial gradient direction set, the gradient cosine similarity between all scoring task pairs is calculated through the following formula to obtain a gradient cosine similarity matrix:

[0030]

[0031] wherein, is the gradient cosine similarity between tasks and , and are the predicted gradient directions of tasks and , respectively;

[0032] S32, based on the gradient cosine similarity matrix and the historical relationship types recorded in the task relationship graph, the real-time conflict index of each scoring task is obtained by the following formula:

[0033]

[0034] wherein, is the conflict index of task , is the gradient cosine similarity between tasks and , is the relationship weight of the task pair obtained based on the task relationship graph;

[0035] S33, based on the real-time conflict index of each task, the first preset threshold and the second preset threshold, the corresponding dynamic parameter freezing operation result is obtained;

[0036] S34, based on the shared layer parameters that are not frozen, the initial gradient direction set is projected and adjusted to obtain a gradient projection result;

[0037] S35, the dynamic parameter freezing operation result and the gradient projection result are integrated to generate the final optimized gradient update direction used to update the shared bottom layer encoder parameters.

[0038] In one of the embodiments, based on the real-time conflict index of each task, the first preset threshold and the second preset threshold, the corresponding dynamic parameter freezing operation result is obtained, including:

[0039] S41, when the real-time conflict index is less than or equal to the first preset threshold, the dynamic parameter freezing operation result is marked as mild conflict, and only the specific neuron subset in the shared layer that contributes most to the conflict is frozen;

[0040] S42, when the real-time conflict index is greater than the first preset threshold and less than or equal to the second preset threshold, the dynamic parameter freezing operation result is marked as moderate conflict, and the entire functional branch related to the high conflict task in the shared layer is frozen;

[0041] S43, when the real-time conflict index is greater than the second preset threshold, the result of the dynamic parameter freezing operation is to mark it as a severe conflict, group the conflict tasks and trigger the alternating training mode.

[0042] In one embodiment, based on the optimized gradient update direction and combined with the multi-objective loss function calibrated by the dynamic weights of each scoring task, the model parameters of the shared underlying encoder and the task-specific top-level expert network are updated to generate an updated oral assessment model, including:

[0043] S51. Calculate the base loss function value for each scoring task in this training batch; the base loss function includes the contrast loss for the pronunciation accuracy task, the segmentation ranking loss for the fluency task, the similarity-based regularization loss for the vocabulary diversity task, and the graph structure loss for the logical coherence task.

[0044] S52. Based on the ratio of the current basic loss value of each task to its recent historical loss value, the loss ratio weight is determined through a dynamic weight calculation model.

[0045] S53. Monitor the Euclidean norm of the gradient magnitude of each task, and perform weight suppression on tasks whose gradient magnitude is consistently higher than the average level to obtain the gradient magnitude weight.

[0046] S54. Combine the loss scaling weights and gradient magnitude weights to obtain the final multi-objective loss weights for each task, and construct a weighted multi-objective loss function; the expression for the multi-objective loss function is:

[0047]

[0048] in, For the multi-objective loss function value, Indicates task The final multi-objective loss weights, Indicates task The base loss value, Indicates the total number of tasks. Represents the regularization coefficient. Represents the set of all model parameters;

[0049] S55. Update the shared encoder parameters based on the optimized gradient update direction, and update the task-specific top-level expert network parameters according to the gradient generated by the weighted multi-objective loss function to generate the updated spoken language evaluation model.

[0050] Secondly, this application also provides an AI-powered oral expression ability assessment device, comprising:

[0051] The data acquisition and feature processing module is configured to perform multi-granularity feature decoupling processing on the collected original speech signal, extract multi-dimensional feature data in an acoustic feature space, a linguistics feature space and a paralanguage feature space, and generate a decoupled feature set.

[0052] The task encoding processing module is configured to perform task-aware encoding processing on the decoupled feature set by using a hierarchical structure of the shared bottom-layer encoder and the task-specific top-layer expert network, and generate specific feature representations for the pronunciation accuracy, fluency, vocabulary diversity and logical coherence scoring tasks.

[0053] The task relationship graph construction module is configured to construct a differentiable scoring rule graph based on the specific feature representations and according to semantic descriptions of the scoring tasks and label co-occurrence relationships in the training data, and generate a task relationship graph based on the differentiable scoring rule graph.

[0054] The initial gradient prediction module is configured to calculate predicted gradient directions of the tasks on the shared layer by forward gradient prediction based on the task relationship graph and model parameters of the shared bottom-layer encoder and the task-specific top-layer expert network, and generate an initial gradient direction set.

[0055] The gradient update optimization module is configured to calculate conflict indexes of the tasks by a conflict detection mechanism based on the initial gradient direction set and the task relationship graph, and implement a dynamic parameter freezing strategy according to the conflict indexes, and generate an optimized gradient update direction.

[0056] The spoken language assessment model updating module is configured to update the model parameters of the shared bottom-layer encoder and the task-specific top-layer expert network based on the optimized gradient update direction and a multi-objective loss function calibrated by dynamic weights of the scoring tasks, and generate an updated spoken language assessment model.

[0057] The spoken language expression ability scoring module is configured to input the speech to be assessed into the updated spoken language assessment model, weight and fuse predicted scores of the scoring tasks, and generate a spoken language expression ability scoring result.

[0058] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the method in the first aspect when executing the computer program.

[0059] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method in the first aspect.

[0060] The aforementioned AI-based method, apparatus, equipment, and medium for evaluating oral expression ability first obtains a set of discriminative features by decoupling multi-granularity features from the original speech signal. Then, it generates specific feature representations for each scoring task using a hierarchical coding structure. Next, it constructs a differentiable scoring rule graph to clarify the relationships between tasks. Based on this, it identifies gradient conflict task pairs through forward gradient prediction and conflict detection and implements dynamic parameter freezing to reconcile and optimize the direction. Finally, it completes model parameter updates by combining dynamic weight calibration of multi-objective loss. This achieves consistency of scoring standards while coordinating gradient conflicts across multiple tasks, thereby improving the accuracy and stability of oral expression ability evaluation. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart illustrating an AI-based method for evaluating oral expression ability in one embodiment.

[0063] Figure 2 This is a schematic diagram of the structure of an AI-based oral expression ability assessment device in one embodiment. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] refer to Figure 1 The application presents a flowchart illustrating the AI-based oral expression ability assessment method provided in this application, which includes the following steps:

[0066] S1. Perform multi-granularity feature decoupling processing on the acquired raw speech signal to extract multi-dimensional feature data from the acoustic feature space, linguistic feature space and paralinguistic feature space, and generate the decoupled feature set.

[0067] Specifically, when performing multi-granularity feature decoupling processing on the acquired raw speech signal, preprocessing is first required to eliminate the impact of environmental noise and equipment interference on subsequent feature extraction. Specifically, a bandpass filter is used to filter low-frequency and high-frequency noise, and an adaptive noise cancellation algorithm is used to suppress steady-state environmental noise. This algorithm, based on the minimum mean square error criterion, tracks noise characteristics by updating filter coefficients in real time, ensuring that the signal-to-noise ratio is improved to above a preset standard. The preprocessed speech signal is then resampled and quantized at a preset sampling rate and preset quantization precision to generate standard speech data segments.

[0068] The process then proceeds to a multi-granularity feature decoupling flow. Acoustic feature space extraction employs a multi-scale Mel-frequency cepstral coefficient (MFCC) combined with prosodic features. The MFCC features are extracted using a Mel filter bank, encompassing static coefficients, first-order difference coefficients, and second-order difference coefficients. Simultaneously, prosodic features such as fundamental frequency, duration, and energy are calculated. The fundamental frequency is extracted using the YIN algorithm, with the extraction window length and step size set according to the speech analysis requirements. Energy features are calculated using the root mean square of the intra-frame signal. Linguistic feature space extraction first requires converting speech into a text sequence using an end-to-end automatic speech recognition model. This model is based on the Transformer architecture, with the encoder and decoder each containing several layers of self-attention and cross-attention modules to ensure high recognition accuracy. Lexical, syntactic, and semantic features are then extracted from the text sequence. Lexical features include word vectors and part-of-speech tagging. Word vectors are generated using a pre-trained language model, and part-of-speech tagging is implemented using conditional random fields. Syntactic features are obtained through dependency parsing to obtain core word dependencies. Semantic features are generated using sentence vectors, which are fused from intra-sentence word vectors through mean pooling. The paralinguistic feature space mainly extracts sentiment tendency and non-linguistic signals. Sentiment tendency is determined by combining the fundamental frequency standard deviation of speech, energy change rate, and sentiment polarity score of text. The sentiment polarity score of text adopts a method combining sentiment dictionary and recurrent neural network. Non-linguistic signals include features such as pause duration and frequency of interjections. Pause duration is obtained by determining consecutive frames through energy threshold.

[0069] Finally, principal component analysis is used to reduce the dimensionality of redundant features in each feature space, retaining feature dimensions whose cumulative variance contribution reaches a preset proportion, and generating a decoupled feature set containing three types of features: acoustic, linguistic, and paralinguistic. The feature dimensions are controlled within a reasonable range to ensure the efficiency of subsequent processing.

[0070] S2. By sharing the hierarchical structure of the underlying encoder and the task-specific top-level expert network, the decoupled feature set is processed by task-aware encoding to generate specific feature representations for scoring tasks such as pronunciation accuracy, fluency, lexical diversity, and logical coherence.

[0071] Specifically, when performing task-aware encoding based on the decoupled feature set, a hierarchical architecture of a shared bottom-level encoder and a task-specific top-level expert network is adopted. This architecture reduces the number of model parameters by sharing feature extraction capabilities, while utilizing the expert network to accurately capture task-specific features. The shared bottom-level encoder adopts a hybrid structure of deep residual network and Transformer. The deep residual network part contains several residual blocks, each consisting of a convolutional layer and a batch normalization layer, used to extract local feature associations. Subsequently, several Transformer encoder layers are connected, each containing a multi-head self-attention mechanism and a fully connected layer. The self-attention mechanism captures global feature dependencies. After processing by the shared bottom-level encoder, the input decoupled feature set generates a fixed-dimensional general feature representation. The task-specific top-level expert network adopts a modular design, constructing independent expert modules for four scoring tasks: pronunciation accuracy, fluency, lexical diversity, and logical coherence. Each expert module contains a fully connected layer and a Dropout layer. The Dropout layer prevents overfitting, and the output dimension of the fully connected layer is gradually adjusted according to the feature processing requirements.

[0072] During the encoding process, general feature representations are first input into a gating mechanism module, which consists of gating units constructed using the Sigmoid activation function. This module dynamically fuses general features with task-specific features by learning the weights required for each task's features. For example, the gating unit of the pronunciation accuracy expert network strengthens the weight of Mel-frequency cepstral coefficients in acoustic features, while the logical coherence expert network enhances the weight of semantic sentence vectors in linguistic features. The output of the gating mechanism serves as the input to the corresponding expert network, which then processes the data to generate task-specific feature representations. Pronunciation accuracy features focus on phoneme-level differences, fluency features highlight the temporal correlation between prosody and pauses, lexical diversity features include quantitative features such as word repetition rate and word length distribution, and logical coherence features emphasize semantic cohesion and topic consistency. All task-specific feature representations undergo L2 normalization to ensure uniform feature scale, laying the foundation for subsequent scoring calculations.

[0073] S3. Based on specific feature representations, construct a differentiable scoring rule graph according to the semantic description of each scoring task and the label co-occurrence relationship in the training data, and generate a task relationship graph based on the differentiable scoring rule graph.

[0074] Specifically, when constructing a differentiable scoring rule graph and generating a task relationship graph based on specific feature representations, two core tasks must first be completed: semantic description parsing and label co-occurrence relationship mining. For the semantic description parsing of each scoring task, natural language processing techniques are used to convert the natural language description of the scoring task into structured semantic features. Specifically, a pre-trained language model is used to encode the semantic description text, generating a fixed-dimensional semantic vector. Then, semantic role labeling is used to identify semantic components such as the evaluation object, evaluation criteria, and core indicators, establishing a mapping relationship between semantic components and feature dimensions. For example, "phoneme pronunciation matching degree" is mapped to the Mel frequency cepstral coefficient difference feature in acoustic features. In the label co-occurrence relationship mining of the training data, a labeled dataset containing a large number of spoken language samples is collected. Each sample contains independent labels for four scoring tasks, and the scoring range is set according to the evaluation requirements. The co-occurrence relationship is quantified by calculating the Pearson correlation coefficient and mutual information value between labels. For example, the correlation coefficient between the pronunciation accuracy label and the fluency label is calculated. The magnitude of the correlation coefficient indicates the degree of correlation between the two, while the mutual information value reflects the degree of dependence between the labels. Based on the semantic parsing results and label co-occurrence relationships, a differentiable scoring rule graph is constructed. This graph uses four scoring tasks as nodes, and the weights of the edges are the weights of the semantic relevance and the label co-occurrence coefficient. The semantic relevance is calculated by the cosine similarity of the semantic vectors. To achieve differentiability, the rule graph adopts a graph convolutional network structure, transforming node features and edge weights into tensors that can participate in gradient calculation. The graph convolutional network contains several graph convolutional layers, with the output dimension of each layer set according to the feature processing requirements. The activation function is ReLU. Finally, the graph convolutional network performs feature aggregation and propagation on the rule graph to generate a task relationship graph. This graph is stored in the form of an adjacency matrix, where the matrix elements are the comprehensive relationship weights between tasks. It also outputs the node feature vectors of each task, providing a task relationship basis for subsequent gradient direction calculation.

[0075] S4. Based on the task relationship graph and the model parameters of the shared bottom encoder and the task-specific top expert network, the predicted gradient direction of each task on the shared layer is calculated by forward gradient prediction to generate an initial gradient direction set.

[0076] Specifically, when performing forward gradient prediction based on the task relationship graph and model parameters, the core is to obtain the gradient directions of each task on the shared layer through forward computation rather than traditional backpropagation, in order to avoid the gradient adversarial problem in real-time backpropagation. First, the parameters of the task-specific top-level expert network are fixed, and gradient prediction is performed only on the parameters of the shared bottom-level encoder. The parameters of the shared bottom-level encoder include the convolutional kernel parameters of the deep residual network, the self-attention weights of the Transformer, and the weights of the fully connected layers. These parameters are organized into a parameter tensor set θ, the dimension of which is determined by the total number of parameters and the parameter dimension. Then, based on the adjacency matrix W and node feature vectors V of the task relationship graph, a task association weight matrix is ​​constructed. The task association feature matrix is ​​obtained through matrix multiplication W×V. This matrix is ​​used to adjust the weights of gradient prediction for each task. The dimension of the adjacency matrix W is determined by the number of tasks, and the dimension of the node feature vector V is set according to the output of the graph convolutional network. Forward gradient prediction uses a parameter perturbation method to adjust the shared layer parameters. Add small perturbations The amplitude and distribution of the disturbance are determined based on the gradient prediction accuracy requirements, thus obtaining the parameters after disturbance. The decoupled feature sets are input into the original parameter model and the perturbation parameter model respectively, and the output differences of the feature representations specific to each task are obtained. , These correspond to four scoring tasks. The calculation represents the L2 distance between the perturbed output feature and the original output feature. According to the Taylor expansion, the gradient direction can be approximated as... and The ratio is calculated using the following formula: . In the formula, Indicates the first Each task has shared layer parameters The gradient vector on the graph is used to characterize the degree of influence of parameter changes on the task output features; Indicates the first The differences in output characteristics of each task before and after parameter perturbation reflect the feature changes brought about by small parameter changes; Indicates the parameters of the shared layer The added small perturbation is a key variable used to approximate the gradient calculation. This approximation yields the initial gradient vector for each task on the shared layer parameters. To improve the accuracy of gradient prediction, a task-related feature matrix is ​​introduced to correct the initial gradient vector. The correction formula is as follows: . In the formula, Indicates the corrected number The gradient vectors of each task, after being corrected, can better reflect the relationships between tasks; For the first time before the correction Initial gradient vectors for each task; For the first The node feature vectors of each task are used to provide feature information about the task itself. The first in the adjacency matrix The weight of the row is used to provide the weight of the first row. The association information between each task and other tasks is used to achieve gradient calibration based on task relationships through the product of the two. Finally, the corrected gradient vectors of the four tasks are organized into an initial gradient direction set, which is stored in tensor form. Its dimension is related to the number of tasks, the total number of parameters, and the parameter dimension. Each dimension corresponds to the gradient direction of a task on each parameter of the shared layer, and the magnitude of each gradient vector is recorded to reflect the gradient strength.

[0077] S5. Based on the initial gradient direction set and task relationship graph, calculate the conflict index of each task through a conflict detection mechanism, and implement a dynamic parameter freezing strategy based on the conflict index to generate optimized gradient update directions.

[0078] Specifically, when calculating the conflict index and implementing a dynamic parameter freezing strategy based on the initial gradient direction set and task relationship graph, a quantitative evaluation mechanism for gradient conflict must first be established. The conflict index is calculated on a task-pair basis, for any two tasks... and Calculate its gradient vector and The cosine similarity is used to quantify the severity of gradient adversarial conflict. When the cosine similarity is less than 0, it indicates that there is adversarial gradient direction conflict between the two tasks. The closer the cosine similarity is to -1, the higher the degree of conflict. A conflict index is defined to quantify the severity of gradient adversarial conflict. When the cosine similarity is greater than or equal to 0, the conflict index is 0, meaning there is no gradient conflict between the two tasks. When the cosine similarity is less than 0, the conflict index is the negative of the cosine similarity, and the larger the value, the more severe the conflict. Based on the adjacency matrix of the task relationship graph, the conflict indices of all task pairs are integrated into a conflict matrix. The dimension of the conflict matrix is ​​determined by the number of tasks, and a conflict threshold is set. This threshold is determined through validation set experiments. When the conflict index is greater than the threshold, it is judged as a severe conflict. Subsequently, a dynamic parameter freezing strategy is initiated. First, the parameter regions where gradient conflict is concentrated in the shared layer are located. By calculating the variance of the gradient vectors of all tasks in each parameter dimension, the parameter dimensions with variance greater than a preset value are judged as conflict-sensitive parameter regions. For task pairs with severe conflicts, the dominant and subordinate tasks are determined based on the importance scores of node feature vectors in the task relationship graph. The importance score of the node feature vector is determined by the magnitude of the node feature vector output by the graph convolutional network. If the importance score of task i is higher than that of task j, the gradient update privileges of task j in the conflict-sensitive parameter region are frozen, while only task i retains the update privileges for parameters in that region. The freezing strategy is implemented using a parameter masking mechanism, setting a binary mask for the conflict-sensitive parameter region. When the mask value is 0, the corresponding parameter is frozen, i.e., the gradient is not updated; when the mask value is 1, the parameter is updated normally. The generation of the mask matrix is ​​jointly controlled by the conflict matrix and the task importance score. Specifically, if a task is the dominant task in the parameter region, the mask value is 1; otherwise, it is 0. Simultaneously, a dynamic unfreezing condition is set. When the conflict index of the task pair in several consecutive training batches is not greater than a threshold, the frozen state is automatically unfrozen, and the parameter update privileges are restored. Through the above conflict detection and dynamic freezing processing, conflict-free gradient directions are selected and integrated to generate optimized gradient update directions. These directions retain the effective gradient information of each task while avoiding the performance degradation caused by gradient adversarial behavior.

[0079] S6. Based on the optimized gradient update direction, and combined with the multi-objective loss function after dynamic weight calibration for each scoring task, update the model parameters of the shared bottom encoder and the task-specific top expert network to generate the updated oral assessment model.

[0080] Specifically, when updating model parameters based on the optimized gradient update direction, the core is to construct a multi-objective loss function with dynamically calibrated weights to achieve balanced optimization of the losses for each task. First, an independent loss function is constructed for each scoring task. The pronunciation accuracy task uses the cross-entropy loss function, as this task can be transformed into a classification problem of correct and incorrect phoneme pronunciation; the loss calculation is based on the category difference between the predicted phoneme and the standard phoneme. Fluency, lexical diversity, and logical coherence tasks are all regression tasks, using the mean squared error loss function. The loss calculation is based on the difference between the predicted score and the manually labeled score. The fluency loss function incorporates temporal attention weights, assigning higher weights to prediction errors in long pause regions. Then, dynamic weight calibration is performed, based on two dimensions: first, task performance metrics, calculated using the validation set to determine the mean absolute error (MAO) for each task. A smaller MAO indicates better task performance, and the weights should be appropriately reduced to avoid overfitting; the weights are positively correlated with the MAO. Second, the degree of task conflict, determined by the average conflict index between this task and other tasks in the conflict matrix. A higher average conflict index requires an appropriate increase in weight to enhance the task's learning priority. Weight calculation employs entropy weighting combined with linear normalization. The specific steps are as follows: First, calculate the mean absolute error (MAE) and average conflict index for each task; then, standardize the MAE and average conflict index to obtain standardized MAE and average conflict index; next, calculate the comprehensive index; finally, calculate the initial weights for each task using entropy weighting, and then perform linear normalization to obtain the final weights, ensuring that the sum of all task weights is 1. A multi-objective loss function guides model parameter updates. Parameter updates utilize the Adam optimizer, with the learning rate... , The weight decay coefficient is set according to the model training requirements. The weight decay coefficient is used to prevent overfitting. During the update process, only the optimized gradient update direction is used to calculate the parameter gradient, and the parameter gradient in the frozen region is set to 0. Model parameter updates are performed in batches. The number of samples in each batch is determined according to training resources and efficiency requirements. After several new training cycles, the convergence of the model is judged by the comprehensive scoring error of the validation set. The comprehensive scoring error is the weighted sum of the average absolute errors of each task. When the rate of change of the comprehensive scoring error is less than a preset value for several consecutive new cycles, training is stopped, and an updated oral assessment model is generated.

[0081] S7. Input the speech to be evaluated into the updated oral assessment model, and perform weighted fusion of the predicted scores of each scoring task to generate the oral expression ability score result.

[0082] Specifically, when inputting the speech to be evaluated into the updated model to generate scoring results, it is necessary to ensure that the preprocessing procedure of the speech to be evaluated is consistent with that of the training data to eliminate scoring bias caused by differences in data distribution. First, the speech to be evaluated undergoes noise suppression, resampling, and quantization to generate standard speech data segments. Then, multi-granularity feature decoupling is performed to extract acoustic, linguistic, and paralinguistic features, generating a set of features to be evaluated that is consistent with the training data format. This set of features is input into the updated spoken language assessment model. A shared bottom-level encoder extracts general features, and then a task-specific top-level expert network generates specific feature representations for each task. The parameters of each expert network have been fixed through training optimization. Based on these specific feature representations, the prediction head for each task outputs preliminary scoring results. The pronunciation accuracy prediction head is a fully connected layer, with the input being the specific feature representation of the corresponding task and the output being a 1D score. A Sigmoid activation function is used to map the output to a preset scoring range. The fluency, lexical diversity, and logical coherence prediction heads have the same structure, all being fully connected layers combined with a ReLU activation function, and their outputs are also mapped to a preset scoring range. The preliminary scores for each task are weighted and fused. The fusion weights are determined through correlation analysis of the scores on the validation set. Specifically, the Pearson correlation coefficient between the preliminary scores for each task and the overall human assessment is calculated. The higher the correlation coefficient, the greater the fusion weight. The calculation method for the fusion weights is consistent with the dynamic weight calibration, ensuring that the fused scores are closer to the human assessment standards. Simultaneously, a score calibration mechanism is introduced. A score deviation correction model is constructed using historical evaluation data. This model employs a linear regression model, with the preliminary scores for each task as input and the corrected scores as output, to eliminate systematic errors. The final oral expression ability score includes two parts: individual scores for each dimension and an overall score. Individual scores for each dimension include pronunciation accuracy, fluency, lexical diversity, and logical coherence. The number of decimal places retained for both the overall score and individual scores is set according to the assessment accuracy requirements. The system also outputs analysis explanations for each dimension's scores, such as providing specific reasons for fluency scores, offering users concrete directions for improvement.

[0083] In the aforementioned AI-based oral expression ability evaluation method, a multi-dimensional feature set is obtained by first decoupling the original speech signal at multiple granularities. Then, a shared encoder and a task expert network are used to generate specific feature representations for each scoring task. Based on this, a differentiable scoring rule graph is constructed to quantify task relationships. Gradient conflict task pairs are identified through forward gradient prediction and conflict detection mechanisms. A dynamic parameter freezing strategy is implemented based on the conflict index to generate optimized gradient directions. Finally, the model parameters are updated by combining a multi-objective loss function with dynamic weight calibration to achieve accurate fusion of multi-task scoring results. This effectively solves the gradient conflict problem in multi-task learning and improves scoring accuracy and cross-scene adaptability.

[0084] In one embodiment, the acquired raw speech signal undergoes multi-granularity feature decoupling processing to extract multi-dimensional feature data from the acoustic feature space, linguistic feature space, and paralinguistic feature space, generating a decoupled feature set, including:

[0085] S11. Perform volume calibration processing on the raw speech signal acquired synchronously through dual channels to obtain standardized speech waveform data.

[0086] The raw speech signals acquired synchronously through dual channels undergo volume calibration processing to obtain standardized speech waveform data. In dual-channel acquisition mode, the main channel is used to acquire the target speech signal, while the auxiliary channel mainly acquires environmental background noise. Differential processing of the dual-channel data can initially suppress environmental interference. The core objective of volume calibration is to eliminate characteristic deviations caused by differences in volume between different acquisition devices and speakers, ensuring that the amplitude range of the standardized speech waveform data conforms to the preset standard. In specific implementation, the short-time energy of the raw speech signal in the main channel is first calculated. The short-time energy is calculated using a sliding window, with the window length and step size set according to the temporal resolution requirements of the speech signal. Typically, the window length covers a complete speech syllable. Using the statistical mean of the short-time energy as a benchmark, an energy normalization algorithm is used to uniformly adjust the amplitude of the speech signal to a preset range. During the normalization process, distortion caused by signal amplitude overflow must be avoided, while preserving the dynamic change characteristics of the speech signal. To address the volume imbalance issue in dual-channel data, the energy ratio of the main and auxiliary channels is calculated, and the gain of the auxiliary channel data is adjusted to ensure that the signal-to-noise ratio of the dual-channel signals remains consistent. The final output standardized speech waveform data must meet the requirements of stable amplitude fluctuation and no obvious clipping distortion, providing a high-quality signal foundation for subsequent feature extraction.

[0087] S12. Extract low-level acoustic features such as Mel-frequency cepstral coefficients, fundamental frequency trajectory, and energy distribution from standardized speech waveform data. Generate acoustic feature space data by calculating vowel format deviation and consonant voicing contrast. The formula for calculating vowel format deviation is:

[0088]

[0089] in, This is due to vowel format deviation. Indicates the first Each format frequency, Represents the original vowel. It represents the standard vowel.

[0090] Specifically, low-level acoustic features such as Mel-frequency cepstral coefficients, fundamental frequency trajectory, and energy distribution are extracted from standardized speech waveform data. Acoustic feature spatial data is generated by calculating vowel format deviation and consonant voicing contrast. The extraction of low-level acoustic features must consider both time-domain and frequency-domain characteristics. Mel-frequency cepstral coefficients are extracted by filtering the speech signal spectrum using a Mel-filter bank, converting the linear spectrum into a Mel-spectrum that conforms to human hearing characteristics, and then obtaining the coefficients through discrete cosine transform. During the extraction process, static coefficients, first-order difference coefficients, and second-order difference coefficients must be retained to fully represent the dynamic changes of acoustic features. The fundamental frequency trajectory is calculated frame-by-frame using the YIN algorithm, and smoothed using a dynamic programming algorithm to eliminate outliers such as fundamental frequency jumps, accurately reflecting the pitch changes of speech. Energy distribution features are represented by the statistical characteristics of frame-by-frame energy, including the mean, variance, and peak distribution of energy. Vowel format deviation is a key feature for evaluating pronunciation accuracy; in the above formula, ... This is the vowel format deviation, used to quantify the degree of difference between the original vowel pronunciation and the standard vowel pronunciation. The smaller the value, the more standard the vowel pronunciation. Indicates the first The format frequencies are important acoustic characteristics of vowel pronunciation; the first three format frequencies are sufficient to distinguish the pronunciation characteristics of different vowels. This indicates that the target vowels extracted from the original speech are located and segmented from the speech waveform using a vowel detection algorithm. The preset standard vowels are represented by format frequency data from authoritative speech acoustics databases to ensure the uniformity of evaluation standards. The consonant voicing contrast is obtained by calculating the spectral centroid shift and energy change rate during the consonant pronunciation period. The spectral feature differences between voiceless and voiced consonants are quantified through contrast. Finally, low-level acoustic features are integrated with high-level acoustic features such as vowel format deviation and consonant voicing contrast to form a unified acoustic feature space data.

[0091] S13. The speech is converted into text using an end-to-end speech recognition engine. Based on the converted text, features of lexical diversity, syntactic complexity, and semantic coherence are extracted. Semantic density is calculated by combining lexical embedding vectors from the pre-trained semantic space, generating linguistic feature space data. The formula for calculating semantic density is:

[0092]

[0093] in, For semantic density, It is a word Embedded vector, This represents the total number of words.

[0094] Specifically, an end-to-end speech recognition engine converts speech into text. Based on the converted text, lexical diversity, syntactic complexity, and semantic coherence features are extracted. Semantic density is calculated by combining lexical embedding vectors from a pre-trained semantic space, generating linguistic feature space data. The end-to-end speech recognition engine employs a Transformer-based encoder-decoder architecture. The encoder captures long-term dependencies in the speech signal through a multi-head self-attention mechanism, while the decoder combines an attention mechanism to achieve accurate mapping of speech features to the text sequence. To improve recognition accuracy, a language model is introduced for decoding optimization. The language model uses a fusion structure of an n-gram-based statistical model and a pre-trained language model. Based on the converted text sequence, lexical diversity features are quantified using three indicators: word repetition rate, word richness, and word difficulty. The word repetition rate is the ratio of repeated words to the total number of words; word richness is calculated using the class-to-form ratio; and word difficulty is obtained by weighting the words according to their levels in a pre-defined difficulty-level dictionary. Syntactic complexity features are measured by the depth of the syntactic tree, the number of nodes, and dependency complexity. A syntactic structure tree is generated using dependency parsing tools, and the average depth of the syntactic tree and the number of child nodes per node are statistically analyzed. Dependency complexity is quantified by the number of dependency arc types. Semantic coherence features are calculated through semantic similarity between sentences, using cosine similarity to measure the degree of association between the semantic vectors of adjacent sentences. Semantic density is a key indicator reflecting the amount of semantic information carried by the text; in the above calculation formula, ... This represents semantic density; a higher value indicates richer semantic information in the text. It is a word The embedding vectors are generated by a pre-trained BERT model and have strong semantic representation capabilities. The total word count of the converted text was obtained by segmenting the text sequence and then counting the words. The extracted linguistic features and semantic density indices were then standardized to ensure that all features were on the same scale, and finally integrated to generate linguistic feature space data.

[0095] S14. Extract pause frequency, pause duration distribution, speech rate variation and prosodic features from the original speech signal, and distinguish between hesitant pauses and thinking pauses through a context-aware pause classification model to generate paralinguistic feature space data.

[0096] Specifically, pause frequency, pause duration distribution, speech rate variation, and prosodic features are extracted from the original speech signal. A context-aware pause classification model distinguishes between hesitant pauses and thoughtful pauses, generating paralinguistic feature space data. Pause feature extraction first uses an energy threshold method to locate pause intervals. When the short-term energy of the speech signal is below a preset threshold and the duration exceeds the minimum pause duration, it is determined as a pause event. The number of all pause events is counted to obtain the pause frequency. The duration of each pause event is statistically analyzed to obtain the distribution characteristics of pause duration, including mean, variance, and quantiles. Speech rate variation features are calculated by the number of syllables per unit time, and combined with a sliding window technique to obtain a dynamic change curve of speech rate. The slope of the curve is calculated to quantify the degree of speech rate fluctuation. Prosodic features include the range of fundamental frequency variation, the mean of the first-order difference of the fundamental frequency, and the coefficient of variation of the duration, comprehensively reflecting the rhythm and intonation characteristics of the speech. The context-aware pause classification model employs a bidirectional long short-term memory network combined with an attention mechanism. The model's input consists of speech feature segments before and after the pause, along with corresponding text semantic features. The speech feature segments are fixed-duration Mel-frequency cepstral coefficient sequences before and after the pause, while the text semantic features are word embedding vectors of the corresponding text. The attention mechanism strengthens the feature weights related to pause classification. The model's output is a probability distribution of pause types; the type with the highest probability value is the final classification result. This model accurately distinguishes between hesitant and thoughtful pauses, providing a basis for subsequent differentiated assessments of fluency and logical coherence. Finally, the pause classification results are integrated with extracted pause frequencies, speech rate variations, and prosodic features to generate paralinguistic feature space data.

[0097] S15. The acoustic feature space data, linguistic feature space data, and paralinguistic feature space data are concatenated according to feature dimensions to generate a decoupled feature set.

[0098] Specifically, acoustic feature space data, linguistic feature space data, and paralinguistic feature space data are concatenated according to feature dimensions to generate a decoupled feature set. Before concatenation, dimensionality consistency checks are performed on the three types of feature space data to ensure that the sample size of each feature space data is completely matched, avoiding feature contamination caused by sample misalignment. For feature dimensions within each feature space, principal component analysis is used to remove redundant features, retaining feature dimensions whose cumulative variance contribution meets preset requirements, thus reducing the computational complexity of subsequent processing while ensuring feature representation capabilities. Feature concatenation is performed in dimensional order, sequentially concatenating the acoustic feature space data, linguistic feature space data, and paralinguistic feature space data into a single high-dimensional feature vector. The concatenated feature vector undergoes L2 normalization to eliminate the influence of scale differences between different feature dimensions. To facilitate feature tracing and debugging in subsequent model processing, each dimension of the concatenated feature vector is labeled to clarify the feature source and specific meaning of each dimension. The final decoupled feature set is stored in tensor form, with a clear feature structure and a unified scale standard, and can be directly input into the subsequent task-aware encoding module for processing.

[0099] In one embodiment, based on the task relationship graph and the model parameters of the shared bottom encoder and the task-specific top expert network, the predicted gradient directions of each task on the shared layer are calculated through forward gradient prediction to generate an initial set of gradient directions, including:

[0100] S21. Based on the current parameters of the shared underlying encoder and the decoupled feature set, the intermediate activation values ​​of each scoring task are calculated through forward propagation to obtain the forward propagation results.

[0101] Specifically, based on the current parameters of the shared underlying encoder and the decoupled feature set, intermediate activation values ​​for each scoring task are calculated through forward propagation to obtain the forward propagation result. The current parameters of the shared underlying encoder include core parameters such as the convolutional kernel parameters, batch normalization layer parameters, and the self-attention weights and fully connected layer weights of the Transformer. The decoupled feature set is used as input in tensor form. After the residual blocks of the shared underlying encoder extract local feature associations and the Transformer layer captures global feature dependencies, the intermediate activation values ​​corresponding to each scoring task are output. These activation values ​​are key feature states in the forward propagation process of the model, fully preserving the transformation information of features from input to task feature representation.

[0102] S22. Based on the forward propagation results and the current parameters of the task-specific top-level expert network, calculate the basic loss value of each scoring task corresponding to its true label, and obtain the task loss set.

[0103] Specifically, based on the forward propagation results and the current parameters of the task-specific top-level expert network, the basic loss value corresponding to the true label for each scoring task is calculated, resulting in a task loss set. The current parameters of the task-specific top-level expert network include the weights of the fully connected layers and the probability of the Dropout layer in each expert module. The intermediate activation values ​​obtained in S41 are input into the corresponding expert network. After the general features and task-specific features are dynamically fused through a gating mechanism, a task-specific feature representation is generated, and then the prediction results are output through the prediction heads of each task. Combining the manually labeled true labels, the pronunciation accuracy task uses cross-entropy to calculate the basic loss, while the fluency, lexical diversity, and logical coherence tasks use mean squared error to calculate the basic loss. Finally, the loss values ​​of each task are integrated into a task loss set.

[0104] S23. Based on the task loss set, the predicted gradients of each task for the shared underlying encoder parameters are approximately calculated using the gradient direction prediction network to generate an initial gradient direction set.

[0105] Specifically, based on the task loss set, a gradient direction prediction network is used to approximately calculate the predicted gradients of each task with respect to the shared underlying encoder parameters, generating an initial gradient direction set. The gradient direction prediction network employs a lightweight Transformer architecture. Its input consists of node feature vectors from the task loss set and the task relationship graph. A multi-head self-attention mechanism captures the intrinsic relationship between loss and task association. The output layer uses a linear activation function to map gradient vectors that match the dimension of the shared underlying encoder parameters. To improve prediction accuracy, a gradient sensitivity coefficient for the task loss is introduced, assigning higher weights to tasks with drastic loss changes. Finally, the predicted gradient vectors of each task are organized into the initial gradient direction set.

[0106] After generating the initial gradient direction set, the parameters of the task-specific top-level expert network are fixed, and subsequent gradient optimization is performed only on the parameters of the shared bottom-level encoder. This is based on the adjacency matrix of the task relationship graph. and node feature vectors Construct a task-related weight matrix and perform matrix multiplication. The task-related feature matrix is ​​obtained, which is used to adjust the weights of gradient predictions for each task, where the adjacency matrix... The dimension is determined by the number of tasks, and the node feature vector The dimension is set according to the output of the graph convolutional network. To further improve the accuracy of gradient prediction, a task-related feature matrix is ​​introduced to correct the initial gradient vector. The correction formula is as follows: . In the formula, Indicates the corrected number The gradient vectors of each task, after being corrected, can better reflect the relationships between tasks; For the first time before the correction Initial gradient vectors for each task; For the first The node feature vectors of each task are used to provide feature information about the task itself. The first in the adjacency matrix The weight of the row is used to provide the weight of the first row. The association information between each task and other tasks is used to achieve gradient calibration based on task relationships through the product of the two. Finally, the corrected gradient vectors of the four tasks are organized into an optimized initial gradient direction set, which is stored in tensor form. Its dimension is related to the number of tasks, the total number of parameters, and the parameter dimension. Each dimension corresponds to the gradient direction of a task on each parameter of the shared layer, and the magnitude of each gradient vector is recorded to reflect the gradient strength.

[0107] In one embodiment, based on the initial gradient direction set and task relationship graph, a conflict index for each task is calculated using a conflict detection mechanism, and a dynamic parameter freezing strategy is implemented based on the conflict index to generate optimized gradient update directions, including:

[0108] S31. Based on the initial gradient direction set, calculate the gradient cosine similarity between all rating task pairs using the following formula to obtain the gradient cosine similarity matrix:

[0109]

[0110] in, For the task and tasks Gradient cosine similarity between them and Tasks and The predicted gradient direction.

[0111] Specifically, based on the initial set of gradient directions, the gradient cosine similarity between all rating task pairs is calculated using the above formula, resulting in a gradient cosine similarity matrix: where, For the task and tasks The gradient cosine similarity between them takes a value in the range of [-1, 1]. The closer the value is to 1, the more consistent the gradient direction is, and the closer it is to -1, the stronger the adversarial relationship is. and Tasks and The predicted gradient directions are derived from the initial set of gradient directions. The numerator is the dot product of two gradient vectors, reflecting the degree of overlap in directions, and the denominator is the product of the magnitudes of the two vectors, used for normalization. The initial set of gradient directions is stored in tensor form. It needs to be split according to the task dimension to obtain the gradient vector of each individual task, and then the gradient vectors of all task pairs are calculated through a double loop. The gradient cosine similarity is calculated, and a 4×4 gradient cosine similarity matrix is ​​finally constructed (corresponding to the four scoring tasks). The matrix elements directly reflect the gradient direction correlation characteristics of any two tasks.

[0112] S32. Based on the gradient cosine similarity matrix and the historical relationship types recorded in the task relationship graph, the real-time conflict index for each scoring task is obtained using the following formula:

[0113]

[0114] in, For the task Conflict index For the task and tasks Gradient cosine similarity between them For task pairs obtained based on task relationship graphs Relationship weights.

[0115] Specifically, based on the gradient cosine similarity matrix and the historical relationship types recorded in the task relationship graph, the real-time conflict index for each scoring task is obtained using the above formula; where, For the task The conflict index, the higher the value, the more challenging the task. The higher the overall degree of conflict with other tasks; As a conflict quantification factor, when The time factor is 0 (no conflict), when Time factor is - (Quantifying the degree of confrontation); For the task The relation weights are directly taken from the adjacency matrix of the task relation graph. The elements of the adjacency matrix are calculated by weighting semantic relevance and label co-occurrence coefficient. Historical relation types (such as strong association and weak association) are already reflected in the adjacency matrix values ​​(strong association tasks). (Larger). The computation requires iterating through all tasks except itself. The conflict quantification factor is multiplied by the corresponding relation weight and then summed to obtain the comprehensive conflict index for each task, ultimately forming a conflict index set containing four elements.

[0116] S33. Based on the real-time conflict index, the first preset threshold, and the second preset threshold for each task, obtain the corresponding dynamic parameter freezing operation results.

[0117] Specifically, based on the real-time conflict index, the first preset threshold, and the second preset threshold for each task, the corresponding dynamic parameter freezing operation results are obtained. The first preset threshold (denoted as T1) is the threshold for mild conflict, and the second preset threshold (denoted as T2) is the threshold for severe conflict, with T2 > T1. These two thresholds are determined through a validation set conflict index distribution and model performance sensitivity analysis: T1 is taken as the 25th percentile of the validation set conflict index distribution (conflict has a relatively small impact), and T2 is taken as the 65th percentile (conflict significantly interferes with model accuracy). Specific values ​​need to be dynamically adjusted based on the task distribution of the actual training data. Different conflict index intervals correspond to differentiated parameter freezing strategies. The shared layer parameter region is divided based on the determined conflict-sensitive parameter region, i.e., the parameter dimensions with variance greater than a preset value. The core parameter region contains key parameters that ensure the basic performance of the model, such as the core weights of the Transformer self-attention mechanism. The non-core parameter region contains auxiliary parameters, such as the bias term of the batch normalization layer. The corresponding dynamic parameter freezing operation result is obtained according to the judgment rules. The freezing operation is implemented through the parameter masking mechanism. A unique binary mask matrix is ​​generated for each task. The mask value 1 indicates that the parameter can be updated, and 0 indicates that it is frozen. The dimension of the mask matrix is ​​consistent with the dimension of the shared underlying encoder parameters.

[0118] S34. Based on the unfrozen shared layer parameters, the initial gradient direction set is projected and adjusted to obtain the gradient projection result.

[0119] Specifically, based on the unfrozen shared layer parameters, the initial gradient direction set is projected and adjusted to obtain the gradient projection result. The core purpose of the projection adjustment is to eliminate the gradient components corresponding to the frozen parameters, ensuring that gradient updates only apply to the updatable parameter region. In practice, the parameter mask matrices of all tasks are first obtained, and the globally effective parameter mask is obtained by performing an element-wise AND operation (the mask value is 1 only when all related tasks do not freeze the parameter); then, the gradient tensor of the initial gradient direction set is multiplied element-wise with the globally effective parameter mask, and the gradient components corresponding to the frozen parameters are set to 0; finally, the adjusted gradient vector is L2 normalized to avoid abnormal gradient magnitudes caused by setting some components to zero, ensuring that the magnitude of the gradient vectors of each task remains consistent, and finally obtaining the gradient projection result with the same dimension as the initial gradient set.

[0120] S35. Integrate the results of the dynamic parameter freezing operation with the gradient projection results to generate the optimized gradient update direction for updating the parameters of the shared underlying encoder.

[0121] Specifically, the results of the dynamic parameter freezing operation and the gradient projection results are integrated to generate the optimized gradient update direction for updating the parameters of the shared underlying encoder. First, the task-specific mask matrix is ​​multiplied again with the corresponding task's projected gradient vector to ensure that the gradient of a single task is only valid within its own updateable region. Second, for conflicting parameter regions, the dominant task (the task with the higher importance score) is determined by combining the importance score of the node feature vectors in the task relationship graph (determined by the magnitude of the output vector of the graph convolutional network). The gradient information of the dominant task in this region is retained, and the corresponding gradient components of subordinate tasks are reset to zero. Finally, the gradient vectors of all tasks are weighted and fused, with the fusion weight positively correlated with the task importance score. After fusion, a unified gradient update direction for each parameter of the shared underlying encoder is obtained. This direction avoids gradient conflict between tasks, prioritizes the retention of gradient information from important tasks, and strictly adheres to the parameter freezing rules. The optimized gradient update direction is stored in tensor form and directly used in subsequent model parameter update processes.

[0122] In one embodiment, based on the real-time conflict index of each task, a first preset threshold, and a second preset threshold, the corresponding dynamic parameter freezing operation result is obtained, including:

[0123] S41, when the real-time conflict index is less than or equal to the first preset threshold, the result of the dynamic parameter freezing operation is marked as mild conflict, and only the specific subset of neurons in the shared layer that contributes the most to the conflict is frozen.

[0124] Specifically, when the real-time conflict index is less than or equal to a first preset threshold, the dynamic parameter freezing operation is marked as mild conflict, and only a specific subset of neurons in the shared layer that contributes the most to the conflict is frozen. The selection criteria for the specific subset of neurons are gradient contribution analysis. The contribution value of each neuron parameter in the shared layer to the gradient conflict between task i and other tasks is calculated. The contribution value is quantified by the absolute value of the dot product between the gradient vector of the neuron parameter and the conflict direction vector. The larger the absolute value of the dot product, the more critical the role of the neuron in the formation of conflict. During the selection process, only neurons whose contribution values ​​meet specific ranking requirements are retained as freezing objects. The freezing method is to fix their current parameter values ​​and stop gradient updates, while the remaining neurons remain in a normal update state. This strategy suppresses mild conflict while maximizing the feature extraction capability of the shared layer and avoiding model performance loss caused by over-freezing.

[0125] S42, when the real-time conflict index is greater than the first preset threshold and the real-time conflict index is less than or equal to the second preset threshold, the result of the dynamic parameter freezing operation is marked as moderate conflict, and the entire functional branch related to the high-conflict task in the shared layer is frozen.

[0126] Specifically, when the real-time conflict index is greater than a first preset threshold and less than or equal to a second preset threshold, the dynamic parameter freezing operation results in a marked moderate conflict, freezing the entire functional branch in the shared layer related to the high-conflict task. The high-conflict task can be directly located using the gradient cosine similarity matrix, determined by its similarity to the current task. The gradient cosine similarity meets specific conditions. The division of shared layer functional branches is based on network structure semantics. The deep residual network part uses residual blocks as functional branch units, and the Transformer part uses multi-head self-attention heads and fully connected sublayers within the encoder layer as functional branches. Branches to be frozen are determined by the task-branch association matrix, which consists of the contribution weights of each functional branch to the task features. The contribution weights are calculated through heatmap analysis and ablation experiments. Functional branches with a high degree of association with high-conflict tasks that meet specific requirements are frozen. The freezing range covers all neuron parameters within the branch, and branch freezing markers are recorded for subsequent training state tracking.

[0127] S43, when the real-time conflict index is greater than the second preset threshold, the result of the dynamic parameter freezing operation is to mark it as a severe conflict, group the conflict tasks and trigger the alternating training mode.

[0128] Specifically, when the real-time conflict index exceeds the second preset threshold, the dynamic parameter freezing operation marks it as a severe conflict, groups the conflict tasks, and triggers an alternating training mode. The task grouping rule is based on the joint analysis of the task relationship graph and the conflict matrix, employing a hierarchical clustering algorithm. Using the comprehensive correlation coefficient between tasks as the distance indicator, the four scoring tasks are divided into a specific number of groups without severe conflicts, ensuring that the gradient cosine similarity between tasks within a group meets specific conditions, achieving a weak adversarial or cooperative state, while maintaining severe conflict isolation between tasks in different groups. The core process of the alternating training mode is to allocate an independent training period to each task group. During training of the current group, the conflict-sensitive branches of the shared layer corresponding to tasks in other groups are frozen, and only the shared layer parameters associated with the current group's tasks and its own expert network parameters are updated. When switching between groups, the basic parameter state of the shared layer is restored through parameter snapshots to avoid gradient interference between groups. The number of training rounds is positively correlated with the number of tasks within a group; single-task groups are allocated more rounds to ensure sufficient learning. The termination condition for alternating training is that the conflict index of all tasks within the group drops below the second preset threshold. The results of the above dynamic parameter freezing operations are all achieved through binary mask matrices and training state markers. A global mask matrix is ​​generated for shared layer parameters to identify updatable parameters, and an independent state marker is generated for each task to record the conflict level and frozen branch ID. Both types of information are stored in real time with the training log, which supports fast mask calculations during parameter updates and provides a basis for conflict tracing in subsequent model tuning.

[0129] In one embodiment, based on the optimized gradient update direction and combined with the multi-objective loss function calibrated by the dynamic weights of each scoring task, the model parameters of the shared underlying encoder and the task-specific top-level expert network are updated to generate an updated oral assessment model, including:

[0130] S51. Calculate the base loss function value for each scoring task in this training batch; the base loss function includes the contrast loss for the pronunciation accuracy task, the segmentation ranking loss for the fluency task, the similarity-based regularization loss for the vocabulary diversity task, and the graph structure loss for the logical coherence task.

[0131] Specifically, the basic loss function needs to be customized based on the evaluation characteristics of each task to ensure that the loss calculation is highly matched with the task objective: the pronunciation accuracy task uses contrastive loss, which strengthens the feature discrimination between correct and incorrect phonemes by constructing positive-negative sample pairs. Positive samples are features corresponding to standard pronunciation, and negative samples are features of common incorrect pronunciation. The loss calculation is based on the difference in feature distance between the two. The fluency task uses segmentation ranking loss, which segments the speech according to semantic units and uses manually labeled fluency levels as the basis to enable the model to learn the ranking relationship of fluency between segments. The loss value increases as the deviation between the predicted ranking and the actual ranking increases. The vocabulary diversity task uses similarity-based regularization loss, which suppresses the use of high-frequency repeated words by calculating the similarity of overlapping pairs of word embedding vectors in the text. The higher the mean similarity, the greater the loss value. The logical coherence task uses graph structure loss, which constructs semantic graphs from text semantic units. The optimization objectives are the rationality of node connections and the completeness of paths. The loss includes two parts: graph structure matching loss and semantic path loss.

[0132] S52. Based on the ratio of the current basic loss value of each task to its recent historical loss value, the loss ratio weight is determined through a dynamic weight calculation model.

[0133] Specifically, the loss ratio is determined by a dynamic weighting model based on the ratio of the current base loss value to its recent historical loss value for each task. The recent historical loss value is the average base loss of the last K training batches, and the value of K is set according to the total number of training batches and data stability. The ratio of the current loss value to the historical loss value... Reflects the current changes in the learning difficulty of the task. This indicates that the task learning difficulty has increased, and the weight needs to be increased to enhance the learning priority; This indicates that task learning is stabilizing, and the weights can be appropriately reduced to avoid overfitting. The dynamic weight calculation model uses the Sigmoid function. Normalization is performed to obtain initial proportional weights, and then linear normalization is performed to ensure that the sum of the proportional weights of all tasks is 1. Finally, the set of output loss proportional weights shows a positive correlation.

[0134] S53. Monitor the Euclidean norm of the gradient magnitude of each task, and suppress the gradient magnitude of tasks that are consistently higher than the average level to obtain the gradient magnitude weight.

[0135] Specifically, the gradient magnitude is calculated by taking the Euclidean norm of the gradient vector of each task on the expert network parameters. The formula for calculating the norm is: ,in For the task gradient magnitude, For the task The gradient vector of the first One portion, This represents the gradient vector dimension. The average gradient magnitude of all tasks is calculated in real-time. If the gradient magnitude of a task is higher than a preset multiple (e.g., 1.5 times) for T consecutive batches, it is considered a gradient-unstable task, and a weighted suppression coefficient is applied. Reduce its weights; suppression coefficient for gradient stabilization tasks Gradient magnitude weight set It is obtained by normalizing the suppression coefficient to ensure that the weight allocation is tilted towards gradient-stable tasks.

[0136] S54. Combine the loss scaling weights and gradient magnitude weights to obtain the final multi-objective loss weights for each task, and construct a weighted multi-objective loss function; the expression for the multi-objective loss function is:

[0137]

[0138] in, For the multi-objective loss function value, Indicates task The final multi-objective loss weights, Indicates task The base loss value, Indicates the total number of tasks. Represents the regularization coefficient. This represents the set of all model parameters.

[0139] Specifically, the fusion adopts a weighted product method, and the final weights are... ,in To compensate for the loss ratio weight, The gradient magnitude weights are used, and the product result is linearly normalized to ensure that the final sum of the weights for all tasks is 1. In the expression of the multi-objective loss function, It represents the value of the multi-objective loss function and is a core indicator guiding the updating of model parameters; Representative task The final multi-objective loss weight is used to adjust the contribution ratio of each task's loss to the total loss; Representative task The base loss value; This represents the total number of scoring tasks, where N=4 in this case. This represents the regularization coefficient, used to control the complexity of model parameters and prevent overfitting. Its value is determined through validation set experiments. It represents the set of all model parameters, covering all parameters of the shared underlying encoder and the task-specific top-level expert network; Representative parameters The L2 norm squared is used to achieve parameter regularization by summing the norm squared of all parameters.

[0140] S55. Update the shared encoder parameters based on the optimized gradient update direction, and update the task-specific top-level expert network parameters according to the gradient generated by the weighted multi-objective loss function to generate the updated spoken language evaluation model.

[0141] Specifically, the shared underlying encoder parameter update uses gradient descent, directly using the optimized gradient update direction as the parameter update gradient. The learning rate is dynamically adjusted using a cosine annealing strategy to ensure rapid convergence in the early stages of training and accurate optimization in the later stages. Task-specific top-level expert network parameter updates obtain the gradient of the expert network parameters by differentiating the multi-objective loss function. The differentiation process uses the backpropagation algorithm, and the gradient calculation is limited to the expert module parameters corresponding to each task, avoiding gradient interference between tasks. During parameter updates, gradient pruning is used to limit the maximum value of the gradient magnitude to prevent gradient explosion. Simultaneously, model performance metrics (such as the validation set comprehensive scoring error) are recorded before and after each parameter update. If there is no performance improvement for multiple consecutive batches, a learning rate decay mechanism is triggered. When the model performance metrics reach the preset convergence condition (such as the comprehensive scoring error being less than a threshold or the error change rate of consecutive batches being less than a threshold), parameter updates are stopped, and the current model parameters are saved as the updated oral assessment model.

[0142] In the aforementioned AI-based oral expression ability evaluation method, a multi-dimensional feature set is obtained by decoupling the original speech signal at multiple granularities. Then, a hierarchical architecture of a shared encoder and a task expert network is used to generate specific feature representations for each scoring task. Based on this, a differentiable scoring rule graph is constructed to quantify the relationships between tasks. After obtaining the initial gradient direction through forward gradient prediction, a conflict detection mechanism is used to dynamically identify gradient conflict task pairs and implement a parameter freezing strategy to generate optimized gradient update directions. Finally, the model parameters are updated through a multi-objective loss function with dynamic weight calibration to achieve collaborative optimization of multi-task scoring prediction. This effectively solves the performance degradation problem caused by gradient conflicts in multi-task learning and improves the consistency and accuracy of oral scoring.

[0143] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0144] Based on the same inventive concept, this application also provides an AI assessment device for oral expression ability to implement the aforementioned AI assessment method for oral expression ability. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more embodiments of the AI ​​assessment device for oral expression ability provided below can be found in the limitations of the AI ​​assessment method for oral expression ability described above, and will not be repeated here.

[0145] In one exemplary embodiment, such as Figure 2 As shown, an AI-powered oral expression ability assessment device 200 is provided, including:

[0146] The data acquisition and feature processing module 201 is used to perform multi-granularity feature decoupling processing on the acquired raw speech signal, extract multi-dimensional feature data from the acoustic feature space, linguistic feature space and paralinguistic feature space, and generate a decoupled feature set;

[0147] The task encoding processing module 202 is used to perform task-aware encoding processing on the decoupled feature set through a hierarchical structure of a shared bottom encoder and a task-specific top expert network, generating specific feature representations for scoring tasks such as pronunciation accuracy, fluency, lexical diversity, and logical coherence.

[0148] The task relationship graph construction module 203 is used to construct a differentiable scoring rule graph based on specific feature representations, according to the semantic description of each scoring task and the label co-occurrence relationship in the training data, and generate a task relationship graph based on the differentiable scoring rule graph.

[0149] The initial gradient prediction module 204 is used to calculate the predicted gradient direction of each task on the shared layer through forward gradient prediction based on the model parameters of the task relationship graph and the shared bottom encoder and task-specific top expert network, and generate an initial gradient direction set.

[0150] The gradient update optimization module 205 is used to calculate the conflict index of each task based on the initial gradient direction set and task relationship graph through a conflict detection mechanism, and to implement a dynamic parameter freezing strategy based on the conflict index to generate optimized gradient update directions.

[0151] The oral assessment model update module 206 is used to update the model parameters of the shared bottom encoder and the task-specific top expert network based on the optimized gradient update direction and the multi-objective loss function after dynamic weight calibration of each scoring task, so as to generate the updated oral assessment model.

[0152] The oral expression ability scoring module 207 is used to take the voice input to be evaluated and update the oral assessment model, and then perform weighted fusion of the predicted scores of each scoring task to generate the oral expression ability scoring result.

[0153] Furthermore, the data acquisition and feature processing module 201 is also used for:

[0154] S11. Perform volume calibration processing on the raw speech signal acquired synchronously through dual channels to obtain standardized speech waveform data; the standardized speech waveform data is used to characterize that the speech signal quality meets the preset standard;

[0155] S12. Extract low-level acoustic features such as Mel-frequency cepstral coefficients, fundamental frequency trajectory, and energy distribution from standardized speech waveform data. Generate acoustic feature space data by calculating vowel format deviation and consonant voicing contrast. The formula for calculating vowel format deviation is:

[0156]

[0157] in, This is due to vowel format deviation. Indicates the first Each format frequency, Represents the original vowel. Represents standard vowels;

[0158] S13. The speech is converted into text using an end-to-end speech recognition engine. Based on the converted text, features of lexical diversity, syntactic complexity, and semantic coherence are extracted. Semantic density is calculated by combining lexical embedding vectors from the pre-trained semantic space, generating linguistic feature space data. The formula for calculating semantic density is:

[0159]

[0160] in, For semantic density, It is a word Embedded vector, Total word count;

[0161] S14. Extract pause frequency, pause duration distribution, speech rate variation and prosodic features from the original speech signal, distinguish between hesitant pauses and thinking pauses through a context-aware pause classification model, and generate paralinguistic feature space data.

[0162] S15. The acoustic feature space data, linguistic feature space data, and paralinguistic feature space data are concatenated according to feature dimensions to generate a decoupled feature set.

[0163] Furthermore, the initial gradient prediction module 204 is also used for:

[0164] S21. Based on the current parameters of the shared underlying encoder and the decoupled feature set, the intermediate activation values ​​of each scoring task are calculated through forward propagation to obtain the forward propagation result.

[0165] S22. Based on the forward propagation results and the current parameters of the task-specific top-level expert network, calculate the basic loss value of each scoring task corresponding to its true label to obtain the task loss set.

[0166] S23. Based on the task loss set, the predicted gradients of each task for the shared underlying encoder parameters are approximately calculated using the gradient direction prediction network to generate an initial gradient direction set.

[0167] Furthermore, the gradient update optimization module 205 is also used for:

[0168] S31. Based on the initial gradient direction set, calculate the gradient cosine similarity between all rating task pairs using the following formula to obtain the gradient cosine similarity matrix:

[0169]

[0170] in, For the task and tasks Gradient cosine similarity between them and Tasks and The predicted gradient direction;

[0171] S32. Based on the gradient cosine similarity matrix and the historical relationship types recorded in the task relationship graph, the real-time conflict index for each scoring task is obtained using the following formula:

[0172]

[0173] in, For the task Conflict index For the task and tasks Gradient cosine similarity between them For task pairs obtained based on task relationship graphs Relationship weights;

[0174] S33. Based on the real-time conflict index, the first preset threshold, and the second preset threshold of each task, obtain the corresponding dynamic parameter freezing operation results.

[0175] S34. Based on the unfrozen shared layer parameters, the initial gradient direction set is projected and adjusted to obtain the gradient projection result;

[0176] S35. Integrate the results of the dynamic parameter freezing operation with the gradient projection results to generate the optimized gradient update direction for updating the parameters of the shared underlying encoder.

[0177] Furthermore, the gradient update optimization module 205 is also used for:

[0178] S41, when the real-time conflict index is less than or equal to the first preset threshold, the result of the dynamic parameter freezing operation is marked as mild conflict, and only the specific subset of neurons in the shared layer that contributes the most to the conflict is frozen.

[0179] S42, when the real-time conflict index is greater than the first preset threshold and the real-time conflict index is less than or equal to the second preset threshold, the result of the dynamic parameter freezing operation is to mark it as moderate conflict and freeze the entire functional branch in the shared layer related to the high-conflict task.

[0180] S43, when the real-time conflict index is greater than the second preset threshold, the result of the dynamic parameter freezing operation is to mark it as a severe conflict, group the conflict tasks and trigger the alternating training mode.

[0181] Furthermore, the oral assessment model update module 206 is also used for:

[0182] S51. Calculate the base loss function value for each scoring task in this training batch; the base loss function includes the contrast loss for the pronunciation accuracy task, the segmentation ranking loss for the fluency task, the similarity-based regularization loss for the vocabulary diversity task, and the graph structure loss for the logical coherence task.

[0183] S52. Based on the ratio of the current basic loss value of each task to its recent historical loss value, the loss ratio weight is determined through a dynamic weight calculation model.

[0184] S53. Monitor the Euclidean norm of the gradient magnitude of each task, and perform weight suppression on tasks whose gradient magnitude is consistently higher than the average level to obtain the gradient magnitude weight.

[0185] S54. Combine the loss scaling weights and gradient magnitude weights to obtain the final multi-objective loss weights for each task, and construct a weighted multi-objective loss function; the expression for the multi-objective loss function is:

[0186]

[0187] in, For the multi-objective loss function value, Indicates task The final multi-objective loss weights, Indicates task The base loss value, Indicates the total number of tasks. Represents the regularization coefficient. Represents the set of all model parameters;

[0188] S55. Update the shared encoder parameters based on the optimized gradient update direction, and update the task-specific top-level expert network parameters according to the gradient generated by the weighted multi-objective loss function to generate the updated spoken language evaluation model.

[0189] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the AI ​​assessment method for spoken expression ability as described above.

[0190] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0191] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0192] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. An AI-based method for evaluating oral expression ability, characterized in that, The method includes: S1. Perform multi-granularity feature decoupling processing on the acquired raw speech signal to extract multi-dimensional feature data from the acoustic feature space, linguistic feature space and paralinguistic feature space, and generate a decoupled feature set; S2. The decoupled feature set is processed by task-aware encoding through a hierarchical structure of a shared bottom encoder and a task-specific top expert network to generate specific feature representations for scoring tasks of pronunciation accuracy, fluency, lexical diversity and logical coherence. S3. Based on the specific feature representation, according to the semantic description of each scoring task and the label co-occurrence relationship in the training data, construct a differentiable scoring rule graph, and generate a task relationship graph based on the differentiable scoring rule graph; S4. Based on the task relationship graph and the model parameters of the shared bottom encoder and the task-specific top expert network, calculate the predicted gradient direction of each task on the shared layer through forward gradient prediction, and generate an initial gradient direction set. S5. Based on the initial gradient direction set and the task relationship graph, calculate the conflict index of each task through the conflict detection mechanism, and implement a dynamic parameter freezing strategy according to the conflict index to generate the optimized gradient update direction. S6. Based on the optimized gradient update direction, and combined with the multi-objective loss function after dynamic weight calibration for each scoring task, update the model parameters of the shared bottom encoder and the task-specific top expert network to generate the updated oral assessment model. S7. Input the speech to be evaluated into the updated oral assessment model, and perform weighted fusion of the predicted scores of each scoring task to generate an oral expression ability score result.

2. The method according to claim 1, characterized in that, The process involves multi-granularity feature decoupling of the acquired raw speech signal to extract multi-dimensional feature data from the acoustic feature space, linguistic feature space, and paralinguistic feature space, generating a decoupled feature set, including: S11. Perform volume calibration processing on the raw speech signal acquired synchronously through dual channels to obtain standardized speech waveform data; the standardized speech waveform data is used to characterize that the speech signal quality meets the preset standard; S12. Extract low-level acoustic features such as Mel-frequency cepstral coefficients, fundamental frequency trajectory, and energy distribution from the standardized speech waveform data. Generate acoustic feature space data by calculating vowel format deviation and consonant voicing contrast. The formula for calculating the vowel format deviation is: in, This is due to vowel format deviation. Indicates the first Each format frequency, Represents the original vowel. Represents standard vowels; S13. The speech is converted into text using an end-to-end speech recognition engine. Based on the converted text, features of lexical diversity, syntactic complexity, and semantic coherence are extracted. Semantic density is calculated by combining the lexical embedding vectors in the pre-trained semantic space to generate linguistic feature space data. The formula for calculating the semantic density is: in, For semantic density, It is a word Embedded vector, Total word count; S14. Extract pause frequency, pause duration distribution, speech rate variation and prosodic features from the original speech signal, distinguish between hesitant pauses and thinking pauses through a context-aware pause classification model, and generate paralinguistic feature space data. S15. The acoustic feature space data, the linguistic feature space data, and the paralinguistic feature space data are concatenated according to feature dimensions to generate the decoupled feature set.

3. The method according to claim 1, characterized in that, Based on the task relationship graph and the model parameters of the shared bottom encoder and the task-specific top expert network, the predicted gradient directions for each task on the shared layer are calculated through forward gradient prediction to generate an initial set of gradient directions, including: S21. Based on the current parameters of the shared underlying encoder and the decoupled feature set, the intermediate activation values ​​of each scoring task are calculated through forward propagation to obtain the forward propagation result. S22. Based on the forward propagation results and the current parameters of the task-specific top-level expert network, calculate the basic loss value of each scoring task corresponding to its true label to obtain the task loss set. S23. Based on the task loss set, the predicted gradient of each task to the shared underlying encoder parameters is approximately calculated through the gradient direction prediction network to generate an initial gradient direction set.

4. The method according to claim 1, characterized in that, The process of calculating the conflict index of each task based on the initial gradient direction set and the task relationship graph, and generating optimized gradient update directions by implementing a dynamic parameter freezing strategy based on the conflict index, includes: S31. Based on the initial set of gradient directions, calculate the gradient cosine similarity between all rating task pairs using the following formula to obtain the gradient cosine similarity matrix: in, For the task and tasks Gradient cosine similarity between them and Tasks and The predicted gradient direction; S32. Based on the gradient cosine similarity matrix and the historical relationship types recorded in the task relationship graph, the real-time conflict index for each scoring task is obtained using the following formula: in, For the task Conflict index For the task and tasks Gradient cosine similarity between them For task pairs obtained based on the task relationship graph Relationship weights; S33. Based on the real-time conflict index, the first preset threshold, and the second preset threshold of each task, obtain the corresponding dynamic parameter freezing operation result; S34. Based on the unfrozen shared layer parameters, the initial gradient direction set is projected and adjusted to obtain the gradient projection result; S35. Integrate the results of the dynamic parameter freezing operation with the gradient projection results to generate the optimized gradient update direction for updating the parameters of the shared underlying encoder.

5. The method according to claim 4, characterized in that, The dynamic parameter freezing operation result obtained based on the real-time conflict index, the first preset threshold, and the second preset threshold for each task includes: S41, when the real-time conflict index is less than or equal to the first preset threshold, the result of the dynamic parameter freezing operation is marked as mild conflict, and only the specific subset of neurons in the shared layer that contributes the most to the conflict is frozen; S42, when the real-time conflict index is greater than the first preset threshold and the real-time conflict index is less than or equal to the second preset threshold, the result of the dynamic parameter freezing operation is marked as moderate conflict, and the entire functional branch related to the high conflict task in the shared layer is frozen. S43, when the real-time conflict index is greater than the second preset threshold, the result of the dynamic parameter freezing operation is marked as severe conflict, the conflict tasks are grouped and the alternating training mode is triggered.

6. The method according to claim 1, characterized in that, The updated oral assessment model is generated by updating the model parameters of the shared bottom encoder and the task-specific top expert network based on the optimized gradient update direction and the multi-objective loss function after dynamic weight calibration for each scoring task, thereby generating the updated oral assessment model. S51. Calculate the base loss function value of each scoring task in this training batch; the base loss function includes the contrast loss of the pronunciation accuracy task, the segmentation ranking loss of the fluency task, the similarity-based regularization loss of the vocabulary diversity task, and the graph structure loss of the logical coherence task. S52. Based on the ratio of the current basic loss value of each task to its recent historical loss value, the loss ratio weight is determined through a dynamic weight calculation model. S53. Monitor the Euclidean norm of the gradient magnitude of each task, and perform weight suppression on tasks whose gradient magnitude is consistently higher than the average level to obtain the gradient magnitude weight. S54. Combine the loss ratio weights and the gradient magnitude weights to obtain the final multi-objective loss weights for each task, and construct a weighted multi-objective loss function; the expression of the multi-objective loss function is: in, For the multi-objective loss function value, Indicates task The final multi-objective loss weights, Indicates task The base loss value, Indicates the total number of tasks. Represents the regularization coefficient. Represents the set of all model parameters; S55. Update the shared encoder parameters based on the optimized gradient update direction, and update the task-specific top-level expert network parameters according to the gradient generated by the weighted multi-objective loss function to generate the updated spoken language evaluation model.

7. An AI-powered oral expression ability assessment device, characterized in that, The device includes: The data acquisition and feature processing module is used to perform multi-granularity feature decoupling processing on the acquired raw speech signal, extract multi-dimensional feature data from the acoustic feature space, linguistic feature space and paralinguistic feature space, and generate a decoupled feature set; The task encoding processing module is used to perform task-aware encoding processing on the decoupled feature set through a hierarchical structure of a shared bottom encoder and a task-specific top expert network, generating specific feature representations for scoring tasks such as pronunciation accuracy, fluency, lexical diversity, and logical coherence. The task relationship graph construction module is used to construct a differentiable scoring rule graph based on the specific feature representation, according to the semantic description of each scoring task and the label co-occurrence relationship in the training data, and generate a task relationship graph based on the differentiable scoring rule graph. The initial gradient prediction module is used to calculate the predicted gradient direction of each task on the shared layer through forward gradient prediction based on the task relationship graph and the model parameters of the shared bottom encoder and the task-specific top expert network, and generate an initial gradient direction set. The gradient update optimization module is used to calculate the conflict index of each task based on the initial gradient direction set and the task relationship graph through a conflict detection mechanism, and to implement a dynamic parameter freezing strategy based on the conflict index to generate optimized gradient update directions. The oral assessment model update module is used to update the model parameters of the shared bottom encoder and the task-specific top expert network based on the optimized gradient update direction and combined with the multi-objective loss function after dynamic weight calibration for each scoring task, so as to generate the updated oral assessment model. The oral expression ability scoring module is used to input the speech to be evaluated into the updated oral assessment model, and to perform weighted fusion of the predicted scores of each scoring task to generate an oral expression ability scoring result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images