Language barrier execution type intervention effect evaluation method based on deep learning

Abstract

The invention relates to the technical field of language barrier evaluation, and discloses a deep learning-based language barrier executive intervention effect evaluation method. The method comprises the following steps: acquiring real-time voice data and facial expression data of a language barrier patient in intervention training through a multi-modal data acquisition device to form an original behavior feature set; performing acoustic feature hierarchical analysis on the voice data by adopting a time sequence feature extraction network to generate a voice time sequence feature vector; performing micro-expression dynamic capture on the expression data through a three-dimensional convolutional neural network to generate an expression state feature vector; inputting the two types of vectors into a multi-modal feature fusion layer to carry out cross-modal correlation analysis, and generating a comprehensive behavior evaluation matrix; on the basis of the matrix, an intervention effect analysis model driven by an attention mechanism is adopted, a behavior improvement degree index of the current intervention stage is calculated, accurate evaluation of the intervention effect is achieved, and support is provided for dynamic adjustment of language barrier rehabilitation intervention.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of language disorder evaluation, in particular to a language disorder executive intervention effect evaluation method based on deep learning. BACKGROUND

[0002] In the field of language disorder rehabilitation, after the patient is trained by the executive intervention, the intervention effect needs to be judged by scientific evaluation to adjust the subsequent training scheme. At present, the evaluation of the effect of the executive intervention of the language disorder mainly depends on the artificial method. The evaluator gives the evaluation result by observing the language expression and facial expression of the patient in the training combined with experience. Such method is greatly influenced by the subjective cognition and professional level difference of the evaluator. There is often deviation in the evaluation results of different evaluators on the same patient. In addition, the artificial observation is difficult to capture the subtle behavior changes of the patient in the training process in real time, resulting in that the evaluation result lacks comprehensiveness and timeliness.

[0003] With the development of technology, part of the evaluation methods begin to introduce single modal data to assist the evaluation. For example, only the voice data of the patient is collected, and the intervention effect is judged by analyzing the acoustic characteristics of the voice. However, the behavior performance of the language disorder patient is not only reflected in the voice level. The facial expression often presents dynamic changes with the language expression. Some patients may have abnormal expression control due to language dysfunction. Only relying on the voice data cannot fully reflect the intervention response of the patient, resulting in that the evaluation dimension is single and it is difficult to cover the comprehensive behavior state of the patient.

[0004] In the feature extraction link, the existing processing of the voice data mostly adopts the traditional acoustic feature extraction method, which cannot effectively capture the dynamic law of the voice signal changing with time and cannot reflect the stage change characteristics of the voice ability of the patient in the intervention process. For the facial expression data, the traditional method mostly adopts the two-dimensional image analysis technology, which can only extract the static expression characteristics and cannot capture the dynamic evolution process of the micro-expression. The information of the emotion and behavior state of the patient contained in the micro-expression has important reference value for the intervention effect evaluation. The static analysis leads to the omission of this part of information.

[0005] In the existing multi-modal evaluation related research, the fusion of different modal data mostly stays in the data splicing level. The internal correlation between the voice and the facial expression is not deeply mined. The complementarity of the two types of data is not fully played. The evaluation basis generated is difficult to fully reflect the behavior improvement of the patient. At the same time, in the intervention effect calculation link, the key behavior characteristics are not focused. The core improvement dimension cannot be focused according to the behavior characteristics of different patients. The pertinence and precision of the evaluation result are insufficient, which is difficult to meet the needs of the fine evaluation of the intervention effect in the clinical rehabilitation. SUMMARY

[0006] The present application aims to provide a language disorder execution intervention effect evaluation method based on deep learning to solve the problems raised in the background art.

[0007] To achieve the above-mentioned purpose, the present application provides a language disorder execution intervention effect evaluation method based on deep learning, which comprises: acquiring real-time speech data and facial expression data of a language disorder patient in an intervention training process through a multi-modal data acquisition device to form an original behavior feature set; performing acoustic feature hierarchical analysis on the speech data in the original behavior feature set by using a time sequence feature extraction network to generate a speech time sequence feature vector; performing micro-expression dynamic capture on the facial expression data in the original behavior feature set by using a three-dimensional convolutional neural network to generate an expression state feature vector; inputting the speech time sequence feature vector and the expression state feature vector into a multi-modal feature fusion layer for cross-modal correlation analysis to generate a comprehensive behavior evaluation matrix; based on the comprehensive behavior evaluation matrix, using an intervention effect analysis model driven by an attention mechanism to calculate a behavior improvement degree index of the current intervention stage.

[0008] Preferably, the acquisition of real-time speech data and facial expression data of a language disorder patient in an intervention training process through a multi-modal data acquisition device specifically comprises: deploying a high-sensitivity microphone array in a standardized intervention training scene to collect continuous speech signals of the patient at a fixed sampling frequency; synchronously starting an infrared thermal imaging camera to record the movement trajectory of the patient's facial muscle group at a millisecond-level time resolution; performing pre-emphasis processing and frame windowing operation on the continuous speech signals to obtain preprocessed speech data; performing key point calibration and motion vector calculation on the movement trajectory of the facial muscle group to obtain standardized expression data.

[0009] Preferably, the acoustic feature hierarchical analysis on the speech data in the original behavior feature set by using a time sequence feature extraction network specifically comprises: constructing a deep feature extractor containing a long short-term memory unit to perform syllable boundary detection on the preprocessed speech data; extracting fundamental frequency trajectory, formant envelope and spectral tilt features respectively within the detected syllable boundaries; performing time scale normalization on the fundamental frequency trajectory, formant envelope and spectral tilt features by hierarchical convolution operation; concatenating the normalized features of different time scales by syllable units to form a speech time sequence feature vector.

[0010] Preferably, the micro-expression dynamic capture of the facial expression data in the original behavior feature set through the three-dimensional convolutional neural network specifically comprises: The standardized expression data is divided into a continuous expression frame sequence in the time dimension; The expression frame sequence is subjected to local muscle movement pattern recognition in the shallow network of the three-dimensional convolutional neural network; The micro-expression change features across frames are extracted through the spatial pyramid pooling operation of the middle layer network; The micro-expression change features are subjected to importance weighting using the spatiotemporal attention module in the deep network, and an expression state feature vector is output.

[0011] Preferably, the speech time sequence feature vector and the expression state feature vector are input into the multi-modal feature fusion layer for cross-modal correlation analysis, specifically comprising: A double-flow feature alignment mechanism is established to synchronize the time stamps of the speech time sequence feature vector and the expression state feature vector; The dynamic correlation weight of the speech features and the expression features is calculated through cross-modal attention; The two feature vectors are subjected to gated fusion operation based on the dynamic correlation weight; The fused features are subjected to dimension reduction processing to generate a comprehensive behavior evaluation matrix.

[0012] Preferably, the behavior improvement degree index of the current intervention stage is calculated based on the comprehensive behavior evaluation matrix using an intervention effect analysis model driven by an attention mechanism, specifically comprising: A pre-trained intervention effect benchmark model is loaded, and the intervention effect benchmark model contains a feature mode library of historical intervention cases; The comprehensive behavior evaluation matrix is subjected to similarity matching with the feature mode library through deformable convolution operation; The difference degree distribution of the current behavior features and each historical mode is calculated using a multi-head attention mechanism; The quantization value of the behavior improvement degree index is calculated according to the area under the difference degree distribution curve.

[0013] Preferably, after the pre-trained intervention effect benchmark model is loaded, it further comprises: Real-time reception of intervention strategy adjustment instructions input from a clinician; Encoding of the intervention strategy adjustment instructions into a strategy feature vector; Embedding of the strategy feature vector into the comprehensive behavior evaluation matrix through feature projection operation; Recomputation of the behavior improvement degree index based on the embedded matrix.

[0014] Preferably, the intervention strategy adjustment instructions are encoded into a strategy feature vector, specifically comprising: constructing an instruction parsing tree containing a policy type, a strength parameter and an execution timing; topological feature extraction on the instruction parsing tree through a graph neural network; comparing the extracted topological features with a historical policy database; outputting a policy feature vector with explainability.

[0015] Preferably, the policy feature vector is embedded into a comprehensive behavior evaluation matrix through a feature projection operation, specifically including: establishing a policy-behavior association mapping table to record the influence coefficients of different policy types on historical behavior features; querying the association mapping table to obtain dynamic projection weights according to the policy feature vector; fusing the policy feature vector into a specified dimension of the comprehensive behavior evaluation matrix by using a weighted summation method; generating an enhanced evaluation matrix containing policy influence information.

[0016] Preferably, after the embedded matrix is used to recalculate the behavior improvement degree index, it further includes: starting a real-time feedback generator to convert the behavior improvement degree index into a visual evaluation report; annotating key improvement feature points and behavior segments to be optimized in the evaluation report; establishing a mapping relationship chain between the current evaluation result and the next stage intervention policy; automatically generating a policy optimization suggestion queue according to the mapping relationship chain.

[0017] Compared with the prior art, the present application has the following advantages: Through the multi-modal data acquisition device, real-time voice data and facial expression data of the language disorder patient during intervention training are synchronously acquired, compared with the traditional single modal data acquisition mode, the voice expression and facial behavior of the patient can be covered at the same time. In the intervention process, the change of the voice signal directly reflects the improvement of the language function, and the dynamic adjustment of the facial expression is often associated with the fluency of the language expression and the emotional state, the synchronous acquisition of the two types of data can record the comprehensive behavior performance of the patient, avoid the one-sidedness of the evaluation caused by the missing of the data dimension, and make the subsequent evaluation basis more in line with the actual intervention response of the patient.

[0018] The time sequence feature extraction network can extract acoustic features of the speech data in a hierarchical manner, and can gradually mine the dynamic change rules in the speech signal. The speech data has obvious time sequence characteristics. During the intervention training, the acoustic features of the speech, such as tone, speech rate and intelligibility, will change in stages with the training process. The time sequence feature extraction network can capture these features that change over time through hierarchical processing, and the generated speech time sequence feature vector can more accurately reflect the dynamic improvement trajectory of the patient's speech ability during the intervention process. Compared with the traditional static feature extraction method, the influence trend of the intervention on the patient's speech function can be better reflected.

[0019] The three-dimensional convolutional neural network can capture the dynamic changes of micro-expression in the time dimension by capturing the dynamic changes of micro-expression in the time dimension. Even if the patient cannot control the subtle changes in the expression, the three-dimensional convolutional neural network can effectively identify and convert the expression state feature vector. These micro-expression information can help to judge the patient's concentration, emotional feedback and adaptation to the training content during the intervention training, and provide more dimensional behavior basis for the intervention effect evaluation, so that the evaluation result can better reflect the real behavior state of the patient.

[0020] The speech time sequence feature vector and the expression state feature vector are input into the multi-modal feature fusion layer to carry out cross-modal correlation analysis. Instead of simply concatenating the two types of data, the internal relationship between speech and facial expression is deeply mined. For example, when the patient's speech fluency improves, the facial tension may be relieved. This kind of cross-modal correlation information can be fully mined through the analysis of the fusion layer, and the comprehensive behavior evaluation matrix can integrate the complementary information of the two types of data, avoid the limitations of single modal data, and fully present the behavior improvement of the patient in the intervention process, providing a more rich and systematic evaluation basis for subsequent effect calculation.

[0021] Based on the comprehensive behavior evaluation matrix, the attention mechanism driven intervention effect analysis model is used to calculate the behavior improvement degree index, which can focus on the core improvement dimension according to the behavior characteristics of different patients. The core problems of different language disorder patients are different. Some patients may focus on improving the intelligibility of speech, and some patients may focus on the coordination ability of expression and language. The attention mechanism can automatically identify the key features that have a greater impact on the evaluation result, and give these features higher attention weight when calculating the improvement degree, so that the final evaluation result is more targeted, and the improvement highlights and improvement direction of the patient in the current intervention stage can be accurately positioned, which provides clear guidance for the individualized adjustment of subsequent intervention plan, and helps the efficient rehabilitation of language disorder patients. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The comprehensive evaluation graph; Figure 2 Flowchart for speech timing feature extraction; Figure 3 Flowchart for multi-modal feature fusion and alignment; Figure 4 Flowchart for pattern matching and difference analysis. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0024] Please refer to Figure 1 The present application provides a language barrier performance intervention effect evaluation method based on deep learning, which comprises the following steps: acquiring real-time speech data and facial expression data of a language barrier patient in an intervention training process through a multi-modal data acquisition device to form an original behavior feature set. An acoustic feature hierarchical analysis is performed on the speech data in the original behavior feature set by using a timing feature extraction network to generate a speech timing feature vector; a micro-expression dynamic capture is performed on the facial expression data in the original behavior feature set by using a three-dimensional convolutional neural network to generate an expression state feature vector. The speech timing feature vector and the expression state feature vector are input into a multi-modal feature fusion layer for cross-modal correlation analysis to generate a comprehensive behavior evaluation matrix; and based on the comprehensive behavior evaluation matrix, an intervention effect analysis model driven by an attention mechanism is used to calculate a behavior improvement degree index of the current intervention stage.

[0025] Embodiment 1: Please refer to Figure 2In the standardized intervention training scenario, a high-sensitivity microphone array is deployed to collect continuous speech signals of patients with speech disorders at a fixed sampling frequency. The high-sensitivity microphone array adopts a linear arrangement, covering the main direction of the patient's pronunciation. The fixed sampling frequency is set to 16 kHz, which can capture the complete spectral information of the speech signal and avoid the loss of high-frequency components. An infrared thermal imaging camera is started synchronously to record the movement trajectory of the patient's facial muscle groups with millisecond-level time resolution. The infrared thermal imaging camera collects facial images at a rate of 100 frames per second, identifies muscle movements through changes in thermal distribution, and ensures the continuity and integrity of data collection. The continuous speech signal is pre-emphasized and framed and windowed. The pre-emphasis processing uses a first-order high-pass filter to boost high-frequency components. The framed and windowed operation uses a Hamming window with a frame length of 25 milliseconds and a frame shift of 10 milliseconds to reduce spectral leakage and obtain preprocessed speech data. The movement trajectory of the facial muscle groups is key-point labeled and motion vector calculated. The key-point labeling is based on the facial anatomy, and the feature points in the eyelid, mouth corner, and eyebrow regions are selected. The motion vector calculation uses the optical flow method to quantify the muscle displacement amplitude and velocity, and obtains standardized expression data. A deep feature extractor containing long short-term memory units is constructed to detect syllable boundaries in the preprocessed speech data. The long short-term memory units in the deep feature extractor are configured in a bidirectional structure to capture the context-dependent relationships of the speech signal. The syllable boundary detection is based on energy mutation and zero-crossing rate changes, and a dynamic threshold is set to distinguish the syllable start and end points. Within the detected syllable boundaries, the fundamental frequency trajectory, formant envelope, and spectral tilt features are extracted. The fundamental frequency trajectory is extracted using the autocorrelation algorithm to calculate the fundamental frequency variation of the speech signal. The formant envelope is obtained through linear predictive coding analysis to represent the vocal tract resonance characteristics. The spectral tilt is calculated using the power spectrum slope to reflect the spectral distribution characteristics of the speech. The fundamental frequency trajectory, formant envelope, and spectral tilt features are time-scale normalized through hierarchical convolution operations. The hierarchical convolution operation uses multi-scale convolution kernels, including 3x3, 5x5, and 7x7 convolution kernels, to extract features at different time scales. Time-scale normalization aligns the feature sequences to a uniform length, eliminating inconsistencies on the time axis. The normalized features at different time scales are concatenated in syllable units to form a speech time sequence feature vector. The final dimension of the speech time sequence feature vector is 256, with each dimension corresponding to a feature point, fully representing the time sequence dynamic characteristics of the speech.

[0026] The deployment position of the high-sensitivity microphone array is optimized to ensure stable capture of the patient's speech signal during intervention training, unaffected by environmental noise. The linear arrangement of the microphone array allows beamforming, enhancing the ability to pick up target speech. The choice of a fixed sampling frequency of 16 kHz is based on the Nyquist sampling theorem, which can cover the typical frequency range of human speech. The first-order high-pass design of the pre-emphasis filter compensates for the high-frequency attenuation of the speech signal during propagation. The application of the Hamming window reduces the spectral distortion caused by framing. The millisecond-level time resolution of the infrared thermal imaging camera is sufficient to capture the rapid changes of facial micro-expressions. The thermal imaging technology indirectly reflects muscle activity based on skin temperature changes, avoiding the influence of lighting conditions. The key point labeling uses an active shape model to accurately locate facial feature points. The motion vector calculation estimates the pixel-level motion through the optical flow method, which uses the Horn-Schunck algorithm to solve the motion field in the image sequence. The standardized expression data contains the displacement vector and timestamp of each key point. The long short-term memory unit of the deep feature extractor has a forget gate, input gate, and output gate structure, which handles long-term dependencies in the speech sequence. The bidirectional design allows the network to consider both the preceding and following context information, improving the accuracy of syllable boundary detection. The dynamic threshold of syllable boundary detection adjusts according to the local statistical properties of the speech signal, avoiding false positives caused by fixed thresholds. The fundamental frequency trajectory extraction uses the autocorrelation function to find the periodic peak value, and the formant analysis uses the inverse filter coefficient of linear predictive coding to derive the formant frequency. The spectral tilt calculation fits the slope of the log power spectrum through linear regression. The multi-scale convolution kernel of the hierarchical convolution operation extracts feature patterns from short to long time. The time scale normalization uses the dynamic time warping algorithm to align the feature sequence. The concatenation operation of the speech timing feature vector organizes the feature blocks in time order, with each syllable corresponding to a feature block. The design of a 256-dimensional vector balances the information density and computational complexity, facilitating subsequent processing.

[0027] The synchronization mechanism of the multi-modal data acquisition device is based on hardware timestamps, ensuring the temporal alignment of speech and expression data, the microphone array and the infrared thermal imaging camera are started synchronously through trigger signals, and the data stream is marked with accurate time. The quality check of pre-processed speech data includes signal-to-noise ratio calculation and abnormal frame rejection, and the standardized expression data is smoothed to remove high-frequency noise, the accuracy of key point labeling is verified by manual labeling, and the result of motion vector calculation is stored as a vector sequence. The training of the deep feature extractor uses a large amount of speech data to optimize the network parameters to minimize the error of syllable boundary detection, the feature extraction process is performed in real time to meet the delay requirements of intervention evaluation. The generation efficiency of speech timing feature vectors is improved through parallel computing, using GPU to accelerate convolution and normalization operations, and the whole process is integrated in an embedded system to realize edge computing. The arrangement of the standardized intervention training scene follows the clinical specifications, reduces the influence of external variables, and the model selection of the high-sensitivity microphone array is based on frequency response and sensitivity indicators, and the resolution of the infrared thermal imaging camera meets the requirements of capturing facial details. The coefficients of the pre-emphasis filter are adjusted according to the speech characteristics, the frame length and frame shift parameters of the frame windowing trade off time resolution and frequency resolution, and the Hamming window function design reduces spectral leakage. The key point labeling algorithm uses a hierarchical regression tree method to improve the labeling speed and accuracy, and the optical flow method of motion vector calculation realizes dense optical flow estimation to capture comprehensive muscle movement. The network structure of the deep feature extractor contains multiple hidden layers, the number of neurons in each layer is optimized, and the threshold learning of syllable boundary detection is completed through supervised training. The median filter is used for smoothing the fundamental frequency trajectory, the order of the linear predictive coding is optimized for extracting the formant envelope, and the calculation range of the spectral tilt covers the effective frequency band. The convolution kernel size of the hierarchical convolution operation is selected based on the speech unit duration, the alignment algorithm of the time scale normalization uses dynamic programming, and the dimension selection of feature concatenation is based on the dimension reduction result of principal component analysis. The application of speech timing feature vectors includes real-time monitoring and offline analysis, vector data is stored in a database for historical queries, and the update of the feature extraction network is realized through online learning to adapt to different patient characteristics. The calibration of the multi-modal data acquisition device is performed regularly to ensure data consistency, the parameters of the pre-processing algorithm are adaptively adjusted according to environmental conditions, and the interpretability of the feature extraction process is enhanced through visualization tools.

[0028] Example 2: see Figure 3The standardized expression data is divided into a continuous expression frame sequence in the time dimension, the division of the expression frame sequence takes 33 milliseconds as a time window, each window contains 5 consecutive image frames, and the time step of the frame sequence is 5 milliseconds, ensuring that the continuity of the dynamic changes of micro-expression is completely retained. Local muscle movement pattern recognition is performed on the expression frame sequence in the shallow network of the three-dimensional convolutional neural network, the shallow network uses a 3x3x3 convolution kernel for three-dimensional convolution operation, and the convolution kernel slides in the spatial and temporal dimensions to extract features, and identifies local muscle movement patterns including mouth corner up amplitude, eyebrow wrinkling frequency, and eyelid opening speed. The spatial pyramid pooling operation of the middle layer network extracts the micro-expression change features across frames, the spatial pyramid pooling sets four pooling windows of different scales, respectively performs 4x4, 8x8, 16x16 and global average pooling on the feature map, and captures cross-scale features from subtle muscle tremor to macro-expression change. In the deep network, the spatio-temporal attention module is used to weight the importance of the micro-expression change features, the spatio-temporal attention module calculates the attention weight matrix in the spatial dimension and the temporal dimension respectively, the spatial attention focuses on the significant area of facial muscle activity, and the temporal attention highlights the key frame of expression change, and outputs a 128-dimensional expression state feature vector. A double-flow feature alignment mechanism is established to synchronize the speech time sequence feature vector and the expression state feature vector, the double-flow feature alignment mechanism is based on the hardware timestamp of the data acquisition device, and uses the dynamic time warping algorithm to align the feature sequences of the two modalities, the path constraint of the dynamic time warping is set as the Sakoe-Chiba band constraint with a slope of 1, and the rationality of the alignment path is ensured. The dynamic association weight between the speech feature and the expression feature is calculated through the cross-modal attention, the cross-modal attention mechanism takes the speech time sequence feature vector as the query vector and the expression state feature vector as the key-value pair, calculates the association degree score between each speech frame and expression frame, and the dimension of the association weight matrix is the product of the number of speech frames and the number of expression frames. The two feature vectors are subjected to a gating fusion operation based on the dynamic association weight, the gating fusion operation uses a sigmoid function as a gating unit, the output value of the sigmoid function adjusts the fusion ratio of the speech feature and the expression feature, the bias term of the gating unit is initialized to zero, and the weight matrix is initialized by the Xavier method. The integrated behavior evaluation matrix is generated by performing dimension reduction processing on the fused features, the dimension reduction processing is realized by two layers of fully connected neural networks, the first layer of fully connected network reduces the feature dimension from 512 to 128, and the second layer of fully connected network further reduces to 64, and each row of the integrated behavior evaluation matrix represents the integrated behavior state at a time point.

[0029] The architecture of the three-dimensional convolutional neural network includes four convolutional layers and two pooling layers, and the filter numbers of the convolutional layers are 32, 64, 128 and 256 respectively. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The pooling layer adopts a maximum pooling operation with a pooling kernel size of 2x2x2. The preprocessing of the expression frame sequence includes grayscale normalization and contrast enhancement. Grayscale normalization maps the pixel value to the range of 0-1, and contrast enhancement uses the histogram equalization method to improve image quality. The training of the local muscle movement pattern recognition uses the labeled facial action unit dataset, and the network parameters are optimized by the back propagation algorithm. The loss function adopts the cross-entropy loss, and the optimizer selects the Adam algorithm with a learning rate of 0.001. The output features of the spatial pyramid pooling layer are spliced, and the dimension of the spliced feature vector is 2688. Then, the dimension is compressed through 1x1 convolution. The spatial attention branch of the spatial-temporal attention module adopts the channel attention mechanism, and the time attention branch uses the gated recurrent unit structure. The attention weight is calculated by the softmax function normalization. The dynamic time warping algorithm of the dual-flow feature alignment mechanism sets the bending cost function as the Euclidean distance, and the path search uses the dynamic programming method. The length of the aligned feature sequence is unified to the length of the longer sequence, and the shorter sequence is expanded by the interpolation method. The query vector, key vector and value vector in the cross-modal attention calculation are obtained by linear transformation. The weight matrix dimension of linear transformation is 64x64, the number of attention heads is set to 8, and the dimension of each attention head is 8. The output value range of the sigmoid function in the gated fusion operation is 0-1. When the output value is close to 1, it means that the speech feature dominates the fusion result, and when it is close to 0, it means that the expression feature dominates the fusion result. The input of the gating unit is the spliced vector of the two feature vectors. The fully connected network of the dimension reduction processing uses the Dropout regularization technique with a Dropout ratio of 0.5 to prevent overfitting. The activation function uses the Tanh function to maintain the symmetric distribution of the output values.

[0030] The generation process of the comprehensive behavior evaluation matrix is performed in real time, and the matrix content is updated every 5 seconds. The number of rows of the matrix corresponds to the number of time points, and the number of columns corresponds to the feature dimension. The matrix data is stored in a ring buffer, supporting historical data backtracking. The inference speed of the three-dimensional convolutional neural network is optimized through model quantization technology, reducing the floating-point precision from 32 bits to 16 bits to meet real-time processing requirements. The timestamp synchronization accuracy of the dual-flow feature alignment mechanism reaches milliseconds, with a synchronization error of less than 10 milliseconds, ensuring the time consistency of multi-modal data. The calculation complexity of the cross-modal attention mechanism is optimized by reducing the dimension of the key-value pair. The original dimension of 128 is reduced to 64 through PCA. The number of parameters of the gating fusion operation is the square of the input dimension, which is reduced through the weight sharing mechanism. Weight sharing is performed in the time dimension. The visualization of the expression state feature vector uses a heat map, which is overlaid on the original facial image. The red area represents significant muscle activity, and the blue area represents a static state, making it easy for clinicians to understand intuitively. The time alignment quality of the speech timing feature vector and the expression state feature vector is evaluated by the correlation coefficient. The correlation coefficient threshold is set to 0.8. If it is lower than the threshold, the realignment process is triggered. The distribution analysis of the dynamic correlation weight uses information entropy calculation. The higher the entropy value, the more complex the correlation between modalities. If the entropy value is lower than the set threshold, the fusion strategy is simplified. The weight update of the reduced dimension full connection network uses the stochastic gradient descent method with momentum, with a momentum coefficient of 0.9. The learning rate decay strategy uses a stepwise decay. The data structure of the comprehensive behavior evaluation matrix is designed in the form of a tensor, supporting GPU accelerated operation. The persistent storage of the matrix uses column storage format to improve read-write efficiency.

[0031] The training data enhancement of the three-dimensional convolutional neural network includes random rotation, translation and scaling transformation, which enhances the generalization ability of the model. The training period is set to 100 rounds. The early stopping mechanism is triggered when the validation set loss does not decrease for 5 consecutive rounds. The abnormal processing of the dual-flow feature alignment mechanism includes interpolation compensation for timestamp loss. The interpolation method uses linear interpolation, and the maximum interpolation interval is limited to 100 milliseconds. The output of the cross-modal attention mechanism is normalized to stabilize the training process. The gain parameter and bias parameter of the normalization are learnable. The gradient flow analysis of the gating fusion operation uses gradient clipping technology to prevent gradient explosion. The clipping threshold is set to 1.0. The feature importance evaluation of the dimension reduction processing is calculated by the L1 norm of the weight matrix. The weight with a large norm corresponds to an important feature dimension. The dimension selection of the comprehensive behavior evaluation matrix is based on feature contribution analysis. The contribution is calculated by the mutual information between the feature and the label. The feature dimensions with mutual information greater than the threshold are retained.

[0032] In the embodiment 3, a pre-trained intervention effect benchmark model is loaded, which contains a feature pattern library of historical intervention cases. The feature pattern library stores strictly screened and labeled historical intervention case data from anonymized case records of multiple rehabilitation centers. The intervention effect benchmark model uses a deep residual network as the basic architecture, which contains 50 convolutional layers, each followed by batch normalization processing and a ReLU activation function. Residual connections span every two convolutional layers to effectively alleviate the gradient vanishing problem of deep networks. Model parameters are optimized through a distributed training framework. The training data set contains more than 100,000 multi-modal feature sequences of intervention sessions. The loss function uses a smooth L1 loss, and the optimization process uses a stochastic gradient descent method with momentum, with a momentum coefficient of 0.9. The feature pattern library uses a hierarchical indexing structure. The first-level index is classified by language barrier type, the second-level index is divided by intervention stage, and the third-level index is grouped by patient age. Each feature pattern corresponds to a 64-dimensional floating-point vector, and the vector values are processed by z-score standardization.

[0033] The comprehensive behavior assessment matrix is matched with the feature pattern library through deformable convolution operation. The deformable convolution layer is set before the standard convolution layer, and the convolution kernel size is fixed at 3x3. Two additional offset parameters are learned at each spatial position, and the initial value of the offset is set to zero. The similarity matching calculation uses the projection distance metric in the feature space. Each feature vector in the comprehensive behavior assessment matrix is projected into the feature space of the pattern library through linear transformation. The dimension of the projection matrix is 64x64. The projected feature vector and the benchmark vector in the pattern library are calculated by Euclidean distance. The distance value is converted into a similarity score through a Gaussian kernel function. The calculation formula of the similarity score is: ; Wherein: represents the similarity score between the feature vector at the th time point in the comprehensive behavior assessment matrix and the th pattern vector in the feature pattern library, is the 64-dimensional feature vector at the th time point in the comprehensive behavior assessment matrix, is the 64-dimensional benchmark vector of the th historical pattern in the feature pattern library, is a 64x64-dimensional projection matrix, is the bandwidth parameter of the Gaussian kernel, which controls the decay rate of the similarity score. The number of rows of the similarity score matrix is equal to the number of time points in the comprehensive behavior assessment matrix, and the number of columns is equal to the number of patterns in the feature pattern library.

[0034] A multi-head attention mechanism is used to calculate the dissimilarity distribution between the current behavioral features and each historical pattern. This mechanism comprises eight independent attention heads, each maintaining its own query, key, and value projection matrices. The query vector is derived from the feature representation of the comprehensive behavioral evaluation matrix after projection transformation. The key and value vectors are derived from baseline vectors in the feature pattern library. The attention weights are calculated using a scaled dot product attention mechanism. The dissimilarity distribution is generated using a kernel density estimation method, with the Epanechnikov kernel chosen as the kernel function. The bandwidth is selected based on empirical rules to ensure the smoothness of the distribution curve. The horizontal axis of the dissimilarity distribution curve represents the dissimilarity value, normalized to the [0,1] interval, and the vertical axis represents the probability density. The total area under the curve is equal to 1.

[0035] The quantitative value of the behavior improvement index is calculated based on the area under the curve of the difference distribution. The area under the curve is calculated using a numerical integration method, with the integration interval covering the entire difference range [0,1]. The numerical integration employs a compound Simpson's law, dividing the integration interval into 1000 equal-length subintervals. A quadratic polynomial is used to approximate the integrand function in each subinterval. Behavior Improvement Index The numerical calculation formula is as follows: ; in: This represents the degree of behavioral improvement, with values ​​ranging from [0,1]. Represents the difference distribution function. The difference value is represented by a numerical method for integral calculation. The improvement index is updated every five minutes using a sliding window mechanism with a window length of 30 minutes. New data is continuously added to the window, while old data is removed from the window in chronological order.

[0036] The parameters of the intervention effect benchmark model are updated using an online learning strategy. Newly generated evaluation data are added to the training set after quality checks, and the model parameters are retrained weekly. The feature pattern library is updated and maintained by adding new patterns and removing old ones. New pattern additions require expert review, while old pattern removal is based on time decay weights; patterns that haven't matched successfully for more than three years are automatically archived. The offset learning for deformable convolutions uses a grouped convolution approach, with each channel learning an independent offset field. Regularization constraints on the offset field are achieved through weight decay. The query projection matrix, key projection matrix, and value projection matrix of the multi-head attention mechanism all use the Xavier initialization method, and the gradient clipping threshold during training is set to 1.0. Real-time generation of the dissimilarity distribution curve uses an incremental kernel density estimation algorithm; adding new data points does not require recalculating the entire distribution. Precision control of numerical integration is achieved by dynamically adjusting the number of sub-intervals; sampling points are automatically increased when the function changes drastically. Inference acceleration of the intervention effect benchmark model uses model quantization technology, quantizing 32-bit floating-point parameters into 8-bit integers, increasing inference speed by three times while maintaining accuracy loss of less than 1%. The feature pattern library retrieval optimization is based on the Locality Sensitive Hashing (LSH) algorithm, which quickly finds nearest neighbor patterns in high-dimensional space. The hardware implementation of deformable convolution uses a dedicated tensor processor to support dynamic shape calculations. A caching mechanism for similarity scores establishes an in-memory cache of recently retrieved results, reducing redundant computation overhead. A visualization analysis tool for the multi-head attention mechanism provides a heatmap display of attention weights to help understand the model's decision-making process. Statistical feature extraction of the difference distribution curve includes mean, variance, and skewness calculations to monitor changes in distribution shape. Time-series analysis of the behavior improvement index uses an autoregressive integral moving average model to predict future trends in improvement. Access security to the feature pattern library is ensured through multi-factor authentication, and data transmission employs end-to-end encryption. The offset field of deformable convolution is smoothed to avoid unreasonable deformation. Confidence evaluation of similarity scores is based on the statistical distribution of nearest neighbor distances, with low-confidence matching results given special labels. The output of the multi-head attention mechanism undergoes layer normalization to improve training stability.

[0037] The version control system of the intervention effect benchmark model records metadata for each parameter update, supporting rapid rollback to historical versions. The feature pattern library backup strategy employs multi-copy off-site storage, achieving 99.99% data reliability. The training efficiency of deformable convolutions is improved through an asynchronous data parallel strategy, supporting simultaneous training on multiple GPUs. Parallel computation of similarity matching leverages the hardware acceleration capabilities of matrix operations, significantly increasing computational speed. Anomaly detection of the difference distribution curve is based on the statistical process control chart method, setting upper and lower control limits to identify abnormal points. Alarm thresholds for behavioral improvement indicators are personalized based on patient baseline levels, triggering an alert when indicator values ​​consistently fall below the threshold. Interpretive analysis of the intervention effect benchmark model uses the SHAP feature importance method to quantify the contribution of each input feature to the final evaluation result. Quality monitoring of the feature pattern library includes data consistency checks and abnormal pattern detection, generating data quality reports periodically. Offset range limitations for deformable convolutions are implemented through activation function pruning, ensuring offsets remain within a reasonable range. Normalization of similarity scores uses a min-max scaling method, mapping scores to the [0,1] interval. The residual connections of the multi-head attention mechanism avoid the performance degradation caused by increasing network depth. Real-time display of the difference distribution curve supports interactive exploration, allowing users to select a specific time range to view distribution changes. Long-term tracking of behavioral improvement indicators utilizes database time series functionality, storing evaluation results for trend analysis. Performance evaluation of the intervention effect benchmark model uses a retained test set, periodically calculating the model's evaluation accuracy and recall.

[0038] See Figure 4This presentation showcases the quantitative evaluation process of intervention effects based on a historical pattern database and deep learning algorithms. Two sub-plots, arranged vertically, systematically present the complete analytical workflow for behavioral feature matching and improvement calculation. The upper heatmap uses a grayscale gradient to display the similarity matching matrix between the current patient's behavioral characteristics at various time points during the intervention and the historical case pattern database. The color depth intuitively reflects the degree of matching; darker areas represent a high degree of consistency between the current behavioral characteristics and patterns from historically successful intervention cases, while lighter areas indicate significant differences. This visualization helps clinicians quickly identify the degree of deviation between the patient's behavioral patterns and the ideal recovery trajectory. The lower probability distribution plot uses a smooth curve to show the distribution of differences between the current behavioral characteristics and the baseline pattern. The curve shape reflects the stability of the patient's behavioral changes during the intervention process, and the skewness of the distribution curve indicates the directionality of the improvement trend. The quantitative calculation of the area under the curve is directly converted into a behavioral improvement index, providing a mathematical basis for the objective evaluation of rehabilitation effects. These two subgraphs together form a complete assessment loop. The pattern matching in the upper graph provides a data foundation for the difference analysis in the lower graph, and the improvement calculation in the lower graph, in turn, verifies the effectiveness of the matching results. The entire chart system reflects the precision and quantitative advantages of deep learning technology in rehabilitation medicine assessment, and provides reliable data support for the adjustment of personalized intervention plans.

[0039] Example 4: Real-time reception of intervention strategy adjustment instructions from clinicians. These instructions are collected through a dedicated input panel in the graphical user interface. The input panel contains three core components: a strategy type selector, an intensity parameter slider, and an execution timing setter. Based on real-time observations of patient performance, clinicians select the appropriate intervention strategy type on the input panel. Strategy types include predefined categories such as adjusting pronunciation training intensity, introducing auxiliary communication tools, and adjusting the difficulty gradient of training tasks. The intensity parameter slider allows clinicians to precisely set the execution intensity of the strategy numerically, with the intensity parameter range normalized to between 0 and 1. The execution timing setter supports both immediate and delayed execution modes; in delayed execution mode, a specific start time point needs to be specified. After format validation, the input instructions are converted into a structured instruction data packet, which contains five fields: timestamp, clinician ID, strategy type code, intensity parameter value, and execution time point.

[0040] Intervention strategy adjustment instructions are encoded into strategy feature vectors, and an instruction parse tree is constructed containing strategy type, strength parameters, and execution timing. The instruction parse tree adopts a binary tree structure, with the root node storing the strategy type encoding, the left subtree storing strength parameter nodes, and the right subtree storing timing parameter nodes. The strategy type node contains a 16-bit type encoding vector, the strength parameter node stores a 32-bit floating-point number, and the timing parameter node contains a time flag and a relative time offset. Topological features are extracted from the instruction parse tree using a graph neural network. The graph neural network adopts a graph convolutional network architecture, with three graph convolutional layers, each followed by a graph pooling operation. Node feature updates use a message passing mechanism. The extracted topological features are compared and retrieved against a historical strategy database, which stores all valid intervention strategy records from the past year. Each record contains a strategy feature vector and a corresponding effect evaluation index. The comparison retrieval is based on the k-nearest neighbor algorithm in the feature space to find the 10 historical strategy cases most similar to the current instruction, outputting interpretable strategy feature vectors. The strategy feature vector dimension is fixed at 32 dimensions. The strategy feature vectors are embedded into the comprehensive behavioral evaluation matrix through feature projection operations, establishing a strategy-behavior association mapping table. This table uses a relational database structure, with each record containing four fields: strategy type ID, behavioral feature dimension ID, influence coefficient, and confidence level. The influence coefficient, calculated using multiple linear regression analysis, represents the average influence of a specific strategy type on a particular behavioral feature dimension. The confidence level is determined based on the statistical significance of historical data. Dynamic projection weights are obtained by querying the association mapping table based on the strategy feature vectors. The query operation uses an index lookup based on the strategy type code, and the matched influence coefficients form a 64-dimensional projection weight vector. A weighted summation method is used to fuse the strategy feature vectors into a specified dimension of the comprehensive behavioral evaluation matrix. The weighted summation formula is: New matrix element = Original matrix element + Strategy feature vector × Projection weight. An enhanced evaluation matrix containing strategy impact information is generated. The dimensions of the enhanced evaluation matrix remain unchanged, but each feature value incorporates the expected impact of strategy adjustments.

[0041] Referring to Table 1, the behavior improvement index is recalculated based on the embedded matrix. The recalculation process uses the same algorithm as the original evaluation, but the input evaluation matrix already includes the expected effect of the policy adjustment. The update frequency of the behavior improvement index remains unchanged, maintaining a rhythm of outputting once every five minutes to ensure the real-time nature of the evaluation results. The embedding operation of the policy feature vector does not change the basic data structure of the matrix, and all subsequent processing modules can be seamlessly integrated. The enhanced evaluation matrix is ​​stored using versioned management, with each version marked with a corresponding policy adjustment instruction ID, supporting result traceability and analysis.

[0042] Table 1: Strategy-Behavior Association Mapping Table Strategy Type ID Behavior Characteristic Dimension ID Influence Coefficient Confidence Last Update Time 0x01A3 12 0.156 0.89 2024-03-15 0x01A3 25 -0.083 0.76 2024-03-15 0x01A5 8 0.214 0.92 2024-03-14 0x01A5 31 0.097 0.81 2024-03-14 0x01B2 15 -0.132 0.78 2024-03-16 0x01B2 42 0.188 0.85 2024-03-16 0x01C7 5 0.305 0.94 2024-03-13 0x01C7 28 0.112 0.82 2024-03-13 The instruction parse tree is constructed using recursive descent analysis. The analyzer identifies policy type keywords, parses strength parameter values, and processes temporal expressions. Node features in the graph neural network are initialized using one-hot encoding, and edge features represent the syntactic relationships between nodes. The number of filters in the graph convolutional layers are 64, 128, and 256, respectively. The vector dimension after topological feature extraction is 512-dimensional, which is reduced to 32-dimensional through a fully connected layer before comparison with the historical policy database. The distance metric in the k-nearest neighbor algorithm uses cosine similarity, and the returned similar cases are sorted from high to low similarity scores. The policy-behavior association mapping table is updated using incremental learning; whenever new policy effect data is generated, the system recalculates the influence coefficient and confidence level. The influence coefficient is calculated using weighted least squares, with more recent data points having higher weights. The confidence level is calculated using a t-test p-value transformation. Smoothing is added during the acquisition of dynamic projection weights to avoid drastic fluctuations between adjacent policy adjustments. The weighted summation operation is executed in parallel on the GPU, with vector addition performed independently for each feature dimension. The enhanced evaluation matrix is ​​validated using cross-validation, dividing historical data into training and test sets to verify the predictive accuracy of the projected weights. The interpretability analysis of the policy feature vectors employs an attention mechanism, displaying which historical policy cases contribute most to the current vector encoding. Recalculated behavior improvement indicators are displayed side-by-side with the original indicators, facilitating physician comparison of the expected effects of policy adjustments. The instruction input panel's interactive design follows ergonomic principles, offering both slider and numerical input methods for important parameter settings, reducing the physician's workload. The policy type selector uses a tree-like classification display, supporting keyword search and quick selection of commonly used policies. The intensity parameter slider's scale undergoes logarithmic transformation, making intensity adjustments more visually linear. The execution timing setter provides a visual timeline interface, supporting drag-and-drop adjustment of execution time points. The training data for the graph neural network comes from a historical instruction parsing tree database, with the training objective being to minimize topological feature reconstruction error; the optimizer uses the AdamW algorithm. The historical policy database uses a B+ tree index structure, supporting millisecond-level similar policy retrieval. Layer normalization is used for policy feature vector normalization to ensure comparability between different policies. The mathematical basis of feature projection operation is vector space transformation theory, which maps the policy feature space to the behavior feature space.

[0043] The enhanced evaluation matrix uses a sparse matrix representation, storing only changed data elements to reduce storage space. Performance optimization during recalculation is achieved through incremental computation, updating only the relevant feature dimensions affected by the strategy. Comparative analysis of behavioral improvement indicators provides trend charts showing the improvement curves before and after strategy adjustments. A strategy effect feedback mechanism allows physicians to score the predicted effect, with the scores used to optimize the parameters of the association mapping table. Error handling in the instruction parse tree includes syntax error detection and semantic rationality checks, alerting physicians when abnormal instructions are detected. Uncertainty estimation is incorporated into the inference process of the graph neural network, marking low-confidence topological feature extraction results. The historical strategy database is cleaned up based on usage frequency, automatically archiving long-unused strategy cases. Strategy feature vector serialization uses Protocol Buffers format to ensure efficient and reliable data transmission. An anomaly recovery mechanism for feature projection operations includes default value handling for missing projection weights, ensuring the system continues to function correctly even with partial data loss. The version comparison function of the enhanced evaluation matrix supports visualization of differences between two versions, highlighting the feature dimensions most affected by the strategy. The recalculation process is logged in detail, recording the calculation parameters for each step to facilitate troubleshooting and effect analysis. The quality assessment of the behavior improvement index is based on the smoothness and consistency of the index values, detecting and filtering abnormal fluctuations. Access control for the instruction input panel is based on role-based access control, with different levels of physicians having different policy adjustment permissions. The encrypted storage of policy feature vectors uses the AES-256 algorithm to protect patient privacy and medical protocol security. Security audits of feature projection operations record the usage of projection weights for each operation, meeting the compliance requirements of the medical system. The transmission of the enhanced evaluation matrix uses digital signature technology to prevent data tampering during transmission.

[0044] Example 5: Establish a strategy-behavior association mapping table. This table uses a relational database table structure and contains five core fields: strategy type code, behavior feature dimension identifier, influence coefficient, confidence level, and last update timestamp. The strategy type code is represented by a 16-bit binary number, covering all predefined intervention strategy categories. The behavior feature dimension identifier corresponds to the 64 feature dimensions of the comprehensive behavior assessment matrix. The influence coefficient is obtained through large-scale historical data analysis, calculated using a multiple linear regression model. Each coefficient represents the expected influence strength of a specific strategy on the corresponding behavioral feature. The confidence level is derived from the p-value transformation of a statistical test, ranging from 0 to 1; a higher value indicates a stronger reliability of the influence coefficient. The last update timestamp records the most recent modification time of each mapping relationship. The system automatically performs consistency checks and data updates on the mapping table monthly. Dynamic projection weights are obtained by querying the association mapping table based on the strategy feature vector. The query operation uses a database index lookup based on the strategy type code, employing a B+ tree index structure to ensure query efficiency. The calculation of dynamic projection weights considers not only the strategy type but also the state information of the current behavior feature matrix, including the distribution range and trend of feature values. The projected weight vector contains 64 weight values, each corresponding to a feature dimension of the behavior evaluation matrix. The weight values ​​are calculated using bilinear interpolation, interpolating along both the policy type and feature state dimensions. The weight vector is normalized to ensure that the sum of the squares of all weight values ​​is 1. The real-time performance of the projected weights is guaranteed through a caching mechanism, with recently used weight combinations cached in memory.

[0045] A weighted summation method is used to fuse policy feature vectors into a specified dimension of the comprehensive behavior evaluation matrix. The weighted summation operation is executed in parallel on the GPU, with the calculation of each feature dimension performed independently. The fusion process consists of two stages: the first stage expands the 32-dimensional policy feature vectors to 64 dimensions through a fully connected layer; the second stage adds the expanded vectors to the original evaluation matrix element-wise. The weighting coefficients are derived from dynamic projection weights, with different weighting coefficients used for different feature dimensions. Important feature dimensions have larger weighting coefficients, while less important feature dimensions have smaller weighting coefficients. The fusion operation preserves the numerical distribution characteristics of the original evaluation matrix, making only moderate adjustments to dimensions with policy influence, the adjustment magnitude controlled by the projection weights. An enhanced evaluation matrix containing policy influence information is generated, with the dimensions of the enhanced evaluation matrix being completely identical to the original matrix, maintaining a 64-dimensional feature vector structure. Each feature value in the matrix contains two components: the original behavior feature and the policy influence, the proportion of which is dynamically adjusted by the projection weights. The enhanced evaluation matrix supports version management, with each version marked with the generation time and the corresponding policy adjustment instruction number. The matrix data uses a columnar storage format to optimize read performance. The enhanced evaluation matrix is ​​validated using cross-validation, which divides historical data into training and test sets to verify the accuracy of the matrix predictions.

[0046] The real-time feedback generator integrates data visualization components and a natural language processing module. The visualization component utilizes WebGL technology for high-performance graphics rendering. The feedback generator triggers an assessment report generation process every five minutes, comprising four steps: data collection, feature extraction, result analysis, and report rendering. The natural language processing module employs a Transformer-based text generation architecture, with model parameters trained on medical text corpora to generate professional reports conforming to clinical standards. The visualization component supports various chart types, including line charts, heatmaps, and radar charts, with color schemes conforming to medical industry visual standards. Behavioral improvement indicators are converted into a visual assessment report. The report uses a hierarchical structure: the top layer displays overall improvement indicators and a trend summary; the middle layer displays detailed analysis of each dimension; and the bottom layer provides a raw data query interface. Key improvement features are highlighted in the report, with annotation information including feature name, improvement magnitude, and statistical significance. Behavioral segments to be optimized are displayed using a timeline, with each segment annotated with its start time, duration, problem type, and improvement suggestions. The assessment report supports interactive exploration, allowing users to view detailed data by clicking on chart elements. The report content is updated in real time, maintaining visual continuity during updates.

[0047] The evaluation report identifies key improvement features and behavioral segments to be optimized. Key improvement features are identified based on feature importance ranking, with importance calculated using the SHAP value analysis method. Each key feature is labeled with an improvement magnitude and confidence interval, calculated as the percentage change between the current value and the baseline value. Behavioral segments to be optimized are detected using an anomaly detection algorithm, employing an isolated forest model to identify anomalous time segments. Segment labeling information includes the main problem type, severity level, and frequency statistics; problem types are selected from a predefined problem library. A mapping chain between the current evaluation results and the next-stage intervention strategy is established, stored in a knowledge graph database. The knowledge graph contains three types of entities: evaluation indicators, behavioral features, and intervention strategies. Relationships between entities include causal, correlational, and recommendation relationships, with relationship strength derived from historical data statistics. Mapping inference uses a graph neural network algorithm, performing multi-hop reasoning on the knowledge graph to find the optimal strategy path from the current evaluation state to the target state. The credibility of the relationship chain is calculated using path weights, which are based on historical strategy effectiveness statistics. A strategy optimization suggestion queue is automatically generated based on the mapping relationship chain. This queue employs a priority queue data structure, with priority determined by a comprehensive evaluation of the strategy's expected effectiveness and implementation cost. Each strategy suggestion includes complete parameter settings, such as strategy type, strength parameter, execution time point, and expected effect indicators. The suggestion queue supports dynamic adjustment; when new evaluation results are generated, the system recalculates the strategy priority and adjusts the queue order. The output format of the strategy suggestions is standardized, including five parts: strategy description, theoretical basis, implementation steps, expected effects, and risk assessment.

[0048] The performance optimization of the real-time feedback generator is achieved through asynchronous rendering technology, with interface rendering and data processing executed in parallel on different threads. Evaluation reports are stored using an incremental storage strategy, storing only changes relative to the previous version to reduce storage space usage. The annotation of key improved feature points supports custom threshold settings, allowing clinicians to adjust annotation sensitivity based on patient-specific conditions. The detection algorithm for behavioral fragments to be optimized incorporates temporal continuity constraints to avoid misclassifying transient noise as behavioral fragments. The maintenance of the mapping relationship chain includes both regular updates and manual verification; the system automatically updates relationship weights weekly, and a team of experts conducts monthly manual reviews. The generation process of the strategy optimization suggestion queue incorporates diversity constraints to ensure that the suggested strategy types are not overly concentrated. Parameter settings for strategy suggestions consider individual differences, adjusting parameter ranges based on patients' historical response data. The output of the natural language generation module is templated to ensure the standardization and readability of the report content. Monitoring of the enhanced evaluation matrix includes data quality checks and outlier detection, automatically triggering a recalculation process when data anomalies are detected. The responsive design of the visualization components adapts to different display devices, displaying correctly on tablets and desktop monitors. The assessment report sharing function supports exporting in multiple formats, including PDF, HTML, and JSON, to meet the needs of different usage scenarios. The historical tracking function for key improvement feature points displays the changing trends of feature values, helping physicians determine the sustainability of improvements. The comparative analysis function for behavioral segments to be optimized supports comparison between multiple segments to identify common problem patterns. The visualization of the mapping relationship chain uses a force-directed graph layout, clearly showing the correlation path between assessment results and strategy recommendations. The interactive interface of the strategy optimization suggestion queue allows physicians to manually adjust strategy priorities; adjustment records are used to optimize the automatic sorting algorithm. The system operation status monitoring panel displays real-time values ​​for various indicators, including data processing latency, model inference speed, and storage space utilization.

[0049] The real-time feedback generator's error handling mechanism includes a fallback plan for missing data, ensuring the system can still generate a basic assessment report even when some data is unavailable. A digital signature mechanism for the assessment report guarantees its integrity and authenticity, preventing tampering. The export function of key improvement feature points supports structured data formats for easy subsequent statistical analysis. The annotation of behavioral segments to be optimized supports physician feedback, allowing physicians to confirm or correct automatically annotated results. Version management of the mapping relationship chain records every structural change, supporting rollback and comparative analysis. The strategy optimization suggestion queue's historical record saves the content of each generated suggestion for effect review and algorithm optimization. The system's access control module controls access permissions for different roles to assessment reports and strategy suggestions, protecting patient privacy data. The performance monitoring system records the execution time of each operation and periodically generates performance analysis reports for system optimization.

[0050] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0051] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for evaluating the effectiveness of executive interventions for language disorders based on deep learning, characterized in that, The method includes: Real-time speech and facial expression data of patients with language disorders during intervention training are acquired through multimodal data acquisition devices to form a set of original behavioral features; A temporal feature extraction network is used to perform hierarchical analysis of acoustic features in the speech data of the original behavioral feature set to generate speech temporal feature vectors. The facial expression data in the original behavioral feature set is dynamically captured by a three-dimensional convolutional neural network to generate an expression state feature vector. The speech temporal feature vector and facial expression state feature vector are input into the multimodal feature fusion layer for cross-modal correlation analysis to generate a comprehensive behavior evaluation matrix; Based on a comprehensive behavioral assessment matrix, an attention-driven intervention effect analysis model is used to calculate the behavioral improvement index at the current intervention stage.

2. The method for evaluating the effectiveness of deep learning-based executive intervention for language disorders according to claim 1, characterized in that, The acquisition of real-time speech and facial expression data of patients with language disorders during intervention training through multimodal data acquisition devices specifically includes: In standardized intervention training scenarios, deploy a high-sensitivity microphone array to collect continuous speech signals from patients at a fixed sampling frequency; Simultaneously activate the infrared thermal imaging camera to record the movement trajectory of the patient's facial muscle groups with millisecond-level time resolution; Pre-emphasis processing and frame-segmentation windowing operations are performed on continuous speech signals to obtain pre-processed speech data; Key points were identified and motion vectors were calculated for the movement trajectories of facial muscle groups to obtain standardized facial expression data.

3. The method for evaluating the effectiveness of deep learning-based executive intervention for language disorders according to claim 2, characterized in that, The step of using a temporal feature extraction network to perform hierarchical acoustic feature parsing of speech data in the original behavioral feature set specifically includes: A deep feature extractor containing long short-term memory units is constructed to perform syllable boundary detection on preprocessed speech data; Fundamental frequency trajectory, formant envelope, and spectral tilt features are extracted within the detected syllable boundaries. Time-scale normalization of fundamental frequency trajectory, formant envelope, and spectral tilt features is performed through hierarchical convolution operations. Normalized features at different time scales are concatenated into syllable units to form a speech temporal feature vector.

4. The method for evaluating the effectiveness of deep learning-based executive intervention for language disorders according to claim 3, characterized in that, The process of dynamically capturing micro-expressions from facial expression data in the original behavioral feature set using a three-dimensional convolutional neural network specifically includes: Standardized facial expression data is divided into a continuous sequence of facial expression frames according to the time dimension; Local muscle movement pattern recognition is performed on facial expression frame sequences in a shallow network of a three-dimensional convolutional neural network. Micro-expression change features across frames are extracted using spatial pyramid pooling operations in the middle layer network; In deep networks, a spatiotemporal attention module is used to weight the importance of micro-expression change features and output an expression state feature vector.

5. The method for evaluating the effectiveness of deep learning-based executive intervention for language disorders according to claim 4, characterized in that, The step of inputting speech temporal feature vectors and facial expression state feature vectors into a multimodal feature fusion layer for cross-modal correlation analysis specifically includes: Establish a dual-stream feature alignment mechanism to synchronize the speech temporal feature vector and facial expression state feature vector with timestamps; The dynamic association weights between speech features and facial expression features are calculated using cross-modal attention. Gated fusion operation is performed on two feature vectors based on dynamic correlation weights; The dimensionality of the fused features is reduced to generate a comprehensive behavior evaluation matrix.

6. The method for evaluating the effectiveness of deep learning-based executive intervention for language disorders according to claim 5, characterized in that, The intervention effect analysis model, based on a comprehensive behavioral assessment matrix and driven by an attention mechanism, calculates the behavioral improvement index for the current intervention stage, specifically including: Load a pre-trained intervention effectiveness benchmark model, which contains a feature pattern library of historical intervention cases; The comprehensive behavior evaluation matrix is ​​matched with the feature pattern library using deformable convolution operations; The multi-head attention mechanism is used to calculate the distribution of the difference between the current behavioral features and each historical pattern; The quantitative value of the behavior improvement index is calculated based on the area under the difference distribution curve.

7. The method for evaluating the effectiveness of deep learning-based executive intervention for language disorders according to claim 6, characterized in that, After loading the pre-trained benchmark model of intervention effect, the following is also included: Receive real-time instructions from clinicians to adjust intervention strategies; The instructions for adjusting the intervention strategy are encoded into a strategy feature vector; The policy feature vectors are embedded into the comprehensive behavior evaluation matrix through feature projection operations; The behavior improvement index is recalculated based on the embedded matrix.

8. The method for evaluating the effectiveness of deep learning-based executive intervention for language disorders according to claim 7, characterized in that, The process of encoding the intervention strategy adjustment instruction into a strategy feature vector specifically includes: Construct an instruction parse tree that includes policy type, strength parameters, and execution timing; Topological features are extracted from the instruction parse tree using a graph neural network. The extracted topological features are compared and retrieved with the historical strategy database; Output interpretable policy feature vectors.

9. The method for evaluating the effectiveness of deep learning-based executive intervention for language disorders according to claim 8, characterized in that, The embedding of the policy feature vector into the comprehensive behavior evaluation matrix through feature projection operation specifically includes: Establish a strategy-behavior association mapping table to record the influence coefficients of different strategy types on historical behavioral characteristics; The dynamic projection weights are obtained by querying the associated mapping table based on the strategy feature vectors. The strategy feature vectors are fused into a specified dimension of the comprehensive behavior evaluation matrix using a weighted summation method. Generate an enhanced evaluation matrix that includes information on the impact of the policy.

10. The method for evaluating the effectiveness of deep learning-based executive intervention for language disorders according to claim 9, characterized in that, After recalculating the behavior improvement index based on the embedded matrix, the method further includes: Launch the real-time feedback generator to convert behavior improvement metrics into visual evaluation reports; The evaluation report should highlight key improvement features and behavioral segments that need optimization. Establish a mapping relationship between current assessment results and the next stage of intervention strategies; A queue of strategy optimization suggestions is automatically generated based on the mapping relationship chain.

Images