Rehabilitation training adjustment method and system based on multi-modal feature fusion and storage medium

Abstract

The invention relates to the technical field of artificial intelligence and human-computer interaction, and discloses a rehabilitation training adjustment method and system based on multi-modal feature fusion and a storage medium, and the method comprises the steps: obtaining voice, face and physiological signals, constructing a multi-modal data stream, extracting features, and outputting a speech error classification identifier and a physiological signal quality index; splitting the facial action net displacement into healthy and affected sides, correcting affected side features based on healthy side features, and constructing visual representation; fusing each feature to output a comprehensive state vector; updating a dynamic baseline in a task gap, mapping a state vector, and screening to obtain an effective action set; calculating a composite reward and storing the composite reward in an experience playback pool to update the strategy network; and based on the action set, re-weighting the large model output probability, and generating a target interaction corpus. According to the method, the state sensing precision is improved by correcting the deviation of the affected side through the uninjured side, and a closed loop of self-adaptive adjustment of the rehabilitation difficulty and safe interaction text generation is realized by combining the dynamic baseline and penalty mechanism optimization reinforcement learning.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, specifically to a rehabilitation training adjustment method, system, and storage medium based on multimodal feature fusion. Background Technology

[0002] Rehabilitation training for speech disorders and other conditions typically relies on human-computer interaction systems for assisted intervention. Current rehabilitation assistance systems mainly advance the training process through preset rule engines, monomodal speech recognition, or fixed corpora. Typically, the system statically adjusts the difficulty of subsequent tasks based on the accuracy of the user's voice responses, or sequentially pushes text and image content in a pre-set order.

[0003] However, conventional interactive intervention methods have significant limitations in practical applications. First, existing systems typically assess user status based solely on a single answer or voice / text, ignoring the unique facial muscle activation asymmetry and physiological signal fluctuations specific to rehabilitation populations. Due to the lack of effective fusion and bias correction of visual and physiological multimodal data, the system often cannot accurately distinguish whether user feedback errors stem from genuine cognitive deficiencies or from physiological fatigue or emotional resistance during training, leading to distorted assessment results.

[0004] Secondly, conventional systems often lag in adjusting training difficulty and lack a reliable physiological baseline for reference. When performing interactive tasks, a user's physiological state changes drastically with cognitive load. If fixed rules are used to intervene in difficulty, it is easy to maintain high-intensity training even when the user experiences frustration. Existing control strategies lack dynamic self-evolution capabilities and cannot iterate strategies in real time based on the user's negative emotional feedback, resulting in a rigid interaction process and severely reducing the user's long-term training adherence.

[0005] Finally, most existing systems rely on fixed question banks with limited capacity, resulting in a lack of targeted interactive content. Even when some systems attempt to introduce text generation models, in the specific context of medical rehabilitation, the conventional model's free decoding process lacks hard constraints on vocabulary difficulty and safety, easily generating corpora that exceed the patient's current cognitive level or are inappropriate, thus failing to achieve accurate and safe adaptation of training content.

[0006] Therefore, this invention proposes a rehabilitation training adjustment method, system, and storage medium based on multimodal feature fusion to address the shortcomings of existing technologies. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a rehabilitation training adjustment method, system, and storage medium based on multimodal feature fusion. This solves the problem that existing assisted rehabilitation systems, due to their single dimension of state perception and static and lagging intervention rules, cannot accurately distinguish between patients' insufficient cognitive ability and emotional fatigue, thus making it difficult to provide real-time, safe, and personalized adaptive interaction strategies.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a rehabilitation training adjustment method based on multimodal feature fusion, comprising the following steps: Acquire users' voice signals, facial video signals, and physiological signals to construct a synchronized multimodal data stream; Speech and physiological features are extracted from multimodal data streams. Speech signals are processed using acoustic coding models and language coding models to output speech error classification labels and calculate physiological signal quality index. The net displacement vector of facial motion units is obtained based on facial video signals. The net displacement vector of facial motion units is split into healthy side feature vector and affected side feature vector. The affected side feature vector is subjected to low-pass filtering. Based on the normalized correlation coefficient between the healthy side feature vector and the filtered affected side feature vector and the activation threshold of the healthy side, the corrected output of the affected side feature vector is generated. The visual representation is constructed by combining the corrected output of the affected side feature vector. The speech features, visual representations and physiological features are fused and processed, and a comprehensive state vector is output by combining the physiological signal quality index. Within the task gap window, the dynamic baseline is updated based on physiological characteristics, or a historical alternative baseline is invoked as the actual baseline if the update conditions are not met; The composite action space is output based on the comprehensive state vector mapping. The effective action set is obtained by filtering the composite action space with the speech error classification label. The composite reward is calculated based on the baseline of the actual call. An allowed vocabulary is constructed based on the set of effective actions. The original probability distribution of candidate words output by the large language model is reweighted to obtain the final probability distribution. The target interaction corpus is generated based on the final probability distribution. The comprehensive state vector, the set of effective actions, the composite reward, and the comprehensive state vector at the next time step are constructed into a transition tuple and stored in the experience replay pool to update the policy network weights.

[0009] Preferably, constructing a synchronous multimodal data stream includes: Establish a basic time axis based on the sensor sampling rate; A sliding window mechanism is used to truncate the multi-source signals to be processed. By using a linear interpolation algorithm, truncated multi-source data is aligned to a unified temporal granularity to construct a synchronous multimodal data stream; Outputting speech error classification labels and calculating the physiological signal quality index includes: extracting acoustic features using an acoustic coding model, extracting high-dimensional speech features containing semantic structure information based on the acoustic features using a language coding model, and mapping and outputting speech error classification labels through a fully connected classification layer. The discrete power spectral density of physiological signals is calculated using the discrete Fourier transform algorithm. The physiological signal quality index is obtained by calculating the ratio of the signal power within the normal heart rate frequency band to the total power within the Nyquist frequency range.

[0010] Preferably, obtaining the net displacement vector of the facial motion unit based on the facial video signal includes: Extract the current three-dimensional absolute coordinates of the facial motion unit; Obtain the rigid body transformation matrix of the pre-calibrated reference anchor point; The net displacement vector of the facial motion unit is obtained by performing vector difference between the current three-dimensional absolute coordinates and the theoretical coordinates containing rigid body motion components. The construction of a visual representation by combining the corrected output of the feature vector from the affected side includes: Extracting micro-expression features and lip-shape consistency features from facial video signals; The corrected output of the feature vector from the affected side, the feature vector from the healthy side, the micro-expression features, and the lip-shape consistency features are concatenated to generate a visual representation.

[0011] Preferably, based on the normalized correlation coefficient between the healthy side feature vector and the filtered affected side feature vector, and the activation threshold of the healthy side, a corrected output of the affected side feature vector is generated, including: Capture the historical sliding time window; Calculate the normalized correlation coefficient between the healthy side feature vector and the filtered affected side feature vector within the historical sliding time window; Calculate the Euclidean norm of the healthy side's eigenvectors; Determine whether the Euclidean norm is greater than the pre-calibrated healthy side activation threshold; When the Euclidean norm is greater than the activation threshold of the healthy side, the feature vector of the healthy side is multiplied by the normalized correlation coefficient to generate the corrected output of the feature vector of the affected side. When the Euclidean norm is not greater than the activation threshold of the healthy side, the zero vector is output as the corrected output of the feature vector of the affected side.

[0012] Preferably, the speech features, visual representations, and physiological features are fused, and a comprehensive state vector is output by combining the physiological signal quality index, including: The attention weights of verbal features, visual representations, and physiological features were calculated using the self-attention mechanism to obtain the enhanced verbal features, enhanced visual representations, and enhanced physiological features. The enhanced visual representation and enhanced physiological features are concatenated along the feature channel dimension to construct auxiliary modal features; Using enhanced speech features as the primary query source and auxiliary modal features as the key source, cross-attention calculation is performed to obtain the cross-attention matrix; The physiological signal quality index, enhanced speech features, and cross-attention matrix are concatenated and input into a fully connected network layer to calculate the dynamic gating factor. The cross-attention matrix and enhanced speech features are weighted and fused using dynamic gating factors to output a comprehensive state vector.

[0013] Preferably, within the task gap window, the dynamic baseline is updated based on physiological characteristics, or a historical alternative baseline is invoked as the actual baseline when the update conditions are not met, including: Lock the task gap window between two task interactions; Calculate the first derivative of the heart rate variability feature sequence contained in the physiological characteristics; When the first derivative is less than a preset derivative threshold, the physiological steady state approaching zero condition is determined to be met. When the physiological steady state approaches zero condition is met, the mean heart rate variability feature within the task interval window is extracted. The dynamic baseline for the next time step is updated by combining the current dynamic baseline with the mean characteristics of heart rate variability using an exponential moving average mechanism. When the physiological steady state approaching zero condition is not met and the preset timeout period is triggered, the historical alternative baseline is called as the actual baseline and a blocking signal is output.

[0014] Preferably, a composite action space is output based on a comprehensive state vector mapping, and the effective action set is obtained by filtering the composite action space using speech error classification labels, including: The comprehensive state vector is input into the policy network and mapped to output a composite action space. The composite action space includes at least difficulty adjustment parameters, prompting strategy parameters, topic shifting parameters, semantic field parameters, syntactic depth parameters, target phoneme parameters, and rest control parameters. By using a prior rule mapping function containing an expert rule matrix and combining it with speech error classification labels, the composite action space is masked and filtered to generate a set of valid actions; The baseline calculation of composite rewards based on actual calls includes: Extract pure negative emotional features; Calculate the numerical offset between the current heart rate variability characteristics and the actual baseline called; The frustration penalty term is obtained by mapping the numerical offset to the pure negative emotional features. The frustration penalty is weighted and aggregated with the accuracy reward, engagement reward, and efficiency reward to generate a composite reward.

[0015] Preferably, the original probability distribution of candidate lexical units output by the large language model is reweighted to obtain the final probability distribution, and the target interactive corpus is generated based on the final probability distribution, including: The set of valid actions includes difficulty adjustment parameters; Determine whether a candidate word belongs to the allowed vocabulary and generate the output value of the mask indicator function; The difficulty score function is obtained by calculating the negative absolute difference between the objective lexical complexity of the allowed words in the vocabulary and the difficulty adjustment parameter in the effective action set. The original probability distribution of candidate words is calculated and normalized by combining the output value of the mask indicator function and the difficulty score function to obtain the final probability distribution. When the weighted sum of all terms at the current time step is detected to be zero, the mask indicator function is disabled, and the preset safe recovery corpus is used as the target interactive corpus.

[0016] This invention also provides a rehabilitation training adjustment system based on multimodal feature fusion, comprising: The perception decoupling module acquires the user's voice signal, facial video signal, and physiological signal to construct a synchronized multimodal data stream. It extracts speech and physiological features from the multimodal data stream, processes the speech signal using acoustic and language coding models to output speech error classification labels, and calculates the physiological signal quality index. Based on the facial video signal, it acquires the net displacement vector of facial motion units, splitting it into healthy-side and affected-side feature vectors. The affected-side feature vector is low-pass filtered, and based on the normalized correlation coefficient between the healthy-side and filtered affected-side feature vectors, as well as the healthy-side activation threshold, a corrected output of the affected-side feature vector is generated. Finally, a visual representation is constructed by combining the corrected output of the affected-side feature vector. The fusion evaluation module is used to fuse speech features, visual representations and physiological features, and output a comprehensive state vector by combining the physiological signal quality index. The baseline decision module is used to update the dynamic baseline based on physiological characteristics within the task gap window, or to call the historical alternative baseline as the actual baseline when the update conditions are not met; it outputs a composite action space based on the comprehensive state vector mapping, and filters the composite action space to obtain a set of effective actions by combining speech error classification labels, and calculates the composite reward based on the actual baseline. The policy evolution module is used to construct an allowed vocabulary based on the set of effective actions, reweight the original probability distribution of candidate words output by the large language model to obtain the final probability distribution, and generate the target interaction corpus based on the final probability distribution. The integrated state vector, the set of effective actions, the composite reward, and the integrated state vector at the next time step are constructed into a transition tuple and stored in the experience replay pool to update the policy network weights.

[0017] The present invention also provides a computer device, including: a processor and a memory, the memory storing a computer program executable by the processor, the computer program performing the method described above when executed by the processor.

[0018] The present invention also provides a storage medium storing a computer program, which is executed by a processor to perform the method described above.

[0019] This invention provides a rehabilitation training adjustment method, system, and storage medium based on multimodal feature fusion. It has the following beneficial effects: 1. This invention constructs a multimodal data stream by acquiring speech, facial, and physiological signals. Specifically, it decomposes the net displacement vector of facial action units into healthy-side and affected-side feature vectors, and uses the healthy-side feature vector to calculate and correct the normalized correlation coefficient of the filtered affected-side feature vector. This technique fully considers the pathological characteristics of asymmetrical facial muscle activation in rehabilitation patients, effectively corrects biases in affected-side action extraction, eliminates misjudgment interference from single modalities, improves the accuracy of visual representation, and provides precise underlying data support for subsequent comprehensive status assessment.

[0020] 2. This invention calculates the first derivative of the heart rate variability feature sequence within a task interval window. When the physiological steady-state condition approaches zero is met, the dynamic baseline is updated using an exponential moving average mechanism. If the condition is not met within a timeout period, a historical alternative baseline is invoked. This technical feature utilizes the gap between two task interactions for data sampling, avoiding physiological signal measurement distortion caused by cognitive load during rehabilitation tasks, ensuring the objectivity and stability of the baseline data, and making subsequent offset calculations more accurate.

[0021] 3. This invention outputs a composite action space based on a comprehensive state vector mapping, combines verbal error classification labels for action selection, and calculates a composite reward including frustration penalties, storing it in an experience replay pool to update network weights. This technical feature transforms extracted negative emotional features and physiological index offsets into quantified penalty feedback, enabling the system to proactively identify and avoid interactive actions that easily cause fatigue or frustration in patients when using reinforcement learning for strategy evolution. This effectively prevents resistance caused by excessive difficulty and improves patient compliance during long-term training.

[0022] 4. This invention constructs an allowed vocabulary based on a set of valid actions. By calculating the absolute difference between the objective vocabulary complexity and the difficulty adjustment parameters, it reweights the original probability distribution of candidate word units output by the large language model and sets up a fallback mechanism for safe rehabilitation corpora. This technical feature strictly masks and redistributes the output probabilities of the large language model during the decoding stage, ensuring that the generated interactive content accurately matches the control instructions issued by the system in terms of semantics and difficulty, completely avoiding the generation of invalid text that is beyond the scope or unsuitable for patients to understand due to the free divergence of the large model. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a schematic diagram of the method flow of the present invention; Figure 3 This is a schematic diagram illustrating the internal working principle of the sensing decoupling module of the present invention; Figure 4 This is a schematic diagram illustrating the internal working principle of the fusion evaluation module of the present invention; Figure 5 This is a schematic diagram of the internal control laws of the baseline decision module and the strategy evolution module of the present invention; Figure 6 This is a schematic diagram comparing the decoupling and filtering timing of the healthy and affected side features in this invention; Figure 7 This is a schematic diagram of the comprehensive state assessment and dynamic baseline tracking curve of the present invention; Figure 8 This is a bar chart comparing the multi-dimensional indicators of the multi-system rehabilitation intervention effect of the present invention, where (a) is the intention recognition accuracy rate, (b) is the average participation time of a single training session, and (c) is the improvement degree of the BDAE aphasia rating scale.

[0024] Among them, 100 is the perception decoupling module; 200 is the fusion evaluation module; 300 is the baseline decision module; and 400 is the strategy evolution module. Detailed Implementation

[0025] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] See attached document Figure 1 This invention provides a rehabilitation training adjustment system based on multimodal feature fusion, comprising: The perception decoupling module 100 acquires speech signals, facial video signals, and physiological signals, establishes a basic timeline, and uses a sliding window mechanism to align the speech signals, facial video signals, and physiological signals to a unified temporal granularity, constructing a multimodal data stream. The perception decoupling module 100 receives the multimodal data stream, extracts speech features, visual features, and physiological features, outputs speech error classification labels, and calculates the physiological signal quality index. The perception decoupling module 100 decomposes the facial motion unit displacement vector, separating the healthy-side feature vector from the affected-side feature vector. The perception decoupling module 100 applies low-pass filtering to the affected-side feature vector, calculates the normalized correlation coefficient between the healthy-side and affected-side feature vectors, combines it with an activation threshold to generate a correction output, and concatenates the correction output with visual features to generate a visual representation.

[0027] The fusion evaluation module 200 performs self-attention calculation based on speech features, visual representations, and physiological features. The fusion evaluation module 200 concatenates the enhancement features of the visual representations and the enhancement features of the physiological features into an auxiliary modality feature, using the enhancement features of the speech features as the query source and the auxiliary modality feature as the key source, to calculate a cross-attention matrix. The fusion evaluation module 200 receives the physiological signal quality index to calculate a dynamic gating factor, and calculates a comprehensive state vector based on the dynamic gating factor, the cross-attention matrix, and the enhancement features of the speech features.

[0028] The baseline decision module 300 extracts heart rate variability feature sequences and calculates the first derivative within the task gap window. When the first derivative satisfies the zero-approach condition, the baseline decision module 300 extracts the mean feature to update the dynamic baseline. When the zero-approach condition is not met, the baseline decision module 300 triggers an interrupt signal, blocking dynamic baseline updates and calling the historical baseline constant as a replacement baseline. The baseline decision module 300 receives the comprehensive state vector and maps it to output a composite action space. The baseline decision module 300 receives verbal error classification labels to filter actions and calculates a composite reward including accuracy reward, engagement reward, efficiency reward, and frustration penalty. The frustration penalty is calculated jointly by negative emotion features and the offset of the dynamic baseline.

[0029] The policy evolution module 400 receives action parameters from the composite action space, constructs a restricted vocabulary, and reweights the original probability distribution of the language model based on the action parameters to generate interactive corpus. The policy evolution module 400 constructs a transition tuple from the comprehensive state vector, the composite action space, the composite reward, and the comprehensive state vector at the next time step and stores it in the experience replay pool. When the set conditions are met, the policy network weights are updated.

[0030] See attached document Figure 2 This invention provides a rehabilitation training adjustment method based on multimodal feature fusion, comprising the following steps: S100 acquires speech signals, facial video signals, and physiological signals. It uses a sliding window mechanism to align the signals and construct a multimodal data stream. It extracts speech features, visual features, and physiological features from the multimodal data stream, calculates the physiological signal quality index, separates the healthy side feature vector and the affected side feature vector, performs low-pass filtering and correlation calculation, and generates a visual representation. S200 calculates the cross-attention matrix of speech features, visual representations and physiological features, calculates the dynamic gating factor of the received physiological signal quality index, and calculates the comprehensive state vector. S300 calculates the first derivative of the heart rate variability feature sequence within the task gap window, updates the dynamic baseline or calls the alternative baseline based on the zero-reaching condition, outputs the composite action space based on the comprehensive state vector mapping, filters actions by combining speech error classification labels, and calculates the composite reward. S400 constructs a restricted vocabulary based on action parameters in the composite action space, reweights the original probability distribution of the language model to generate interactive corpus, and constructs a transition tuple from the comprehensive state vector, composite action space, composite reward and the comprehensive state vector of the next time step into the experience replay pool to update the network weights.

[0031] To further clarify the implementation of each technical aspect of the present invention, the following will provide a detailed description of the implementation of each functional module involved above and its internal processing flow.

[0032] See attached document Figure 2 Step S100 is specifically executed by the perception decoupling module 100 in the system architecture, mainly completing the time alignment of multimodal signals, basic feature extraction, and pathological decoupling processing based on the correlation between healthy and affected sides. In this embodiment, to ensure the logical consistency of multi-source heterogeneous data in the spatiotemporal dimension and to remove pathological noise interference, this step is further divided into the following sub-steps for execution: S110, the basic sampling rate standard for the multimodal signal is set and a multimodal data stream is constructed. The perception decoupling module 100 acquires the user's voice signal, facial video signal, and physiological signal through a sensor array. Specifically, the sampling rate of the voice signal is set to 16kHz or 44.1kHz to preserve acoustic details; the frame rate of the facial video signal is set to 30fps or 60fps to capture subtle facial muscle movements; the physiological signal preferably uses photoplethysmography (rPPG) or electrocardiogram (ECG) signals, with a sampling rate set to 256Hz or 512Hz. As the optimal implementation method for the specific rehabilitation scenario of this invention, the physiological signal is continuously estimated from the region of interest of the facial video using remote photoplethysmography (rPPG) technology, and its sampling rate is mapped and resampled to 25Hz to 60Hz to ensure the accuracy of heart rate variability analysis. Due to the inherent differences in sampling frequencies among various sensors, to eliminate time misalignment caused by asynchronous sampling, the perception decoupling module 100 establishes a basic time axis based on the aforementioned sampling rate and employs a sliding window mechanism to truncate the signal to be processed. The sliding window's time window length is set to 2 seconds, and the sliding step size is set to 0.5 seconds. Furthermore, the perception decoupling module 100 aligns the truncated multi-source data to a unified time granularity using a linear interpolation algorithm, thereby constructing a synchronous multimodal data stream.

[0033] S120 extracts speech and physiological features and calculates the physiological signal quality index in real time. The perception decoupling module 100 receives the aligned multimodal data stream and processes the audio data stream using an acoustic coding model and a language coding model. As a preferred approach, the acoustic coding model employs a feature extraction network containing multiple one-dimensional convolutional layers and relative position encoding to map the original audio waveform into an acoustic feature sequence; the language coding model uses a bidirectional Transformer encoder to receive the aforementioned acoustic feature sequence and perform contextual self-attention calculation, outputting high-dimensional speech features containing semantic structural information. As another preferred implementation, the acoustic coding model can use a Wav2Vec2-type pre-trained speech representation network, and the language coding model can use a RoBERTa-type bidirectional language coding network. Based on this, a fully connected classification layer is cascaded at the end of the model, using a Softmax activation function to output a speech misclassification label. This label specifically indicates the type of obstacle the user faces in pronunciation or semantic understanding, such as phoneme distortion or word-finding difficulty, using a one-hot encoding format. To ensure the model can accurately identify disorder types, a rehabilitation corpus containing various pathological pronunciations and corresponding authoritative medical annotations is used as training samples during the training phase of the acoustic coding model and the language coding model. The authoritative medical annotations are preprocessed before training to be one-hot encoded true labels consistent with the speech error classification categories. The acoustic coding model and the language coding model are then subjected to end-to-end supervised joint training using the cross-entropy loss function. Simultaneously, the perception decoupling module 100 extracts physiological features from the physiological signals. These features preferably include time-domain indices of heart rate variability, such as the standard deviation of all normal sinus intervals and the root mean square of the difference between adjacent NN intervals. To filter out motion artifacts from the physiological signals, the perception decoupling module 100 calculates the physiological signal quality index from the physiological signals in real time based on the frequency domain signal-to-noise ratio. In general signal processing principles, the effective components of physiological signals are usually concentrated in specific low-frequency narrow bands, while environmental noise and motion artifacts exhibit a broadband spectrum. Based on this characteristic, the specific formula for calculating the physiological signal quality index is as follows: ; In the formula: The quality index of the physiological signal extracted at the current moment; The discrete power spectral density of the physiological signal is calculated using the discrete Fourier transform. to The normal heart rate frequency band is set to 0.7Hz to 3.5Hz. This is the physiological signal sampling rate. This formula quantifies the proportion of the effective heart rate signal in the overall signal by calculating the ratio of the signal power within the normal heart rate frequency band to the total power within the Nyquist frequency range.

[0034] S130: Extract the absolute coordinates of the facial motion units and remove high-frequency pathological spasms on the affected side. The perception decoupling module 100 performs keypoint tracking on the facial video signal in the multimodal data stream and extracts the current three-dimensional absolute coordinates of the facial motion units. To eliminate errors caused by head movement, the perception decoupling module 100 uses pre-calibrated reference anchor points to calculate the rigid body transformation matrix, and performs vector difference between the current absolute coordinates and the theoretical coordinates containing rigid body motion components to obtain the net displacement vector. The specific formula for calculating the net displacement vector is as follows: ; In the formula: This represents the net displacement vector of the facial motion unit at the current moment; These are the original three-dimensional absolute coordinates of the key points extracted at the current moment; This is the head rotation matrix; The pre-calibrated three-dimensional point cloud coordinates of the face in a resting state; This is the head translation vector.

[0035] After obtaining the net displacement vector, the perception decoupling module 100 splits it into a healthy side feature vector and an affected side feature vector along the vertical midline of the face. Considering that the affected side of patients with facial paralysis or hemiplegia is often accompanied by involuntary high-frequency spasms, the perception decoupling module 100 applies a first-order exponential smoothing low-pass filter to the affected side feature vector to remove high-frequency pathological spasm components. The specific formula for the filter calculation is as follows: ; In the formula: This is the filtered feature vector of the affected side; The smoothing coefficient is set to a value between 0.1 and 0.3. This range can filter out high-frequency jitter while retaining the low-frequency true facial deformation trend. This is the feature vector of the affected side at the current time before filtering; This is the filtered feature vector of the affected side from the previous time step.

[0036] S140, calculate the correlation between the healthy and affected sides and generate the corrected output for the affected side. Human facial muscle movements exhibit strong bilateral symmetry in a healthy state. Based on this principle, the perception decoupling module 100 calculates the normalized correlation coefficient between the feature vector of the healthy side and the smoothed feature vector of the affected side to quantify the degree of coordination between the two sides' muscle movements. To ensure the mathematical conditions for statistical calculations are met, the perception decoupling module 100 truncates the historical length, including the current moment. The sliding time window constitutes the feature sequence. The specific formula for calculating the normalized correlation coefficient is as follows: ; In the formula: The normalized correlation coefficient between the feature vectors of the healthy side and the feature vectors of the affected side within the sliding window; The function for calculating covariance; The sequence of healthy-side feature vectors within the sliding window; This is the function for calculating standard deviation; To prevent extremely small constants with a denominator of 0, their preferred value is 10. -6 ; The number of frames per window is preferably 15 to 30.

[0037] Based on the obtained normalized correlation coefficient, the perception decoupling module 100 constructs a conditional gating equation in conjunction with the healthy side activation threshold, generating a corrected output that reflects the user's true motion intention. The specific formula of the conditional gating equation is as follows: ; In the formula: This is the corrected output of the feature vector of the affected side; Let be the Euclidean norm of the eigenvectors of the healthy side, representing the absolute amplitude of motion of the healthy side; The activation threshold for the healthy side is pre-calibrated based on the user's resting baseline. When the movement intensity on the healthy side exceeds this threshold, the perception decoupling module 100 determines that there is a valid movement intention and uses the state of the healthy side combined with the correlation coefficient to reconstruct the state of the affected side in a compensatory manner; otherwise, it outputs a 0 vector to avoid misjudging noise in the resting state as movement.

[0038] S150, extract visual base features and generate visual representations. The perceptual decoupling module 100 uses a deep learning model to extract micro-expression features and lip-sync features from facial video signals. Specifically, micro-expression features are preferably extracted by a three-dimensional convolutional neural network combined with an attention-based Transformer model. In the model architecture design of this embodiment, the perceptual decoupling module 100 inputs continuous video facial cropped frames into a C3D network containing multiple layers of three-dimensional convolutional kernels to extract spatiotemporal local feature maps. Then, the flattened feature sequence is input into a multi-head Transformer encoding layer to capture global temporal dependencies, and outputs a micro-expression feature vector after global average pooling. The micro-expression category includes at least six basic emotions such as joy, sadness, anger, fear, disgust, and surprise, as well as two cognitive load states: confusion and effort. To ensure that different models can effectively backpropagate in this step, when building the model, it is preferred to use a large-scale facial expression dataset and a facial video dataset with manually supplemented annotations of confusion and effort states to jointly construct a joint training set containing eight state category labels, and supervised training is performed using the cross-entropy loss function. In addition, for lip-shape consistency features, the perceptual decoupling module 100 extracts a sequence of mouth region images from the facial video signal and inputs it into a Visual Geometric Group Network (VGG network) to extract lip movement features. These features are then compared temporally with the acoustic feature sequence output from the aforementioned acoustic coding model using a Dynamic Time Warping (DTW) algorithm, resulting in a lip-shape consistency feature vector representing the degree of matching between the intended pronunciation and the actual lip shape. Finally, the perceptual decoupling module 100 concatenates the corrected affected-side features, healthy-side features, and extracted micro-expression features and lip-shape consistency features to generate a complete visual representation. The specific concatenation formula for generating the visual representation is as follows: ; In the formula: This is the final visual representation output to downstream modules; Micro-expression features; This is a characteristic of consistent mouth shape. This indicates a tensor splicing operation.

[0039] See attached document Figure 2 Step S200 is specifically executed by the fusion evaluation module 200 in the system architecture. It mainly transforms heterogeneous multimodal features into a unified high-dimensional state representation and introduces a smooth degradation mechanism for physical quality perception. In this embodiment, to ensure information complementarity and system robustness against interference during multimodal fusion, this step is further divided into the following sub-steps: S210, self-attention networks are applied to speech features, visual representations, and physiological features respectively for intramodal context enhancement. The fusion evaluation module 200 receives sequences of speech features, visual representations, and physiological features from the current time and historical time windows. Considering the long-distance temporal dependence of single-source signals in natural interaction, as a preferred approach, the fusion evaluation module 200 pre-assigns fixed positional codes to each modal feature to preserve temporal order information, and then uses a multi-head self-attention mechanism to calculate the attention weights within each modal sequence to enhance the information expression at key time nodes. The calculation formula for intramodal context enhancement is as follows: ; ; ; In the formula: For enhanced speech features; For enhanced visual representation; These are enhanced physiological characteristics; For self-attention network mapping functions; These are primitive speech characteristics; The original visual representation; These are the original physiological characteristics. For the query, key-value matrix multiplication, and activation processes within the self-attention network mapping function, those skilled in the art can implement them using the standard Transformer coding layer structure, which is a well-known technique in the field and will not be elaborated upon here.

[0040] S220, Constructing Auxiliary Modal Features. In human-computer interaction scenarios, the user's verbal expression usually carries the core task intent, while facial movements and physiological states serve more as secondary evidence of emotion and cognitive load. Based on this physical relationship, the fusion evaluation module 200 performs tensor concatenation of the enhanced visual representation and enhanced physiological features along the feature channel dimension to construct unified auxiliary modal features. The specific formula for tensor concatenation is as follows: ; In the formula: For auxiliary modal features; This represents a tensor splicing operation along the feature channel dimension. In this embodiment, before performing the splicing operation, the fusion evaluation module 200 ensures that the visual feature sequence and the physiological feature sequence are strictly aligned in the time step dimension based on the basic time axis established by the front end. Since the splicing is performed in the channel dimension, this operation does not require the channel sizes of the two sets of tensors to be strictly consistent, but it preserves their respective independent physical feature expression spaces, avoiding premature loss of information.

[0041] S230 performs cross-modal feature mapping based on a cross-attention mechanism. In the principle of multi-source information fusion, cross-attention can use the dominant signal of one modality as a guide to adaptively retrieve highly relevant supplementary information from other modalities. The fusion evaluation module 200 uses the enhanced speech features as the dominant query source and the auxiliary modal features as the key source to calculate a cross-attention matrix containing a scaling factor and a time-step alignment mask. This operation can adaptively retrieve relevant state cues from visual and physiological signals based on the semantic focus of the current speech content. The fusion evaluation module 200 maps the input features to the same latent space through the corresponding linear transformation matrix. The projection calculation formula is as follows: ; ; ; In the formula: To query the feature matrix; The key feature matrix; The characteristic matrix is ​​a value; , , These are the corresponding learnable linear mapping weight matrices.

[0042] After obtaining the above mapping matrix, the fusion evaluation module 200 calculates the cross-attention matrix. The calculation formula for the standard attention matrix is ​​as follows: ; In the formula: This is a cross-attention matrix; To query the inner product of the feature matrix and the transpose of the key feature matrix; is the feature dimension of the latent space. Since this dimension is fixed to be greater than 0 during the model design phase, its square root is used to numerically scale the dot product result to prevent gradient vanishing. This is a time step alignment mask matrix used to mask invalid time steps or prevent information leakage from future time steps; It is a normalized exponential function.

[0043] S240, Calculate the dynamic gating factor. To avoid low-quality physiological signals misleading subsequent state assessments, the fusion assessment module 200 directly uses the physiological signal quality index calculated in the previous step as an input parameter, and jointly concatenates it with the enhanced speech features and cross-attention matrix into the fully connected network layer to calculate the dynamic gating factor that determines the proportion of auxiliary modality information inflow. To unify the spatial dimension difference between the three-dimensional temporal matrix and the one-dimensional scalar index, in this embodiment, the system pre-performs global average pooling along the time series dimension of the feature matrix, flattens it into a one-dimensional feature vector, and then performs tensor concatenation. The specific formula for calculating the dynamic gating factor is as follows: ; In the formula: It is a dynamic gating factor, and its value ranges from 0 to 1; Use the Sigmoid activation function; This is the learnable weight matrix for this network layer; The enhanced speech features are a one-dimensional feature vector obtained by global average pooling; This is a one-dimensional feature vector obtained by global average pooling of the cross-attention matrix; The quality index of the physiological signal extracted at the current moment; This is a bias term. By introducing a physiological signal quality index, the dynamic gating factor gains the ability to perceive the signal-to-noise ratio of the underlying hardware signal, thus enabling an explicit assessment of feature reliability.

[0044] S250 utilizes dynamic gating factors to output a comprehensive state vector and establishes a smooth degradation mechanism. The fusion evaluation module 200 uses dynamic gating factors to perform weighted fusion of the cross-attention matrix and the enhanced speech features across feature dimensions, thereby outputting a comprehensive state vector. The calculation formula for the dynamic fusion mechanism is as follows: ; In the formula: This is the comprehensive state vector output at the current moment; This represents the Hadamard product operation of tensor elements. The fusion assessment module 200 maps and decodes the comprehensive state vector through the downstream classification layer network, clarifying that the comprehensive state vector semantically includes at least three dimensions: cognitive ability assessment, fatigue level, and frustration level. Specifically, the downstream classification layer adopts a multi-task feedforward neural network architecture with three independent regression heads, each outputting a continuous numerical index between 0 and 1, providing a direct quantitative basis for adjusting subsequent interaction strategies. In the model construction and training phase, a clinical rehabilitation dataset containing labeled data is extracted as the sample library. The assessment scores given by professional rehabilitation physicians according to the standard scale are normalized and used as the true labels. The mean squared error loss function is preferably used to complete the gradient backpropagation training of the network.

[0045] As a preferred embodiment, the fusion evaluation module 200 incorporates a physical fallback mechanism at its underlying layer. When the physiological signal is severely interfered with by the external environment or the user's violent movements, causing the extracted physiological signal quality index to fall below a preset safety threshold, this preset safety threshold is preferably set to 0.3. Since the neural network parameters are optimized under supervision during the training phase using a penalty loss function that includes low signal-to-noise ratio samples, the dynamic gating factor... It will be forced to approach 0. According to the dynamic fusion formula above, the cross-attention matrix at this time... The proportion of [something] was significantly suppressed, while the enhanced speech features [were more prominent]. The majority of weights are retained. In this way, the overall state vector smoothly degenerates into a state representation dominated by verbal features. This processing logic effectively isolates the negative impact of poor-quality multimodal data on the decision-making process and avoids the overall system collapse caused by the temporary failure of a single modality sensor.

[0046] See attached document Figure 2 Step S300 is specifically executed by the baseline decision module 300 in the system architecture, mainly implementing the physiological baseline state machine control law based on task intervals, as well as the reinforcement learning compound action mapping and fine-grained reward calculation logic. In this embodiment, this step is further divided into the following sub-steps for execution: In S310, the baseline sampling trigger window is locked, and the physiological steady-state zero-reaching condition is determined. In the rehabilitation interaction scenario, the human body experiences significant physiological stress during task execution, and the autonomic nervous system only gradually recovers to its baseline level during the rest period after the task ends. Based on this physical and physiological law, the baseline decision module 300 strictly locks the baseline sampling trigger window within the task gap window between two task interactions. The task gap refers to the rest period between the end of the previous interaction task and the start of the next interaction task. The baseline decision module 300 continuously calculates the first derivative of the heart rate variability feature sequence and determines the physiological steady-state zero-reaching condition in conjunction with the time window. The formula for determining the physiological steady-state zero-reaching condition is as follows: ; In the formula: Heart rate variability features extracted at the current moment; The preset derivative threshold is used to define the tolerance for fluctuations in physiological indicators. Its specific value range is preferably set to 0.01 to 0.05 to ensure that the heart rate variability is in a relatively stable state and is not affected by minor noise. The current moment; The length of the time window is defined as 30 to 60 seconds, which is preferably set according to the normal physiological recovery cycle. Satisfying this formula indicates that the physiological state tends to be stable within the time window, meeting the conditions for baseline sampling.

[0047] S320 executes dynamic baseline update and timeout blocking logic. When the physiological steady-state condition reaches zero is met, the baseline decision module 300 extracts the mean heart rate variability within the task interval window and updates the dynamic baseline using an exponential moving average mechanism. The specific formula for dynamic baseline update is as follows: ; In the formula: The updated dynamic baseline for the next time step; This serves as the dynamic baseline for the current moment. The smoothing coefficient is preferably set to 0.8 to 0.95 to control the weight of historical baselines and prevent baseline abrupt changes caused by a single abnormal sampling. This represents the mean characteristic of heart rate variability within the task gap window.

[0048] To prevent users from being under continuous high load, causing the system to wait indefinitely for baseline updates, the baseline decision module 300 introduces timeout blocking logic. If the zeroing condition is not met within the preset maximum waiting time (set to 120 seconds in this embodiment), a timeout mechanism is triggered, and the baseline decision module 300 calls a historical alternative baseline. When the timeout mechanism is triggered, in addition to calling the historical alternative baseline, the baseline decision module 300 also outputs a high-priority blocking signal to the action decision branch to suppress unnecessary task progression or trigger extended rest actions. The specific formula for the explicit state machine call is as follows: ; In the formula: This serves as the baseline for actual system calls; The historical baseline is the one that was most recently and successfully updated. This indicates that the above-mentioned physiological steady-state condition for approaching zero is met; This indicates a timeout trigger. This logic ensures that the system has a reliable reference baseline under any abnormal operating conditions.

[0049] S330 performs composite action mapping and prior rule constraints. The baseline decision module 300 inputs the comprehensive state vector output from the preceding steps into the policy network for nonlinear mapping. As a preferred approach, the policy network adopts an Actor-Critic architecture based on a multilayer perceptron, where the actor network branch receives the state vector and outputs a composite action space covering various intervention dimensions. This composite action space specifically consists of an enumeration set of seven action dimensions: difficulty adjustment, cue strategy, topic shift, semantic field, syntactic depth, target phoneme, and forced rest control. The mathematical expression of the composite action space is as follows: ; In the formula: This represents the initial composite action space output by the policy network at the current moment; each item inside the curly braces corresponds to a discrete action scalar in one of the seven specific dimensions mentioned above.

[0050] Based on this, the baseline decision module 300 combines verbal error classification labels to apply prior rules to complex actions and perform action screening to prevent the output of harmful actions that violate clinical rehabilitation guidelines. The specific formula for the prior rule constraints is as follows: ; In the formula: This is the set of valid actions after filtering. This is a priori rule mapping function. The function has a pre-set expert rule matrix for different verbal errors. By assigning a negative infinite logical penalty value to taboo actions, hard mask filtering is achieved. This is the initial composite action space output by the policy network; The speech error classification labels extracted prior to the perception decoupling module 100 are used for this function mapping. Through this function mapping, the baseline decision module 300 can block inappropriate actions from being issued.

[0051] S340 calculates a fine-grained reward mechanism and aggregates it to generate a composite reward. To guide the policy network to learn interaction strategies that conform to the principles of rehabilitation medicine, a precise reward feedback mechanism must be established. The baseline decision module 300 utilizes pure negative emotional features extracted from the visual modality, combined with the dynamic baseline offset calculated from the baseline actually invoked by the system, to calculate the frustration penalty term under pathological constraints. The core control equation for calculating the frustration penalty term is as follows: ; In the formula: For frustration and punishment items; The Sigmoid activation function non-linearly maps the result to the interval between 0 and 1, maintaining dimensional consistency. The baseline between current heart rate variability characteristics and actual system usage. The numerical difference between them reflects the degree of physiological stress; The emotional weighting coefficient is preferably set to 0.5; The pure negative emotional characteristics are quantified by the intensity of actions such as frowning recognized by the facial action unit.

[0052] Furthermore, the baseline decision module 300 constructs fine-grained sub-items of reinforcement learning composite rewards. In addition to frustration penalties, these fine-grained sub-items also include accuracy rewards, engagement rewards, and efficiency rewards. These three indicators constitute the positive incentives for measuring the quality of rehabilitation task completion, and the corresponding sub-item calculation formulas are as follows: ; ; ; In the formula: Rewards are given based on accuracy. The text matching score between the speech recognition text and the target corpus for the current task; The error severity coefficient ranges from 0 to 1. This coefficient is derived from the speech error classification labels received by the baseline decision module 300. The baseline decision module 300 is configured with a rehabilitation expert assessment matrix at its bottom layer. Based on the different types of obstacles represented by the unique hot coding (e.g., phoneme distortion is assigned 0.3, word finding difficulty is assigned 0.6, and semantic breakdown is assigned 0.9), the severity weight is automatically assigned by looking up a table. Rewards for participation; This is the scaling factor; A preset reference response time for a specific task; This refers to the actual response time for the user. Reward for efficiency; This is a Boolean indicator variable for task success, set to 1 if successful and 0 otherwise. and This is the task gain coefficient; It is a very small constant, preferably 10. -5 This is used to avoid computational overflow where the denominator of a division operation approaches zero due to extremely short actual response times.

[0053] Finally, the baseline decision module 300, combined with preset weighting coefficients, weights and summarizes each fine-grained sub-item to generate a scalar signal guiding the optimization of the policy network. The overall formula for the composite reward is as follows: ; In the formula: The composite reward for the current moment; Rewards are given based on accuracy. Rewards for participation; Reward for efficiency; , , , These are the corresponding preset weight coefficients. In this embodiment, to highlight the importance of rehabilitation accuracy and participation, the preset weight reference values ​​are set to 0.4, 0.3, 0.2, and 0.1, respectively. The output result of this general formula is determined based on multi-dimensional weighted logic, avoiding one-sided judgments that rely solely on a single interaction extreme value, thereby driving the system to output dynamic intervention actions that better reflect the user's actual state. During the system's model training phase, the strategy evolution module 400 stores the scalar composite reward calculated by multi-dimensional weighting into the experience replay pool, calculates the advantage function in conjunction with the proximal strategy optimization algorithm, and adopts a gradient ascent mechanism that maximizes the cumulative expected reward to achieve continuous iteration and optimization of the weight parameters in the strategy network.

[0054] See attached document Figure 2Step S400 is specifically executed by the policy evolution module 400 in the system architecture, mainly implementing the domain decoding rules of the large language model under the constraints of reinforcement learning parameters, as well as the experience replay and network update mechanism. In this embodiment, this step is further divided into the following sub-steps for execution: S410, Receive action parameters and construct a permitted vocabulary. Under conditions where the system does not trigger a forced rest signal, the strategy evolution module 400 receives the filtered set of valid actions output from the preceding steps and constructs a permitted vocabulary based on the action scalar parameters within it. To ensure that the generated interactive corpus matches the patient's current cognitive level and speech recovery stage, the constraints on the permitted vocabulary include at least target phonemes, semantic fields, word frequency thresholds, sentence length limits, syntactic depth, and topic shift constraints. Specifically, the strategy evolution module 400 limits the pronunciation retrieval range based on the target phonemes in the action set, locks highly context-related words through the semantic field, and filters extremely low-frequency, obscure words using the word frequency threshold. Simultaneously, the sentence length limit and syntactic depth are used to strictly limit the dependency syntax tree depth of the generated sentences, preventing the system from outputting complex long sentences that exceed the patient's current comprehension load.

[0055] In S420, the original probability distribution of candidate words in the large language model is reweighted and masked. As a preferred approach, the large language model used by the policy evolution module 400 is based on a standard Transformer decoder architecture, containing multiple layers of masked multi-head self-attention layers and a feedforward neural network. It receives historical interactive text as input and outputs the probability distribution for predicting the next word in the vocabulary dimension. After obtaining the allowed vocabulary, the policy evolution module 400 mathematically fuses the candidate word distribution output by the large language model during the decoding stage with specific physical constraints. In this control principle, the policy evolution module 400 does not directly modify the fixed network weights at the bottom layer of the large language model. Instead, during the inference output stage, it reweights the original probability distribution of candidate words by introducing a hard mask indicator function and a difficulty score function. The specific formula for reweighting control is as follows: ; In the formula: This represents the final probability distribution of candidate lexical units after reweighting. This indicates a direct proportional relationship, prompting the system to perform regular probability normalization on the result on the right side of the equation in subsequent calculations; The original probability distribution of candidate lexical units output by the large language model; Candidate words for the current decoding time step; For indicator functions, when candidate words Belongs to the allowed vocabulary When the condition is met, the value of the indicator function is 1; otherwise, it is 0, thus achieving the filtering and truncation of non-compliant words. The allowed vocabulary constructed for the preceding steps; This is an adjustment coefficient used to balance the weight between the basic fluency of the language model and the rehabilitation intervention goals; its value is preferably set to be between 0.1 and 0.5. is the difficulty score function for candidate lexical terms.

[0056] To achieve objective quantitative evaluation and avoid the difficulty score becoming a black-box parameter that is difficult to reproduce, in this embodiment, the strategy evolution module 400 pre-labels the objective lexical complexity of each word in the vocabulary using natural language processing tools, and defines the difficulty score function as the negative absolute difference between the lexical complexity and the difficulty adjustment parameter issued in the effective action set. The smaller the difference, the closer the word is to the difficulty index issued by the baseline decision module 300, and the greater the probability weight assigned after exponential operation. Furthermore, to prevent the vocabulary from having no overlap with the high-confidence candidate words of the current large language model, which would lead to the reweighted result of all words being 0 and thus triggering a computational dead zone where the denominator equals 0 during probability normalization, the strategy evolution module 400 sets up a safety fallback logic. When it is detected that the weighted sum of all words in the current time step is 0, the mask indicator function temporarily fails, and the strategy evolution module 400 rolls back to call the pre-set standardized safety recovery corpus as the interactive output of the current round. This reweighting mechanism allows the system to guide the generation distribution to conform to specific medical intervention settings without disrupting the underlying semantic space of the large language model.

[0057] S430, decoding sampling based on the final probability distribution is performed to generate the target interactive corpus. After distribution recombination is completed, the policy evolution module 400 performs lexical sampling based on the final probability distribution. As a preferred approach, the policy evolution module 400 uses kernel sampling technology to select lexical units from the truncated space of the final probability distribution, and then concatenates the sequences generated in consecutive time steps into complete text, ultimately generating the target interactive corpus output to the user. The cumulative probability ranking and boundary truncation mechanisms within kernel sampling can be implemented using standard natural language processing generation algorithms, which are well-known technologies in the field and will not be elaborated upon here.

[0058] S440 constructs transition tuples and performs network updates and visualization export. After completing a single round of human-computer dialogue, the policy evolution module 400 needs to structurally encapsulate the state transitions and behavioral intervention data during the interaction process to support the continuous evolution of the underlying policy. The policy evolution module 400 constructs transition tuples strictly according to the Markov decision process specifications, using the current comprehensive state vector, the filtered set of effective actions, the current composite reward, and the new state vector extracted for the next time step. The formula for constructing transition tuples is as follows: ; In the formula: This is the transition tuple corresponding to the current interaction time step; This is the current state vector; This is the set of valid actions selected from the preceding steps. The composite reward calculated at the current moment; This generates a new integrated state vector by capturing and mapping features for the system at the next time step.

[0059] The policy evolution module 400 continuously stores the generated transition tuples into the experience replay pool within the underlying storage space. When the total number of samples accumulated in the experience replay pool reaches a preset cumulative sample threshold, the policy evolution module 400 triggers the reinforcement learning update control law. To ensure the statistical stability of the network gradient backpropagation process and avoid gradient update fluctuations caused by uneven local sample distribution, the aforementioned cumulative sample threshold is preferably set to 1024 to 2048 tuple units. The policy evolution module 400 extracts data from the pool in batches and iteratively updates the connection weights in the aforementioned policy network using a proximal policy optimization algorithm. After completing the weight parameter replacement, the policy evolution module 400 extracts statistical features based on the output variance and action preferences of the policy network and exports a two-dimensional decision graph, which is then pushed to the backend interface for visualization. This process enables the self-correction and evolution of the system's interaction strategy and provides intuitive engineering data support for clinicians to review the historical intervention tendencies of the system.

[0060] The present invention also provides a computer device, including: a processor and a memory, the memory storing a computer program executable by the processor, the computer program performing the method described above when executed by the processor.

[0061] The present invention also provides a storage medium storing a computer program, which is executed by a processor to perform the method described above.

[0062] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0063] To further aid in understanding the technical solution of the present invention and its practical application effects, the following description is provided in conjunction with specific application embodiments and comparative experimental data.

[0064] This embodiment targets a stroke patient with hemiplegia, accompanied by right-sided facial central paralysis and moderate motor aphasia, characterized by phoneme distortion during speech and involuntary high-frequency muscle spasms on the affected side of the face during speech. The target user interacts with the rehabilitation training and adjustment system based on multimodal feature fusion provided by this invention via a terminal device equipped with a high-definition camera and microphone.

[0065] When the target user performs a reading task, the perception decoupling module 100 collects voice signals, facial video signals, and photoplethysmography waves estimated based on the facial video signals in real time.

[0066] See attached document Figure 6 The system comprises three curves: the dashed line represents the original affected side feature curve, showing high-frequency fluctuations caused by pathological spasms; the thin solid line represents the affected side feature curve after low-pass filtering, where high-frequency fluctuations are filtered out while low-frequency motion trends are retained; and the thick solid line represents the healthy side feature curve. Comparison reveals a temporal correlation between the filtered affected side feature curve and the healthy side feature curve. The perception decoupling module 100 calculates this correlation using the healthy side activation threshold, generating a visual representation stripped of pathological noise and incorporating motion intent.

[0067] When the target user completes the current reading task and enters the task gap window, the baseline decision module 300 continuously extracts and monitors physiological characteristics.

[0068] See attached document Figure 7This includes a heart rate variability curve (line graph) and a dynamic baseline (stepped horizontal line). During the task execution period (0 to 45 seconds in the figure), due to the target user's speech impairment, combined with the negative emotional features extracted from the visual modality by the perception decoupling module 100, the heart rate variability deviates from the dynamic baseline and decreases. Consequently, the representation values ​​of cognitive load and frustration dimensions in the comprehensive state vector output by the fusion evaluation module 200 increase.

[0069] During the task interval window (45 to 75 seconds in the figure), the baseline decision module 300 determines whether the condition of approaching zero is met based on the first derivative of the heart rate variability feature sequence, extracts the mean feature to update the dynamic baseline, and the step-like horizontal line in the figure shifts downward. This logic establishes a new reference benchmark, isolating the interference of long-term cumulative fatigue on the evaluation of the state of a single task.

[0070] Based on the comprehensive state vector containing cognitive load and frustration representations, the baseline decision module 300 maps and outputs a composite action space, which includes action parameters such as difficulty reduction, enabling lip-sync cues, and semantic field constraints.

[0071] The strategy evolution module 400 receives action parameters from the composite action space and constructs a restricted vocabulary. When generating the next round of interactive text, the strategy evolution module 400 reweights and masks the original probability distribution of candidate lexical units output by the large language model during the decoding phase. It applies masking truncation to complex sentence lexical units exceeding the upper limit of syntactic depth and word frequency threshold, and performs decoding sampling based on the final probability distribution to generate simplified interactive corpus that conforms to the target user's current cognitive and vocal load. This process ensures the coherence of the rehabilitation task logic and the compliance of the intervention actions.

[0072] To verify the application effect of the technical solution of the present invention, 60 stroke patients with speech and facial nerve disorders were recruited and randomly divided into three groups (20 people in each group) to carry out a comparative experiment for 8 weeks (5 times a week, 30 minutes each time).

[0073] The experimental group settings are as follows: Control group A: A single-modal system was used, with step-by-step difficulty adjustment based solely on speech recognition accuracy, without any visual or physiological feature input.

[0074] Control group B: An undecoupled multimodal system was used to receive multimodal signals, but the sensing link did not perform feature separation and low-pass filtering of the healthy and affected sides, and a fixed physiological baseline was used.

[0075] Experimental group: The rehabilitation training adjustment system based on multimodal feature fusion provided by this invention was used.

[0076] See attached document Figure 8 The chart shows a comparison of the mean values ​​of various core indicators for three groups after the 8-week experimental period. The specific analysis is as follows: In control group B, the accuracy rate of intent recognition was 74.2% due to interference from pathological spasms and physiological signal artifacts on the affected side. In the experimental group, the perception decoupling module 100 performed feature separation and low-pass filtering on the healthy and affected sides, and the fusion evaluation module 200 introduced a dynamic gating factor to perform smooth degradation. The accuracy rate of recognition of the target user's interaction intent reached 92.5%, which verified the filtering effect of the perception decoupling module 100 on heterogeneous pathological noise.

[0077] The average training session duration for control group A was 15.2 minutes, while for control group B it was 18.6 minutes. In the experimental group, the baseline decision module 300 calculated the frustration penalty term using pure negative emotion features and dynamic baseline offset, and the strategy evolution module 400 generated a reweighted interactive corpus based on composite rewards. The average training session duration for target users in the experimental group reached 28.4 minutes, improving the sustainability of human-computer interaction.

[0078] Double-blind assessments were conducted by professional rehabilitation physicians before and after the experiment. After 8 weeks of training, the average comprehensive score of control group A improved by 12.4%, and that of control group B by 16.1%. The average comprehensive score of the target users in the experimental group improved by 27.8%. The data indicate that the composite action space output by the baseline decision module 300 and the target interaction corpus generated by the strategy evolution module 400 have intervention effectiveness in the clinical rehabilitation dimension.

[0079] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A rehabilitation training adjustment method based on multimodal feature fusion, characterized in that, Includes the following steps: Acquire users' voice signals, facial video signals, and physiological signals to construct a synchronized multimodal data stream; Speech and physiological features are extracted from multimodal data streams. Speech signals are processed using acoustic coding models and language coding models to output speech error classification labels and calculate physiological signal quality index. The net displacement vector of facial motion units is obtained based on facial video signals. The net displacement vector of facial motion units is split into healthy side feature vector and affected side feature vector. The affected side feature vector is subjected to low-pass filtering. Based on the normalized correlation coefficient between the healthy side feature vector and the filtered affected side feature vector and the activation threshold of the healthy side, the corrected output of the affected side feature vector is generated. The visual representation is constructed by combining the corrected output of the affected side feature vector. The speech features, visual representations and physiological features are fused and processed, and a comprehensive state vector is output by combining the physiological signal quality index. Within the task gap window, the dynamic baseline is updated based on physiological characteristics, or a historical alternative baseline is invoked as the actual baseline if the update conditions are not met; The composite action space is output based on the comprehensive state vector mapping. The effective action set is obtained by filtering the composite action space with the speech error classification label. The composite reward is calculated based on the baseline of the actual call. An allowed vocabulary is constructed based on the set of valid actions. The original probability distribution of candidate word elements output by the large language model is reweighted to obtain the final probability distribution. The target interactive corpus is generated based on the final probability distribution. The integrated state vector, the set of effective actions, the composite reward, and the integrated state vector at the next moment are constructed into a transition tuple and stored in the experience replay pool to update the policy network weights.

2. The rehabilitation training adjustment method based on multimodal feature fusion according to claim 1, characterized in that, Constructing a synchronous multimodal data stream includes: Establish a basic time axis based on the sensor sampling rate; A sliding window mechanism is used to truncate the multi-source signals to be processed. By using a linear interpolation algorithm, truncated multi-source data is aligned to a unified temporal granularity to construct a synchronous multimodal data stream; Outputting speech error classification labels and calculating the physiological signal quality index includes: extracting acoustic features using an acoustic coding model, extracting high-dimensional speech features containing semantic structure information based on the acoustic features using a language coding model, and mapping and outputting speech error classification labels through a fully connected classification layer. The discrete power spectral density of physiological signals is calculated using the discrete Fourier transform algorithm. The physiological signal quality index is obtained by calculating the ratio of the signal power within the normal heart rate frequency band to the total power within the Nyquist frequency range.

3. The rehabilitation training adjustment method based on multimodal feature fusion according to claim 1, characterized in that, The net displacement vector of the facial motion unit is obtained based on the facial video signal, including: Extract the current three-dimensional absolute coordinates of the facial motion unit; Obtain the rigid body transformation matrix of the pre-calibrated reference anchor point; The net displacement vector of the facial motion unit is obtained by performing vector difference between the current three-dimensional absolute coordinates and the theoretical coordinates containing rigid body motion components. The construction of a visual representation by combining the corrected output of the feature vector from the affected side includes: Extracting micro-expression features and lip-shape consistency features from facial video signals; The corrected output of the feature vector from the affected side, the feature vector from the healthy side, the micro-expression features, and the lip-shape consistency features are concatenated to generate a visual representation.

4. The rehabilitation training adjustment method based on multimodal feature fusion according to claim 1, characterized in that, Based on the normalized correlation coefficient between the healthy side feature vector and the filtered affected side feature vector, and the activation threshold of the healthy side, a corrected output of the affected side feature vector is generated, including: Capture the historical sliding time window; Calculate the normalized correlation coefficient between the healthy side feature vector and the filtered affected side feature vector within the historical sliding time window; Calculate the Euclidean norm of the healthy side's eigenvectors; Determine whether the Euclidean norm is greater than the pre-calibrated healthy side activation threshold; When the Euclidean norm is greater than the activation threshold of the healthy side, the feature vector of the healthy side is multiplied by the normalized correlation coefficient to generate the corrected output of the feature vector of the affected side. When the Euclidean norm is not greater than the activation threshold of the healthy side, the zero vector is output as the corrected output of the feature vector of the affected side.

5. The rehabilitation training adjustment method based on multimodal feature fusion according to claim 1, characterized in that, The speech features, visual representations, and physiological features are fused and processed, and a comprehensive state vector is output by combining the physiological signal quality index, including: The attention weights of verbal features, visual representations, and physiological features were calculated using the self-attention mechanism to obtain the enhanced verbal features, enhanced visual representations, and enhanced physiological features. The enhanced visual representation and enhanced physiological features are concatenated along the feature channel dimension to construct auxiliary modal features; Using enhanced speech features as the primary query source and auxiliary modal features as the key source, cross-attention calculation is performed to obtain the cross-attention matrix; The physiological signal quality index, enhanced speech features, and cross-attention matrix are concatenated and input into a fully connected network layer to calculate the dynamic gating factor. The cross-attention matrix and enhanced speech features are weighted and fused using dynamic gating factors to output a comprehensive state vector.

6. The rehabilitation training adjustment method based on multimodal feature fusion according to claim 1, characterized in that, Within the task gap window, the dynamic baseline is updated based on physiological characteristics, or a historical alternative baseline is invoked as the actual baseline when the update conditions are not met, including: Lock the task gap window between two task interactions; Calculate the first derivative of the heart rate variability feature sequence contained in the physiological characteristics; When the first derivative is less than a preset derivative threshold, the physiological steady state approaching zero condition is determined to be met. When the physiological steady state approaches zero condition is met, the mean heart rate variability feature within the task interval window is extracted. The dynamic baseline for the next time step is updated by combining the current dynamic baseline with the mean characteristics of heart rate variability using an exponential moving average mechanism. When the physiological steady state approaching zero condition is not met and the preset timeout period is triggered, the historical alternative baseline is called as the actual baseline and a blocking signal is output.

7. The rehabilitation training adjustment method based on multimodal feature fusion according to claim 1, characterized in that, Based on the comprehensive state vector mapping, a composite action space is output. This space is then filtered using speech error classification labels to obtain a set of effective actions, including: The comprehensive state vector is input into the policy network and mapped to output a composite action space. The composite action space includes at least difficulty adjustment parameters, prompting strategy parameters, topic shifting parameters, semantic field parameters, syntactic depth parameters, target phoneme parameters, and rest control parameters. By using a prior rule mapping function containing an expert rule matrix and combining it with speech error classification labels, the composite action space is masked and filtered to generate a set of valid actions; The baseline calculation of composite rewards based on actual calls includes: Extract pure negative emotional features; Calculate the numerical offset between the current heart rate variability characteristics and the actual baseline called; The frustration penalty term is obtained by mapping the numerical offset to the pure negative emotional features. The frustration penalty is weighted and aggregated with the accuracy reward, engagement reward, and efficiency reward to generate a composite reward.

8. The rehabilitation training adjustment method based on multimodal feature fusion according to claim 1, characterized in that, The original probability distribution of candidate lexical units output by the large language model is reweighted to obtain the final probability distribution. Based on this final probability distribution, the target interactive corpus is generated, including: The set of valid actions includes difficulty adjustment parameters; Determine whether a candidate word belongs to the allowed vocabulary and generate the output value of the mask indicator function; The difficulty score function is obtained by calculating the negative absolute difference between the objective lexical complexity of the allowed words in the vocabulary and the difficulty adjustment parameter in the effective action set. The original probability distribution of candidate words is calculated and normalized by combining the output value of the mask indicator function and the difficulty score function to obtain the final probability distribution. When the weighted sum of all terms at the current time step is detected to be zero, the mask indicator function is disabled, and the preset safe recovery corpus is used as the target interactive corpus.

9. A rehabilitation training adjustment system based on multimodal feature fusion, applied to the method described in any one of claims 1-8, characterized in that, include: The perception decoupling module is used to acquire the user's voice signals, facial video signals, and physiological signals to construct a synchronized multimodal data stream; Speech and physiological features are extracted from multimodal data streams. Speech signals are processed using acoustic and language coding models to output speech error classification labels and calculate the physiological signal quality index. The net displacement vector of facial motion units is obtained based on facial video signals and is split into healthy and affected side feature vectors. The affected side feature vector is low-pass filtered. Based on the normalized correlation coefficient between the healthy and filtered affected side feature vectors and the healthy side activation threshold, a corrected output of the affected side feature vector is generated. A visual representation is constructed by combining the corrected output of the affected side feature vector. The fusion evaluation module is used to fuse speech features, visual representations and physiological features, and output a comprehensive state vector by combining the physiological signal quality index. The baseline decision module is used to update the dynamic baseline based on physiological characteristics within the task gap window, or to call the historical alternative baseline as the actual baseline when the update conditions are not met. The composite action space is output based on the comprehensive state vector mapping. The effective action set is obtained by filtering the composite action space with the speech error classification label. The composite reward is calculated based on the baseline of the actual call. The strategy evolution module is used to construct an allowed vocabulary based on the set of effective actions, reweight the original probability distribution of candidate lexical units output by the large language model to obtain the final probability distribution, and generate the target interactive corpus based on the final probability distribution. The integrated state vector, the set of effective actions, the composite reward, and the integrated state vector at the next moment are constructed into a transition tuple and stored in the experience replay pool to update the policy network weights.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images