Multimodal gesture interaction control method and system for postoperative care of neurointerventional procedures
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-29
- Publication Date
- 2026-08-14
AI Technical Summary
[0004]然而，上述现有技术在神经介入术后护理的临床应用中，存在如下核心技术问题：现有手势交互控制技术未构建适配病房复杂环境与患者术后生理特征的多模态手势交互全流程闭环技术体系，存在单模态识别鲁棒性不足、多模态融合机制与临床场景适配性差的核心缺陷，导致复杂临床场景下手势识别的准确率与稳定性不足，无法实现精准、可靠的护理指令交互控制
[0047]根据本发明公开的神经介入术后护理用多模态手势交互控制方法及系统，能够解决单模态手势识别鲁棒性不足的缺陷，大幅提升复杂临床场景下识别的准确率与稳定性，实现精准可靠的术后护理手势交互控制。
Smart Images
Figure CN122569746A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of smart medical care and human-computer interaction technology, specifically to a multimodal gesture interaction control method and system for postoperative care after neurointervention. Background Technology
[0002] Neurointerventional surgery is a mainstream minimally invasive treatment for central nervous system vascular diseases such as intracranial aneurysms and ischemic stroke. Postoperatively, patients require strict bed rest and immobilization, and often experience speech and limb movement difficulties due to the surgery or underlying diseases, making it impossible to communicate nursing needs through traditional contactless call devices or touch terminals. Non-contact gesture interaction, as a low-load interaction method that conforms to natural human behavior, is a core technological path to address the essential interaction needs in this clinical scenario. Multimodal signal fusion gesture recognition technology is a key research direction in this field.
[0003] Existing gesture interaction control technologies for post-neurointerventional nursing scenarios mainly fall into two mainstream categories: The first category is a single visual modality gesture recognition scheme. This scheme acquires image sequences of the patient's hand through image acquisition equipment, performs image segmentation to locate and extract the hand region, and then extracts core features such as hand contours and key points. It completes gesture semantic recognition through a classification model and finally maps it to nursing control commands. This scheme is easy to deploy and is currently the most widely used gesture interaction scheme in clinical nursing scenarios. The second category is a simple multimodal fusion gesture recognition scheme. This scheme simultaneously acquires multiple signals related to limb movement on the basis of visual signals. After performing basic preprocessing and feature extraction on each single modality signal, it completes multimodal feature fusion by directly concatenating feature vectors and averaging them with fixed weights. Then, it performs gesture classification and command mapping based on the fused features to improve the recognition effect.
[0004] However, the existing technologies mentioned above have the following core technical problems in the clinical application of postoperative care after neurointervention: the existing gesture interaction control technology has not built a multimodal gesture interaction closed-loop technology system that is adapted to the complex ward environment and the postoperative physiological characteristics of patients. It has core defects such as insufficient robustness of single-modal recognition and poor adaptability of multimodal fusion mechanism to clinical scenarios. As a result, the accuracy and stability of gesture recognition in complex clinical scenarios are insufficient, and it is impossible to achieve precise and reliable nursing instruction interaction control. Summary of the Invention
[0005] The present invention aims to at least partially solve the technical problems in the above-mentioned technologies.
[0006] Therefore, this invention discloses a multimodal gesture interaction control method for postoperative care after neurointervention, comprising:
[0007] S1: Synchronously acquire multimodal raw signals, mark each single-modal raw signal in the multimodal raw signals with a globally unified timestamp and perform time axis interpolation alignment; preprocess each single-modal raw signal to generate a corresponding valid signal; perform quality assessment on the valid signal to generate a corresponding prior reliability weight;
[0008] S2: Extract high-level semantic features from the effective signals in independent feature spaces to generate corresponding single-modal high-level feature vectors of the same dimension;
[0009] S3: Based on the prior reliability weights, perform prior weighted embedding on the single-modal high-level feature vectors to generate multimodal features; model the cross-modal attention mechanism between the multimodal features to generate enhanced multimodal features; based on the prior reliability weights and the multimodal features, calculate the dynamic fusion weights of each of the single-modal high-level feature vectors; based on the dynamic fusion weights, perform weighted aggregation on the multimodal features to generate a global fusion feature vector;
[0010] S4: Based on the global fusion feature vector, perform binary classification of active intention gestures and postoperative unconscious movements, remove the global fusion feature vectors corresponding to postoperative unconscious movements, and retain the global fusion feature vectors corresponding to active intention gestures;
[0011] S5: Perform semantic classification on the global fusion feature vector determined to be an active intention gesture to generate a preliminary classification result, perform confidence judgment and ambiguity resolution on the preliminary classification result, and map the preliminary classification result into standardized nursing instructions.
[0012] In addition, the multimodal gesture interaction control method for postoperative care of neurointerventional procedures disclosed in this invention may also have the following additional technical features:
[0013] Furthermore, the synchronous acquisition of the multimodal raw signal in step S1 includes, but is not limited to:
[0014] Depth map sequences acquired by a ToF camera, multi-channel electromyography signals acquired by an sEMG array, and attitude angle data acquired by an IMU sensor.
[0015] Furthermore, the time axis interpolation alignment in step S1 specifically involves:
[0016] Using the single-mode original signal with the largest sampling rate among the multimodal original signals as the reference time axis, the remaining single-mode original signals are time axis interpolated and aligned using an interpolation algorithm.
[0017] Further, in step S1, the preprocessing of each of the single-mode raw signals to generate the corresponding effective signal specifically involves:
[0018] The depth map sequence is segmented by thresholding to filter out the depth range corresponding to the hand region, and then small noise points are removed and holes are filled.
[0019] The power frequency interference, ECG interference, and motion artifacts of the multi-channel electromyography signals are filtered sequentially.
[0020] The attitude angle data is filtered by moving average.
[0021] Further, in step S1, the effective signal is subjected to quality evaluation to generate the corresponding prior reliability weight, specifically as follows:
[0022] Calculate the integrity index of the depth map sequence according to the formula. Generate the corresponding prior reliability weights;
[0023] Calculate the signal-to-noise ratio of the multi-channel electromyography signal according to the formula. Generate the corresponding prior reliability weights;
[0024] Calculate the variance of the attitude angle fluctuation according to the formula. Generate the corresponding prior reliability weights.
[0025] Further, step S2 specifically includes:
[0026] S2.1: Perform preliminary spatial feature extraction and dynamic temporal feature encoding on the depth map sequence, and generate the corresponding single-modal high-level feature vector after dimensionality reduction and normalization.
[0027] S2.2: Perform multi-scale local feature extraction and channel temporal dual attention encoding on the multi-channel electromyography signal. After feature aggregation, dimensionality reduction and normalization, generate the corresponding single-modality high-level feature vector.
[0028] S2.3: Perform kinematic feature enhancement and temporal motion pattern encoding on the attitude angle data, and after dimensionality reduction and normalization, generate the corresponding single-modal high-level feature vector.
[0029] Furthermore, in step S3, modeling the cross-modal attention mechanism between the multimodal features to capture the complementary correlation between modalities specifically involves:
[0030] S3.1.1: Normalize the prior reliability weights, and then weight the normalized prior reliability weights and the corresponding single-modal high-level feature vectors element by element to generate the multimodal features;
[0031] S3.1.2: Convert the multimodal features into a modal sequence;
[0032] S3.1.3: Map the modal sequences into query matrices, key matrices, and value matrices, respectively;
[0033] S3.1.4: The query matrix, the key matrix, and the value matrix are each split into multiple attention heads;
[0034] S3.1.5: For each attention head, calculate the attention score and weight the value matrix;
[0035] S3.1.6: The outputs of all the attention heads are concatenated and processed through a linear layer to generate the enhanced multimodal features.
[0036] Further, in step S3, based on the prior reliability weights and the complementary correlation, the dynamic fusion weights of each of the single-modal high-level feature vectors are calculated, specifically as follows:
[0037] S3.2.1: After averaging the attention score matrices of all the attention heads, average each column of the attention score matrix to obtain the dynamic importance vector of each modality in the multimodal features;
[0038] S3.2.2: Multiply the normalized prior reliability weights and the dynamic importance scores element by element to generate intermediate weights;
[0039] S3.2.3: Perform softmax normalization on the intermediate weights to generate the final dynamic fusion weights.
[0040] Further, step S5 specifically includes:
[0041] S5.1: Input the global fusion feature vector into the multi-classifier and output the preliminary classification result and the corresponding probability distribution;
[0042] S5.2: Set a confidence threshold. If the confidence of the preliminary classification result reaches the confidence threshold, the classification result is directly confirmed. If the confidence threshold is not reached, the ambiguity is resolved by combining context information, and the classification result is re-output. If the confidence threshold is still not reached after ambiguity resolution, a recognition failure message is displayed.
[0043] S5.3: The final classification result after ambiguity resolution is mapped into standardized nursing instructions through a semantic instruction mapping function.
[0044] Furthermore, another aspect of the present invention discloses a multimodal gesture interaction control system for postoperative care after neurointerventional surgery, comprising:
[0045] Memory, used to store computer programs;
[0046] A processor, configured to implement, when executing the computer program, the multimodal gesture interaction control method for postoperative care of neurointerventional procedures as described in any one of claims 1 to 9.
[0047] The multimodal gesture interaction control method and system for postoperative care of neurointerventional procedures disclosed in this invention can overcome the shortcomings of insufficient robustness of single-modal gesture recognition, significantly improve the accuracy and stability of recognition in complex clinical scenarios, and achieve precise and reliable postoperative care gesture interaction control.
[0048] Additional features and advantages of this invention will be set forth in the description which follows, or may be learned by practicing the invention. Attached Figure Description
[0049] The technical solution and beneficial effects of the present invention will become apparent and readily understood from the following description in conjunction with the accompanying drawings, wherein:
[0050] Figure 1 This is a flowchart of a multimodal gesture interaction control method for postoperative care of neurointerventional procedures according to the present invention.
[0051] Figure 2 This is another flowchart of the multimodal gesture interaction control method for postoperative care of neurointerventional procedures according to the present invention;
[0052] Figure 3 This is another flowchart of the multimodal gesture interaction control method for postoperative care of neurointerventional procedures according to the present invention. Detailed Implementation
[0053] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0054] The multimodal gesture interaction control method and system for postoperative care of neurointerventional procedures disclosed in this invention will now be described with reference to the accompanying drawings.
[0055] like Figure 1 , Figure 2 and Figure 3 As shown, a multimodal gesture interaction control method for postoperative care after neurointerventional surgery includes:
[0056] S1: Synchronously acquire multimodal raw signals, mark each single-mode raw signal in the multimodal raw signals with a globally unified timestamp and perform time axis interpolation and alignment; preprocess each single-mode raw signal to generate the corresponding effective signal; perform quality assessment on the effective signal to generate the corresponding prior reliability weight;
[0057] S2: Extract high-level semantic features from the effective signals in independent feature spaces to generate corresponding single-modal high-level feature vectors of the same dimension;
[0058] S3: Based on prior reliability weights, perform prior weighted embedding of single-modal high-level feature vectors to generate multimodal features; model cross-modal attention mechanisms between multimodal features to generate enhanced multimodal features; calculate dynamic fusion weights for each single-modal high-level feature vector based on prior reliability weights and multimodal features; perform weighted aggregation of multimodal features based on dynamic fusion weights to generate a global fusion feature vector;
[0059] S4: Based on the global fusion feature vector, perform binary classification of active intention gestures and postoperative unconscious movements, remove the global fusion feature vectors corresponding to postoperative unconscious movements, and retain the global fusion feature vectors corresponding to active intention gestures.
[0060] S5: Perform semantic classification on the global fusion feature vectors that are determined to be active intention gestures to generate preliminary classification results. Perform confidence judgment and ambiguity resolution on the preliminary classification results, and map the preliminary classification results into standardized nursing instructions.
[0061] The synchronous acquisition of multimodal raw signals in step S1 includes, but is not limited to:
[0062] Depth map sequences acquired by a ToF camera, multi-channel electromyography signals acquired by an sEMG array, and attitude angle data acquired by an IMU sensor.
[0063] Example 1
[0064] The time axis interpolation alignment in step S1 is specifically as follows:
[0065] Using the single-mode original signal with the highest sampling rate in the multimodal original signal as the reference time axis, the remaining single-mode original signals are time axis interpolated and aligned using an interpolation algorithm.
[0066] In step S1, the raw signals of each single mode are preprocessed to generate the corresponding effective signals, specifically as follows:
[0067] Threshold segmentation is performed on the depth map sequence to filter out the depth range corresponding to the hand area, and then small noise points are removed and holes are filled.
[0068] Sequentially filter out power frequency interference, ECG interference, and motion artifacts from multi-channel electromyography signals;
[0069] Apply a moving average filter to the attitude angle data.
[0070] In step S1, the quality of the valid signal is evaluated to generate the corresponding prior reliability weights, specifically as follows:
[0071] Calculate the integrity index of the depth map sequence according to the formula. Generate the corresponding prior reliability weights;
[0072] Calculate the signal-to-noise ratio of multichannel electromyography signals according to the formula. Generate the corresponding prior reliability weights;
[0073] Calculate the variance of attitude angle fluctuations using the formula. Generate the corresponding prior reliability weights.
[0074] Specifically:
[0075] In the synchronous acquisition, a ToF camera was used to acquire depth map sequences at a frame rate of 30fps with a resolution of 640×576. An sEMG array was used to acquire 8-channel electromyography signals at a sampling rate of 2000Hz. An IMU sensor was used to acquire three-axis attitude angle data at a sampling rate of 100Hz. All sensors were triggered to acquire data through the same motherboard clock synchronization signal of the industrial control computer, and a globally unified timestamp was marked for each frame of data.
[0076] Due to the different sampling rates of the various sensors, time axis interpolation and alignment are required. Using the time axis of the sEMG signal with the highest sampling rate as the reference, cubic spline interpolation is applied to the ToF depth map and IMU attitude angle data respectively, interpolating them to each time point of the sEMG to ensure strict temporal alignment of the three modal data. After interpolation, each time point contains synchronized depth map, electromyography signal, and attitude angle.
[0077] In the preprocessing stage, for the depth map sequence, the hand region is first extracted by threshold segmentation. The depth values are then filtered according to the preset range of the distance between the patient's hand and the camera when the patient is lying down (e.g., 0.3m~0.8m) to generate a binary mask. Then, morphological opening operation is used to remove isolated noise points. Next, a hole filling algorithm is used to repair the holes in the hand region caused by reflections, etc. Finally, the processed depth map is converted into a hand depth value matrix.
[0078] For multi-channel sEMG signals, a 50Hz notch filter was used to filter out power frequency interference, an adaptive template matching method was used to remove ECG interference (because postoperative patients may have ECG monitoring), and a high-pass filter (cutoff frequency 20Hz) and a high-order cumulant method were used to suppress motion artifacts to obtain a pure electromyography signal.
[0079] For IMU attitude angle data (pitch angle, roll angle, yaw angle), a moving average filter with a sliding window length of 5 is used for smoothing to reduce high-frequency jitter.
[0080] For quality assessment and prior reliability weight generation, the depth map integrity index Defined as the proportion of effective hand pixels to the total number of pixels in the preset hand area, calculated using the following formula: ,in The inherent confidence level of the sensor is set to 0.95. An adjustment factor (set to 1.2) is used to enhance the weighting of high integrity; sEMG signal-to-noise ratio. The weights are obtained by calculating the ratio of signal energy to noise energy, with the noise segment using resting-state data. ,in Maximum signal-to-noise ratio in historical data (default 40dB); attitude angle fluctuation variance Calculate the weights within a sliding window (window length 1 second). ,in Choosing 5 ensures that the larger the variance in volatility, the faster the weight decays.
[0081] The remaining technical details of this embodiment are described above and will not be repeated here.
[0082] Example 2
[0083] Step S2, specifically:
[0084] S2.1: Perform preliminary spatial feature extraction and dynamic temporal feature encoding on the depth map sequence. After dimensionality reduction and normalization, generate the corresponding single-modal high-level feature vector.
[0085] S2.2: Multi-scale local feature extraction and channel temporal dual attention encoding are performed on multi-channel electromyography signals. After feature aggregation, dimensionality reduction and normalization, the corresponding single-modality high-level feature vector is generated.
[0086] S2.3: Perform kinematic feature enhancement and temporal motion pattern encoding on the attitude angle data. After dimensionality reduction and normalization, generate the corresponding single-modal high-level feature vector.
[0087] Specifically:
[0088] The preprocessed depth map sequence (16 consecutive frames as a sample) is input into a lightweight 3D convolutional neural network. First, spatial and spatiotemporal features are extracted through a 3D convolutional layer, resulting in a feature map of dimension [T, C, H, W] (T is the temporal length, C is the number of channels). Then, a temporal attention module is applied, calculating attention weights along the time axis for each spatial location's features, and summing these weights to obtain the temporal dynamic encoding. Finally, a global average pooling layer compresses the feature map into a 512-dimensional vector, which is then reduced to 256 dimensions through a fully connected layer and L2 normalized to obtain the high-level feature vector of the depth modality. .
[0089] Multi-scale feature extraction was performed on the preprocessed sEMG signal (sampling window length 1 second, i.e., 2000 sampling points, 8 channels). First, local temporal features were extracted using three different sliding windows (50ms, 100ms, and 200ms) to form a multi-scale local feature sequence. This sequence was then input into a channel-based temporal dual-attention network. The importance weights of each channel were first learned and weighted using a channel attention module. Then, a temporal convolutional network was used to capture long-term dependencies, and a multi-head self-attention layer was added to weight the temporal steps. Finally, the obtained feature vectors were globally pooled and then reduced to 256 dimensions through a fully connected layer, followed by L2 normalization to obtain the high-level feature vectors of the sEMG modality. .
[0090] Kinematic feature enhancement was performed on the preprocessed IMU attitude angle data (three-axis angles, 1 second window length, 100 sampling points). The first-order difference (angular velocity) and second-order difference (angular acceleration) were calculated and concatenated with the original angles to form 9-dimensional temporal data. This data was then input into a bidirectional LSTM network with a hidden layer dimension of 128. The output at the last time step was used as the temporal motion mode encoding, which was then reduced to 256 dimensions through a fully connected layer and L2 normalized to obtain the high-level feature vectors of the IMU modes. .
[0091] The remaining technical details of this embodiment are described above and will not be repeated here.
[0092] Example 3
[0093] In step S3, the cross-modal attention mechanism is modeled for the multimodal features to capture the complementary relationships between modalities. Specifically:
[0094] S3.1.1: Normalize the prior reliability weights, and then weight the normalized prior reliability weights and the corresponding single-modal high-level feature vectors element by element to generate multimodal features;
[0095] S3.1.2: Convert multimodal features into modal sequences;
[0096] S3.1.3: Map the modal sequences to query matrices, key matrices, and value matrices respectively;
[0097] S3.1.4: Split the query matrix, key matrix, and value matrix into multiple attention heads;
[0098] S3.1.5: For each attention head, calculate the attention score and add a weighted matrix;
[0099] S3.1.6: The outputs of all attention heads are concatenated and processed through a linear layer to generate enhanced multimodal features.
[0100] In step S3, based on prior reliability weights and complementary correlations, the dynamic fusion weights of each single-modal high-level feature vector are calculated, specifically as follows:
[0101] S3.2.1: After averaging the attention score matrices of all attention heads, average each column of the attention score matrix to obtain the dynamic importance vector of each modality in the multimodal features;
[0102] S3.2.2: Multiply the normalized prior reliability weights and dynamic importance scores element by element to generate intermediate weights;
[0103] S3.2.3: Perform softmax normalization on the intermediate weights to generate the final dynamic fusion weights.
[0104] Specifically:
[0105] First, the prior reliability weights obtained in step S1 are... Perform softmax normalization to obtain normalized weights. To make the sum of the three equal to 1, the normalized weights are then multiplied element-wise by the corresponding single-modal high-level feature vectors to obtain the weighted multimodal feature vector set: These vectors form a modal sequence .
[0106] modal sequences The input to the cross-modal multi-head attention module is first processed through three linear layers. The matrix is mapped to a query matrix Q, a key matrix K, and a value matrix V, each with a dimension of 3×256. Four attention heads are set, and Q, K, and V are each split into four heads, each with a dimension of 3×64. The outputs of all heads are concatenated and then reprojected to 3×256 through a linear layer to obtain the enhanced multimodal features. .
[0107] Dynamic importance is calculated based on the attention score matrix. The attention score matrices of all heads are averaged to obtain the average attention matrix (representing mutual attention between modalities). Then, the average of each column of the average attention matrix (corresponding to the modality being attended to) is taken to obtain the dynamic importance score of each modality. The normalized prior weights are multiplied element-wise with the dynamic importance scores to obtain the intermediate weights. Finally, the intermediate weights are normalized to obtain the final dynamic fusion weights.
[0108] Based on the dynamic fusion weights, the enhanced multimodal features are weighted and summed to obtain the global fusion feature vector, which integrates complementary information between modalities and is adaptively adjusted by the dynamic weights.
[0109] The remaining technical details of this embodiment are described above and will not be repeated here.
[0110] Example 4
[0111] Step S5 is as follows:
[0112] S5.1: Input the global fused feature vector into the multi-classifier and output the preliminary classification results and corresponding probability distribution;
[0113] S5.2: Set a confidence threshold. If the confidence of the preliminary classification result reaches the confidence threshold, the classification result is directly confirmed. If the confidence threshold is not reached, the ambiguity is resolved by combining the context information, and the classification result is re-output. If the confidence threshold is still not reached after ambiguity resolution, the recognition failure is indicated.
[0114] S5.3: The final classification result after ambiguity resolution is mapped into standardized nursing instructions through a semantic instruction mapping function.
[0115] Specifically:
[0116] The global fusion feature vectors corresponding to the active intention gestures retained after filtering in step S4 are input into a multi-classifier (a three-layer fully connected network with 512, 256, and 128 hidden layer nodes, and the number of output layer nodes equal to the preset number of gesture categories N). The classifier outputs the probability distribution of each category, and the category with the highest probability is taken as the preliminary classification result, and its confidence level is the maximum probability value.
[0117] If the confidence level is greater than or equal to 0.85, the classification result is directly confirmed as the preliminary classification result; if the confidence level is less than 0.85, the ambiguity resolution process begins.
[0118] Maintain a sliding window of length 5 to record the most recent 5 classification results and their confidence scores;
[0119] If the current classification result matches the most frequent category in the window, and the average confidence of that category in the window is higher than another threshold (0.6), then that category is taken as the result after resolution.
[0120] Otherwise, if the current classification result is the same as the most recent result in the window, it will also be retained.
[0121] If the threshold still cannot be reached after the above elimination process, the system will prompt an identification failure and record the event.
[0122] Finally, the final classification results are mapped into standardized nursing instructions through a predefined semantic instruction mapping function.
[0123] The remaining technical details of this embodiment are described above and will not be repeated here.
[0124] A multimodal gesture interaction control system for postoperative care after neurointerventional surgery includes:
[0125] Memory, used to store computer programs;
[0126] A processor for implementing a multimodal gesture interaction control method for postoperative care of neurointerventional procedures while executing computer programs.
[0127] In summary, the multimodal gesture interaction control method and system for postoperative care of neurointerventional procedures disclosed in this invention can overcome the shortcomings of insufficient robustness in single-modal gesture recognition, significantly improve the accuracy and stability of recognition in complex clinical scenarios, and achieve precise and reliable postoperative care gesture interaction control.
[0128] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A multimodal gesture interaction control method for postoperative care after neurointerventional surgery, characterized in that, include: S1: Synchronously acquire multimodal raw signals, mark each single-mode raw signal in the multimodal raw signals with a globally unified timestamp and perform time axis interpolation and alignment; Each of the original single-mode signals is preprocessed to generate a corresponding effective signal; the effective signal is then evaluated for quality to generate a corresponding prior reliability weight. S2: Extract high-level semantic features from the effective signals in independent feature spaces to generate corresponding single-modal high-level feature vectors of the same dimension; S3: Based on the prior reliability weights, perform prior weighted embedding on the single-modal high-level feature vector to generate multimodal features; A cross-modal attention mechanism is modeled for the multimodal features to generate enhanced multimodal features; Based on the prior reliability weights and the multimodal features, the dynamic fusion weights of each of the single-modal high-level feature vectors are calculated; the multimodal features are weighted and aggregated based on the dynamic fusion weights to generate a global fusion feature vector. S4: Based on the global fusion feature vector, perform binary classification of active intention gestures and postoperative unconscious movements, remove the global fusion feature vectors corresponding to postoperative unconscious movements, and retain the global fusion feature vectors corresponding to active intention gestures; S5: Perform semantic classification on the global fusion feature vector determined to be an active intention gesture to generate a preliminary classification result, perform confidence judgment and ambiguity resolution on the preliminary classification result, and map the preliminary classification result into standardized nursing instructions.
2. The multimodal gesture interaction control method for postoperative care of neurointerventional procedures as described in claim 1, characterized in that, The synchronous acquisition of the multimodal raw signal in step S1 includes, but is not limited to: Depth map sequences acquired by a ToF camera, multi-channel electromyography signals acquired by an sEMG array, and attitude angle data acquired by an IMU sensor.
3. The multimodal gesture interaction control method for postoperative care of neurointerventional procedures as described in claim 1, characterized in that, The time axis interpolation alignment in step S1 specifically includes: Using the single-mode original signal with the largest sampling rate among the multimodal original signals as the reference time axis, the remaining single-mode original signals are time axis interpolated and aligned using an interpolation algorithm.
4. The multimodal gesture interaction control method for postoperative care of neurointerventional procedures as described in claim 2, characterized in that, In step S1, the preprocessing of each of the single-mode raw signals to generate the corresponding effective signal specifically involves: The depth map sequence is segmented by thresholding to filter out the depth range corresponding to the hand region, and then small noise points are removed and holes are filled. The power frequency interference, ECG interference, and motion artifacts of the multi-channel electromyography signals are filtered sequentially. The attitude angle data is filtered by moving average.
5. The multimodal gesture interaction control method for postoperative care of neurointerventional procedures as described in claim 4, characterized in that, In step S1, the effective signal is evaluated for quality to generate the corresponding prior reliability weight, specifically as follows: Calculate the integrity index of the depth map sequence according to the formula. Generate the corresponding prior reliability weights; Calculate the signal-to-noise ratio of the multi-channel electromyography signal according to the formula. Generate the corresponding prior reliability weights; Calculate the variance of the attitude angle fluctuation according to the formula. Generate the corresponding prior reliability weights.
6. The multimodal gesture interaction control method for postoperative care of neurointerventional procedures as described in claim 4, characterized in that, Step S2 specifically includes: S2.1: Perform preliminary spatial feature extraction and dynamic temporal feature encoding on the depth map sequence, and generate the corresponding single-modal high-level feature vector after dimensionality reduction and normalization. S2.2: Perform multi-scale local feature extraction and channel temporal dual attention encoding on the multi-channel electromyography signal. After feature aggregation, dimensionality reduction and normalization, generate the corresponding single-modality high-level feature vector. S2.3: Perform kinematic feature enhancement and temporal motion pattern encoding on the attitude angle data, and after dimensionality reduction and normalization, generate the corresponding single-modal high-level feature vector.
7. The multimodal gesture interaction control method for postoperative care of neurointerventional procedures as described in claim 4, characterized in that, In step S3, modeling the cross-modal attention mechanism between the multimodal features to capture the complementary correlations between modalities specifically involves: S3.1.1: Normalize the prior reliability weights, and then weight the normalized prior reliability weights and the corresponding single-modal high-level feature vectors element by element to generate the multimodal features; S3.1.2: Convert the multimodal features into a modal sequence; S3.1.3: Map the modal sequences into query matrices, key matrices, and value matrices, respectively; S3.1.4: The query matrix, the key matrix, and the value matrix are each split into multiple attention heads; S3.1.5: For each attention head, calculate the attention score and weight the value matrix; S3.1.6: The outputs of all the attention heads are concatenated and processed through a linear layer to generate the enhanced multimodal features.
8. The multimodal gesture interaction control method for postoperative care of neurointerventional procedures as described in claim 7, characterized in that, In step S3, based on the prior reliability weights and the complementary correlation, the dynamic fusion weights of each of the single-modal high-level feature vectors are calculated, specifically as follows: S3.2.1: After averaging the attention score matrices of all the attention heads, average each column of the attention score matrix to obtain the dynamic importance vector of each modality in the multimodal features; S3.2.2: Multiply the normalized prior reliability weights and the dynamic importance scores element by element to generate intermediate weights; S3.2.3: Perform softmax normalization on the intermediate weights to generate the final dynamic fusion weights.
9. The multimodal gesture interaction control method for postoperative care of neurointerventional procedures as described in claim 1, characterized in that, Step S5 specifically includes: S5.1: Input the global fusion feature vector into the multi-classifier and output the preliminary classification result and the corresponding probability distribution; S5.2: Set a confidence threshold. If the confidence of the preliminary classification result reaches the confidence threshold, the classification result is directly confirmed. If the confidence threshold is not reached, the ambiguity is resolved by combining contextual information, and the classification result is re-output. If the confidence threshold is still not reached after ambiguity resolution, an identification failure message will be displayed. S5.3: The final classification result after ambiguity resolution is mapped into standardized nursing instructions through a semantic instruction mapping function.
10. A multimodal gesture interaction control system for postoperative care after neurointerventional surgery, characterized in that, include: Memory, used to store computer programs; A processor is configured to implement, when executing the computer program, the multimodal gesture interaction control method for postoperative care of neurointerventional procedures as described in any one of claims 1 to 9.