Multi-modal intelligent decision and device control method and system for intensive care unit critical recovery

Abstract

The application discloses an ICU critical recovery multi-modal intelligent decision and device control method and system, and relates to the technical field of medical rehabilitation; the method comprises the following steps: acquiring static medical records and multi-modal data; standardizing the multi-modal data and generating a missing mask, and then inputting the multi-modal fusion network to obtain fusion features; identifying the disease stage and risk assessment based on the fusion features, generating a rehabilitation path and determining an uncertainty quantization value; performing safety verification according to the static medical record data and the uncertainty quantization value, and triggering manual review if the safety verification fails; based on the rehabilitation path after the safety verification passes or the manual review, generating a device control instruction and issuing the device control instruction for execution, simultaneously collecting device operation state data and patient physiological feedback data; dynamically adjusting the rehabilitation path according to the operation state data and the patient physiological feedback data, and updating the multi-modal fusion network parameters by using incremental learning and federated learning. The application can realize intelligent closed-loop management of the whole process of critical recovery diagnosis and treatment.

Description

Technical Field

[0001] This invention relates to the field of medical rehabilitation technology, and in particular to a multimodal intelligent decision-making and equipment control method and system for ICU critical care rehabilitation. Background Technology

[0002] Existing technologies for ICU rehabilitation systems and products exhibit significant deficiencies in information collection and fusion. Medical systems (HIS / EMR) and rehabilitation systems are disconnected, data collection standards are inconsistent, and multimodal data suffers from problems such as large differences in sampling rates, high random missing data rates, and strong noise interference. Traditional splicing or weighted averaging fusion methods struggle to effectively align heterogeneous data and are highly sensitive to data gaps. Regarding rehabilitation decision-making, existing solutions rely excessively on physicians' personal experience, lack standardized rehabilitation pathways, and multimodal fusion algorithms specific to the ICU scenario. They cannot accurately identify disease stages by combining physiological signals, neurological function scores, and other multidimensional data, and rehabilitation pathways lack dynamic adjustment capabilities. At the equipment control level, rehabilitation equipment such as APS, isokinetic devices, and balance training devices are controlled in a fragmented manner without a unified protocol interface. Most operations are manual and stand-alone, lacking intelligent parameter tuning and closed-loop control mechanisms, and real-time emergency handling is impossible when equipment malfunctions.

[0003] Regarding efficacy assessment and expert collaboration, rehabilitation training data is isolated and difficult to trace back; efficacy lacks real-time quantitative assessment methods; expert participation in rehabilitation diagnosis and treatment is often delayed; remote collaboration is not real-time, communication costs are high, and there is a lack of dedicated collaborative operation and instruction issuance platforms. At the algorithm model level, traditional rehabilitation decision-making models are "black box" models, lacking confidence output and interpretability. Doctors have low trust in model decisions, model training does not fully consider the high-risk characteristics of the ICU scenario, prediction accuracy for high-risk samples is insufficient, and data privacy protection measures are lacking. From the perspective of system closed loop, most ICU rehabilitation platforms only reach the stage of data recording or simple connection with the HIS system, failing to form a complete closed loop from assessment, pathway formulation, execution, tracking to feedback optimization. The linkage of rehabilitation equipment and pathway recommendation capabilities are also relatively lacking. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a multimodal intelligent decision-making and equipment control method and system for ICU critical care rehabilitation, which can realize intelligent closed-loop management of the entire process of critical care rehabilitation diagnosis and treatment.

[0005] To achieve the above objectives, the present invention provides a multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation, comprising: Acquire static medical record data and multimodal data characterizing the health information of critically ill ICU patients; the multimodal data includes continuous physiological signal data, discrete manual scoring data, and feedback data from rehabilitation equipment; The multimodal data is standardized to generate standardized data and corresponding missing masks; The standardized data and missing mask are input into a pre-constructed multimodal fusion network to obtain fusion features; Based on the fusion features, disease stage identification and risk assessment are performed to generate a rehabilitation path and determine the uncertainty quantification value corresponding to the rehabilitation path; the rehabilitation path includes equipment combination, operation mode and control parameters; The rehabilitation pathway is subjected to security verification based on the static medical record data and uncertainty quantification value, and manual review is triggered when the security verification fails. Based on the rehabilitation path that has passed security verification or been manually reviewed, standardized equipment control instructions are generated and sent to the corresponding rehabilitation equipment for execution. At the same time, the operating status data of the rehabilitation equipment and the corresponding patient physiological feedback data are collected. The rehabilitation pathway is dynamically adjusted based on the device's operating status data and physiological feedback data, and the parameters of the multimodal fusion network are updated using incremental learning and federated learning mechanisms.

[0006] Optionally, the multimodal fusion network includes: The tower encoder is used to extract features from the multimodal data to obtain physiological signal features, discrete scoring features, and device feedback features. A cross-modal attention layer is used to perform attention calculation and feature alignment using the physiological signal features as query vectors and the discrete scoring features and device feedback features as keys, to obtain cross-modal features; A modal missing compensation layer is used to gating and adjusting the attention weights of the cross-modal features in conjunction with the missing mask; The fusion output layer is used to fuse the gated cross-modal features with the physiological signal features to generate fused features.

[0007] Optionally, based on the fusion features, disease stage identification and risk assessment are performed to generate a rehabilitation path, including: The fused features are interacted with a preset disease stage embedding vector through cross-stage attention to obtain stage-enhanced features; the disease stage embedding vector corresponds to three stage labels: acute phase, subacute phase, and recovery phase. The enhanced stage features are input into a pre-constructed stage classifier to identify the current stage of the disease; The stage enhancement features are input into a pre-constructed path decision network to generate an initial rehabilitation path corresponding to the current stage of the disease. The fused features are input into the risk estimator to obtain a risk score; The control parameters in the initial rehabilitation path are adjusted based on the risk score to obtain the rehabilitation path.

[0008] Optionally, determining the uncertainty quantification value corresponding to the rehabilitation path includes: The attention weights in the multimodal fusion network are sampled multiple times using a Bayesian cross-modal attention mechanism; The uncertainty quantification value corresponding to the rehabilitation path is calculated based on the attention weight distribution obtained from sampling.

[0009] Optionally, triggering manual review includes: The rehabilitation pathway, static medical record data, and uncertainty quantification values ​​are pushed to the remote expert collaboration platform, and manual review prompts are provided. The system receives review and adjustment instructions from experts through the remote expert collaboration platform, and updates the rehabilitation path according to the review and adjustment instructions to obtain the rehabilitation path after manual review.

[0010] Optionally, the method further includes: When the patient's physiological feedback data exceeds a preset safety threshold, and / or when the device's operating status data contains an abnormal status code, an emergency shutdown sequence is executed.

[0011] Optionally, the method further includes: The rehabilitation pathway, static medical record data, uncertainty quantification values, and patient physiological feedback data are pushed to a remote expert collaboration platform for visualization. When an adjustment instruction is received from an expert through the remote expert collaboration platform, the rehabilitation path is updated according to the adjustment instruction.

[0012] Optionally, the method further includes: An attention heatmap is generated based on the mean of the attention weight distribution; The attention heatmap is then pushed to a remote expert collaboration platform for visualization.

[0013] Optionally, the standardized equipment control instructions adopt a 16-byte standard instruction frame format, which includes a header code, instruction code, axis position parameters, operating parameters, and a verification field.

[0014] This invention also provides a multimodal intelligent decision-making and equipment control system for ICU critical care rehabilitation, comprising: The data acquisition unit is used to acquire static medical record data and multimodal data that characterize the health information of critically ill patients in the ICU; the multimodal data includes continuous physiological signal data, discrete manual scoring data, and feedback data from rehabilitation equipment. The preprocessing unit is used to standardize the multimodal data to generate standardized data and corresponding missing masks. A multimodal fusion unit is used to input the standardized data and missing mask into a pre-constructed multimodal fusion network to obtain fusion features; The rehabilitation decision-making unit is used to identify disease stages and assess risks based on the fusion features, so as to generate a rehabilitation path and determine the uncertainty quantification value corresponding to the rehabilitation path; the rehabilitation path includes equipment combination, operation mode and control parameters; The security verification unit is used to perform security verification on the rehabilitation path based on the static medical record data and uncertainty quantification value, and to trigger manual review when the security verification fails. The equipment control unit is used to generate standardized equipment control commands and send them to the corresponding rehabilitation equipment for execution based on the rehabilitation path that has passed safety verification or been manually reviewed. At the same time, it collects the operating status data of the rehabilitation equipment and the corresponding patient physiological feedback data. The closed-loop optimization unit is used to dynamically adjust the rehabilitation path based on the device operating status data and physiological feedback data, and to update the parameters of the multimodal fusion network using incremental learning and federated learning mechanisms.

[0015] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: The multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation provided by this invention acquires static medical record data and multimodal data, and performs standardized processing on the multimodal data to generate a missing mask. This transforms heterogeneous data with large differences in sampling rate, high random missing rate, and strong noise interference into unified format data with integrity identifiers, providing a reliable data foundation for subsequent fusion and decision-making. It solves the problems of disconnect between medical systems and rehabilitation systems, inconsistent data acquisition standards, and difficulty in effective alignment in the prior art.

[0016] By inputting standardized data and a missing mask into a pre-constructed multimodal fusion network to obtain fusion features, and then using these features to identify disease stages and assess risks to generate rehabilitation pathways, this approach achieves accurate identification of disease stages in ICU patients and automated generation of rehabilitation pathways. This overcomes the shortcomings of existing solutions, such as over-reliance on physicians' personal experience, lack of standardized pathways, and inability to dynamically adjust based on multi-dimensional data. Simultaneously, the corresponding uncertainty quantification value is determined during rehabilitation pathway generation. In subsequent steps, the rehabilitation pathway is validated based on static medical record data and this uncertainty quantification value. If the validation fails, manual review is triggered. This mechanism explicitly quantifies the confidence level of the model's decisions and incorporates it into safety constraints. This addresses the lack of interpretability and low physician trust in traditional "black box" models, while also ensuring the safety of rehabilitation training in clinically contraindicated and high-uncertainty scenarios through dual safety checks.

[0017] This invention generates standardized equipment control commands based on rehabilitation pathways that have passed security verification or manual review, and issues them for execution. Simultaneously, it collects real-time data on equipment operation status and patient physiological feedback, dynamically adjusting the rehabilitation pathway accordingly. This achieves a complete closed loop from decision-making to execution and feedback optimization, changing the current situation of decentralized equipment control, manual operation, and lack of closed-loop control, and improving the intelligence level and real-time emergency response capability of rehabilitation equipment control. Furthermore, it utilizes incremental learning and federated learning mechanisms to update the parameters of the multimodal fusion network, enabling the model to continuously optimize while protecting data privacy. Local incremental learning quickly adapts to new patient characteristics, and multi-center federated learning achieves cross-center model generalization. This solves the problems of existing models not considering high-risk characteristics, lacking data privacy protection, and being unable to continuously evolve, providing a safe, accurate, and dynamically evolving intelligent management solution for the entire process of ICU critical care rehabilitation. Attached Figure Description

[0018] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same parts.

[0019] Figure 1 This is a schematic diagram of the method flow of the multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation, as shown in an embodiment of the present invention; Figure 2 This is a schematic diagram of a multimodal fusion network structure shown in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the linkage structure between the SAFE stage enhancement module and the multi-task network in an embodiment of the present invention. Figure 4This is a schematic diagram illustrating the interface layout and functions of a remote expert collaboration platform according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the mapping between the device control command frame format and the emergency status code in an embodiment of the present invention; Figure 6 This is a visualization diagram of Bayesian-attention fusion as shown in an embodiment of the present invention; Figure 7 This is a schematic diagram of the modular structure of a multimodal intelligent decision-making and equipment control system for ICU critical care rehabilitation, as shown in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 , Figure 1 A flowchart illustrating the multimodal intelligent decision-making and equipment control methods for ICU critical care rehabilitation.

[0022] Multimodal intelligent decision-making and equipment control methods for ICU critical care rehabilitation include: S101: Acquire static medical record data and multimodal data that characterize the health information of critically ill ICU patients.

[0023] Referring to Table 1 below, static medical record data comes from the hospital information system and electronic medical record system, including patient admission records, imaging reports, diagnostic ICD codes, etc., and is usually retrieved once through system integration when the patient is admitted to the ICU. Multimodal data includes continuous physiological signal data, discrete manual assessment data, and rehabilitation equipment feedback data. Continuous physiological signal data is collected in real time by bedside monitors at a sampling frequency of 125Hz. Core data items include heart rate, electrocardiogram waveform, blood pressure, blood oxygen saturation, respiratory rate, and body temperature, forming a continuous data stream with a 30-second time window. Discrete manual assessment data is entered by medical staff via mobile terminals and recorded every 8 hours, including the Glasgow Coma Scale (GCS), National Institutes of Health Scale (NIHSS), and Barthel Activities of Daily Living (ADL) scores, used to reflect the dynamic changes in the patient's neurological function and daily living abilities. Rehabilitation equipment feedback data comes from rehabilitation training equipment, including surface electromyography (EMG) signals, joint angles, torques, training compliance, and movement trajectories, with a sampling frequency of 100Hz, also collected in 30-second windows.

[0024] Table 1 All the aforementioned multimodal data were timestamped during acquisition to ensure accurate association with patient IDs, providing complete and traceable raw data for subsequent standardized processing and fusion analysis. It should be noted that data loss may occur during acquisition due to factors such as device status, patient cooperation, or communication failures. For example, physiological signals may be missing due to electrode detachment, manual scores may be missing due to assessment omissions, and device feedback data may be lost due to communication interruptions. This missing information will be identified and processed using a subsequently generated missing information mask to ensure the robustness of multimodal data fusion.

[0025] S102: Standardize the multimodal data to generate standardized data and corresponding missing masks.

[0026] After acquiring multi-source, heterogeneous static medical record data and multimodal data, the data undergoes unified standardization to eliminate differences between different sources, sampling frequencies, and data structures. Specifically, the standardization process first assigns a unique patient identifier and a precise timestamp to all collected data, generating standardized data in JSON format to ensure that each data point is traceable to a specific patient's health status at a specific time. For example: {Patient ID: "P2023001", Data Type: "NIHSS", Value: 12, Timestamp: "2025-10-01T08:30:00"}. For continuous physiological signal data and rehabilitation equipment feedback data, data is sliced ​​according to a preset 30-second time window, with each window containing a historical sequence of physiological signal data and equipment feedback data. For discrete manual scoring data, due to its long sampling period, it is mapped to the corresponding physiological window by timestamp, forming the alignment basis of multimodal data in the time dimension.

[0027] During the standardization process, a corresponding missing mask Mt (0 = missing / invalid, 1 = valid) is generated for each frame of continuous data. The missing mask is a binary matrix with the same dimension as the original data, where 0 indicates that the corresponding data is missing or invalid, and 1 indicates that the data is valid. The generation of the missing mask is based on the actual state at the time of data acquisition: for physiological signals, when data is missing due to electrode detachment or signal interruption, the mask at the corresponding position is marked as 0; for discrete scores, when data is missing due to score omissions or input delays, the mask is marked as 0; for device feedback data, when data is missing due to communication failures or sensor malfunctions, the mask is marked as 0. This missing mask serves as an additional input to the subsequent multimodal fusion network, enabling the network to explicitly perceive the integrity status of each modality's data. This allows for reasonable weight reduction of missing modalities during the fusion process, preventing missing data from contaminating the fusion results. Through the above standardization process, the heterogeneous multimodal data, which originally had large differences in sampling rates, high random missing rates, and strong noise interference, is transformed into unified format data with integrity identifiers, providing a high-quality data foundation for subsequent cross-modal feature extraction and fusion.

[0028] S103: Input standardized data and missing mask into a pre-built multimodal fusion network to obtain fusion features.

[0029] The multimodal fusion network includes: The tower encoder is used to extract features from multimodal data to obtain physiological signal features, discrete scoring features, and equipment feedback features. The cross-modal attention layer is used to perform attention calculation and feature alignment using physiological signal features as query vectors and discrete rating features and device feedback features as keys, thereby obtaining cross-modal features; A modal missing compensation layer is used to gating and adjusting the attention weights of cross-modal features by combining the missing mask; The fusion output layer is used to fuse the gated cross-modal features with physiological signal features to generate fused features.

[0030] In the application, after data standardization and the generation of corresponding missing masks, the standardized data and missing masks are input together into a pre-built multimodal fusion network (CMAC-Net). This network enables sampling rate-independent, missing-robust multimodal feature extraction and fusion to obtain fused features for subsequent decision-making. See also Figure 2 The multimodal fusion network adopts a split-tower parallel architecture, consisting of a split-tower encoder, a cross-modal attention layer, a modal missing compensation layer, and a fusion output layer connected in series.

[0031] The three-tower encoder adopts a parallel structure, which performs targeted feature extraction for different types of data. The output dimension of each tower is forcibly unified to 256 dimensions, which lays the foundation for subsequent cross-modal attention calculation. The functions of each tower can be found in Table 2 below.

[0032] Table 2

[0033] The physiological signal tower takes continuous physiological signal data within a 30-second time window as input. It extracts temporal-morphological joint features such as short-term heart rate variability, blood pressure fluctuation patterns, and peak and trough blood oxygen levels using a multi-scale one-dimensional convolutional neural network combined with dilated convolution, outputting a physiological signal feature vector. The discrete scoring tower takes three scalar values—GCS coma score, NIHSS neurological function score, and Barthel Activities of Daily Living score—as input. It maps discrete scores to feature vectors containing semantic information about disease severity while preserving temporal relationships through a learnable embedding layer combined with positional encoding, outputting a discrete score feature vector. The device feedback tower takes rehabilitation device feedback data within a 30-second time window as input. It uses a Transformer encoder combined with relative positional encoding to extract action-physiological coupling features such as rehabilitation movement execution quality, muscle activation patterns, and patient active participation, outputting a device feedback feature vector. Because each tower independently processes its own modality of data, there is no need for unified resampling of the original data, thus avoiding data distortion caused by sampling rate conversion.

[0034] The cross-modal attention layer receives three types of feature vectors from the tower encoder output. It uses the physiological signal feature vector as the query vector and the discrete scoring feature vector and device feedback feature vector as both the key and value vectors. Cross-modal attention weights are calculated using a scaled dot product attention mechanism. Specifically, the attention weights can be calculated using the formula: in, Physiological signal characteristics, This involves concatenating discrete scoring features with device feedback features. and This is a learnable mapping matrix. This mechanism can automatically increase the weight of scoring data and device feedback data related to the current physiological abnormality, while effectively suppressing noise channels and irrelevant information, achieving dynamic alignment of heterogeneous multimodal data at the feature level, and outputting attention-weighted cross-modal features.

[0035] A modal missing data compensation layer, combined with a previously generated missing data mask, gates the attention weights of cross-modal features to address the common problem of random missing data in ICU scenarios. During training, this layer randomly masks the input of any sub-array, enabling the model to learn and generate existence probability vectors for each modality. These vectors represent the completeness of each modality's data at the current time step. Subsequently, using these existence probability vectors as gating coefficients, the attention weights output by the cross-modal attention layer are gated, resulting in the gated weights, mathematically expressed as follows: ,in There exists a probability vector for the mode. These are the original attention weights. Through this mechanism, the attention weights corresponding to missing modalities are significantly reduced, and the missing values ​​only affect the features of this modality without polluting the features of other modalities, thus achieving a fusion effect of reduced weighting for missing values ​​and effective enhancement.

[0036] The fusion output layer performs a weighted summation of the gated cross-modal features according to attention weights, and then performs residual fusion with the original physiological signal features output from the physiological signal tower. After layer normalization and random inactivation processing, the final fused feature is generated. The fusion process can be described by the following formula: in, For the value vector in the cross-modal attention layer, This is a physiological signal feature. This fusion feature integrates complementary information from multimodal data, preserving the sensitivity of physiological signals to the patient's real-time status while incorporating long-term disease trends reflected by discrete scores and the quality of action execution reflected by device feedback. This provides a rich feature foundation for subsequent disease stage identification, risk assessment, and rehabilitation pathway generation.

[0037] S104: Based on fusion features, identify disease stages and assess risks to generate rehabilitation pathways and determine the corresponding uncertainty quantification values ​​for the rehabilitation pathways.

[0038] After obtaining the fusion characteristics, disease stage identification and risk assessment are performed simultaneously based on these characteristics to generate a personalized rehabilitation path and determine the corresponding uncertainty quantification value. The rehabilitation path includes equipment combinations, operating modes, and control parameters.

[0039] Specifically, based on the fusion characteristics, disease stages are identified and risks are assessed to generate rehabilitation pathways, including: The fused features are interacted with the pre-defined disease stage embedding vectors through cross-stage attention to obtain stage-enhanced features; the disease stage embedding vectors correspond to the three stage labels of acute phase, subacute phase, and recovery phase. The stage enhancement features are input into a pre-built stage classifier to identify the current stage of the disease; The stage-enhanced features are input into a pre-constructed path decision network to generate an initial rehabilitation path corresponding to the current stage of the disease. The fused features are input into the risk estimator to obtain a risk score; The rehabilitation pathway is obtained by adjusting the control parameters in the initial rehabilitation pathway based on the risk score.

[0040] Specifically, the fused features are first subjected to cross-stage attention interaction with pre-defined disease stage embedding vectors. The disease stage embedding vectors are learnable semantic mappings of the three stage labels: acute, subacute, and recovery. Each stage label corresponds to a high-dimensional embedding vector, used to explicitly guide the model to perceive the current disease stage of the patient. In the cross-stage attention mechanism, the disease stage embedding vector is used as the query vector, and the fused features are used as the key and value vectors. Attention calculations strengthen feature channels relevant to the current disease stage and suppress irrelevant or interfering feature information, thereby generating stage-enhanced features. These stage-enhanced features retain the multimodal information of the original fused features while incorporating explicit perception of the disease stage, enabling subsequent disease stage identification and rehabilitation path generation to fully integrate with the specific disease stage the patient is in.

[0041] Subsequently, the stage-enhanced features are input into a pre-built stage classifier and a path decision network, respectively. The stage classifier identifies the patient's current disease stage and outputs a classification label indicating acute, subacute, or recovery phase, providing clinicians with a quantitative reference for the patient's disease progression. The path decision network generates an initial rehabilitation path corresponding to the current disease stage based on the stage-enhanced features. This rehabilitation path includes equipment combinations, operating modes, and control parameters. The equipment combinations specify the types of rehabilitation equipment to be used in this rehabilitation training, such as isokinetic trainers, APS rehabilitation systems, or balance trainers; the operating modes define the operating methods of the equipment, such as passive mode, active assisted mode, or resistance mode; and the control parameters include specific values ​​such as joint range of motion, movement speed, and upper torque limit, which directly determine the intensity and range of movement when the rehabilitation equipment performs training.

[0042] Simultaneously, the original fusion features are input into a risk estimator to obtain a risk score. The risk estimator quantifies the patient's immediate risk using a regression model. Its input is the fusion features, and its output is a risk score between 0 and 1. This score comprehensively reflects the patient's current physiological stability, the potential risk of performing rehabilitation movements, and any abnormalities that occurred during historical training. A higher risk score indicates a greater risk for the patient during rehabilitation training. Subsequently, the control parameters in the initial rehabilitation pathway are adjusted based on this risk score: when the risk score is low, the training intensity is maintained or appropriately increased; when the risk score is high, the training intensity is reduced or the range of motion of the joints is limited to ensure training safety. Only the risk-adjusted rehabilitation pathway can be used for subsequent rehabilitation.

[0043] While generating the rehabilitation path, a Bayesian cross-modal attention mechanism is used to determine the uncertainty quantification value corresponding to the rehabilitation path. The determination of the uncertainty quantification value corresponding to the rehabilitation path includes: The attention weights in the multimodal fusion network are sampled multiple times using a Bayesian cross-modal attention mechanism; The uncertainty quantification value corresponding to the rehabilitation path is calculated based on the attention weight distribution obtained from sampling.

[0044] Specifically, the mapping matrix of the cross-modal attention layer in the multimodal fusion network is modeled as a probability distribution, rather than a traditional deterministic parameter. Multiple Monte Carlo samplings are performed on this probability distribution to obtain the attention weight distribution corresponding to each sampling. Based on this attention weight distribution, its mean and variance are calculated, and further, an uncertainty quantification value is calculated through a weighted sum of entropy and variance. This uncertainty quantification value characterizes the model's confidence in the current decision outcome: a smaller value indicates greater confidence in the generated rehabilitation path, while a larger value indicates greater uncertainty in the model's decision. Through this mechanism, while outputting personalized rehabilitation paths, a reliable quantification index for clinicians' decisions is also provided, making the model's decision-making process more transparent and interpretable, effectively solving the clinical trust barrier caused by the black-box problem of traditional rehabilitation decision-making models.

[0045] The aforementioned multimodal fusion network, stage classifier, path decision network, and risk estimator require joint training before deployment. During training, a risk-aware weighted loss function (RAWL) is used as the optimization objective to enhance the model's prediction accuracy for high-risk samples. This loss function employs a multi-task joint learning framework, simultaneously optimizing the three tasks of disease stage classification, rehabilitation path generation, and risk prediction. The total loss can be expressed as: in, For the first The cross-entropy loss for classifying the disease stage of each sample. For the first The gradient loss of the rehabilitation path recommendation strategy for each sample. For the first Mean squared error loss for risk prediction of each sample; As the first weighting coefficient, The second weighting coefficient can be obtained automatically through multi-task uncertainty weighting or manually configured according to clinical priorities. For the first The risk weight of each sample is used to assign a higher loss weight to high-risk samples, and its calculation formula is as follows: in, This is the risk weight adjustment coefficient; The risk score is the sample's instantaneous risk score, output by the risk estimator during training, and comprehensively reflects the patient's physiological stability and the risk of performing rehabilitation movements at the current moment. The specific calculation can be expressed as: In the formula, As an indicator of heart rate variability, For the range of blood pressure fluctuations, Fatigue characteristics of surface electromyography signals. , and The third, fourth, and fifth weighting coefficients are listed in order, and these coefficients can be determined by prior knowledge from ICU experts or by Bayesian optimization methods. Through this risk-aware weighted loss function, the model assigns higher learning weights to high-risk samples during training, thereby enhancing the accuracy of predictions for high-risk scenarios and effectively reducing the clinical risks of ICU rehabilitation training.

[0046] The following section provides a further introduction to the stage classifier, path decision network, and risk estimator mentioned above. It should also be noted that the aforementioned processing mechanism of "interacting with the fused features and the preset disease stage embedding vector through cross-stage attention to obtain stage-enhanced features" can be handled by the SAFE (Stage-Adaptive Feature Enhancement) module. This module is used to enhance feature channels related to the current disease stage, outputting stage-enhanced features.

[0047] In the application, the stage classifier can adopt a lightweight classification network architecture. Its input is the stage enhancement feature output by the SAFE module, which has a dimension of 512. The core structure of the stage classifier includes two fully connected layers, a batch normalization layer, a ReLU activation function, and a Softmax classification layer. Specifically, the first fully connected layer maps the 512-dimensional stage enhancement feature to 128 dimensions. After batch normalization and ReLU activation, the second fully connected layer maps the 128-dimensional feature to 3 dimensions. Finally, the Softmax layer outputs the probability distribution of the three classification labels corresponding to the acute, subacute, and recovery phases, and the maximum probability is taken as the current disease stage label. During training, the stage classifier uses cross-entropy loss as the optimization objective and incorporates Dropout and L2 regularization to prevent overfitting. Compared to existing classifiers, the core improvement of the stage classifier lies in the fact that its input is a stage-enhanced feature after being enhanced by the SAFE module. This feature explicitly incorporates the disease stage embedding vector, strengthens the feature channels related to the current stage, and effectively improves the problem of the blurry boundary of the disease stage identification of ICU patients in existing models. At the same time, the classifier adopts a lightweight design, and the inference latency can be controlled within 10 milliseconds, which meets the real-time requirements of ICU scenarios.

[0048] The path decision network can adopt a hybrid architecture combining a rule layer and a data-driven layer. Its input is the stage enhancement features output by the SAFE module, and its output is the action space of the rehabilitation path. This action space includes three-dimensional action combinations of equipment combinations, operating modes, and control parameters. The rule layer embeds ICU rehabilitation clinical guidelines to form a set of action space constraints, used to filter ineffective or dangerous actions. The data-driven layer uses a reinforcement learning policy network, consisting of a fully connected layer for feature extraction, a policy head, and a value head. The fully connected layer for feature extraction maps the 512-dimensional stage enhancement features to 256 dimensions. The policy head outputs the probability distribution of the action space, and the value head evaluates the benefit value of the current action to assist in policy optimization. During training, the path decision network uses policy gradient loss as the optimization objective and designs a reward function based on the principle of "maximizing rehabilitation benefit and minimizing risk." This reward function is expressed as R = 0.6 × efficacy improvement - 0.4 × risk value, where efficacy improvement is measured by the NIHSS score decline rate, the risk value is output by the risk estimator, and the weight coefficients can be adjusted according to the clinical scenario. Compared to existing path recommendation models, the core improvements of the path decision network are: adopting a dual-driven architecture of rules and data, which solves the problems of pure data-driven models easily generating dangerous paths that do not conform to clinical guidelines and pure rule models lacking personalization; introducing the risk estimator output into the reward function to achieve a dynamic balance between efficacy and risk; the action space is customized for ICU rehabilitation equipment, including a three-dimensional action space of equipment combination, operation mode and control parameters; and online incremental optimization is performed in conjunction with the real-time feedback of the ICU-DDL closed loop, so that the rehabilitation path can adapt to the dynamic changes in the patient's condition.

[0049] The risk estimator employs a regression network architecture. Its input is the original fused features output from the multimodal fusion network, which has 256 dimensions. The output is a continuous risk score between 0 and 1. The core structure of the risk estimator includes three fully connected layers, a batch normalization layer, a LeakyReLU activation function, and a Sigmoid regression layer. Specifically, the first fully connected layer maps the 256-dimensional fused features to 128 dimensions, the second fully connected layer maps the 128 dimensions to 64 dimensions, and the third fully connected layer maps the 64 dimensions to 1 dimension. Finally, the Sigmoid layer normalizes the features to the 0-1 range, resulting in the final risk score, where 0 represents low risk and 1 represents high risk. During training, the risk estimator uses mean squared error loss as the optimization objective and employs RAWL risk-aware weighted loss to assign higher loss weights to high-risk samples, thereby enhancing the model's prediction accuracy for high-risk scenarios. Compared to existing risk prediction models, the core improvement of the risk estimator lies in the following: the input is the original multimodal fusion features, which retains complete information from physiological signals, discrete scores, and device feedback, overcoming the one-sidedness of existing models that predict risk based on only a single physiological indicator; the output is a continuous risk score from 0 to 1, rather than a simple binary classification result, allowing doctors to flexibly configure risk intervention thresholds according to clinical needs; the risk score is directly used to adjust the control parameters in the initial rehabilitation path generated by the path decision network, achieving seamless linkage between risk assessment and path adjustment.

[0050] See Figure 3 The stage classifier, path decision network, and risk estimator are jointly trained with the multimodal fusion network and the SAFE stage enhancement module, and uniformly incorporated into the RAWL risk perception weighted loss function framework to achieve multi-task collaborative optimization of "stage identification-path generation-risk assessment". All three components employ a unified loss function. Joint training is conducted. Through this unified loss framework, the gradients of the three components synergize during backpropagation: the gradient of the risk estimator guides the fusion feature layer to focus more on risk-related feature channels; the gradient of the stage classifier guides the SAFE module to strengthen the extraction of stage-discriminative features; and the gradient of the path decision network prompts the stage-enhanced features to optimize towards higher returns. This results in a virtuous cycle of multi-task collaborative learning. At the feature level, the stage classifier and path decision network share the SAFE stage-enhanced features, enabling stage information to simultaneously serve disease identification and path generation; the risk estimator uses the original fusion features, retaining more comprehensive multimodal information for accurate risk assessment. At the output level, the risk score output by the risk estimator is directly fed back to the path decision network to adjust the initially generated rehabilitation path; the disease stage label output by the stage classifier provides prior constraints on the action space for the path decision network, for example, only opening the passive training action space during the acute phase and opening the active training action space during the recovery phase. These three components form an overall architecture of stage identification, risk assessment, and path decision-making, achieving end-to-end collaboration from patient state perception to rehabilitation intervention generation.

[0051] S105: Based on static medical record data and uncertainty quantification values, perform safety verification on the rehabilitation pathway, and trigger manual review when the safety verification fails.

[0052] After generating a rehabilitation pathway and determining its corresponding uncertainty quantification value, a safety verification is performed on the rehabilitation pathway based on static medical record data and this uncertainty quantification value to ensure that rehabilitation training is conducted within clinical safety boundaries. The safety verification employs a dual safety constraint mechanism, including hard rule verification based on static medical record data and soft rule verification based on the uncertainty quantification value. Hard rule verification and soft rule verification are performed in parallel; failure of either verification results in a failed safety verification.

[0053] The hard rule verification, based on the patient's clinical information recorded in static medical records, conducts a compliance review of the equipment combinations, operating modes, and control parameters in the rehabilitation pathway. Specifically, key information such as the patient's diagnosis, imaging reports, past medical history, and clinical contraindications are extracted from the static medical record data to form a set of constraint rules. Each parameter in the generated rehabilitation pathway is compared with these rules item by item. If any equipment combination or control parameter involved in the rehabilitation pathway conflicts with the contraindications in the patient's medical record, such as using active training equipment for a patient with an unhealed fracture, or including significant positional changes in rehabilitation parameters for a patient with increased intracranial pressure, the safety verification is deemed to have failed. Conversely, if the rehabilitation pathway fully complies with the safety constraints recorded in the medical record, the hard rule verification is deemed to have passed.

[0054] Soft rule validation assesses the safety of the rehabilitation pathway based on the uncertainty quantification value. The uncertainty quantification value is compared to a preset uncertainty threshold. When the uncertainty quantification value is below or equal to the threshold, it indicates that the model has high confidence in the current rehabilitation pathway, and the soft rule validation passes. When the uncertainty quantification value exceeds the threshold, it indicates that the model has low certainty about the current rehabilitation pathway, posing a potential risk. In this case, the soft rule validation fails, and the control parameters in the rehabilitation pathway are automatically adjusted. Specifically, this includes reducing equipment operating intensity, slowing down exercise speed, or reducing joint range of motion. The adjusted rehabilitation pathway is then used as a new pathway to be validated. The adjusted rehabilitation pathway undergoes soft rule validation again; if it still exceeds the threshold, manual review is triggered.

[0055] If any hard rule check or soft rule check fails, the security check is deemed failed, and manual review is triggered. Specifically, triggering manual review includes: The rehabilitation pathway, static medical record data, and uncertainty quantification values ​​are pushed to the remote expert collaboration platform, and manual review prompts are provided. Receive review and adjustment instructions from experts through a remote expert collaboration platform, update the rehabilitation path according to the review and adjustment instructions, and obtain the rehabilitation path after manual review.

[0056] See Figure 4 , Figure 4This diagram illustrates the interface layout and functions of a remote expert collaboration platform. The patient information displayed is for illustrative purposes only and does not represent actual patient information. In practice, manual review can be completed through the platform. The current rehabilitation pathway, relevant static medical record data, and uncertainty quantification values ​​are pushed to the platform, prompting experts for manual review via pop-ups, voice messages, and vibrations. Experts can comprehensively assess the safety of the rehabilitation pathway based on medical record information and uncertainty indicators, and make review decisions based on clinical experience. Specifically, experts can confirm the adoption of the current rehabilitation pathway, at which point the system marks the pathway as approved for manual review. Experts can also modify control parameters such as equipment operation mode, training intensity, or joint range of motion based on the system-push rehabilitation pathway adjustment area, using methods such as intensity sliders, equipment combination selections, or direct parameter editing, generating custom adjustment instructions. The system automatically updates the rehabilitation pathway upon receiving these instructions. Experts can also directly reject the current rehabilitation pathway, pausing the issuance of instructions for this rehabilitation training, and waiting for the expert to create a new pathway before execution. After the expert makes a decision, their electronic signature is obtained through the instruction confirmation area. The signature information, decision time, and path parameters before and after adjustments are then bound and stored to form a complete review record. After manual review, the resulting rehabilitation path becomes the manually reviewed rehabilitation path, serving as the basis for generating subsequent equipment control instructions. This manual review mechanism enables effective supervision and intervention by clinical experts in the model's decision-making process, ensuring the safety of rehabilitation training while fully leveraging the synergistic advantages of expert clinical experience and the model's intelligent decision-making.

[0057] S106: Based on the rehabilitation path that has passed security verification or been manually reviewed, generate standardized equipment control instructions and send them to the corresponding rehabilitation equipment for execution, while collecting the operating status data of the rehabilitation equipment and the corresponding patient physiological feedback data.

[0058] The standardized equipment control instructions adopt a 16-byte standard instruction frame format, which includes a header code, instruction code, axis position parameters, operating parameters, and a verification field.

[0059] See Figure 5 , Figure 5This diagram illustrates the mapping between the device control command frame format and emergency status codes. In application, after safety verification or manual review, standardized device control commands are generated based on the confirmed rehabilitation path. The rehabilitation path includes device combinations, operating modes, and control parameters, specifically joint angles, movement speeds, and torque limits. These parameters are filled into a preset standard command frame format (see Table 3 below). This standard command frame has a unified 16-byte structure, including a header code, command code, axis parameters, operating parameters, and a verification field. The header code uses the fixed two bytes 0x5A and 0xA5 to identify the start of the command. The command code indicates the device's operation type. The axis parameters specify the joint or movement axis to be controlled. The operating parameters include specific values ​​such as movement direction, target position, and speed. The verification field verifies the integrity of the command transmission. This unified command frame format allows for adaptation to different types of rehabilitation equipment, enabling unified control of heterogeneous devices.

[0060] Table 3

[0061] After generating standardized device control commands, these commands are sent to the corresponding rehabilitation devices for execution. A timeout mechanism, such as 100 milliseconds, can be set during the sending process. If no response is received from the device within the timeout period, the command will automatically retry, up to two times. Upon receiving the command, the device executes the corresponding rehabilitation action and transmits the device's operating status code in real time. The status code can be defined in four levels: 0x01 indicates normal device operation, 0x02 indicates communication timeout or device operation exceeding a safety threshold, 0x03 indicates force value exceeding a preset threshold, and 0x04 indicates abnormal patient physiological signals. This operating status data is collected in real time and recorded in the device log, which includes information such as operation time, execution parameters, and completion rate.

[0062] While rehabilitation training is being conducted, physiological feedback data from patients is collected simultaneously. Bedside monitors continuously monitor physiological parameters such as heart rate, blood pressure, blood oxygen saturation, and respiratory rate, with a sampling frequency of 125 Hz, forming a data stream every 30 seconds. This physiological feedback data, along with equipment operating status data, is recorded in real time for subsequent dynamic adjustments and efficacy evaluation.

[0063] S107: The rehabilitation pathway is dynamically adjusted based on equipment operation status data and physiological feedback data, and the parameters of the multimodal fusion network are updated using incremental learning and federated learning mechanisms.

[0064] Based on the training of rehabilitation equipment and the real-time collection of operational status data and patient physiological feedback data, the rehabilitation path is dynamically adjusted according to this feedback data. Incremental learning and federated learning mechanisms are used to update the parameters of the multimodal fusion network, thereby achieving continuous optimization of rehabilitation strategies and cross-center generalization of the model.

[0065] The dynamic adjustment mechanism consists of two levels: periodic re-decision and real-time emergency response. Real-time emergency response takes precedence over periodic re-decision; when an emergency shutdown is triggered, periodic re-decision is temporarily suspended. At the periodic re-decision level, multimodal data within the latest time window is reacquired according to a preset time period, such as every 5 minutes. This includes continuous physiological signals from the last 30 seconds, the latest human assessment for the day, and device feedback data recorded from the last training session. This data is then used to re-execute the aforementioned multimodal fusion network processing and rehabilitation pathway generation process, outputting an updated rehabilitation pathway. This periodic re-decision mechanism allows the rehabilitation pathway to be adjusted in real-time according to the dynamic changes in the patient's condition, achieving precise and personalized rehabilitation tailored to each individual and each specific timeframe.

[0066] At the real-time emergency response level, the system continuously monitors equipment operation status data and patient physiological feedback data during rehabilitation equipment operation. When patient physiological feedback data exceeds a preset safety threshold, and / or when equipment operation status data contains abnormal status codes, an emergency shutdown sequence is executed. Specifically, when abnormal status codes are detected in equipment operation status data, such as communication timeout, exceeding safety thresholds, or force exceeding limits, or when patient physiological feedback data exceeds preset safety thresholds, such as heart rate exceeding 150 beats per minute or systolic blood pressure exceeding 180 mmHg, an emergency shutdown sequence is immediately executed, a shutdown command is issued, the equipment power circuit is cut off, an audible and visual alarm is triggered, and information such as the abnormality type, time, and equipment parameters are recorded in the fault log. This real-time emergency response mechanism ensures that training can be terminated immediately in the event of equipment failure or patient physiological abnormalities, guaranteeing patient safety.

[0067] At the model parameter update level, incremental learning and federated learning mechanisms are combined to continuously optimize the multimodal fusion network. Incremental learning is mainly performed locally, with the system maintaining an experience pool of 5000 data records and using a first-in, first-out (FIFO) strategy to manage data updates. After every 100 new local training data records, an elastic weight consolidation algorithm is used to fine-tune the multimodal fusion network. This algorithm introduces a Fisher information matrix into the loss function to constrain the updates of important network parameters, thereby retaining existing knowledge while learning new data features and effectively avoiding the catastrophic forgetting problem.

[0068] Federated learning is used for global model optimization in multi-center deployment scenarios. Each center node deploys a local federated learning node, and all raw data remains locally at each center. Only model gradient momentum is uploaded to the core server via a VPN encrypted channel. The core server uses a federated averaging algorithm to aggregate the gradients uploaded from each center, calculates the global gradient, and updates the global model parameters. The updated model parameters are then distributed to each center node. Data transmission employs dual encryption using TLS 1.3 and the Chinese national cryptographic standard SM4, with the encryption key dynamically updated hourly to ensure data sovereignty and medical data privacy. Through federated learning, centers can collaboratively optimize the global model without sharing original patient data, achieving cross-center generalization improvement in model performance.

[0069] In one embodiment, the above method further includes: The rehabilitation pathway, static medical record data, uncertainty quantification values, and patient physiological feedback data are pushed to a remote expert collaboration platform for visualization. When an adjustment instruction is received from an expert through a remote expert collaboration platform, the rehabilitation path is updated according to the adjustment instruction.

[0070] In actual operation, the remote expert collaboration platform serves as the human-machine interface, providing clinical experts with real-time monitoring and remote intervention capabilities. The system pushes rehabilitation pathways, static medical record data, uncertainty quantification values, and patient physiological feedback data to the remote expert collaboration platform for visualization, enabling experts to intuitively understand the patient's current rehabilitation plan, basic medical history, the confidence level of the model's decisions, and real-time physiological status. The platform adopts a multi-area layout design. The real-time monitoring panel, using a combination of dashboards and line graphs, refreshes patient physiological parameters and equipment operation curves every 5 seconds, displaying the changing trends of key indicators such as heart rate, blood pressure, and blood oxygen saturation. The rehabilitation pathway adjustment area displays the equipment combinations, operating modes, and control parameters included in the current rehabilitation pathway in a structured format, providing interactive controls such as pathway template selection, intensity slider adjustment, and equipment combination selection. The historical trend comparison area displays the patient's neurological function score trend and rehabilitation training completion trend over the past 7 days, providing data support for experts' remote assessment. Uncertainty quantification values ​​are presented in numerical or color-coded form; when uncertainty exceeds a preset threshold, the panel automatically highlights it, reminding experts to pay close attention.

[0071] After reviewing the data through the remote expert collaboration platform, if experts believe the current rehabilitation pathway needs adjustment, they can directly issue adjustment instructions through the platform interface. The platform supports multiple adjustment methods: experts can apply recommended pathway templates with a single click; they can also adjust parameters such as training intensity, speed, or torque limits in real time by dragging the intensity slider, with the platform displaying the adjusted parameter values ​​in real time; experts can also manually select or deselect specific equipment combinations, or directly edit the specific values ​​of control parameters. After adjustment, the platform automatically generates an adjustment instruction, which includes a comparison of parameters before and after modification, the adjustment time, and the expert's identity information. Experts use a handwriting pad to electronically sign the instruction in the confirmation area; the signature information is bound to the adjustment instruction and stored, forming a complete operation record.

[0072] Upon receiving adjustment instructions from experts, the system updates the current rehabilitation pathway accordingly and uses the updated pathway as the basis for generating subsequent equipment control instructions. If the adjustment instruction involves modifying equipment operating parameters, the system will regenerate standardized equipment control instructions based on the updated rehabilitation pathway and issue them to the rehabilitation equipment for execution. If the adjustment instruction involves changing the equipment combination, the new equipment combination will be activated in the next training cycle. Through this remote expert collaboration mechanism, a deep integration of expert clinical experience and model-based intelligent decision-making is achieved. Expert intervention response time is reduced from the traditional hours to minutes, significantly improving the efficiency of cross-center sharing of high-quality rehabilitation medical resources and the real-time response capability of clinical diagnosis and treatment.

[0073] In one embodiment, the above method further includes: An attention heatmap is generated based on the mean of the attention weight distribution; The attention heatmap is pushed to a remote expert collaboration platform for visualization.

[0074] See Figure 6 , Figure 6 This is a visualization of Bayesian attention fusion. In the multimodal fusion network processing, the attention weights are sampled multiple times using a Bayesian cross-modal attention mechanism to obtain their distribution. The mean of this distribution is then calculated as the final predicted attention weight. Based on this, an attention heatmap is generated according to the mean of the attention weight distribution. Specifically, the attention heatmap is presented as a matrix or image, where the color intensity of each pixel or cell represents the contribution of that modality or time point to the rehabilitation decision; a darker color indicates a higher contribution.

[0075] On the interface of the remote expert collaboration platform, attention heatmaps are displayed alongside rehabilitation pathways, uncertainty quantification values, and patient physiological feedback data. A dual-channel visualization design is employed, overlaying an uncertainty information layer on top of the attention heatmap; for example, darker red areas indicate higher uncertainty at that location. When an expert clicks on a highlighted area in the heatmap, a pop-up window displays the corresponding original signal segment. For instance, clicking on a highlighted heart rate-related area displays the ECG waveform for that time period, and clicking on a highlighted electromyography-related area displays the corresponding surface electromyography curve. The response time for the visualization is controlled within 200 milliseconds, meeting the real-time interaction requirements of ICU scenarios. Through this mechanism, doctors can intuitively understand why the model makes specific rehabilitation decisions, thereby enhancing their trust in the model's decisions. It also provides traceable evidence for experts to conduct manual review and adjust rehabilitation pathways.

[0076] Corresponding to the aforementioned application function implementation method embodiments, the present invention also provides a multimodal intelligent decision-making and equipment control system for ICU critical care rehabilitation and corresponding embodiments.

[0077] Please see Figure 7 , Figure 7 A schematic diagram of the modular structure of a multimodal intelligent decision-making and equipment control system for ICU critical care rehabilitation.

[0078] A multimodal intelligent decision-making and equipment control system for ICU critical care rehabilitation, including: The data acquisition unit 71 is used to acquire static medical record data and multimodal data that characterize the health information of critically ill patients in the ICU; the multimodal data includes continuous physiological signal data, discrete manual scoring data, and feedback data from rehabilitation equipment; Preprocessing unit 72 is used to standardize multimodal data and generate standardized data and corresponding missing masks; The multimodal fusion unit 73 is used to input standardized data and missing masks into a pre-constructed multimodal fusion network to obtain fusion features; The rehabilitation decision-making unit 74 is used to identify disease stages and assess risks based on fusion features, in order to generate a rehabilitation path and determine the uncertainty quantification value corresponding to the rehabilitation path; the rehabilitation path includes equipment combination, operation mode and control parameters; The security verification unit 75 is used to perform security verification on the rehabilitation pathway based on static medical record data and uncertainty quantification value, and to trigger manual review when the security verification fails. The equipment control unit 76 is used to generate standardized equipment control commands and send them to the corresponding rehabilitation equipment for execution based on the rehabilitation path after safety verification or manual review, and at the same time collect the operating status data of the rehabilitation equipment and the corresponding patient physiological feedback data. The closed-loop optimization unit 77 is used to dynamically adjust the rehabilitation path based on equipment operation status data and physiological feedback data, and to update the parameters of the multimodal fusion network using incremental learning and federated learning mechanisms.

[0079] In one embodiment, in identifying disease stages and assessing risks based on fusion features to generate a rehabilitation pathway, the rehabilitation decision-making unit 74 is specifically used for: The fused features are interacted with the pre-defined disease stage embedding vectors through cross-stage attention to obtain stage-enhanced features; the disease stage embedding vectors correspond to the three stage labels of acute phase, subacute phase, and recovery phase. The stage enhancement features are input into a pre-built stage classifier to identify the current stage of the disease; The stage-enhanced features are input into a pre-constructed path decision network to generate an initial rehabilitation path corresponding to the current stage of the disease. The fused features are input into the risk estimator to obtain a risk score; The rehabilitation pathway is obtained by adjusting the control parameters in the initial rehabilitation pathway based on the risk score.

[0080] In one embodiment, the rehabilitation decision-making unit 74 is specifically used for determining the uncertainty quantification value corresponding to the rehabilitation path in the following ways: The attention weights in the multimodal fusion network are sampled multiple times using a Bayesian cross-modal attention mechanism; The uncertainty quantification value corresponding to the rehabilitation path is calculated based on the attention weight distribution obtained from sampling.

[0081] In one embodiment, the security verification unit 75 is specifically used for triggering manual review in the following ways: The rehabilitation pathway, static medical record data, and uncertainty quantification values ​​are pushed to the remote expert collaboration platform, and manual review prompts are provided. Receive review and adjustment instructions from experts through a remote expert collaboration platform, update the rehabilitation path according to the review and adjustment instructions, and obtain the rehabilitation path after manual review.

[0082] In one embodiment, the system further includes: The emergency response unit is used to execute an emergency shutdown sequence when the patient's physiological feedback data exceeds a preset safety threshold, and / or when the equipment operating status data contains an abnormal status code.

[0083] In one embodiment, the system further includes: The remote expert collaboration unit is used for: The rehabilitation pathway, static medical record data, uncertainty quantification values, and patient physiological feedback data are pushed to a remote expert collaboration platform for visualization. When an adjustment instruction is received from an expert through a remote expert collaboration platform, the rehabilitation path is updated according to the adjustment instruction.

[0084] In one embodiment, the system further includes: Attention visualization unit, used for: An attention heatmap is generated based on the mean of the attention weight distribution; The attention heatmap is pushed to a remote expert collaboration platform for visualization.

[0085] Regarding the system in the above embodiments, the specific manner in which each unit module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0086] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation, characterized in that, include: Acquire static medical record data and multimodal data characterizing the health information of critically ill ICU patients; the multimodal data includes continuous physiological signal data, discrete manual scoring data, and feedback data from rehabilitation equipment; The multimodal data is standardized to generate standardized data and corresponding missing masks; The standardized data and missing mask are input into a pre-constructed multimodal fusion network to obtain fusion features; Based on the fusion features, disease stage identification and risk assessment are performed to generate a rehabilitation path and determine the uncertainty quantification value corresponding to the rehabilitation path; the rehabilitation path includes equipment combination, operation mode and control parameters; The rehabilitation pathway is subjected to security verification based on the static medical record data and uncertainty quantification value, and manual review is triggered when the security verification fails. Based on the rehabilitation path that has passed security verification or been manually reviewed, standardized equipment control instructions are generated and sent to the corresponding rehabilitation equipment for execution. At the same time, the operating status data of the rehabilitation equipment and the corresponding patient physiological feedback data are collected. The rehabilitation pathway is dynamically adjusted based on the device's operating status data and physiological feedback data, and the parameters of the multimodal fusion network are updated using incremental learning and federated learning mechanisms.

2. The multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation according to claim 1, characterized in that, The multimodal fusion network includes: The tower encoder is used to extract features from the multimodal data to obtain physiological signal features, discrete scoring features, and device feedback features. A cross-modal attention layer is used to perform attention calculation and feature alignment using the physiological signal features as query vectors and the discrete scoring features and device feedback features as keys, to obtain cross-modal features; A modal missing compensation layer is used to gating and adjusting the attention weights of the cross-modal features in conjunction with the missing mask; The fusion output layer is used to fuse the gated cross-modal features with the physiological signal features to generate fused features.

3. The multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation according to claim 1, characterized in that, Based on the fusion features, disease stage identification and risk assessment are performed to generate a rehabilitation pathway, including: The fused features are interacted with a preset disease stage embedding vector through cross-stage attention to obtain stage-enhanced features; the disease stage embedding vector corresponds to three stage labels: acute phase, subacute phase, and recovery phase. The enhanced stage features are input into a pre-constructed stage classifier to identify the current stage of the disease; The stage enhancement features are input into a pre-constructed path decision network to generate an initial rehabilitation path corresponding to the current stage of the disease. The fused features are input into the risk estimator to obtain a risk score; The control parameters in the initial rehabilitation path are adjusted based on the risk score to obtain the rehabilitation path.

4. The multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation according to claim 1, characterized in that, Determining the uncertainty quantification value corresponding to the rehabilitation path includes: The attention weights in the multimodal fusion network are sampled multiple times using a Bayesian cross-modal attention mechanism; The uncertainty quantification value corresponding to the rehabilitation path is calculated based on the attention weight distribution obtained from sampling.

5. The multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation according to claim 1, characterized in that, The triggering of manual review includes: The rehabilitation pathway, static medical record data, and uncertainty quantification values ​​are pushed to the remote expert collaboration platform, and manual review prompts are provided. The system receives review and adjustment instructions from experts through the remote expert collaboration platform, and updates the rehabilitation path according to the review and adjustment instructions to obtain the rehabilitation path after manual review.

6. The multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation according to claim 1, characterized in that, The method further includes: When the patient's physiological feedback data exceeds a preset safety threshold, and / or when the device's operating status data contains an abnormal status code, an emergency shutdown sequence is executed.

7. The multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation according to claim 1, characterized in that, The method further includes: The rehabilitation pathway, static medical record data, uncertainty quantification values, and patient physiological feedback data are pushed to a remote expert collaboration platform for visualization. When an adjustment instruction is received from an expert through the remote expert collaboration platform, the rehabilitation path is updated according to the adjustment instruction.

8. The multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation according to claim 4, characterized in that, The method further includes: An attention heatmap is generated based on the mean of the attention weight distribution; The attention heatmap is then pushed to a remote expert collaboration platform for visualization.

9. The multimodal intelligent decision-making and equipment control method for ICU critical care rehabilitation according to claim 1, characterized in that, The standardized equipment control instructions adopt a 16-byte standard instruction frame format, which includes a header code, instruction code, axis position parameters, operating parameters, and a verification field.

10. A multimodal intelligent decision-making and equipment control system for ICU critical care rehabilitation, characterized in that, include: The data acquisition unit is used to acquire static medical record data and multimodal data that characterize the health information of critically ill patients in the ICU; the multimodal data includes continuous physiological signal data, discrete manual scoring data, and feedback data from rehabilitation equipment. The preprocessing unit is used to standardize the multimodal data to generate standardized data and corresponding missing masks. A multimodal fusion unit is used to input the standardized data and missing mask into a pre-constructed multimodal fusion network to obtain fusion features; The rehabilitation decision-making unit is used to identify disease stages and assess risks based on the fusion features, so as to generate a rehabilitation path and determine the uncertainty quantification value corresponding to the rehabilitation path; the rehabilitation path includes equipment combination, operation mode and control parameters; The security verification unit is used to perform security verification on the rehabilitation path based on the static medical record data and uncertainty quantification value, and to trigger manual review when the security verification fails. The equipment control unit is used to generate standardized equipment control commands and send them to the corresponding rehabilitation equipment for execution based on the rehabilitation path that has passed safety verification or been manually reviewed. At the same time, it collects the operating status data of the rehabilitation equipment and the corresponding patient physiological feedback data. The closed-loop optimization unit is used to dynamically adjust the rehabilitation path based on the device operating status data and physiological feedback data, and to update the parameters of the multimodal fusion network using incremental learning and federated learning mechanisms.

Images