Medical monitoring method and equipment based on meta-learning and multi-modal monitoring information fusion

Abstract

The invention discloses a medical monitoring method and device based on meta-learning and multi-modal monitoring information fusion, and relates to the technical field of medical monitoring instruments.The method comprises the steps that a multi-modal monitoring information time sequence of a monitored person is input into a multi-modal fusion network, and a multi-modal signal time sequence is obtained; and inputting the multi-modal signal time sequence into the medical monitoring model to obtain a monitoring result. The monitoring result comprises the state and the transition probability of the disease trajectory and whether the illness state is worsened or not; the medical monitoring model is obtained by training an initial medical monitoring model according to multi-modal monitoring information by using meta-learning and hidden Markov principles. The method is applied to the medical monitoring system based on meta-learning and multi-modal monitoring information fusion, so that the medical monitoring system can complete disease trajectory prediction, and the intelligent level and the automation level of the medical monitoring system are further improved.

Description

Technical Field

[0001] This application relates to the technical field of medical monitoring devices, and particularly to a medical monitoring method and device based on meta-learning and multi-modal monitoring information fusion. Background Art

[0002] Stroke has currently become the second leading cause of death and disability worldwide. Compared with the management of ordinary acute strokes, critically ill cerebrovascular disease patients require more meticulous and personalized monitoring strategies. Research on exploring the trajectory prediction of critically ill patients and its correlation with clinical outcomes based on clinical and physiological information is particularly important. Existing research has found that imaging information, biochemical indicators, and physiological monitoring information are closely related to the trajectory prediction of severe strokes.

[0003] However, in the actual clinical environment, although complex information and multi-modal monitoring data are monitored in real time and limitedly recorded, only a small part of the data is screened for clinical analysis. These multi-dimensional and complex data usually require experienced doctors to interpret, but still face the risk of misjudgment. In particular, the individual characteristics of critically ill patients are significant, making it difficult to apply a generally applicable evaluation standard to the analysis of these massive data, further increasing the analysis difficulty. Currently, doctors often have difficulty fully understanding the relationship between continuously monitored electrophysiological data and the changes in the patient's condition through the existing knowledge framework. Traditional data analysis methods cause multi-dimensional and continuous monitoring data to lose its authenticity and integrity during the analysis process, and are regarded as "noise" or processed discontinuously. Therefore, there is an urgent need to develop new analysis tools to help clinicians deeply analyze these data and reveal the true meaning behind them. Summary of the Invention

[0004] The purpose of this application is to provide a medical monitoring method and device based on meta-learning and multi-modal monitoring information fusion, which can be applied to a medical monitoring system, enabling the medical monitoring system to complete disease trajectory prediction, thereby improving the intelligent level and automation level of the medical monitoring system.

[0005] To achieve the above purpose, this application provides the following solutions: In the first aspect, this application provides a medical monitoring method based on meta-learning and multi-modal monitoring information fusion. The medical monitoring method based on meta-learning and multi-modal monitoring information fusion is applied to a medical monitoring system; the medical monitoring system is used to obtain multi-modal monitoring information of the monitored person; The medical monitoring method based on meta-learning and multi-modal monitoring information fusion includes: Obtain the time series of multi-modal monitoring information of the monitored person; Input the time series of multimodal monitoring information of the monitored person into a multimodal fusion network to obtain a multimodal signal time series; the multimodal fusion network is obtained by training an initial multimodal fusion network according to multimodal monitoring information. Input the multimodal signal time series into a medical monitoring model to obtain a monitoring result; the monitoring result includes a first classification task result and a second classification task result; the first classification task result is the state and transition probability of a disease trajectory; the second classification task result is whether the condition will deteriorate; the medical monitoring model is obtained by training an initial medical monitoring model according to multimodal monitoring information using meta-learning and the implicit Markov principle.

[0006] Optionally, the multimodal monitoring information includes: imaging data, physiological monitoring data, and structured / unstructured electronic medical record data.

[0007] Optionally, before obtaining the time series of multimodal monitoring information of the monitored person, it further includes: Obtain the initial time series of multimodal monitoring information of multiple historical monitored persons. Clean the physiological monitoring data in the initial time series of multimodal monitoring information to obtain the historical time series of multimodal monitoring information of multiple historical monitored persons. Select the historical time series of multimodal monitoring information corresponding to multiple historical monitored persons without the target disease to construct a support set. Select the historical time series of multimodal monitoring information corresponding to multiple historical monitored persons without the target disease, and the historical time series of multimodal monitoring information corresponding to multiple historical monitored persons with the target disease to construct a query set. Use the support set to train the initial multimodal fusion network to obtain a multimodal fusion network; the multimodal fusion network includes a deep autoencoder and a deep auto-decoder; both the deep autoencoder and the deep auto-decoder are embedded with a cross-modal attention mechanism. Use the support set and the query set to train the initial medical monitoring model using meta-learning and the implicit Markov principle to obtain a medical monitoring model.

[0008] Optionally, using the support set to train the initial multimodal fusion network to obtain a multimodal fusion network includes: Preprocess each of the multiple historical multimodal monitoring information in the historical time series of multimodal monitoring information in the support set to obtain multiple preprocessed historical multimodal monitoring information. Input the preprocessed historical multimodal monitoring information into the deep autoencoder to obtain historical multimodal signals. Input the historical multimodal signals into the deep auto-decoder to obtain the historical multimodal reconstruction information; Traverse the support set, and determine the multimodal fusion network loss according to multiple preprocessed historical multimodal monitoring information and multiple historical multimodal reconstruction information; Based on the multimodal fusion network loss, use the gradient descent method to adjust the parameters of the initial multimodal fusion network; Return to the step "Input the preprocessed historical multimodal monitoring information into the deep auto-encoder to obtain the historical multimodal signals" until the multimodal fusion network loss is within the preset multimodal fusion network loss range; Determine that the deep auto-encoder is the multimodal fusion network.

[0009] Optionally, the deep auto-encoder is: ; Wherein, represents the historical multimodal signal; represents the deep auto-encoder; represents the historical multimodal signal, represents the neural network activation function; represents the weight matrix of the deep auto-encoder; represents the bias vector of the deep auto-encoder; The deep auto-decoder is: ; Wherein, represents the historical multimodal reconstruction information; represents the deep auto-decoder; represents the weight matrix of the deep auto-decoder; represents the bias vector of the deep auto-decoder.

[0010] Optionally, use the support set and the query set to train the initial medical monitoring model by using meta-learning and the implicit Markov principle to obtain the medical monitoring model, including: According to the support set and the deep auto-encoder, obtain multiple historical multimodal signal sequences; Obtain the first classification task result and the second classification task result corresponding to each historical multimodal signal sequence in the support set; According to the query set and the deep auto-encoder, obtain multiple historical multimodal signal sequences; Obtain the first classification task result and the second classification task result corresponding to each historical multimodal signal sequence in the query set; Construct the initial medical monitoring model; Input multiple historical multimodal signal sequences in the support set into the initial medical monitoring model to obtain the predicted values of the first classification task result and the second classification task result for each historical multimodal signal sequence in the support set; Determine the loss value of the first classification task based on the predicted value of the first classification task result and the result of the first classification task for each historical multimodal signal sequence in the support set; Determine the loss value of the second classification task based on the predicted value of the second classification task result and the result of the second classification task for each historical multimodal signal sequence in the support set; Determine the sum of the loss value of the first classification task and the loss value of the second classification task as the total loss value; Adjust the parameters of the Transformer encoder in the initial medical monitoring model based on the total loss value, and return to the step "Input multiple historical multimodal signal sequences in the support set into the initial medical monitoring model to obtain the predicted values of the first classification task result and the second classification task result for each historical multimodal signal sequence in the support set" until the total loss value is within the preset total loss value range, and complete the inner loop training stage of meta-learning; Input multiple historical multimodal signal sequences in the query set into the initial medical monitoring model to obtain the predicted values of the first classification task result and the second classification task result for each historical multimodal signal sequence in the query set; Determine the loss value of the first classification task based on the predicted value of the first classification task result and the result of the first classification task for each historical multimodal signal sequence in the query set; Determine the loss value of the second classification task based on the predicted value of the second classification task result and the result of the second classification task for each historical multimodal signal sequence in the query set; Determine the sum of the loss value of the first classification task and the loss value of the second classification task as the total loss value; Adjust the parameters of the Transformer encoder in the initial medical monitoring model based on the total loss value, and return to the step "Input multiple historical multimodal signal sequences in the query set into the initial medical monitoring model to obtain the predicted values of the first classification task result and the second classification task result for each historical multimodal signal sequence in the query set" until the total loss value is within the preset total loss value range, and complete the outer loop training stage of meta-learning, and determine the Transformer encoder as the meta-learning adaptive model; Train the meta-learning adaptive model using the support set and adjust the parameters of the meta-learning adaptive model to obtain the medical monitoring model.

[0011] Optionally, inputting multiple historical multimodal signal sequences in the support set into the initial medical monitoring model to obtain the predicted values of the first classification task result and the second classification task result for each historical multimodal signal sequence in the support set includes: Determine any historical multi-modal signal sequence as the current historical multi-modal signal sequence; Input the historical multi-modal signal sequence into the Transformer encoder to obtain information encoding; Input the information encoding into the position encoding layer to obtain the position encoding of the information encoding; Monitor the information encoding to obtain statistical features, and construct an automatic early warning model based on the statistical features and upper and lower limits; the automatic early warning model is used to output the prediction value of the second classification task result; Perform clustering processing on the information encoding using a clustering algorithm; Monitor the information encoding after clustering processing, and construct a disease trajectory transition model using a hidden Markov model; the disease trajectory transition model is used to output the prediction value of the first classification task result; Input the position encoding, the prediction value of the first classification task result, and the prediction value of the second classification task result into the Transformer decoder to obtain a decoding result; the Transformer decoder includes a multi-head attention network and a feed-forward network connected in sequence.

[0012] Optionally, the loss function of the first classification task is: ; Wherein, represents the loss function of the first classification task; N represents the number of samples, represents the number of label categories; represents the true label of the i-th sample in the c-th category, represents the probability that the disease trajectory transition model predicts the i-th sample in the c-th category.

[0013] Optionally, the loss function of the second classification task is: ; Wherein, represents the loss function of the second classification task; is the true label of the i-th sample, is the probability that the automatic early warning model predicts the deterioration of the condition of the i-th sample.

[0014] In a second aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the above-mentioned medical monitoring method based on meta-learning and multi-modal monitoring information fusion.

[0015] According to the specific embodiments provided by the present application, the following technical effects are disclosed in the present application: The present application provides a medical monitoring method and device based on meta-learning and multi-modal monitoring information fusion. The innovative combination of static multi-dimensional clinical and imaging information, unstructured medical record information, and high-volume dynamic monitoring data is a new direction for the development of future intelligent intensive care. Expanding from a static prediction model to a dynamic continuous real-time prediction + early warning model attempts to break through traditional medical models. Using meta-learning and adaptive technologies for adaptive variable selection and model construction can be applied to the update of patients and different ICU scenarios, adapting to different monitoring levels, different intensive care units, and different information input sources, and having good model generalizability. The constructed multi-task interpretable medical model with multi-level outcome prediction, multi-trajectory description, and early warning information recognition better meets the needs of clinical applications. Expanding from the model after endovascular treatment to the model of acute severe cerebrovascular diseases, the model adapts to a wider range of scenarios. In summary, the acute severe cerebrovascular disease trajectory prediction model constructed in the present application has good scientific research prospects and clinical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 It is a flowchart of a medical monitoring method based on meta-learning and multi-modal monitoring information fusion in an embodiment of the present application; Figure 2 It is a structural diagram of a multi-modal fusion network in an embodiment of the present application; Figure 3 It is a structural diagram of a Transformer network in an embodiment of the present application; Figure 4 It is a schematic diagram of a statistical process control early warning model in an embodiment of the present application; Figure 5 It is a structural diagram of a meta-learning algorithm network in an embodiment of the present application; Figure 6 It is a flowchart of meta-learning adaptive model training and testing in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0019] To make the above objects, features, and advantages of the present application more apparent and understandable, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The early exploration of automation engineering and artificial intelligence technologies in the medical field has brought new possibilities for the prediction and treatment of acute and critical cerebrovascular diseases, especially in the applications of disease trajectory prediction, personalized treatment, and the rehabilitation process. AI technology can process and analyze a large amount of complex medical data, thereby identifying disease patterns, optimizing treatment plans, and monitoring the disease progression of patients. However, many scholars have also pointed out that although the models combining multi-modal monitoring data and AI have shown superior performance on the data set, their generalization ability and application effects in the actual clinical environment still need to be further verified and optimized. The currently applied technologies have limitations in accurately monitoring and predicting the changes in the condition of acute and critical cerebrovascular diseases, especially in the early identification of potential neurological deterioration. Despite the significant progress in medical imaging technologies and physiological monitoring devices in recent years, challenges still exist in real-time data analysis, cross-modal data integration, and decision support based on these data. There are a series of challenges when translating these technologies into useful tools in clinical practice. Although some promising critical care models have been developed by researchers currently, such as the early assessment of sepsis, fluid management, and the prediction model of the mortality of critically ill patients, etc., the successful application of these published critical disease trajectory models depends on high-quality retrospective data sets. In the actual clinical environment, due to the heterogeneity of the environment and parameters, the acquisition and integration of data face huge challenges, and few models are truly applied to the ICU clinical practice. This phenomenon is called the "AI gap", which refers to the fact that offline, pure computational verification models cannot successfully achieve real-world clinical deployment.

[0021] The reasons are as follows. First of all, the training and application of AI models require a large amount of high-quality data. In the actual medical environment, collecting such data faces problems such as privacy protection, data standardization, and data integrity. In addition, the physiological and pathological differences between different monitoring environments and different patients make it difficult for AI models to generalize to all individuals, especially when facing complex and changing disease states. Secondly, the interpretability issue of AI models is also an important challenge. The "black box" nature of the AI decision-making process makes it difficult for doctors and patients to understand the reasons for the recommended diagnosis and treatment decisions. This lack of transparency may affect doctors' trust in and adoption of AI technology. Despite these challenges, the potential of AI in the management of acute and critical cerebrovascular diseases is huge. Improving data collection and processing methods, enhancing the interpretability of AI models, and developing adaptive algorithms that can adapt to environmental and individual differences may be important directions for the future development of the medical AI field. The development and application of new-generation artificial intelligence algorithms such as meta-learning, multi-modal fusion, and multi-task learning in the medical field, as well as integrating more extensive and real multi-scenario data sources, have opened up new ideas for diagnosis, treatment, and disease management. The application of these algorithms can more effectively process and analyze large medical data sets, such as electronic health records, medical images, genomic data, and dynamic multi-modal physiological signals from different monitoring devices, improve the early detection ability of diseases and the degree of personalized treatment, and enhance the generalization ability and clinical application value of the model through training and iteration in a wider patient population. The combination of meta-learning and multi-modal information fusion further enhances the accuracy and robustness of the prediction model, providing strong support for clinical decision-making and personalized treatment. In addition, attempting to combine adaptive control with the prediction model can, compared with traditional prediction models, dynamically adjust learning strategies and parameter configurations according to new data, adapt to individual differences and complex disease patterns in the medical field, and thus improve the personalization and accuracy of prediction. This flexibility and precision are the keys to processing complex and dynamic medical data, opening up new paths for improving treatment effects and patient management levels.

[0022] In view of the critical and changeable conditions of acute and severe cerebrovascular diseases and the limitations of existing diagnosis and treatment methods, accurate prediction of disease trajectories and individualized decision-making have become key requirements and important challenges for improving the prognosis of patients. Previous studies have shown that combining multimodal monitoring information with deep learning has the potential to predict the disease trajectories of acute and severe cerebrovascular diseases, but there are problems such as difficult data processing, poor self-adaptability, and lack of clinical interpretability. Previous studies have found that methods such as meta-learning and multimodal information fusion are expected to overcome the above problems. Accordingly, in an exemplary embodiment, a medical monitoring method based on the fusion of meta-learning and multimodal monitoring information is provided. Using a deep neural network developed with a meta-learning strategy as the core, complex multidimensional clinical image data and electrophysiological signals are extracted and integrated. At the same time, the implicit Markov model and statistical process control are used for multi-task learning to achieve the goals of accurately predicting the multi-level outcomes and multi-stage trajectories of patients with severe cerebrovascular diseases and timely identifying the deterioration of the condition. By integrating technologies such as meta-learning and multimodal information, this method is expected to achieve the real-time automatic prediction of the disease trajectories of acute and severe cerebrovascular diseases for the first time, overcome the technical problems in the prediction of the trajectories of acute and severe cerebrovascular diseases, and promote the research progress and clinical practice of the diagnosis and treatment of acute and severe cerebrovascular diseases.

[0023] The medical monitoring method based on the fusion of meta-learning and multimodal monitoring information is applied to a medical monitoring system. The medical monitoring system is used to obtain the multimodal monitoring information of the monitored person.

[0024] Such as Figure 1 , the medical monitoring method based on the fusion of meta-learning and multimodal monitoring information includes: Step 101: Obtain the time series of multimodal monitoring information of the monitored person. The multimodal monitoring information includes: image data, physiological monitoring data, and structured / unstructured electronic medical record data.

[0025] Step 102: Input the time series of multimodal monitoring information of the monitored person into the multimodal fusion network to obtain the time series of multimodal signals. The multimodal fusion network is obtained by training the initial multimodal fusion network according to the multimodal monitoring information.

[0026] Step 103: Input the time series of multimodal signals into the medical monitoring model to obtain the monitoring results. The monitoring results include the results of the first classification task and the results of the second classification task. The results of the first classification task are the status and transition probability of the disease trajectory. The results of the second classification task are whether the condition will deteriorate. The medical monitoring model is obtained by training the initial medical monitoring model according to the multimodal monitoring information using meta-learning and the implicit Markov principle.

[0027] Before step 101, it further includes: Step 104: Obtain the initial time series of multimodal monitoring information of multiple historical monitored persons.

[0028] Step 105: Clean the physiological monitoring data in the initial multi-modal monitoring information time series to obtain the historical multi-modal monitoring information time series of multiple historical subjects.

[0029] Step 106: Select the historical multi-modal monitoring information time series corresponding to multiple historical subjects without the target disease to construct a support set.

[0030] Step 107: Select the historical multi-modal monitoring information time series corresponding to multiple historical subjects without the target disease and the historical multi-modal monitoring information time series corresponding to multiple historical subjects with the target disease to construct a query set.

[0031] The clinical and imaging data collected in the study have good consistency (the preliminary data cleaning has been completed) and can be directly used as the dataset for analysis. Mainly, data cleaning is performed on the electrophysiological monitoring information. Data cleaning is mainly carried out from two aspects. The first step is to process the missing data, which can be completed based on clinical analysis and model parameter optimization. The second step is to process the discrete and "noise" using the Binning method. The third step is to normalize all the data. Clean all the retrospective data information and match and establish a modeling dataset. Select the part of the normal group patients (no deterioration during the NICU period) in the Neurological Intensive Care Unit (NICU) as the model support set, and select the abnormal group (deterioration occurred during the NICU period and trajectory transition occurred) and part of the normal group to construct a mixed group as the model query set.

[0032] Step 108: Use the support set to train the initial multi-modal fusion network to obtain a multi-modal fusion network. The multi-modal fusion network includes a deep autoencoder and a deep decoder. Both the deep autoencoder and the deep decoder are embedded with a cross-modal attention mechanism.

[0033] Step 108 includes: Step 108-1: Preprocess each of the multiple historical multi-modal monitoring information in each historical multi-modal monitoring information time series in the support set to obtain multiple preprocessed historical multi-modal monitoring information.

[0034] Step 108-2: Input the preprocessed historical multi-modal monitoring information into the deep autoencoder to obtain a historical multi-modal signal. The deep autoencoder is: .

[0035] Wherein, represents the historical multi-modal signal. represents the deep autoencoder. represents historical multimodal signals, represents the neural network activation function. represents the weight matrix of the deep autoencoder. represents the bias vector of the deep autoencoder.

[0036] Step 108-3: Input the historical multimodal signals into the deep auto-decoder to obtain historical multimodal reconstruction information. The deep auto-decoder is: .

[0037] where, represents the historical multimodal reconstruction information. represents the deep auto-decoder. represents the weight matrix of the deep auto-decoder. represents the bias vector of the deep auto-decoder.

[0038] Step 108-4: Traverse the support set, and determine the multimodal fusion network loss according to multiple preprocessed historical multimodal monitoring information and multiple historical multimodal reconstruction information.

[0039] Step 108-5: Based on the multimodal fusion network loss, use the gradient descent method to adjust the parameters of the initial multimodal fusion network.

[0040] Step 108-6: Return to Step 108-2 until the multimodal fusion network loss is within the preset multimodal fusion network loss range.

[0041] Step 108-7: Determine that the deep autoencoder is the multimodal fusion network.

[0042] Combine different modalities of data in the modeling data support set, effectively utilize the complementary, redundant, and collaborative features of different modalities, embed the cross-modal attention mechanism as a whole into the multi-modal fusion network, and process the multi-modal fusion network. Then integrate the data of different modalities and generate the corresponding spatial relationship of the feature information between multiple modalities of data, so as to map this feature information to the subsequent early warning and prediction models. The specific steps of multi-modal fusion are as follows: First, construct a multi-modal fusion network to preprocess the data of four modalities, namely image data, monitoring data, structured data, and unstructured electronic medical records, to ensure the same size and data type. Then, extract features and perform cross-modal attention weighting to better fuse the feature information between different modalities and avoid unbalanced modal information weights when fusing features. To further process feature fusion, the generated feature volume blocks will be output as individual channels respectively. Finally, the feature set after batch normalization operation will be convolved. For the multi-modal data of this application, convolution can reduce the computational complexity, effectively improve the computational efficiency of the network and the overall performance of the model, and input it into the subsequent early warning and prediction models. After the above steps, the cross-modal fusion network will effectively fuse the multi-modal data support set, make up for the heterogeneity differences of different modalities, output a multi-modal signal with a unified feature representation, and improve the performance and performance of subsequent model construction and training. The structure diagram of the multi-modal fusion network is as Figure 2 shown.

[0043] Train an autoencoder model based on high-dimensional and high-volume multi-modal signals. In the support set (normal group), use a deep autoencoder as the basic learning model to extract features from high-dimensional input signals, so as to obtain an integrated corrected information code. In order to effectively and quickly adapt to new tasks, construct an optimization objective and automatically adjust the model structure. In the decoding and encoding of the basic learning model, the encoder maps the input signal of the multi-modal network to the hidden layer, where are the vector forms of image data, monitoring data, and the vector form after splicing structured data and unstructured data respectively. The hidden layer outputs low-dimensional feature quantities , where are the reduced-dimensional vectors of image data, monitoring data, and the spliced structured data and unstructured data respectively, completing the dimensionality reduction of the input data; the decoder maps the feature quantities to the output layer to obtain the reconstructed signal , where are the reconstructed vectors of image data, monitoring data, and the spliced structured data and unstructured data respectively, completing the restoration of the input signal. The encoding and decoding processes of the deep autoencoder can be expressed as follows: .

[0044] 。

[0045] In the formula, are respectively the weight matrix and bias vector of the encoder are respectively the weight matrix and bias vector of the decoder, The neural network activation function adopts the Sigmoid function, and the formula is as follows: 。

[0046] The deep autoencoder aims to minimize the loss function and continuously optimizes the network parameters using the gradient descent method, ultimately minimizing the error between the reconstructed signal y and the input signal x. The loss function of the deep autoencoder is expressed as: 。

[0047] Among them, c is the dimension of the input signal, that is, the number of features of the input data, is the input signal the th eigenvalue, is the input signal the th eigenvalue, λ is the weight decay parameter, is the number of hidden layers of the deep autoencoder, is the number of neurons in the a-th layer, is the weight coefficient between the q-th neuron in the a-th layer and the r-th neuron in the a+1-th layer.

[0048] Step 109: Use the support set and the query set to train the initial medical monitoring model using meta-learning and the implicit Markov principle to obtain the medical monitoring model.

[0049] Step 109 includes: Step 109-1: Obtain multiple historical multi-modal signal sequences according to the support set and the deep autoencoder.

[0050] Step 109-2: Obtain the first classification task result and the second classification task result corresponding to each historical multi-modal signal sequence in the support set.

[0051] Step 109-3: Obtain multiple historical multi-modal signal sequences according to the query set and the deep autoencoder.

[0052] Step 109-4: Obtain the first classification task result and the second classification task result corresponding to each historical multi-modal signal sequence in the query set.

[0053] Step 109-5: Construct the initial medical monitoring model.

[0054] Step 109-6: Input multiple historical multimodal signal sequences in the support set into the initial medical monitoring model to obtain the predicted values of the first classification task result and the second classification task result for each historical multimodal signal sequence in the support set.

[0055] Step 109-6 includes: Step 109-6-1: Determine any historical multimodal signal sequence as the current historical multimodal signal sequence.

[0056] Step 109-6-2: Input the historical multimodal signal sequence into the Transformer encoder to obtain information encoding.

[0057] Step 109-6-3: Input the information encoding into the position encoding layer to obtain the position encoding of the information encoding.

[0058] Step 109-6-4: Monitor the information encoding to obtain statistical features, and construct an automatic early warning model based on the statistical features and upper and lower limits. The automatic early warning model is used to output the predicted value of the second classification task result.

[0059] Step 109-6-5: Perform clustering processing on the information encoding using a clustering algorithm.

[0060] Step 109-6-6: Monitor the information encoding after clustering processing, and construct a disease trajectory transition model using a hidden Markov model. The disease trajectory transition model is used to output the predicted value of the first classification task result.

[0061] Step 109-6-7: Input the position encoding, the predicted value of the first classification task result, and the predicted value of the second classification task result into the Transformer decoder to obtain a decoding result. The Transformer decoder includes a multi-head attention network and a feed-forward network connected in sequence. The input of the Transformer decoder directly depends on the outputs of two models, where the automatic early warning model provides the predicted value of the real-time abnormal early warning signal, which is generated based on the statistical features and threshold judgment. The disease trajectory transition model provides the predicted value of the disease stage or state transition probability result, which is obtained by analyzing the clustered information encoding through a hidden Markov model. The decoder realizes the unified representation of multi-task information by integrating two types of prediction results (early warning + trajectory) and the position encoding of the original signal.

[0062] Step 109-7: Based on the predicted value of the first classification task result and the first classification task result for each historical multimodal signal sequence in the support set, determine the first classification task loss value. The first classification task loss function is: .

[0063] Where, represents the first classification task loss function. N represents the number of samples, Indicates the number of label categories. Indicates the true label of the i-th sample in the c-th category, Indicates the probability of the i-th sample predicted by the disease trajectory transition model in the c-th category.

[0064] Step 109-8: Determine the second classification task loss value based on the predicted value and the result of the second classification task for each historical multimodal signal sequence in the support set. The second classification task loss function is: .

[0065] Where, Indicates the second classification task loss function. is the true label of the i-th sample, is the probability of the i-th sample's condition deterioration predicted by the automatic early warning model.

[0066] Step 109-9: Determine the sum of the first classification task loss value and the second classification task loss value as the total loss value.

[0067] Step 109-10: Adjust the parameters of the Transformer encoder in the initial medical monitoring model based on the total loss value, and return to Step 109-6 until the total loss value is within the preset total loss value range, completing the inner loop training stage of meta-learning.

[0068] Step 109-11: Input multiple historical multimodal signal sequences in the query set into the initial medical monitoring model to obtain the predicted values and results of the first classification task for each historical multimodal signal sequence in the query set.

[0069] Step 109-12: Determine the first classification task loss value based on the predicted value and the result of the first classification task for each historical multimodal signal sequence in the query set.

[0070] Step 109-13: Determine the second classification task loss value based on the predicted value and the result of the second classification task for each historical multimodal signal sequence in the query set.

[0071] Step 109-14: Determine the sum of the first classification task loss value and the second classification task loss value as the total loss value.

[0072] Step 109-15: Adjust the parameters of the Transformer encoder in the initial medical monitoring model based on the total loss value, and return to Step 109-11 until the total loss value is within the preset total loss value range, completing the outer loop training stage of meta-learning, and determining the Transformer encoder as the meta-learning adaptive model.

[0073] Step 109-16: Use the support set to train the meta-learning adaptive model, adjust the parameters of the meta-learning adaptive model, and obtain the medical monitoring model.

[0074] Verify and improve the model effectiveness using the query set patients (mixed group). This application uses the contribution degree of relevant features in the dataset and defines the output score Score. The disease is defined as different degrees according to the physiological conditions and clinical manifestations of patients at different stages. There may also be sub-stages under each stage. The entire stage may be upward (regression or improvement), downward (deterioration or death), or even horizontal (stable). The model performs multi-outcome prediction to draw the disease trajectory. At the same time, the clinical event trajectories that lead to different clinical trajectory changes are also depicted and predicted as a whole (taking cerebral edema and multiple organ dysfunction as examples). Develop a Transformer-based supervision method, perform a time series clustering algorithm according to information encoding, and finally use the hidden Markov state for sequence modeling to obtain the probability value of state transition. The position encoding layer of the Transformer is used to capture position or time information, the decoding layer of the Transformer is used to calculate the correlation of different sub-vectors, and pays more attention to the key sub-vectors. The classification layer of the clustering module is used to generate the final classification output. The structure diagram is as Figure 3 shown.

[0075] Let be the encoder result. If it is immediately input into the decoder, the sequential information or time information will be lost. This application uses the sine function to represent the odd sub-vectors and the cosine function to represent the even sub-vectors, as follows: .

[0076] .

[0077] where m represents the position. The input of the decoder is defined as plus the position encoding process, as shown in the formula: 。

[0078] This application only uses a decoder layer consisting of two sub-networks. One sub-network is a multi-head attention network, and the other sub-network is a feed-forward network. Several special properties of the attention mechanism contribute greatly to its excellent performance. Another advantage is that it can capture global connections, which means it can make full use of discontinuous sub-vectors to improve accuracy. The attention mechanism adopted is scaled dot-product attention. This is done by calculating the weighted sum of sub-vectors, where the weights are determined by the softmax function and applied to a compatibility function that measures the similarity between the current sub-vector and other sub-vectors. The formula for the attention mechanism is as follows: 。

[0079] where, . Although self-attention can use adaptive weights and focus on all sub-vectors, there are still some non-linear features that are not captured. Therefore, the feed-forward network is used to increase non-linearity. The feed-forward layer contains two linear layers with a rectified linear activation function as the activation function.

[0080] Finally, based on multi-modal data as input, a disease trajectory transition model is established using the hidden Markov model, which describes the process of randomly generating an unobservable state random sequence by a hidden Markov chain, and then generating an observation from each state to produce an observation random sequence. The sequence of states randomly generated by the hidden Markov chain is called the state sequence; each state generates a rule, and the resulting random sequence of observations is called the observation sequence. Through the trained deep autoencoder, transfer learning is used to extract the relevant features of different possible factors (such as cerebral edema, infection) that cause the deterioration of the disease state, and the estimation of the changes in the state of multiple clinical events is extended to the overall disease state estimation of critically ill cerebrovascular patients. The obtained comprehensive score Score is used as a parameter to detect abnormalities, and it is used to predict the development of the disease. A prediction layer is added to the disease trajectory transition model to construct a trajectory transition warning model. The formula is as follows: .

[0081] where, As the encoder result, is the prediction function, are the parameters of the prediction layer. At the same time, this application uses the exponentially weighted moving-average control chart (EWMA) of statistical process control to estimate the control limits of the warning, and obtains the confidence prediction intervals UCL (upper control limit) and LCL (lower control limit). When the model reaches the limit in the confidence prediction interval, an alarm (warning) is issued, that is When the value is higher than the UCL or lower than the LCL, it indicates a potential deterioration of the future condition. Continue to repeat the above steps until the sample reaches the theoretical optimum of the model. The statistical process control warning model is as Figure 4 shown.

[0082] The defining formula for the exponentially weighted moving average control chart is:

[0083] where the value range of the constant λ is 0 < λ ≤ 1, is the EWMA statistic, which is the weighted average of all previous sample means. The initial value of this formula (when i = 1) takes the target value of the process (i.e., in ), and sometimes the mean of the initial data is also used as the initial value, i.e., . If the observed value is an independent random variable with variance , then the variance of is: .

[0084] Therefore, the vertical axis of the EWMA control chart is , the horizontal axis is the sample number or time, and the calculation formulas for the center line and control limits are: .

[0085]

[0086] .

[0087] Note the part in the formula. When gradually increases, will quickly converge to 0. Therefore, when increases, the UCL and LCL will stabilize to the following two values, which is the control limit convergence of EWMA: .

[0088] .

[0089] This model adopts Multi-Task Learning (MTL), aiming to learn multiple related tasks simultaneously to improve the generalization performance of each task. It mainly includes two tasks: one is to predict the state and transition probability of the disease trajectory, and the other is to warn of the risk of disease deterioration. This method lies in leveraging the commonalities between tasks to improve learning efficiency and effectiveness through shared representations. Task 1 is a classification task that predicts the state and transition probability of the disease. This application uses the cross-entropy loss function to measure the difference between the model's prediction and the true label. Suppose this application has C categories (disease states), is the true label of the one-hot encoding of the i-th sample, is the probability distribution predicted by the model, then the loss function of Task 1 is expressed as: .

[0090] where: N is the number of samples, is the true label (0 or 1) of the i-th sample in the c-th category, is the probability of the i-th sample predicted by the model in the c-th category.

[0091] Task 2 is a binary classification task that predicts whether the disease condition will deteriorate. This application uses the binary cross-entropy loss function to measure the difference between the model's prediction and the true label. Suppose is the true label (0 or 1) of the i-th sample, is the probability predicted by the model (i.e., the probability of disease deterioration), then the loss function of Task 2 can be expressed as: .

[0092] where: N is the number of samples, is the true label (0 or 1) of the i-th sample, is the probability of the i-th sample predicted by the model for disease deterioration.

[0093] The goal of the multi-task learning model is to minimize the total loss of all tasks as: .

[0094] where, is the loss function of the -th task, is the parameter specific to the -th task, is the parameter shared across tasks, is the The weight of each task indicates the importance of the task. The key to designing this multi-task learning lies in parameter sharing and the design of task-specific parameters. Parameter sharing enables the backbone network to learn a common representation across tasks, while task-specific parameters enable the model to be optimized for the characteristics of each task.

[0095] With the deep autoencoder as the backbone network, multi-task learning of trajectory state prediction model and disease trajectory transition warning model is carried out. Meta-learning algorithm is introduced to build a meta-learning adaptive model, and the gradient descent and parameter update of the autoencoder are adjusted to improve the adaptive ability. The network structure diagram is shown in the figure below. Figure 5 The specific process is as shown in Figure 6 shown.

[0096] First, in the meta-learning framework, the training process involves a two-layer loop structure, namely an inner loop and an outer loop, to achieve knowledge transfer and generalization from the source domain to the target domain. In the inner loop of meta-learning, the initial step includes randomly initializing the model parameters to facilitate subsequent rapid adaptation. Next, a batch of support sets are selected from the source domain dataset, which are used to simulate different learning tasks. Next, the training gradients for each task are calculated through the back-propagation algorithm. These gradients are then used to update the model parameters to minimize the loss function. This process is repeated in multiple iterations to optimize the parameters of the model. After completing a round of inner loop training, the model will select a query set from the source domain dataset to verify the effectiveness of the parameter update in the inner loop and improve the model's generalization ability between different tasks. When all the scheduled training rounds are completed, a meta-learning adaptive model is formed, which can show excellent rapid adaptation when facing new tasks. In the testing phase of meta-learning, the meta-model uses a new dataset from the target domain for performance evaluation. Subsequently, the intermediate representation layer (encoding layer) of the trained meta-learning adaptive model is extracted and integrated into the backbone network as a feature extractor to build a feature learning framework. By cascading different decoder components, each decoder is usually responsible for processing a specific task or feature. As the feature extractor of the backbone network, high-dimensional features are gradually parsed and reconstructed for multi-tasks, and the trajectory transition warning model and trajectory state prediction model are trained and optimized. In the framework of cascading different decoder components, each decoder is usually responsible for processing a specific task or feature. The differences between these decoders are mainly reflected in the following aspects: 1. Different task objectives; 2. Different input features; 3. Different output forms; 4. Different model structures.

[0097] Through the training process of the autoencoder, a set of optimal initialization parameters (such as weights and biases) can be obtained. These parameters are usually used as the initial parameters of the meta-model. The meta-model is usually a model that can generate task-specific models. The meta-model learns how to share knowledge between different tasks, so as to quickly adapt to new tasks. The meta-model can generate a task-specific model according to the requirements of the new task. The initial parameters of this task-specific model are usually provided by the meta-model, and these parameters are based on the optimal initialization parameters learned by the autoencoder. In the meta-learning framework, the task-specific model will be fine-tuned according to the data of the new task. Since the initial parameters are already close to the optimal, the model only needs a small amount of data and iteration times to achieve good performance. Finally, the meta-model can generate a task-specific model according to the new task and fine-tune the model through an adaptive learning process. Since the initial parameters have been optimized, the task-specific model can quickly adapt to the new task and perform well on a small amount of data.

[0098] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a medical monitoring method based on meta-learning and multi-modal monitoring information fusion.

[0099] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0100] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0102] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the various embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0103] The databases involved in the various embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the various embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0104] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0105] In this article, specific examples are used to illustrate the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A medical monitoring method based on meta-learning and multimodal monitoring information fusion, characterized in that: The medical monitoring method based on meta-learning and multimodal monitoring information fusion is applied to a medical monitoring system; The medical monitoring system is used to obtain multimodal monitoring information of the monitored person; The medical monitoring method based on meta-learning and multimodal monitoring information fusion includes: Obtaining the time series of multimodal monitoring information of the monitored person; Inputting the multimodal monitoring information time series of the monitored person into a multimodal fusion network to obtain a multimodal signal time series; the multimodal fusion network is obtained by training an initial multimodal fusion network based on the multimodal monitoring information; The multimodal signal time series is input into the medical monitoring model to obtain a monitoring result; the monitoring result includes a first classification task result and a second classification task result; the first classification task result is the state and transition probability of the disease trajectory; the second classification task result is whether the condition will worsen; the medical monitoring model is obtained by training the initial medical monitoring model based on the multimodal monitoring information using meta-learning and hidden Markov principle.

2. The medical monitoring method based on meta-learning and multimodal monitoring information fusion according to claim 1, characterized in that: The multimodal monitoring information includes: imaging data and physiological monitoring data, and structured / unstructured electronic medical record data.

3. The medical monitoring method based on meta-learning and multimodal monitoring information fusion according to claim 1, characterized in that: Before obtaining the multimodal monitoring information time series of the monitored person, it also includes: Obtaining initial multimodal monitoring information time series of multiple historical monitored persons; Cleaning the physiological monitoring data in the initial multimodal monitoring information time series to obtain historical multimodal monitoring information time series of multiple historical monitored persons; Select multiple historical multimodal monitoring information time series corresponding to historical monitored persons who do not suffer from the target disease to construct a support set; Selecting a plurality of historical multimodal monitoring information time series corresponding to historical monitored persons who do not suffer from the target disease, and a plurality of historical multimodal monitoring information time series corresponding to historical monitored persons who suffer from the target disease, to construct a query set; The support set is used to train the initial multimodal fusion network to obtain a multimodal fusion network; the multimodal fusion network includes a deep autoencoder and a deep autodecoder; the deep autoencoder and the deep autodecoder are both embedded with a cross-modal attention mechanism; The support set and the query set are used to train the initial medical monitoring model using meta-learning and hidden Markov principle to obtain a medical monitoring model.

4. The medical monitoring method based on meta-learning and multimodal monitoring information fusion according to claim 3, characterized in that: The initial multimodal fusion network is trained using the support set to obtain a multimodal fusion network, including: Preprocessing the multiple historical multimodal monitoring information in each historical multimodal monitoring information time series in the support set respectively to obtain multiple preprocessed historical multimodal monitoring information; Input the preprocessed historical multimodal monitoring information into a deep autoencoder to obtain a historical multimodal signal; Input the historical multimodal signal into the deep automatic decoder to obtain the historical multimodal reconstruction information; Traversing the support set, and determining a multimodal fusion network loss according to a plurality of preprocessed historical multimodal monitoring information and a plurality of historical multimodal reconstruction information; Based on the multimodal fusion network loss, adjusting the parameters of the initial multimodal fusion network using a gradient descent method; Return to step "inputting the preprocessed historical multimodal monitoring information into the deep autoencoder to obtain the historical multimodal signal" until the multimodal fusion network loss is within the preset multimodal fusion network loss range; Identify deep autoencoders as multimodal fusion networks.

5. The medical monitoring method based on meta-learning and multimodal monitoring information fusion according to claim 4, characterized in that: The deep autoencoder is: ； in, Represents historical multimodal signals; represents a deep autoencoder; represents the historical multimodal signal, represents the neural network activation function; Represents the weight matrix of the deep autoencoder; represents the bias vector of the deep autoencoder; The deep auto-decoder is: ； in, Represents historical multimodal reconstruction information; represents a deep autodecoder; represents the weight matrix of the deep autodecoder; Represents the bias vector of the deep autodecoder.

6. The medical monitoring method based on meta-learning and multimodal monitoring information fusion according to claim 4, characterized in that: The initial medical monitoring model is trained by using the support set and the query set, using meta-learning and the hidden Markov principle, to obtain a medical monitoring model, including: According to the support set and the deep autoencoder, multiple historical multimodal signal sequences are obtained; Obtaining the first classification task result and the second classification task result corresponding to each historical multimodal signal sequence in the support set; According to the query set and the deep autoencoder, multiple historical multimodal signal sequences are obtained; Obtaining the first classification task result and the second classification task result corresponding to each historical multimodal signal sequence in the query set; constructing an initial medical surveillance model; Inputting multiple historical multimodal signal sequences in the support set into the initial medical monitoring model to obtain a first classification task result prediction value and a second classification task result prediction value for each historical multimodal signal sequence in the support set; Determine a first classification task loss value based on a first classification task result prediction value and a first classification task result of each historical multimodal signal sequence in the support set; Determine the second classification task loss value based on the second classification task result prediction value and the second classification task result of each historical multimodal signal sequence in the support set; Determine the sum of the first classification task loss value and the second classification task loss value as the total loss value; Adjust the parameters of the Transformer encoder in the initial medical monitoring model based on the total loss value, and return to step "input multiple historical multimodal signal sequences in the support set into the initial medical monitoring model to obtain the first classification task result prediction value and the second classification task result prediction value of each historical multimodal signal sequence in the support set" until the total loss value is within the preset total loss value range, completing the meta-learning inner loop training stage; Inputting multiple historical multimodal signal sequences in the query set into the initial medical monitoring model to obtain a first classification task result prediction value and a second classification task result prediction value for each historical multimodal signal sequence in the query set; Determine a first classification task loss value based on a first classification task result prediction value and a first classification task result of each historical multimodal signal sequence in the query set; Determine a second classification task loss value based on a second classification task result prediction value and a second classification task result of each historical multimodal signal sequence in the query set; Determine the sum of the first classification task loss value and the second classification task loss value as the total loss value; Adjust the parameters of the Transformer encoder in the initial medical monitoring model based on the total loss value, and return to the step "inputting multiple historical multimodal signal sequences in the query set into the initial medical monitoring model to obtain the first classification task result prediction value and the second classification task result prediction value of each historical multimodal signal sequence in the query set" until the total loss value is within the preset total loss value range, completing the meta-learning outer loop training phase, and determining that the Transformer encoder is a meta-learning adaptive model; The support set is used to train the meta-learning adaptive model, and the parameters of the meta-learning adaptive model are adjusted to obtain a medical monitoring model.

7. The medical monitoring method based on meta-learning and multimodal monitoring information fusion according to claim 6, characterized in that: Input multiple historical multimodal signal sequences in the support set into the initial medical monitoring model to obtain the first classification task result prediction value and the second classification task result prediction value of each historical multimodal signal sequence in the support set, including: Determining any historical multimodal signal sequence as a current historical multimodal signal sequence; Input the historical multimodal signal sequence into the Transformer encoder to obtain information encoding; Input the information code into the position coding layer to obtain the position code of the information code; Monitor the information coding to obtain statistical features, and construct an automatic warning model based on the statistical features and upper and lower limits; the automatic warning model is used to output a predicted value of the second classification task result; Performing clustering processing on the information coding by using a clustering algorithm; Monitoring the information encoding after clustering processing, and constructing a disease trajectory transition model using a hidden Markov model; the disease trajectory transition model is used to output a predicted value of the first classification task result; The position code, the first classification task result prediction value and the second classification task result prediction value are input into a Transformer decoder to obtain a decoding result; the Transformer decoder includes a multi-head attention network and a feedforward network connected in sequence.

8. The medical monitoring method based on meta-learning and multimodal monitoring information fusion according to claim 7, characterized in that: The loss function for the first classification task is: ; in, represents the first classification task loss function; N represents the number of samples, Indicates the number of label categories; represents the true label of the i-th sample in the c-th category, Represents the probability of the i-th sample in the c-th category predicted by the disease trajectory transition model.

9. The medical monitoring method based on meta-learning and multimodal monitoring information fusion according to claim 8, characterized in that: The loss function for the second classification task is: ； in, Represents the loss function of the second classification task; is the true label of the i-th sample, is the probability of worsening of the condition of the i-th sample predicted by the automatic early warning model.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the medical monitoring method based on meta-learning and multimodal monitoring information fusion as described in any one of claims 1 to 9.

Images