Perioperative delirium risk prediction method and equipment and storage medium

Abstract

The invention relates to a perioperative delirium risk prediction method and device and a storage medium, and belongs to the technical field of big data processing. The method comprises the steps of obtaining to-be-predicted multi-modal data; encoding the multi-modal data, and extracting global state features; fusing information of different modes through a multi-mode cross attention mechanism to generate fusion features; the fusion features are combined with coding information to be input into a recursive reasoning module, and global state features at future time points are gradually reasoned; inputting the global state features at the future time point into a perioperative period delirium risk prediction model, and generating a perioperative period delirium risk probability at the future time point; and generating a perioperative period delirium risk probability sequence in a time sequence form through recursive reasoning. According to the method, multi-dimensional perioperative delirium risk factors are covered; the capturing capability of feature interaction and association between modals is improved; the change trend of the perioperative period risk can be dynamically predicted, and the interpretability and practicability of the model are enhanced.

Description

Technical Field

[0001] The present application belongs to the field of big data processing technology, and specifically relates to a perioperative delirium risk prediction method, device and storage medium. Background Art

[0002] Perioperative surgical trauma, anesthetic drugs and other factors may damage the already fragile but still able to maintain homeostasis brain function, leading to abnormal postoperative consciousness state conversion and cognitive impairment. The most common and most difficult complications in elderly patients during the perioperative period are central nervous system complications, such as abnormal changes in consciousness after surgical anesthesia, including loss of consciousness caused by anesthetic drugs, delayed recovery of consciousness during postoperative recovery, abnormal consciousness state in the early postoperative period, postoperative delirium (POD), and cognitive dysfunction in the medium and long term. Perioperative central nervous system complications are highly correlated with mortality, such as POD can increase mortality by 40%. Therefore, perioperative central nervous system complications have become a major clinical problem that seriously affects the perioperative safety and postoperative outcomes of elderly patients.

[0003] Therefore, designing a perioperative delirium risk prediction scheme to effectively and accurately predict the risk of perioperative central nervous system complications is one of the important research topics for technical personnel in this field. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a perioperative delirium risk prediction method, device and storage medium to achieve accurate and effective prediction of perioperative delirium risk.

[0005] In a first aspect of the present application, a method for predicting perioperative delirium risk is provided, comprising:

[0006] Acquire multimodal data to be predicted; the multimodal data includes image feature data related to the occurrence of perioperative delirium extracted from preoperative and intraoperative images, time series feature data related to the occurrence of perioperative delirium extracted from time series data, and text feature data extracted from clinical texts;

[0007] The Transformer architecture is used to encode the multimodal data and extract the global state feature F modality (F image ,F EEG ,F text );

[0008] The information of different modalities is fused through the multimodal cross attention mechanism to generate the fusion feature F fusion =CrossAttention(F image ,F EEG ,F text );

[0009] The fusion features are combined with the encoding information and input into the recursive reasoning module to gradually infer the global state features F' at the future time point modality (F' image ,F' EEG ,F' text );

[0010] The global state feature F' at the future time point modality (F' image ,F' EEG ,F' text ) is input into the perioperative delirium risk prediction model to generate the perioperative delirium risk probability at a future time point;

[0011] Through recursive reasoning, a perioperative delirium risk probability sequence in the form of a time series is generated;

[0012] The perioperative delirium risk prediction model is obtained by using large-scale multimodal data as a training data set and performing multi-step reasoning training on the training data set using recursive reasoning.

[0013] Optionally, it also includes:

[0014] For the image feature data, each value in the corresponding attention weight matrix represents the importance of different blocks in the image feature data;

[0015] Extract the weight corresponding to each block from the attention weight matrix, and map the weight to the position of the block corresponding to the original image;

[0016] A heat map is generated by superimposing the original image, and different colors are used to distinguish and display the importance of the areas corresponding to different blocks.

[0017] Optionally, it also includes:

[0018] For time series data, each value in the corresponding attention weight matrix represents the importance of each time point;

[0019] The attention weight is displayed in the form of a heat map on the timeline, and the importance of different time points is shown by different colors.

[0020] Optionally, it also includes:

[0021] For text data, each value of the corresponding attention weight matrix represents the importance of each word or phrase in the text;

[0022] The words or phrases in the text are highlighted according to the attention weights, and the importance of different words or phrases in the text are displayed through different colors or transparency.

[0023] Optionally, it also includes:

[0024] Obtaining the prediction result of the perioperative delirium risk prediction model for a given sample as a benchmark prediction result;

[0025] Calculate the changed prediction results by removing or combining different input features;

[0026] Based on the benchmark prediction result and the changed prediction result, respectively obtaining Shapley values ​​of the image feature data, the time series feature data and the text feature data;

[0027] By sorting the Shapley values ​​of image feature data, time series feature data, and text feature data, key features that contribute greatly to the prediction of perioperative delirium and redundant features that contribute little to the prediction of perioperative delirium are identified;

[0028] The training process of the perioperative delirium risk prediction model is optimized based on the key features and the redundant features.

[0029] Optionally, it also includes:

[0030] Obtaining a cross-attention weight matrix, wherein the cross-attention weight matrix represents an association relationship between the first feature data and the second feature data;

[0031] Retrieving high weights greater than a preset threshold from the cross-attention weight matrix;

[0032] The association relationship between the first feature data and the second feature data corresponding to the high weight is pushed to the user.

[0033] Optionally, the perioperative delirium risk prediction model adopts a self-supervised learning mechanism of self-learning reasoning technology to perform self-adjustment according to the degree of match between the generated reasoning steps and the prediction results; when the reasoning steps contribute to the correct prediction, the model strengthens the corresponding reasoning path through gradient updating; if the reasoning steps contain redundant or erroneous logic, the model weakens the weight of the corresponding reasoning path; after multiple iterations, the reasoning model is gradually optimized.

[0034] Optionally, after generating the perioperative delirium risk probability sequence in the form of a time series, the method further includes: evaluating the performance of perioperative delirium risk prediction using the perioperative delirium risk time series prediction error;

[0035] Among them, the Token representation of time series is designed from the perspectives of long-term and short-term trends, pattern matching, spectrum distribution, and extreme points of time series. Metric learning technology is used to train the neural network through contrast loss to obtain a metric function for comparing the similarity of time series waveforms and evaluate the similarity between the predicted time series and the actual value.

[0036] In a second aspect of the present application, a perioperative delirium risk prediction device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the perioperative delirium risk prediction method according to any one of the above-described methods is implemented.

[0037] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the perioperative delirium risk prediction method according to any one of the above-mentioned methods is implemented.

[0038] The perioperative delirium risk prediction method provided in the present application obtains multimodal data to be predicted; uses the Transformer architecture to encode the multimodal data and extract global state features; fuses information of different modalities through a multimodal cross-attention mechanism to generate fusion features; inputs the fusion features into the recursive reasoning module in combination with the encoded information, and gradually infers the global state features of future time points; inputs the global state features of the future time points into the perioperative delirium risk prediction model to generate the perioperative delirium risk probability at future time points; and generates a perioperative delirium risk probability sequence in the form of a time series through recursive reasoning. The present application combines imaging, time series and text data to cover multi-dimensional perioperative delirium risk factors; the multimodal cross-attention mechanism improves the ability to capture the interaction and association of features between modalities; through recursive reasoning combined with time series modeling, it can dynamically predict the changing trend of perioperative risk. In addition, through feature alignment and modal association analysis, the contribution of different modalities to the prediction results can be explained. In addition, the present application also provides a perioperative delirium risk prediction device and storage medium having the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0040] Figure 1 A flowchart of a specific implementation method of the perioperative delirium risk prediction method provided in this application;

[0041] Figure 2 This is a schematic diagram of the process of interpreting image feature data in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the feature importance evaluation process for inputting an input feature in an embodiment of the present application;

[0043] Figure 4 This is a structural block diagram of the perioperative delirium risk prediction device provided in this application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0045] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0046] Most of the existing perioperative delirium risk (hereinafter referred to as: POD) prediction schemes use statistical methods such as multivariate logistic regression, combined with clinical experience and known risk factors (age, preoperative cognitive status, type of surgery, operation time, etc.) to develop POD prediction tools to assess patients' postoperative risks. In recent years, machine learning technology has been gradually applied to POD risk prediction. Machine learning models have strong nonlinear data processing capabilities, can integrate multidimensional data and realize automatic learning and optimization, and have shown higher prediction accuracy than traditional models. The machine learning models currently used for POD prediction include logistic regression (LR), random forest (RF), support vector machine (SVM) and extreme gradient boosting (XGBoost), and the results show that XGBoost and RF technologies have relatively good effects. Although the application of machine learning technology in POD prediction has shown good prospects, the current POD risk prediction methods face the following challenges, which limit the application and promotion of the models in actual clinical settings:

[0047] (1) Ignoring temporal dynamic characteristics: The occurrence of POD is dynamic, but most current models still tend to use simple statistical summaries rather than complex time series analysis, which may lead to limitations in prediction results.

[0048] (2) Insufficient use of multimodal data: Most current POD prediction models rely on a single type of data, such as demographic data or surgical records, and fail to make full use of multimodal data, such as imaging data (e.g., fMRI), time series data (e.g., EEG), biomarkers, metabolomics, etc., resulting in low prediction performance of the model;

[0049] (3) Low model interpretability: Many deep learning or machine learning models currently used, such as RF and XGBoost, have good prediction effects but lack interpretability, which limits their widespread application in actual clinical practice. Clinicians need to understand the prediction logic of the model in order to intervene effectively, and black box models have limitations in this regard.

[0050] Based on this, the present application provides a method for predicting the risk of perioperative delirium, such as Figure 1 The present application provides a flowchart of a specific implementation of the method for predicting the risk of perioperative delirium, which specifically includes:

[0051] S101: Acquire multimodal data to be predicted.

[0052] The multimodal data includes image feature data related to the occurrence of perioperative delirium extracted from preoperative and intraoperative images, time series feature data related to the occurrence of perioperative delirium extracted from time series data, and text feature data extracted from clinical texts.

[0053] Image feature data: Deep learning technology can be used to extract fine-grained features related to the occurrence of perioperative delirium from preoperative and intraoperative images (such as fMRI-fNIRS data), including but not limited to subtle changes in brain structure and patterns of brain area activity.

[0054] Time series feature data: Using fast Fourier transform, wavelet transform and other technologies, key signals related to the occurrence of perioperative delirium are extracted from complex time series signals (such as EEG data), including but not limited to EEG microstates, spectral features, etc.

[0055] Text feature data: Natural language processing technologies such as MetaMap are used to extract key text data such as patients’ medical history, surgical records, and postoperative care reports from clinical texts such as electronic medical records.

[0056] In addition, multimodal data can also include: metabolite and physiological feature extraction. Through the analysis of metabolomics data, metabolite markers related to the occurrence of perioperative delirium are extracted, including but not limited to lactate, creatinine, etc. Metabolite and physiological feature data can be used as part of text data.

[0057] S102: Encode the multimodal data using the Transformer architecture to extract the global state feature F modality (F image ,F EEG ,F text ).

[0058] Before encoding the multimodal data, operations of time, space, and feature alignment of the multimodal feature data may also be performed.

[0059] Time alignment: POD risk prediction involves multimodal data collected at multiple times, including preoperative, intraoperative and postoperative physiological data, imaging data, and clinical text information. The process of time alignment ensures that multimodal data is synchronized along the time axis, so that features from different modalities can be aligned to the same time point or time period.

[0060] Among them, physiological data such as EEG, blood pressure, and heart rate usually have precise timestamps and are aligned through time windows. Imaging data such as fMRI have specific time points or timestamps. The imaging data is mapped to the corresponding time window and combined with other time series data to ensure that the imaging data can reflect the specific stage during or after surgery. Clinical text data usually contains timestamps or specific time nodes. Time information is extracted through natural language processing technology and aligned with data from other modalities.

[0061] Spatial alignment: Establishing spatial correspondence between modalities, especially for imaging and EEG modalities, ensures spatial alignment of various multimodal features.

[0062] Among them, imaging data such as fMRI carry standardized spatial coordinates to map the patient's brain image into a standard space, align each patient's image into a standardized coordinate system, and uniformly process the imaging and non-imaging data information of different patients. EEG data carries brain region information, aligns the location of brain electrodes and imaging data, aligns the spatial information in EEG with specific areas of the brain, and helps understand the spatial distribution characteristics of POD. For clinical text data that does not contain spatial coordinate information, natural language processing technology is used to extract descriptive information related to specific brain regions or body parts from medical records and align them with the corresponding areas in the imaging data.

[0063] Feature alignment: Unify the feature representations of different modalities so that they can be directly compared and fused in the feature space.

[0064] Image data such as fMRI are high-dimensional spatial data (three-dimensional or four-dimensional), time series data such as EEG are one-dimensional time data, and clinical text data are usually discrete symbolic information. Deep learning technology is used to extract high-level features of image and time series data and map them to the same vector space. Natural language processing technology is used to extract key information in clinical texts (such as descriptions of postoperative complications, details of surgical procedures, etc.) as specific medical entities or vector representations, and multimodal fusion technology is used to align text feature data with image feature data and time series feature data.

[0065] In addition, the embodiments of the present application can further perform missing data completion operations for multimodal data. For some patients who may lack complete imaging data or physiological data, the missing data completion technology based on data interpolation and deep learning can generate approximate values ​​of the missing data to ensure the integrity of feature alignment.

[0066] Through time alignment, space alignment and feature alignment, comprehensive alignment of multimodal data can be achieved. This process provides a solid foundation for multimodal fusion and performance improvement of the model, and provides clearer logical support for subsequent explanatory analysis.

[0067] Multimodal data is divided into small, processable blocks to obtain multimodal blocks (patches, windows or tokens) in order to capture fine-grained spatial and temporal features and establish cross-modal correspondence.

[0068] For image data, the image data is decomposed into fixed-size image patches. The three-dimensional temporal characteristics or four-dimensional spatiotemporal characteristics contained in fMRI and other image data involve brain area structure analysis. Combining gradient information guidance and morphological operations (edge ​​detection, contour extraction), a Tokenizer with spatiotemporal adaptive capabilities is designed. For key time periods and key local area data related to POD (areas with drastic spatiotemporal changes), the block granularity is adaptively adjusted to achieve accurate capture of key features.

[0069] For time series data, the time series data is divided into time windows of fixed length. For example, time series data such as EEG is segmented, and tokenization is performed in each time window according to the changes in neural activity to accurately capture the abnormal changes in neural activity before and after postoperative delirium.

[0070] For clinical texts, natural language processing technologies such as MetaMap are used for word segmentation to identify clinical symptoms, diagnostic information, and drug use related to postoperative delirium. These tokenized text information can be temporally and spatially associated with other modal data.

[0071] After the image data, time series data, and text data are segmented, corresponding spatiotemporal coding is generated for each multimodal segment based on the spatiotemporal dependency of each modal data (such as spatial position and timestamp).

[0072] In order to influence the feature data, multi-scale block embedding technology is used to provide spatiotemporal position coding for the blocks of image feature data. Position coding (such as sine-cosine coding) is introduced to represent spatiotemporal information and output image feature F image .

[0073] For time series data, the time series Transformer is used to capture the global and local dependencies of time series features. The sliding window position encoding technology is used to provide spatiotemporal position encoding for the blocks of time series feature data, and the time series feature F is output. EEG .

[0074] For text feature data, sentence-level and word-level embeddings are extracted through pre-trained language models (such as BERT), and sine-cosine position encoding technology is used to provide spatiotemporal position encoding for the blocks of text feature data, and the text data encoding F is output. text .

[0075] The Transformer architecture uses a multi-head attention mechanism to encode each modal feature and extract global information. Multimodal data is independently encoded to form a feature representation F modality . Through multimodal segmentation and spatiotemporal encoding, heterogeneous data such as images, time series and text are mapped into a unified spatiotemporal framework. This process can not only capture the fine-grained features within the modality, but also model the complex relationship between modalities through cross-modal interaction, providing a rich multimodal representation for perioperative delirium (POD) risk prediction, which helps the POD risk prediction model understand the pathological changes in different brain regions and different time periods.

[0076] S103: Fuse information from different modalities through a multimodal cross-attention mechanism to generate fusion features F fusion =CrossAttention(F image ,F EEG ,F text ).

[0077] For the cross attention of image feature data and time series feature data: the image feature data blocks are used as query, and the time series feature data blocks are used as key and value, and the correlation between the features of different brain regions in the image data and the time series signals is calculated:

[0078] It helps capture the relationship between brain structure and functional activity, such as whether structural abnormalities in specific brain regions are accompanied by changes in brain electrical activity.

[0079] Cross-attention for image feature data and text feature data: Use image feature data blocks as query, text feature data blocks as key and value, and calculate the association between brain structure features in image data and medical history or diagnosis information in text:

[0080] For example, through the description in the text (such as preoperative brain atrophy or cerebral hemorrhage), the corresponding area in the image will receive extra attention.

[0081] Cross-attention for time series and clinical text: The time series feature data is divided into blocks as query, and the text feature data is divided into blocks as key and value, and the relationship between the EEG activity in different time periods in the time series data and the clinical description in the text is calculated:

[0082] For example, one can capture whether certain drug use (extracted from text) affects the fluctuations of the EEG signal through a cross-attention mechanism.

[0083] Perform cross-attention operations on image-time series, image-text, and time series-text in sequence, and finally fuse all modal features into F fusion .

[0084] S104: Input the fusion feature and the coding information into the recursive reasoning module, and gradually reason out the global state feature F' at the future time point modality (F' image ,F' EEG ,F' text ).

[0085] The recursive reasoning module is based on the decoder structure of Transformer and inputs the fusion feature F at the current time point. fusion and past states, inferring the characteristics of the next time point.

[0086] Specifically, in the reasoning mechanism, the key brain region features at future time points are predicted for imaging data, signal trends are predicted for time series data, and changes in risk factors that may be caused (such as the impact of postoperative medication) are predicted for text data.

[0087] S105: The global state feature F' at the future time point modality (F' image ,F' EEG ,F' text ) is input into the perioperative delirium risk prediction model to generate the perioperative delirium risk probability at future time points.

[0088] The perioperative delirium risk prediction model is obtained by using large-scale multimodal data as a training data set and performing multi-step reasoning training on the training data set using recursive reasoning. The perioperative delirium risk prediction model inputs the global state feature F' modality , output the perioperative delirium risk level (such as high risk, medium risk, low risk) or specific probability value.

[0089] The perioperative delirium risk prediction model adopts a self-supervised learning mechanism of self-learning reasoning technology, and adjusts itself according to the matching degree between the generated reasoning steps and the prediction results. When the reasoning steps contribute to the correct prediction, the model strengthens the corresponding reasoning path through gradient updating. If the reasoning steps contain redundant or erroneous logic, the model weakens the weight of the corresponding reasoning path. After multiple iterations, the reasoning model is gradually optimized.

[0090] This application adopts Self-Taught Reasoner (STaR) technology, which relies on its self-supervised learning mechanism to gradually optimize the reasoning ability by evaluating the reasoning steps generated by the model itself. During the model training process, STaR technology will adjust itself according to the effect of reasoning. For example, if the explanation generated by the model (such as the relationship between abnormal fluctuations in a certain section of EEG signals and POD) is consistent with the final correct prediction result, the model will further strengthen this type of reasoning through feedback. In addition, STaR technology iterates the reasoning steps, making the reasoning process of the model more and more accurate. As the model is trained, the generated explanatory reasoning will be more in line with real clinical phenomena, thereby helping the model to better capture the associations between multimodal data.

[0091] S106: Generate a perioperative delirium risk probability sequence in the form of a time series through recursive reasoning.

[0092] Use the recursive reasoning module to gradually predict the risk probability at each time point. The risk probability P in the form of a time series t Represents the predicted values ​​at different time points.

[0093] In perioperative delirium risk prediction models, interpretability is crucial because the medical field requires a clear reasoning process and prediction basis to enhance the trust and understanding of model decisions by relevant personnel.

[0094] Based on any of the above embodiments, Figure 2 As shown in the schematic diagram of the interpretability process of image feature data, the perioperative delirium risk prediction model provided in this application also includes:

[0095] S201: For the image feature data, each value in the corresponding attention weight matrix represents the importance of different blocks in the image feature data.

[0096] The attention weight matrix is ​​one of the key outputs in the multimodal model, which indicates the importance of the model to different input blocks. In image feature data, blocks usually correspond to small areas of fixed size (such as the grid of the image) into which the original image is divided. The weight value reflects the degree of attention paid by the model to each block. The larger the value, the more important the area is to the final decision.

[0097] S202: Extract the weight corresponding to each block from the attention weight matrix, and map the weight to the position of the block corresponding to the original image.

[0098] The original image is divided into multiple small blocks (such as 16×16 or 32×32 image grids), and each row / column of the attention weight matrix corresponds to the weight of a block. After the weight of each block is extracted from the attention weight matrix, it is indexed to the specific position on the original image. If a block is located in the upper left corner of the image (such as the block in the first row and first column), its weight will directly affect the color intensity of the upper left corner of the heat map.

[0099] S203: Generate a heat map by overlaying the original image, and display the importance of the areas corresponding to different blocks by different colors.

[0100] The weight of each block is superimposed with its corresponding image area to obtain the attention distribution of the entire image. For example, if a certain brain region feature (block) has a higher weight in predicting delirium risk, a highlight color will be displayed on the brain region. Different weight values ​​are mapped to different color shades. For example: red represents high weight (high attention area), and blue represents low weight (low attention area). Use color gradients (such as red-yellow-blue) to more intuitively represent the attention distribution of the model.

[0101] Extract the weight matrix of each block of image feature data from the self-attention mechanism

[0102]

[0103] As can be seen, these weights are mapped back to the spatial position of the original image and visualized as a heat map that matches the resolution of the original image, showing the areas that the model focuses on. Then, using a heat map or a superimposed transparent layer, the attention weights are intuitively displayed on the original image, marking which brain areas and specific image features the model pays more attention to, and highlighting the brain areas that the model pays the most attention to. For example, highlighting brain areas with high attention on preoperative MRI images.

[0104] In this way, relevant personnel can intuitively see through the heat map which image areas the model pays more attention to. For example, atrophy of a certain area of ​​the brain is given a high weight by the model, suggesting that it may be related to perioperative delirium. Abnormal blood vessels or tissue areas may receive higher attention, explaining the prediction basis of the model. This embodiment explains the decision-making basis of the model in the image feature data by visualizing the attention weight matrix. It can provide medical staff with a reference for medical diagnosis, thereby enhancing its interpretability and practicality.

[0105] Based on any of the above embodiments, the perioperative delirium risk prediction model provided by the present application also includes: for time series data, each value in the corresponding attention weight matrix represents the importance of each time point; the attention weight is displayed in the form of a heat map on the time axis, and the importance of different time points is displayed by different colors.

[0106] Each time point is represented by a color block to represent its weight, and all time points are arranged in order into a heat map to intuitively show the distribution of the model's attention to different time points in the time series. Highlight the key time periods in the time series signal (such as abnormal bands of heart rate or EEG) to show how the model associates time series features. By generating a heat map on the timeline to show the attention weight of each time point, it is possible to clearly reflect the focus of the model on the time series features.

[0107] Based on any of the above embodiments, the perioperative delirium risk prediction model provided by the present application also includes: for text data, each value of the corresponding attention weight matrix represents the importance of each word or phrase in the text; the words or phrases in the text are highlighted according to the attention weight, and the importance of different words or phrases in the text is displayed through different colors or transparencies.

[0108] Use attention weights to annotate important terms in clinical texts (such as drug names and medical history descriptions). For example, highlight the weight of "postoperative cognitive impairment" or "chronic disease history" in the prediction. The process of highlighting words or phrases through the attention weight matrix is ​​intended to explain the model's decision-making basis in text features and help us intuitively understand which words or phrases contribute most to the model's predictions.

[0109] In addition, by evaluating the contribution of each mode or each feature to the model output, it is possible to identify which data plays a key role in the prediction. Figure 3 The schematic diagram of the input feature importance evaluation process is shown in the figure. The process specifically includes:

[0110] S301: Obtain the prediction result of the perioperative delirium risk prediction model for a given sample as a benchmark prediction result.

[0111] The complete multimodal feature data (images, time series, text) are input into the model to obtain the prediction results of the model output (such as the risk probability or classification results of perioperative delirium), which serve as a reference point for subsequent changes after feature removal or combination.

[0112] S302: Calculate the changed prediction result by removing or combining different input features.

[0113] By systematically removing or combining different features, observe the impact on the model's prediction results to assess the importance of the features. Remove a single feature: For example, remove MRI image data from the multimodal features and observe the changes in the prediction results. Combine features, such as retaining only one modality feature (such as time series data) or combining two modalities (such as image + text) to input the model and observe the changes in the prediction. After each feature is removed or combined, record the changes in the predicted value output by the model.

[0114] S303: Based on the benchmark prediction result and the changed prediction result, Shapley values ​​of the image feature data, the time series feature data and the text feature data are obtained respectively.

[0115] The Shapley value is used to measure the average marginal contribution of each input feature to the model prediction.

[0116] S304: Identify key features that contribute greatly to the prediction of perioperative delirium and redundant features that contribute little to the prediction of perioperative delirium by sorting the Shapley values ​​of the image feature data, the time series feature data, and the text feature data.

[0117] According to the Shapley value, the features are sorted from high to low. A threshold is set (e.g. Shapley value > Y% is a key feature, and the value of Y can be determined according to the actual situation) to divide all features into key features and redundant features.

[0118] S305: Optimizing the training process of the perioperative delirium risk prediction model based on the key features and the redundant features.

[0119] Use feature importance information to optimize the model structure and training process to improve model performance. Optimizing training can specifically include: increasing the sampling weight of key features (such as focusing on learning abnormalities in the hippocampus in the image). Reduce the sampling of redundant features or completely eliminate them to reduce model complexity. It can also include: improving the structure of the multimodal model to assign higher weights to key modalities or key features. For example, in the fusion layer, increase the cross-attention weights for the imaging modality (hippocampus) and the time series modality (postoperative alpha band). In addition, the trained Shapley value is used to highlight the model's focus area, and the optimized model prediction logic is presented to relevant personnel in a more transparent manner.

[0120] The Shapley value is a feature importance calculation method based on cooperative game theory, which can measure the contribution of each input feature to the model prediction. The Shapley value method is used to calculate the marginal contribution of each feature in different feature combinations to evaluate the importance of the feature. For the multimodal POD prediction model, Shapley value analysis is performed on the features of images, time series, and texts. First, the prediction results of the POD prediction model for a given sample are obtained; by removing or combining different input features (such as local brain areas in imaging data, specific time points in EEG time series, and key terms in text), the changes in the model prediction results are calculated; for each feature, the Shapley value measures the average marginal contribution of the feature to the prediction result. The larger the Shapley value, the more important the feature is to POD prediction.

[0121] Through the above process, we can focus on the key features that contribute greatly to the prediction, eliminate redundant features, reduce the noise interference of the model, and the optimized model can more accurately capture the features related to delirium risk, greatly improving the performance of the model. Through Shapley value analysis, the key basis for model decision-making (such as a certain brain area in the image, a certain band in the time series) is clearly pointed out, which enhances the interpretability of the model. In addition, key risk factors (such as EEG abnormalities in a certain period of time after surgery and the use of specific drugs) can also be identified to provide medical staff with a basis for diagnosis and intervention.

[0122] In addition, a cross-attention weight matrix can be obtained, which represents the association relationship between the first feature data and the second feature data; high weights greater than a preset threshold are retrieved from the cross-attention weight matrix; and the association relationship between the first feature data and the second feature data corresponding to the high weights is pushed to the user.

[0123] Calculate the attention distribution between multimodal data to reveal the correlation between each modality and the reasoning process of the POD model between different modalities. Taking the cross attention of image and time series data as an example, the cross attention weight matrix is ​​extracted by the following formula

[0124]

[0125] The high weight values ​​in this matrix indicate that the model believes that the structural changes in a certain brain region in the image may affect the brain function performance in a certain time period or frequency band in the EEG. For example, the cross-attention matrix shows that the hippocampus region in the image has a high weight distribution with the alpha band in the EEG signal, which may mean that the structural damage or atrophy of the region is related to abnormal EEG activity, indicating a high risk of POD. The same is true for the cross-attention interpretation method between other modalities.

[0126] The cross-attention weight matrix is ​​a two-dimensional matrix that represents the strength of association between each block in the first modality (such as image features) and each feature in the second modality (such as time series features). Feature pairs with higher weight values ​​(i.e., features of the first modality and the second modality with strong correlation) are selected from the cross-attention weight matrix to identify the inter-modal relationship that the model focuses on. According to the actual task requirements, a threshold for attention weight can be set. By traversing the cross-attention weight matrix, high weights greater than the threshold are retrieved, and the first modal features and second modal features corresponding to the high weights are extracted and pushed to the user, intuitively showing how the model combines the correlation between modalities for risk prediction.

[0127] Furthermore, we can generate explanatory descriptions based on the meaning of high-weight features, for example: "The structural atrophy of the hippocampus in the imaging feature is highly correlated with the abnormal alpha band on the second day after surgery in the time series feature (weight = 0.85), which may be a key factor in the risk of delirium." In this way, we can clearly see which modal relationships the model focuses on when fusing multimodal data. At the same time, we can verify whether the model's focus is reasonable and improve the model's interpretability and trustworthiness.

[0128] After generating a perioperative delirium risk probability sequence in the form of a time series, the present application further includes: evaluating the performance of perioperative delirium risk prediction using the perioperative delirium risk time series prediction error. By comparing the similarity between the predicted time series and the actual time series, the prediction ability and performance of the model are quantified.

[0129] The prediction error of the time series of perioperative delirium risk is studied. From the perspectives of long-term and short-term trends, pattern matching, spectrum distribution, and extreme points of the time series, a tokenized representation of the time series is designed. The metric learning technology is used to train the neural network through contrast loss to obtain a metric function for comparing the similarity of time series waveforms, and to evaluate the similarity between the predicted time series and the actual value.

[0130] In addition, the performance of perioperative delirium risk prediction can also be evaluated by using traditional indicators with an augmented time dimension. This method reflects the importance of different time periods by time weighting, especially the early postoperative period, which may be more critical to clinical decision-making. By introducing time weights, the contribution of each time period to the final score is dynamically adjusted in the calculation of traditional indicators (such as AUC-ROC, accuracy, recall, F1 score, etc.), highlighting the importance of the early critical period.

[0131] Taking the AUC-ROC curve as an example, considering that the POD risk prediction in the early postoperative period is more important than in other time periods, the time-weighted AUC method is used to introduce time weights when calculating AUC. Different weights are assigned to the prediction results of each time period, and the data weight of the early postoperative period is larger to reflect its importance in time. The time-weighted versions of accuracy, recall, precision, and F1 score are designed in the same way.

[0132] This application makes a comprehensive evaluation of the POD risk time series prediction results by constructing a POD risk prediction performance evaluation system with an augmented time dimension.

[0133] The following is a further detailed description of the process of fusing and modeling multimodal data in the perioperative delirium risk prediction method provided in this application.

[0134] Multimodal feature extraction process:

[0135] Electronic health record (EHR) feature processing: EHR data is converted into Token representation through feature mapping:

[0136]

[0137] Among them, h (1) is the output of the first hidden layer; h (2) is the output of the second hidden layer; h (l) is the output of the lth hidden layer, σ is the activation function, W (1) is the weight matrix of the first layer; W (2) is the weight matrix of the second layer; W (l) is the weight matrix of the lth layer; b (1) is the bias vector of the first layer; b (2) is the bias vector of the second layer; b (l) is the bias vector of the lth layer. The features are mapped to the high-dimensional embedding space through linear transformation. f(y) represents the activation function of the output layer, which converts the output of the last layer into the required format. Token is the basic unit of text.

[0138] Image feature processing includes the following processes:

[0139] Blocking: Divide the image into patches of the same size:

[0140] Patch flattening and linear mapping: Flatten each patch into a vector and map it to a high-dimensional space through linear mapping:

[0141] Position encoding: Add position information to each patch:

[0142] E i For location information.

[0143] Input the representation of all patches into the Transformer encoder:

[0144]

[0145] After L layers of Transformer encoders, each layer contains a multi-head self-attention mechanism and a feedforward neural network.

[0146] Z (l+1) = TransformerLater(Z (l) ), l = 0, 1, ..., L-1

[0147] Final embedding:

[0148] Among them, W cls Represents the weight matrix of the classification layer, which is used to transform the final hidden layer features into the classification space. cls Represents the original output of the classification layer, which is cls The value after the linear transformation and before the activation function is applied. cls Represents the bias vector of the classification layer, adding a learnable offset to the classification decision.

[0149] The text data processing process includes the following:

[0150] First, it needs to be tokenized and then converted into token word embedding.

[0151] Then add the positional encoding. Positional encoding is needed to let the model understand the position information of words in the text.

[0152] The text information is processed by the Transformer encoder and passed through multiple layers of self-attention and feedforward networks to obtain context-aware text representation.

[0153] Text, EHR features H EHR , and image features H image Concatenate into a joint representation:

[0154] x combined =[x text ;h EHR ;h image ]

[0155] Based on the joint representation X combined , for the target sequence y t Perform joint modeling:

[0156] Among them, P Φ represents a parameterized policy network, Φ is the parameter of the policy network. t represents the output of the model at time step t. y<t means y t Previous output.

[0157] The method of optimizing deep learning models by using parameter-efficient fine-tuning (PEFT) is used to fine-tune the model based on parameter efficiency. The optimization method is:

[0158]

[0159] Among them, △ represents the update amount or change amount, which is the adjustment size of the parameter during the optimization process. Θ represents the parameter set of the model, including all the parameters in the model that need to be learned and optimized.

[0160] The modeling method is:

[0161]

[0162] This application uses multimodal data from imaging data, time series data, and clinical texts to capture the complex correlations of perioperative delirium. Each modality provides a unique source of information: imaging features reflect abnormalities in the patient's brain structure or function (such as hippocampal atrophy); time series features describe the dynamics of brain function before and during surgery (such as abnormal fluctuations in EEG signals); text features are used to extract semantic information from medical records or postoperative records to capture risk factors (such as past medical history, drug use). Multimodal fusion significantly improves the model's ability to capture the mechanism of delirium and improves prediction accuracy compared to single-modal models.

[0163] This application uses a time alignment mechanism to align preoperative, intraoperative, and postoperative data in time steps, so that the model can model the dynamic evolution of delirium risk. A spatial alignment mechanism is used to unify the anatomical spatial position of the imaging modality (such as brain area) and the spatial distribution of the EEG signal, so that the model can understand the abnormalities of structure-function coupling. It is further combined with a feature alignment mechanism to standardize multimodal features so that imaging, time series, and text features can be compared and fused in the same feature space. These alignment operations eliminate the heterogeneity between modalities and improve the consistency and effectiveness of data fusion. This application decomposes multimodal data into smaller units, allowing the model to focus on fine-grained features. Adding embeddings of time and spatial position to each block allows the model to perceive the spatiotemporal structure of the data and capture spatiotemporal dependencies (such as the spatiotemporal correlation between intraoperative brain region changes and delirium risk).

[0164] The multimodal large model can process images, time series and text blocks at the same time, and learn the interactive relationship between modalities. This application captures key features across modalities and within modalities through the attention mechanism. Through a unified spatiotemporal framework, the model's ability to process multimodal information is improved. Based on the input multimodal data, the possibility of delirium at different risk levels (such as low risk, medium risk, and high risk) is predicted, which enhances the robustness and accuracy of the prediction.

[0165] In addition, through multimodal segmentation, spatiotemporal coding and attention mechanism, the model can generate interpretable prediction results. This interpretability enhances the credibility of the model in clinical practice and helps doctors understand the basis of the model's predictions.

[0166] In addition, the present application also provides a perioperative delirium risk prediction device, such as Figure 4 The structural block diagram of the perioperative delirium risk prediction device provided in the present application includes a memory 41 and a processor 42, wherein the memory 41 stores a computer program, and when the computer program is executed by the processor 42, the perioperative delirium risk prediction method according to any one of the above-mentioned methods is implemented.

[0167] In addition, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the perioperative delirium risk prediction method according to any one of the above-described methods is implemented.

[0168] Computer readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0169] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0170] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0171] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for predicting perioperative delirium risk, characterized in that: include: Acquire multimodal data to be predicted; the multimodal data includes image feature data related to the occurrence of perioperative delirium extracted from preoperative and intraoperative images, time series feature data related to the occurrence of perioperative delirium extracted from time series data, and text feature data extracted from clinical texts; The Transformer architecture is used to encode the multimodal data and extract the global state feature F modality (F image ,F EEG ,F text ); The information of different modalities is fused through the multimodal cross attention mechanism to generate the fusion feature F fusion =CrossAttention(F image ,F EEG ,F text ); The fusion features are combined with the encoding information and input into the recursive reasoning module to gradually infer the global state features F' at the future time point modality (F' image ,F' EEG ,F' text ); The global state feature F' at the future time point modality (F' image ,F' EEG ,F' text ) is input into the perioperative delirium risk prediction model to generate the perioperative delirium risk probability at a future time point; Through recursive reasoning, a perioperative delirium risk probability sequence in the form of a time series is generated; The perioperative delirium risk prediction model is obtained by using large-scale multimodal data as a training data set and performing multi-step reasoning training on the training data set using recursive reasoning.

2. The method for predicting perioperative delirium risk according to claim 1, characterized in that: Also includes: For the image feature data, each value in the corresponding attention weight matrix represents the importance of different blocks in the image feature data; Extract the weight corresponding to each block from the attention weight matrix, and map the weight to the position of the block corresponding to the original image; A heat map is generated by superimposing the original image, and different colors are used to distinguish and display the importance of the areas corresponding to different blocks.

3. The perioperative delirium risk prediction method according to claim 1, characterized in that: Also includes: For time series data, each value in the corresponding attention weight matrix represents the importance of each time point; The attention weight is displayed in the form of a heat map on the timeline, and the importance of different time points is shown by different colors.

4. The perioperative delirium risk prediction method according to claim 1, characterized in that: Also includes: For text data, each value of the corresponding attention weight matrix represents the importance of each word or phrase in the text; The words or phrases in the text are highlighted according to the attention weights, and the importance of different words or phrases in the text are displayed through different colors or transparency.

5. The method for predicting perioperative delirium risk according to any one of claims 1 to 4, characterized in that: Also includes: Obtaining the prediction result of the perioperative delirium risk prediction model for a given sample as a benchmark prediction result; Calculate the changed prediction results by removing or combining different input features; Based on the benchmark prediction result and the changed prediction result, respectively obtaining Shapley values ​​of the image feature data, the time series feature data and the text feature data; By sorting the Shapley values ​​of image feature data, time series feature data, and text feature data, key features that contribute greatly to the prediction of perioperative delirium and redundant features that contribute little to the prediction of perioperative delirium are identified; The training process of the perioperative delirium risk prediction model is optimized based on the key features and the redundant features.

6. The method for predicting perioperative delirium risk according to any one of claims 1 to 4, characterized in that: Also includes: Obtaining a cross-attention weight matrix, wherein the cross-attention weight matrix represents an association relationship between the first feature data and the second feature data; Retrieving high weights greater than a preset threshold from the cross-attention weight matrix; The association relationship between the first feature data and the second feature data corresponding to the high weight is pushed to the user.

7. The method for predicting perioperative delirium risk according to claim 1, characterized in that: The perioperative delirium risk prediction model adopts a self-supervised learning mechanism of self-learning reasoning technology, and performs self-adjustment according to the matching degree between the generated reasoning steps and the prediction results; when the reasoning steps contribute to the correct prediction, the model strengthens the corresponding reasoning path through gradient updating; when the reasoning steps contain redundant or erroneous logic, the model weakens the weight of the corresponding reasoning path; After multiple iterations, the inference model is gradually optimized.

8. The perioperative delirium risk prediction method according to claim 7, characterized in that: After generating the perioperative delirium risk probability sequence in the form of a time series, the method further includes: evaluating the performance of perioperative delirium risk prediction using the perioperative delirium risk time series prediction error; Among them, the Token representation of time series is designed from the perspectives of long-term and short-term trends, pattern matching, spectrum distribution, and extreme points of time series. Metric learning technology is used to train the neural network through contrast loss to obtain a metric function for comparing the similarity of time series waveforms and evaluate the similarity between the predicted time series and the actual value.

9. A perioperative delirium risk prediction device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the perioperative delirium risk prediction method according to any one of claims 1-8 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for predicting the risk of perioperative delirium according to any one of claims 1-8 is implemented.

Images