Construction method of perioperative delirium risk prediction model, prediction method and equipment

Abstract

The invention relates to a construction method of a perioperative delirium risk prediction model, a prediction method and equipment, and belongs to the technical field of big data processing. The method comprises the following steps: performing feature extraction to obtain multi-modal feature data by acquiring different types of multi-modal data from different sources collected at multiple times; performing time, space and feature alignment on the multi-modal feature data; decomposing the multi-modal characteristic data to obtain multi-modal blocks; a corresponding space-time code is generated for each multi-mode block, so that different data modes are fused under a unified space-time framework; the multi-modal blocks and the space-time codes are input into a multi-modal large model for training, a perioperative delirium risk prediction model is constructed, and after multi-modal data of a to-be-predicted object is received, a prediction result of the perioperative delirium risk level is output. According to the method, the multi-modal data is combined, so that the prediction performance of the perioperative delirium risk prediction model is improved, and accurate prediction and early warning of the perioperative delirium risk are realized.

Description

Technical Field

[0001] The present application belongs to the technical field of big data processing, and specifically relates to a method for constructing a perioperative delirium risk prediction model, a prediction method and a device. Background Art

[0002] In recent years, the number of elderly people who need surgical treatment has increased year by year. Ensuring the safe outcome of anesthesia and perioperative period for the elderly, especially perioperative brain health management, has received much attention. In addition to physiological brain aging, the elderly are often accompanied by underlying diseases. This special physiological and pathological state makes the brain fragile and the compensatory ability of brain function decreases. Perioperative surgical trauma, anesthetic drugs and other factors may destroy the already fragile but still able to maintain steady-state brain function, leading to abnormal postoperative consciousness state conversion and cognitive impairment. The most common and most difficult complications in elderly patients during the perioperative period are central nervous system complications, such as abnormal changes in consciousness state after surgical anesthesia, including loss of consciousness caused by anesthetic drugs, delayed recovery of consciousness during postoperative recovery, abnormal consciousness state in the early postoperative period, postoperative delirium (POD) and cognitive dysfunction in the medium and long term. Perioperative central nervous system complications are highly correlated with mortality. For example, POD can increase mortality by 40%. Therefore, perioperative central nervous system complications have become a major clinical problem that seriously affects the perioperative safety and postoperative outcome of elderly patients.

[0003] Therefore, constructing an effective and feasible perioperative delirium risk prediction model to predict the risk of perioperative central nervous system complications is one of the important research topics for technical personnel in this field. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a method for constructing a perioperative delirium risk prediction model, a prediction method and a device, so as to achieve accurate and effective prediction of perioperative delirium risk.

[0005] In a first aspect of the present application, a method for constructing a perioperative delirium risk prediction model is provided, comprising:

[0006] Acquire multimodal data of different types from different sources collected at multiple times, perform feature extraction on the multimodal data, and obtain multimodal feature data; wherein the multimodal feature data includes image feature data related to the occurrence of perioperative delirium extracted from preoperative and intraoperative image data, time series feature data related to the occurrence of perioperative delirium extracted from time series data, and text feature data extracted from clinical texts;

[0007] Performing time, space, and feature alignment on the multimodal feature data;

[0008] Decomposing the multimodal feature data to obtain multimodal blocks; and generating corresponding spatiotemporal codes for each multimodal block, so that different data modalities can be integrated under a unified spatiotemporal framework;

[0009] The multimodal segmentation and spatiotemporal coding are input into a multimodal large model for training to construct a perioperative delirium risk prediction model. The perioperative delirium risk prediction model is used to output a prediction result of the perioperative delirium risk level of the object to be predicted after receiving the multimodal data of the object to be predicted.

[0010] Optionally, during the training process of the perioperative delirium risk prediction model, a self-attention mechanism is combined to guide the model;

[0011] Among them, the self-attention of image feature data is used to capture the interaction characteristics between specific brain regions and establish the connection between local abnormalities and changes in the whole brain structure;

[0012] Self-attention of time series feature data is used to capture the dynamic changes of neural activity and physiology between different time periods and their dependencies, and to identify the potential relationship between abnormal neural activity in specific time periods before, during and after surgery and the occurrence of perioperative delirium;

[0013] Self-attention of text feature data is used to capture semantic relationships at different levels and extract content related to the occurrence of perioperative delirium.

[0014] Optionally, during the training process of the perioperative delirium risk prediction model, a multi-head attention mechanism is combined to guide the model;

[0015] The multi-head attention mechanism includes:

[0016] Image local and global spatial patterns: used to capture the correlation between adjacent areas in the image and the dependencies between medium and long distance areas;

[0017] Image structure and edge mode: used to capture edge or structural information in images, focusing on the boundaries between different tissues or regions;

[0018] Image multi-scale mode: used to capture features at different scales, large-scale global information and small-scale local details;

[0019] Image Symmetry Mode: used to capture the symmetry or asymmetry between the left and right brain in the image;

[0020] Short-term and long-term time-dependent patterns of time series: used to capture the short-term and long-term time-dependent features in EEG signals and the time series changes of brain activity;

[0021] Frequency-dependent mode: used to capture the characteristics and interactions of EEG signals in different frequency bands;

[0022] Functional connectivity patterns across brain regions: used to capture the functional connectivity patterns between different brain regions;

[0023] Waveform mutation mode: used to capture sudden waveform changes in EEG signals;

[0024] Symptom and diagnosis model: used to capture key symptom descriptions, diagnostic results and medical history related to perioperative delirium in clinical texts;

[0025] Medication and Dosage Patterns: used to capture descriptions of medication use, dosage, and timing in clinical text;

[0026] Timeline mode: used to capture timeline information in clinical texts;

[0027] Significant Events and Risk Factors Model: used to capture risk factors and significant events associated with perioperative delirium in the medical record.

[0028] Optionally, during the training process of the perioperative delirium risk prediction model, a cross-attention mechanism is combined to guide the model;

[0029] Also included before applying the criss-cross attention mechanism:

[0030] Construct an adaptive Tokenizer to align the dimensions of data of different modalities through matrix semi-tensor product and generate a unified representation:

[0031] For the fusion of image feature data and time series feature data, the following formula is used to obtain the correlation between the features of different brain regions in the image data and the time series features:

[0032]

[0033] Among them, Q′ image K′ is the Query feature dimension of the image feature data, and is represented after alignment by the adaptive Tokenizer; EEG V′ is the Key feature dimension of the time series feature data, after being aligned by the adaptive Tokenizer; EEG It is the representation of the Value feature dimension of the time series feature data after alignment by the adaptive Tokenizer; d k is the dimension of the key vector.

[0034] For the fusion of text feature data and image feature data, the following formula is used to calculate the correlation between the brain structure features in the image feature data and the text features in the clinical text:

[0035]

[0036] Among them, Q′ text K′ is the Query feature dimension of the text feature data, and is represented after alignment by the adaptive Tokenizer; image V′ is the Key feature dimension of the image feature data, after alignment by adaptive Tokenizer; image It is the representation of the Value feature dimension of the image feature data after alignment by the adaptive Tokenizer;

[0037] For the fusion of text feature data and time series feature data, the following formula is used to calculate the correlation between the EEG activity in different time periods in the time series data and the text features in the clinical text:

[0038]

[0039] Among them, Q′ text K′ is the Query feature dimension of the text feature data, and is represented after alignment by the adaptive Tokenizer; EEG V′ is the Key feature dimension of the time series feature data, after being aligned by the adaptive Tokenizer; EEG It is the representation of the Value feature dimension of the time series feature data after alignment by the adaptive Tokenizer.

[0040] Optionally, in the process of applying the cross-attention mechanism, it also includes:

[0041] Assign corresponding weights to each mode;

[0042] Performing weighted fusion on the data of each modality according to the weights to obtain a final multimodal representation;

[0043] The weights are determined by adopting a learnable dynamic learning mechanism, and the weight generation is optimized by the loss function, and the weights corresponding to each mode are automatically adjusted according to the task requirements.

[0044] Optionally, generating corresponding spatiotemporal coding for each multimodal block includes:

[0045] The multi-scale block embedding technology is used to provide spatiotemporal position encoding for the blocks of image feature data;

[0046] The sliding window position coding technology is used to provide spatiotemporal position coding for the blocks of time series feature data;

[0047] The sine-cosine position coding technique is used to provide spatiotemporal position coding for the blocks of text feature data.

[0048] Optionally, during the training process of the perioperative delirium risk prediction model, a causal attention mechanism is combined to guide the model;

[0049] When image feature data, time series feature data, and text feature data carry time sequence information, the calculation scope of attention is limited by introducing a causal mask so that each time step only focuses on the current and previous time step data, and cannot see future time steps.

[0050] Optionally, the perioperative delirium risk prediction model combines the multimodal features encoded by Transformer with a recursive reasoning module:

[0051] The image feature data, time series feature data and text feature data are encoded through the Transformer encoder, and the multimodal cross-attention mechanism is used to fuse the multimodal feature data;

[0052] The encoded multimodal feature data is input into the recursive reasoning module to gradually infer the EEG state at future time points or the changes in brain structure after surgery, thereby obtaining the prediction results of the perioperative delirium risk level;

[0053] Among them, the recursive reasoning module supports multi-step reasoning, and the recursive reasoning output of each step serves as the conditional input for the prediction of the subsequent time step.

[0054] The second aspect of the present application provides a method for predicting perioperative delirium risk, comprising:

[0055] Obtain multimodal data to be predicted;

[0056] Inputting the multimodal data to be predicted into a pre-built perioperative delirium risk prediction model, and outputting a prediction result of the perioperative delirium risk;

[0057] Among them, the perioperative delirium risk prediction model is a model obtained according to any of the above-mentioned methods for constructing a perioperative delirium risk prediction model.

[0058] According to a third aspect of the present application, a perioperative delirium risk prediction device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the perioperative delirium risk prediction method described above is implemented.

[0059] The method for constructing a perioperative delirium risk prediction model provided in the present application is to obtain multimodal data of different types from different sources collected at multiple times, extract features from the multimodal data, and obtain multimodal feature data; perform time, space, and feature alignment on the multimodal feature data; decompose the multimodal feature data to obtain multimodal blocks; and generate corresponding spatiotemporal coding for each multimodal block so that different data modalities can be integrated under a unified spatiotemporal framework; input the multimodal blocks and spatiotemporal coding into a large multimodal model for training to construct a perioperative delirium risk prediction model. The perioperative delirium risk prediction model is used to output the prediction result of the perioperative delirium risk level for the object to be predicted after receiving the multimodal data of the object to be predicted. The present application combines multimodal data to improve the prediction performance of the perioperative delirium risk prediction model, and realizes accurate prediction and early warning of perioperative delirium risk. In addition, the present application also provides a perioperative delirium risk prediction method and device with the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0061] Figure 1 A flowchart of a specific implementation method of the method for constructing a perioperative delirium risk prediction model provided in this application;

[0062] Figure 2 A flowchart of the perioperative delirium risk prediction method provided for this application;

[0063] Figure 3 This is a structural block diagram of the perioperative delirium risk prediction device provided in this application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0065] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0066] Most of the existing perioperative delirium risk (hereinafter referred to as: POD) prediction models use statistical methods such as multivariate logistic regression, combined with clinical experience and known risk factors (age, preoperative cognitive status, type of surgery, operation time, etc.) to develop POD prediction tools to assess patients' postoperative risks. In recent years, machine learning technology has gradually been applied to POD risk prediction. Machine learning models have strong nonlinear data processing capabilities, can integrate multidimensional data and realize automatic learning and optimization, and have shown higher prediction accuracy than traditional models. The machine learning models currently used for POD prediction include logistic regression (LR), random forest (RF), support vector machine (SVM) and extreme gradient boosting (XGBoost), and the results show that XGBoost and RF technologies have relatively good effects. Although the application of machine learning technology in POD prediction has shown good prospects, the current POD risk prediction methods face the following challenges, which limit the application and promotion of the models in actual clinical settings:

[0067] (1) Ignoring temporal dynamic characteristics: The occurrence of POD is dynamic, but most current models still tend to use simple statistical summaries rather than complex time series analysis, which may lead to limitations in prediction results.

[0068] (2) Insufficient use of multimodal data: Most current POD prediction models rely on a single type of data, such as demographic data or surgical records, and fail to make full use of multimodal data, such as imaging data (e.g., fMRI), time series data (e.g., EEG), biomarkers, metabolomics, etc., resulting in low prediction performance of the model;

[0069] (3) Low model interpretability: Many deep learning or machine learning models currently used, such as RF and XGBoost, have good prediction effects but lack interpretability, which limits their widespread application in actual clinical practice. Clinicians need to understand the prediction logic of the model in order to intervene effectively, and black box models have limitations in this regard.

[0070] Based on this, the present application provides a method for constructing a perioperative delirium risk prediction model, such as Figure 1 The present application provides a method for constructing a perioperative delirium risk prediction model as shown in a flowchart of a specific implementation method, the method comprising:

[0071] S101: Acquire multimodal data of different types and from different sources collected at multiple times, perform feature extraction on the multimodal data, and obtain multimodal feature data.

[0072] Among them, the multimodal feature data includes imaging feature data related to the occurrence of perioperative delirium extracted from preoperative and intraoperative imaging data, time series feature data related to the occurrence of perioperative delirium extracted from time series data, and text feature data extracted from clinical texts.

[0073] Image feature data: Deep learning technology can be used to extract fine-grained features related to the occurrence of perioperative delirium from preoperative and intraoperative images (such as fMRI-fNIRS data), including but not limited to subtle changes in brain structure and patterns of brain area activity.

[0074] Time series feature data: Using fast Fourier transform, wavelet transform and other technologies, key signals related to the occurrence of perioperative delirium are extracted from complex time series signals (such as EEG data), including but not limited to EEG microstates, spectral features, etc.

[0075] Text feature data: Natural language processing technologies such as MetaMap are used to extract key text data such as patients’ medical history, surgical records, and postoperative care reports from clinical texts such as electronic medical records.

[0076] In addition, multimodal data can also include: metabolite and physiological feature extraction. Through the analysis of metabolomics data, metabolite markers related to the occurrence of perioperative delirium are extracted, including but not limited to lactate, creatinine, etc. Metabolite and physiological feature data can be used as part of text data.

[0077] S102: Perform time, space, and feature alignment on multimodal feature data.

[0078] Time alignment: POD risk prediction involves multimodal data collected at multiple times, including preoperative, intraoperative and postoperative physiological data, imaging data, and clinical text information. The process of time alignment ensures that multimodal data is synchronized along the time axis, so that features from different modalities can be aligned to the same time point or time period.

[0079] Among them, physiological data such as EEG, blood pressure, and heart rate usually have precise timestamps and are aligned through time windows. Imaging data such as fMRI have specific time points or timestamps. The imaging data is mapped to the corresponding time window and combined with other time series data to ensure that the imaging data can reflect the specific stage during or after surgery. Clinical text data usually contains timestamps or specific time nodes. Time information is extracted through natural language processing technology and aligned with data from other modalities.

[0080] Spatial alignment: Establishing spatial correspondence between modalities, especially for imaging and EEG modalities, ensures spatial alignment of various multimodal features.

[0081] Among them, imaging data such as fMRI carry standardized spatial coordinates to map the patient's brain image into a standard space, align each patient's image into a standardized coordinate system, and uniformly process the imaging and non-imaging data information of different patients. EEG data carries brain region information, aligns the location of brain electrodes and imaging data, aligns the spatial information in EEG with specific areas of the brain, and helps understand the spatial distribution characteristics of POD. For clinical text data that does not contain spatial coordinate information, natural language processing technology is used to extract descriptive information related to specific brain regions or body parts from medical records and align them with the corresponding areas in the imaging data.

[0082] Feature alignment: Unify the feature representations of different modalities so that they can be directly compared and fused in the feature space.

[0083] Image data such as fMRI are high-dimensional spatial data (three-dimensional or four-dimensional), time series data such as EEG are one-dimensional time data, and clinical text data are usually discrete symbolic information. Deep learning technology is used to extract high-level features of image and time series data and map them to the same vector space. Natural language processing technology is used to extract key information in clinical texts (such as descriptions of postoperative complications, details of surgical procedures, etc.) as specific medical entities or vector representations, and multimodal fusion technology is used to align text feature data with image feature data and time series feature data.

[0084] In addition, the embodiments of the present application can further perform missing data completion operations for multimodal data. For some patients who may lack complete imaging data or physiological data, the missing data completion technology based on data interpolation and deep learning can generate approximate values ​​of the missing data to ensure the integrity of feature alignment.

[0085] This step can achieve comprehensive alignment of multimodal data through time alignment, space alignment and feature alignment. This process provides a solid foundation for multimodal fusion and performance improvement of the model, and provides clearer logical support for subsequent explanatory analysis.

[0086] S103: Decomposing the multimodal feature data to obtain multimodal blocks; and generating corresponding spatiotemporal coding for each multimodal block, so that different data modalities can be integrated under a unified spatiotemporal framework.

[0087] Multimodal data is divided into small, processable blocks to obtain multimodal blocks (patches, windows or tokens) in order to capture fine-grained spatial and temporal features and establish cross-modal correspondence.

[0088] For image data, the image data is decomposed into fixed-size image patches. The three-dimensional temporal characteristics or four-dimensional spatiotemporal characteristics contained in fMRI and other image data involve brain area structure analysis. Combining gradient information guidance and morphological operations (edge ​​detection, contour extraction), a Tokenizer with spatiotemporal adaptive capabilities is designed. For key time periods and key local area data related to POD (areas with drastic spatiotemporal changes), the block granularity is adaptively adjusted to achieve accurate capture of key features.

[0089] For time series data, the time series data is divided into time windows of fixed length. For example, time series data such as EEG is segmented, and tokenization is performed in each time window according to the changes in neural activity to accurately capture the abnormal changes in neural activity before and after postoperative delirium.

[0090] For clinical texts, natural language processing technologies such as MetaMap are used for word segmentation to identify clinical symptoms, diagnostic information, and drug use related to postoperative delirium. These tokenized text information can be temporally and spatially associated with other modal data.

[0091] After the image data, time series data, and text data are segmented, corresponding spatiotemporal coding is generated for each multimodal segment based on the spatiotemporal dependency of each modal data (such as spatial position and timestamp).

[0092] Multi-scale block embedding technology is used to provide spatiotemporal position coding for blocks of image feature data; sliding window position coding technology is used to provide spatiotemporal position coding for blocks of time series feature data; sine-cosine position coding technology is used to provide spatiotemporal position coding for blocks of text feature data.

[0093] Through multimodal segmentation and spatiotemporal coding, heterogeneous data such as images, time series and text are mapped into a unified spatiotemporal framework. This process can not only capture the fine-grained features within the modality, but also model the complex relationship between modalities through cross-modal interaction, providing a rich multimodal representation for perioperative delirium (POD) risk prediction, which helps the POD risk prediction model understand the pathological changes in different brain regions and different time periods.

[0094] S104: Inputting the multimodal segmentation and spatiotemporal coding into a multimodal large model for training to construct a perioperative delirium risk prediction model.

[0095] The perioperative delirium risk prediction model is used to output a prediction result of the perioperative delirium risk level of the object to be predicted after receiving multimodal data of the object to be predicted.

[0096] As a specific implementation, the perioperative delirium risk prediction model in the present application adopts self-attention and multi-head attention mechanisms.

[0097] In terms of the self-attention mechanism, the traditional weight calculation method is used Evaluate self-attention on image data, time series data, and text data.

[0098] Among them, the self-attention of image feature data mainly captures the interaction characteristics between specific brain regions (such as prefrontal lobe, hippocampus, etc.), establishes the connection between local abnormalities (such as brain damage, atrophy, etc.) and whole-brain structural changes, and more accurately identifies structural changes related to postoperative delirium.

[0099] Time series data self-attention is used to capture the dynamic changes in neural activity and physiology between different time periods and their dependencies, and to identify the potential relationship between abnormal neural activity in certain specific time periods before, during, and after surgery and the occurrence of perioperative delirium.

[0100] The self-attention of text feature data is used to capture semantic relationships at different levels and extract content related to the occurrence of perioperative delirium. It helps the POD risk prediction model understand the context of text data, identify the relationship between key information and other descriptive information, and extract the core content related to delirium prediction.

[0101] In terms of multi-head attention mechanism, combined with POD's actual clinical experience, the following types of attention modes are set:

[0102] Local and global spatial patterns in images: used to capture correlations between adjacent regions in the image, such as the interactions between neighboring regions of the brain; and used to capture dependencies between medium and long-range regions, such as the long-range correlations between the prefrontal cortex and the hippocampus.

[0103] Image structure and edge mode: used to capture edge or structural information in images, focusing on the boundaries between different tissues or regions, such as the boundaries of ventricular dilatation or brain atrophy. Image multi-scale mode: used to capture features at different scales, large-scale global information (such as the overall structure of the brain) and small-scale local details (such as tiny lesions or blood vessels in the brain).

[0104] Image Symmetry Mode: Used to capture the symmetry or asymmetry between the left and right brain in the image, such as differences in gray matter thickness, ventricle size, or function of specific areas between the left and right sides.

[0105] Short-term and long-term time dependency patterns of time series: used to capture the short-term and long-term time dependency features in EEG signals and capture the time series changes of brain activity.

[0106] Frequency-dependent mode: used to capture the characteristics and interactions of EEG signals in different frequency bands (such as delta waves, theta waves, alpha waves, beta waves, etc.).

[0107] Functional connectivity patterns across brain regions: used to capture functional connectivity patterns between different brain regions, such as synchronous or asynchronous activities across brain regions in EEG signals.

[0108] Waveform mutation mode: used to capture sudden waveform changes in EEG signals, such as abnormal spikes or long periods of low voltage.

[0109] Symptom and diagnosis schema: used to capture key symptom descriptions, diagnostic results, and medical history related to perioperative delirium in clinical text.

[0110] Medication and dosage patterns: used to capture descriptions of medication use, dosage, and medication schedule in clinical texts to identify potential effects of medications on POD.

[0111] Timeline mode: used to capture timeline information in clinical texts, especially important nodes such as operation time, postoperative monitoring time, and delirium occurrence time.

[0112] Important events and risk factor model: used to capture risk factors and important events related to perioperative delirium in the medical records, such as the patient's previous psychiatric history, serious infection, length of hospital stay, etc.

[0113] In addition, the cross-attention mechanism is used to guide the model during the training process of the perioperative delirium risk prediction model. This includes:

[0114] For the cross attention of image feature data and time series feature data: the image feature data blocks are used as query, and the time series feature data blocks are used as key and value, and the correlation between the features of different brain regions in the image data and the time series signals is calculated:

[0115] It helps capture the relationship between brain structure and functional activity, such as whether structural abnormalities in specific brain regions are accompanied by changes in brain electrical activity.

[0116] Cross-attention for image feature data and text feature data: Use image feature data blocks as query, text feature data blocks as key and value, and calculate the association between brain structure features in image data and medical history or diagnosis information in text:

[0117] For example, through the description in the text (such as preoperative brain atrophy or cerebral hemorrhage), the corresponding area in the image will receive extra attention.

[0118] Cross-attention for time series and clinical text: The time series feature data is divided into blocks as query, and the text feature data is divided into blocks as key and value, and the relationship between the EEG activity in different time periods in the time series data and the clinical description in the text is calculated:

[0119] For example, one can capture whether certain drug use (extracted from text) affects the fluctuations of the EEG signal through a cross-attention mechanism.

[0120] Data of different modalities usually have different Query, Key, and Value feature dimensions. This application uses matrix semi-tensor product (STP) for dimension alignment to avoid the inability to perform cross-attention weight calculations due to inconsistent dimensions.

[0121] The role of the adaptive Tokenizer is to map data from different modalities to a unified representation space so that they can interact and perform attention calculations under the same framework. Specifically, the Tokenizer is implemented in the following ways: Aligning the data dimensions of different modalities: Through the matrix semi-tensor product (MHTP) tool, the feature representations of different modalities are unified so that each modality can perform effective attention operations in the same spatial dimension. According to the characteristics of the input data (such as the length and dimension of images, time series, or text), the representation and encoding method of each modality is dynamically adjusted so that they can perform cross-attention calculations.

[0122] The matrix semi-tensor product is a mathematical operation that can be used to handle the combination of two sets of matrices or tensors of different dimensions. Specifically, the multiplication of two matrices is expanded to a product operation on partial tensors, allowing adaptive fusion of inputs with inconsistent dimensional alignment.

[0123] Through the matrix semi-tensor product, these features are expanded to the same dimension d so that they can be calculated interactively:

[0124] X′ * =STP(X * ,d)

[0125] Where X refers to K, Q, V, and the subscript * refers to image, EEG, or txt. The expanded feature matrix is ​​used to perform cross-attention calculations between modalities.

[0126] That is, before applying the cross-attention mechanism, it also includes: building an adaptive Tokenizer, aligning the dimensions of data of different modalities through matrix semi-tensor product, and generating a unified representation.

[0127] For the fusion of image feature data and time series feature data, the following formula is used to obtain the correlation between the features of different brain regions in the image data and the time series feature data:

[0128]

[0129] Among them, Q′ image K′ is the Query feature dimension of the image feature data, and is represented after alignment by the adaptive Tokenizer; EEG V′ is the Key feature dimension of the time series feature data, after being aligned by the adaptive Tokenizer; EEG It is the representation of the Value feature dimension of the time series feature data after alignment by the adaptive Tokenizer; d k is the dimension of the key vector.

[0130] For the fusion of text feature data and image feature data, the following formula is used to calculate the correlation between the brain structure features in the image feature data and the text features in the clinical text:

[0131]

[0132] Among them, Q′ text K′ is the Query feature dimension of the text feature data, and is represented after alignment by the adaptive Tokenizer; image V′ is the Key feature dimension of the image feature data, after alignment by adaptive Tokenizer; imageIt is the representation of the Value feature dimension of the image feature data after alignment by the adaptive Tokenizer;

[0133] For the fusion of text feature data and time series feature data, the following formula is used to calculate the correlation between the EEG activity in different time periods in the time series data and the text features in the clinical text:

[0134]

[0135] Among them, Q′ text K′ is the Query feature dimension of the text feature data, and is represented after alignment by the adaptive Tokenizer; EEG V′ is the Key feature dimension of the time series feature data, after being aligned by the adaptive Tokenizer; EEG It is the representation of the Value feature dimension of the time series feature data after alignment by the adaptive Tokenizer.

[0136] In the cross-attention mechanism, features of different modalities may have different importance. Adopting learnable weighting technology, an adaptive weighting mechanism is designed to dynamically adjust the weights of different modalities in the fusion process. For example, for some patients, changes in EEG signals may better reflect the risk of POD, while for other patients, imaging or text information may be more important. The model can automatically decide which modality is more important in different situations by learning adaptive weights.

[0137] The process specifically includes the following steps: assigning a corresponding weight to each modality, performing weighted fusion on the data of each modality according to the weight, and obtaining a final multimodal representation.

[0138] The weights are determined by adopting a learnable dynamic learning mechanism, and the weight generation is optimized by a loss function so that the modal weights are automatically adjusted according to task requirements.

[0139] In addition, in the embodiments of the present application, if the image, time series, and text data carry time sequence information, a masking mechanism can be introduced to ensure that the information flow is limited to the causal order. During the training process of the perioperative delirium risk prediction model, the causal attention mechanism is combined to guide the model; when the image feature data, time series feature data, and text feature data carry time sequence information, the causal mask is introduced to limit the calculation scope of attention, so that each time step can only focus on the current and previous time step data, but cannot see the future time step.

[0140] Taking EEG data as an example, we first divide the EEG data into several time steps (For example, the EEG data per second is regarded as a time step), and each time step contains multiple frequency band features of the EEG. Then the causal mask M(t) is introduced to limit the calculation scope of attention, ensuring that each time step t can only focus on the current and previous time step data, but cannot see the future time step data.

[0141]

[0142] Attention weights are calculated based only on past and current EEG signals.

[0143]

[0144] The causal attention mechanism for image and text data, as well as the causal cross-attention mechanism for image, time series, and text data are designed in the same way.

[0145] The causal attention mechanism is introduced to ensure that the model's reasoning process conforms to the temporal causal relationship and avoid the problem of "peek into the future" when processing time series or multimodal data with time order characteristics (such as images and text).

[0146] The perioperative delirium risk prediction model combines the multimodal features encoded by Transformer with the recursive reasoning module to achieve joint prediction of global features and time series features. The specific steps are as follows: Encode the image feature data, time series feature data, and text feature data through the Transformer encoder, and use the multimodal cross-attention mechanism to fuse the information of each modality; Input the encoded multimodal feature data into the recursive reasoning module, gradually infer the future EEG state or postoperative brain structure changes, and then obtain the prediction results of perioperative delirium risk.

[0147] Among them, the recursive reasoning module supports multi-step reasoning, and the recursive reasoning output of each step is used as the conditional input for the prediction of the subsequent time step. In this way, this application not only predicts the risk of POD occurrence at a certain moment after surgery, but also predicts the evolution of POD risk during the entire postoperative recovery process.

[0148] The following is a further detailed description of the process of fusing and modeling multimodal data in the method for constructing a perioperative delirium risk prediction model provided in this application.

[0149] Multimodal feature extraction process:

[0150] Electronic health record (EHR) feature processing: EHR data is converted into Token representation through feature mapping:

[0151]

[0152] Among them, h (1) is the output of the first hidden layer; h(2) is the output of the second hidden layer; h (l) is the output of the lth hidden layer, σ is the activation function, W (1) is the weight matrix of the first layer; W (2) is the weight matrix of the second layer; W (l) is the weight matrix of the lth layer; b (1) is the bias vector of the first layer; b (2) is the bias vector of the second layer; b (l) is the bias vector of the lth layer. The features are mapped to the high-dimensional embedding space through linear transformation. f(y) represents the activation function of the output layer, which converts the output of the last layer into the required format. Token is the basic unit of text.

[0153] Image feature processing includes the following processes:

[0154] Blocking: Divide the image into patches of the same size:

[0155] Patch flattening and linear mapping: Flatten each patch into a vector and map it to a high-dimensional space through linear mapping:

[0156] Position encoding: Add position information to each patch:

[0157] E i For location information.

[0158] Input the representation of all patches into the Transformer encoder:

[0159]

[0160] After L layers of Transformer encoders, each layer contains a multi-head self-attention mechanism and a feedforward neural network.

[0161] Z (l+1) = TransformerLayer(Z (l) ), l = 0, 1, ..., L-1

[0162] Final embedding:

[0163] Among them, W cls Represents the weight matrix of the classification layer, which is used to transform the final hidden layer features into the classification space. cls Represents the original output of the classification layer, which is cls The value after the linear transformation and before the activation function is applied. clsRepresents the bias vector of the classification layer, adding a learnable offset to the classification decision.

[0164] The text data processing process includes the following:

[0165] First, it needs to be tokenized and then converted into token word embedding.

[0166] Then add the positional encoding. Positional encoding is needed to let the model understand the position information of words in the text.

[0167] The text information is processed by the Transformer encoder and passed through multiple layers of self-attention and feedforward networks to obtain context-aware text representation.

[0168] Text, EHR features H EHR , and image features H image Concatenate into a joint representation:

[0169] x combined =[x text ;h EHR ;h image ]

[0170] Based on the joint representation X combined , for the target sequence y t Perform joint modeling:

[0171] Among them, P Φ represents a parameterized policy network, Φ is the parameter of the policy network. t represents the output of the model at time step t. y<t means y t Previous output.

[0172] The method of optimizing deep learning models by using parameter-efficient fine-tuning (PEFT) is used to fine-tune the model based on parameter efficiency. The optimization method is:

[0173]

[0174] Among them, △ represents the update amount or change amount, which is the adjustment size of the parameter during the optimization process. Θ represents the parameter set of the model, including all the parameters in the model that need to be learned and optimized.

[0175] The modeling method is:

[0176]

[0177] This application uses multimodal data from imaging data, time series data, and clinical texts to capture the complex correlations of perioperative delirium. Each modality provides a unique source of information: imaging features reflect abnormalities in the patient's brain structure or function (such as hippocampal atrophy); time series features describe the dynamics of brain function before and during surgery (such as abnormal fluctuations in EEG signals); text features are used to extract semantic information from medical records or postoperative records to capture risk factors (such as past medical history, drug use). Multimodal fusion significantly improves the model's ability to capture the mechanism of delirium and improves prediction accuracy compared to single-modal models.

[0178] This application uses a time alignment mechanism to align preoperative, intraoperative, and postoperative data in time steps, so that the model can model the dynamic evolution of delirium risk. A spatial alignment mechanism is used to unify the anatomical spatial position of the imaging modality (such as brain area) and the spatial distribution of the EEG signal, so that the model can understand the abnormalities of structure-function coupling. It is further combined with a feature alignment mechanism to standardize multimodal features so that imaging, time series, and text features can be compared and fused in the same feature space. These alignment operations eliminate the heterogeneity between modalities and improve the consistency and effectiveness of data fusion. This application decomposes multimodal data into smaller units, allowing the model to focus on fine-grained features. Adding embeddings of time and spatial position to each block allows the model to perceive the spatiotemporal structure of the data and capture spatiotemporal dependencies (such as the spatiotemporal correlation between intraoperative brain region changes and delirium risk).

[0179] The multimodal large model can process images, time series and text blocks at the same time, and learn the interactive relationship between modalities. This application captures key features across modalities and within modalities through the self-attention mechanism. Through a unified spatiotemporal framework, the model's ability to process multimodal information is improved. Based on the input multimodal data, the possibility of delirium at different risk levels (such as low risk, medium risk, and high risk) is predicted, which enhances the robustness and accuracy of the prediction.

[0180] In addition, through multimodal segmentation, spatiotemporal coding and attention mechanism, the model can generate interpretable prediction results. This interpretability enhances the credibility of the model in clinical practice and helps doctors understand the basis of the model's predictions.

[0181] In addition, this application also provides a perioperative delirium risk prediction method, such as Figure 2 The flowchart of the perioperative delirium risk prediction method is shown in Figure 1. The method specifically includes:

[0182] S201: Acquire multimodal data to be predicted.

[0183] S202: Inputting the multimodal data to be predicted into a pre-built perioperative delirium risk prediction model, and outputting a prediction result of the perioperative delirium risk.

[0184] Among them, the perioperative delirium risk prediction model is a model obtained according to any of the above-mentioned methods for constructing a perioperative delirium risk prediction model.

[0185] In addition, the present application also provides a perioperative delirium risk prediction device, such as Figure 3 As shown in the structural block diagram of the perioperative delirium risk prediction device, the device includes a memory 31 and a processor 32, wherein the memory 31 stores a computer program, and when the computer program is executed by the processor 32, the perioperative delirium risk prediction method described above is implemented.

[0186] In addition, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the perioperative delirium risk prediction method according to any one of the above-described methods is implemented.

[0187] Computer readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0188] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0189] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0190] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for constructing a perioperative delirium risk prediction model, characterized in that: include: Acquire multimodal data of different types from different sources collected at multiple times, perform feature extraction on the multimodal data, and obtain multimodal feature data; wherein the multimodal feature data includes image feature data related to the occurrence of perioperative delirium extracted from preoperative and intraoperative image data, time series feature data related to the occurrence of perioperative delirium extracted from time series data, and text feature data extracted from clinical texts; Performing time, space, and feature alignment on the multimodal feature data; Decomposing the multimodal feature data to obtain multimodal blocks; and generating corresponding spatiotemporal codes for each multimodal block, so that different data modalities can be integrated under a unified spatiotemporal framework; The multimodal segmentation and spatiotemporal coding are input into a multimodal large model for training to construct a perioperative delirium risk prediction model. The perioperative delirium risk prediction model is used to output a prediction result of the perioperative delirium risk level of the object to be predicted after receiving the multimodal data of the object to be predicted.

2. The method for constructing a perioperative delirium risk prediction model according to claim 1, characterized in that: During the training process of the perioperative delirium risk prediction model, a self-attention mechanism is combined to guide the model; Among them, the self-attention of image feature data is used to capture the interaction characteristics between specific brain regions and establish the connection between local abnormalities and changes in the whole brain structure; Self-attention of time series feature data is used to capture the dynamic changes of neural activity and physiology between different time periods and their dependencies, and to identify the potential relationship between abnormal neural activity in specific time periods before, during and after surgery and the occurrence of perioperative delirium; Self-attention of text feature data is used to capture semantic relationships at different levels and extract content related to the occurrence of perioperative delirium.

3. The method for constructing a perioperative delirium risk prediction model according to claim 1, characterized in that: During the training process of the perioperative delirium risk prediction model, a multi-head attention mechanism is combined to guide the model; The multi-head attention mechanism includes: Image local and global spatial patterns: used to capture the correlation between adjacent areas in the image and the dependencies between medium and long distance areas; Image structure and edge mode: used to capture edge or structural information in images, focusing on the boundaries between different tissues or regions; Image multi-scale mode: used to capture features at different scales, large-scale global information and small-scale local details; Image Symmetry Mode: used to capture the symmetry or asymmetry between the left and right brain in the image; Short-term and long-term time-dependent patterns of time series: used to capture the short-term and long-term time-dependent features in EEG signals and the time series changes of brain activity; Frequency-dependent mode: used to capture the characteristics and interactions of EEG signals in different frequency bands; Functional connectivity patterns across brain regions: used to capture the functional connectivity patterns between different brain regions; Waveform mutation mode: used to capture sudden waveform changes in EEG signals; Symptom and diagnosis model: used to capture key symptom descriptions, diagnostic results and medical history related to perioperative delirium in clinical texts; Medication and Dosage Patterns: used to capture descriptions of medication use, dosage, and timing in clinical text; Timeline mode: used to capture timeline information in clinical texts; Significant Events and Risk Factors Model: used to capture risk factors and significant events associated with perioperative delirium in the medical record.

4. The method for constructing a perioperative delirium risk prediction model according to claim 1, characterized in that: During the training process of the perioperative delirium risk prediction model, a cross-attention mechanism is combined to guide the model; Also included before applying the criss-cross attention mechanism: Construct an adaptive Tokenizer to align the dimensions of data of different modalities through matrix semi-tensor product and generate a unified representation: For the fusion of image feature data and time series feature data, the following formula is used to obtain the correlation between the features of different brain regions in the image data and the time series features: Among them, Q′ image K′ is the Query feature dimension of the image feature data, and is represented after alignment by the adaptive Tokenizer; EEG V′ is the Key feature dimension of the time series feature data, after being aligned by the adaptive Tokenizer; EEG It is the representation of the Value feature dimension of the time series feature data after alignment by the adaptive Tokenizer; d k is the dimension of the key vector; For the fusion of text feature data and image feature data, the following formula is used to calculate the correlation between the brain structure features in the image feature data and the text features in the clinical text: Among them, Q′ text K′ is the Query feature dimension of the text feature data, and is represented after alignment by the adaptive Tokenizer; image V′ is the Key feature dimension of the image feature data, after alignment by adaptive Tokenizer; image It is the representation of the Value feature dimension of the image feature data after alignment by the adaptive Tokenizer; For the fusion of text feature data and time series feature data, the following formula is used to calculate the correlation between the EEG activity in different time periods in the time series data and the text features in the clinical text: Among them, Q′ text K′ is the Query feature dimension of the text feature data, and is represented after alignment by the adaptive Tokenizer; EEG V′ is the Key feature dimension of the time series feature data, after being aligned by the adaptive Tokenizer; EEG It is the representation of the Value feature dimension of the time series feature data after alignment by the adaptive Tokenizer.

5. The method for constructing a perioperative delirium risk prediction model according to claim 4, characterized in that: In the process of applying the cross-attention mechanism, it also includes: Assign corresponding weights to each mode; Performing weighted fusion on the data of each modality according to the weights to obtain a final multimodal representation; The weights are determined by adopting a learnable dynamic learning mechanism, and the weight generation is optimized by the loss function, and the weights corresponding to each mode are automatically adjusted according to the task requirements.

6. The method for constructing a perioperative delirium risk prediction model according to any one of claims 1 to 5, characterized in that: Generating corresponding spatiotemporal coding for each multimodal block includes: The multi-scale block embedding technology is used to provide spatiotemporal position encoding for the blocks of image feature data; The sliding window position coding technology is used to provide spatiotemporal position coding for the blocks of time series feature data; The sine-cosine position coding technique is used to provide spatiotemporal position coding for the blocks of text feature data.

7. The method for constructing a perioperative delirium risk prediction model according to any one of claims 1 to 5, characterized in that: During the training process of the perioperative delirium risk prediction model, a causal attention mechanism is combined to guide the model; When image feature data, time series feature data, and text feature data carry time sequence information, the calculation scope of attention is limited by introducing a causal mask so that each time step only focuses on the current and previous time step data, and cannot see future time steps.

8. The method for constructing a perioperative delirium risk prediction model according to any one of claims 1 to 5, characterized in that: The perioperative delirium risk prediction model combines the multimodal features encoded by Transformer with a recursive reasoning module: The image feature data, time series feature data and text feature data are encoded through the Transformer encoder, and the multimodal cross-attention mechanism is used to fuse the multimodal feature data; The encoded multimodal feature data is input into the recursive reasoning module to gradually infer the EEG state at future time points or the changes in brain structure after surgery, thereby obtaining the prediction results of the perioperative delirium risk level; Among them, the recursive reasoning module supports multi-step reasoning, and the recursive reasoning output of each step serves as the conditional input for the prediction of the subsequent time step.

9. A method for predicting perioperative delirium risk, characterized in that: include: Obtain multimodal data to be predicted; Inputting the multimodal data to be predicted into a pre-built perioperative delirium risk prediction model, and outputting a prediction result of the perioperative delirium risk; Wherein, the perioperative delirium risk prediction model is a model obtained according to the method for constructing a perioperative delirium risk prediction model according to any one of claims 1 to 8.

10. A perioperative delirium risk prediction device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the perioperative delirium risk prediction method according to claim 9 is implemented.

Images