Multi-modal fusion ERCP intra-operative risk early warning method and system

Abstract

The invention discloses a risk early warning method in a multi-modal fusion ERCP operation. The method comprises the following steps: synchronizing multi-modal data; preprocessing the multi-modal data by adopting gray correction, Kalman filtering and other algorithms, and removing noise and abnormal data; image features are extracted through CNN, pressure time sequence features are extracted through TCN, and a fusion feature vector is output in combination with an attention fusion algorithm; inputting the fusion feature vector into a Transform dichotomy model, calculating a risk probability through a sigmoid activation function, judging a risk level based on a preset threshold value, and triggering early warning; feedback information is received, and the model is iteratively optimized through a parameter updating algorithm. The system comprises a corresponding data acquisition module, a preprocessing module, a feature fusion module, a risk early warning module, a feedback iteration module and a storage module. According to the method, the accuracy and timeliness of risk early warning in the ERCP operation are remarkably improved, the occurrence rate of complications is reduced, the operation safety is improved, and the workload of medical staff is relieved.

Description

Technical Field

[0001] This invention relates to the field of medical auxiliary information processing technology, specifically to a multimodal fusion method for intraoperative risk warning during ERCP. Background Technology

[0002] Endoscopic retrograde cholangiopancreatography (ERCP) is an important minimally invasive surgery for the diagnosis and treatment of pancreatic and biliary diseases. It has advantages such as minimal trauma and rapid recovery. However, the surgical area has a complex anatomical structure and involves sensitive tissues such as bile ducts, pancreatic ducts, and blood vessels. Serious complications such as bleeding and perforation are prone to occur during the operation. If these complications are not detected and intervened in time, they may lead to the deterioration of the patient's condition or even endanger their life.

[0003] Currently, intraoperative risk monitoring during ERCP mainly relies on the experience and judgment of medical staff, supplemented by single-modality monitoring methods, which presents the following technical challenges: First, existing monitoring methods are mostly single-modality, such as relying solely on cholangiography to observe anatomical abnormalities or solely on pressure monitoring to determine changes in pressure within the pancreaticobiliary duct. This fails to integrate multi-dimensional information for comprehensive judgment, easily leading to missed or incorrect diagnoses. For example, when no obvious abnormalities are shown on imaging but pressure changes suddenly, it is difficult to quickly associate the risk. Second, manual monitoring depends on the operational experience and concentration of medical staff. ERCP surgery is difficult and time-consuming, and medical staff are under constant high stress, which can easily lead to delayed risk warnings due to fatigue and judgment errors, missing the optimal intervention opportunity. Third, the few existing warning solutions that attempt to combine AI mostly focus on feature recognition of a single imaging modality, without combining the surgical operation sequence information, failing to match the dynamic changes of the surgical process, resulting in insufficient timeliness and accuracy of warnings. Fourth, existing systems lack standardized risk grading mechanisms and clear intervention prompts. Even when warnings are issued, medical staff find it difficult to quickly determine the risk level and corresponding treatment measures, further affecting intervention efficiency.

[0004] In summary, existing ERCP intraoperative risk monitoring technologies suffer from problems such as incomplete information integration, poor accuracy and timeliness of early warnings, reliance on human experience, and lack of effective intervention guidance. They cannot meet the clinical surgical needs for accurate and real-time risk warnings, and there is an urgent need for a technical solution that can integrate multimodal information and achieve intelligent and dynamic risk warnings. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention discloses a multimodal fusion-based intraoperative risk warning method for ERCP, in order to solve the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal fusion-based intraoperative risk warning method for ERCP, comprising the following steps:

[0007] S1. First, acquire multimodal data during ERCP, including pancreaticobiliary imaging data, pancreaticobiliary pressure monitoring data, and surgical operation timing data.

[0008] S2. Standardize the multimodal data to remove noise and abnormal data, and achieve temporal consistency alignment of each modality data.

[0009] S3. Perform multimodal feature fusion, extract the features of each modal data after preprocessing, and fuse the features of each modality through an attention mechanism to obtain a fused feature vector;

[0010] S4. Input the fused feature vector into a preset risk classification model, calculate the risk probability and determine the risk level in real time, and trigger the corresponding early warning signal and intervention prompt according to the risk level.

[0011] S5. Receive early warning feedback information from medical staff, supplement training samples, and iteratively optimize the risk classification model and feature fusion related algorithms.

[0012] Preferably, in step S1, the specific steps of data acquisition are as follows: Multimodal data is simultaneously acquired intraoperatively through the image acquisition module, pressure sensor module, and operation timing recording module of the ERCP surgical equipment. The multimodal data includes pancreaticobiliary imaging data, pancreaticobiliary pressure monitoring data, and surgical operation timing data; wherein, the pancreaticobiliary imaging data is a sequence of pancreaticobiliary images captured in real-time during the operation, denoted as:

[0013] ;

[0014] Where T is the total number of frames captured, x img，t Let represent the image data of frame t, where t = 1, 2, ..., T; the pressure monitoring data consists of continuous pressure values ​​from 1 to 10 frames per second, denoted as:

[0015] ;

[0016] Where x pre,t This represents the pressure value at time t; the operation timing data includes the start time, duration, and operation type of key operations such as instrument insertion, contrast agent injection, and sphincter incision, denoted as:

[0017] ;

[0018] Where, x seq,t Indicates the operation type at time t.

[0019] Preferably, in step S2, the pancreaticobiliary contrast imaging data preprocessing involves sequentially performing grayscale correction, image enhancement, and region of interest (ROI) extraction, using the following specific formulas:

[0020] 1) Grayscale correction: A linear grayscale correction algorithm is used, and the correction formula is as follows:

[0021] ;

[0022] Where x' img,t (i,j) represents the original grayscale value of pixel (i,j) in the t-th frame of the image, x' img,t (i,j) represents the corrected grayscale value, a represents the contrast adjustment coefficient, and b represents the brightness adjustment coefficient. The optimal values ​​of a and b are determined by least squares fitting.

[0023] 2) Image Enhancement: A histogram equalization algorithm is used. First, the image grayscale histogram is calculated:

[0024] ;

[0025] n k Let k be the number of pixels with a grayscale value of k, and M×N be the image resolution; then calculate the cumulative distribution function. Finally, through x'' img,t (i,j) = round(255·cdf (x' img,t (i,j))) Implements grayscale value mapping to improve image contrast;

[0026] 3) ROI Extraction: The Otsu thresholding algorithm is used to calculate the optimal segmentation threshold.

[0027] ;

[0028] Where w0(T) and w1(T) are the pixel proportions of the background and target regions after threshold T segmentation, respectively; u0(T) and u1(T) are the average grayscale values ​​of the background and target regions, respectively; and u is the average grayscale value of the entire image; This can be achieved using the following formula:

[0029] ;

[0030] Then, the ROI region is extracted.

[0031] Preferably, in step S2, the pressure monitoring data preprocessing involves using Kalman filtering to remove outlier data and linear interpolation to supplement missing data. The specific formula is as follows:

[0032] 1) Kalman filter noise reduction:

[0033] The state equation is: x pre,k = A·x pre,k-1 + w k-1

[0034] The observation equation is: zpre,k = H·x pre,k + v k ,

[0035] Where, x pre,k Let A represent the true pressure value at time k, H be the state transition matrix, and W be the observation matrix. k-1 For process noise, v k To observe noise;

[0036] Filtering update steps:

[0037] ① Prediction:

[0038] MRI-based automated brain tissue segmentation method and system pre, k|k- 1 =A· pre , k- 1|k- 1 ;

[0039] P k|k-1= A·P k-1|k-1 · A T +Q;

[0040] ② Update:

[0041] K k =P k|k-1 ·H T · (H·P k|k-1 · H T +R) - ¹;

[0042] pre ,k|k = pre,K | k-1 +K k ·(Z pre,k -H· pre,k|k-1 );

[0043] P k | k =(IK k ·H)·P k|k-1 ;

[0044] Where I is the identity matrix; filtering removes residual outliers; P k | k K represents the covariance of the optimal estimate after filtering at time k. k This represents the Kalman gain at time k.

[0045] After filtering, if the difference between the filtered values ​​at adjacent time points exceeds a set threshold, it is determined to be a residual outlier and replaced. The determination formula is as follows:

[0046] | pre,k | k pre ,k-1|k-1 |>10mmHg / s;

[0047] That is, a pressure change exceeding 10 mmHg per second is considered abnormal.

[0048] 2) Linear interpolation completion: For missing data points x pre, m Through adjacent valid data points x pre,i and x pre,j Interpolation calculation, the formula is:

[0049] ;

[0050] Where, x pre,m This represents the missing pressure data point at time m;

[0051] 3) Standardization: The min-max standardization algorithm is used, and the formula is:

[0052] ;

[0053] Where x''' pre, t X represents the standardized pressure value at time t; pre, filter This is the filtered pressure data set, and the standardized data is mapped to the 0-1 interval.

[0054] Preferably, in step S2, the operation timing data preprocessing involves standardizing the operation type using one-hot encoding, with the following formula:

[0055] x''' seq,t = [0,...,1,...,0]

[0056] Where x''' seq,t This represents the operation timing vector after one-hot encoding at time t; the data of each modality is synchronized using a timestamp alignment algorithm, with the formula: t sync = max(t img , t pre , t seq ), where t sync t represents the unified timestamp after synchronization. img The timestamp of the image data, t pre The timestamp representing the stress data, t seq Indicates the time of operation timing data; interpolates and aligns the data of each modality to the synchronization timestamp t. sync This ensures timing consistency.

[0057] Preferably, in step S3, a multimodal feature fusion model is constructed, features of each preprocessed data are extracted, and fused through an attention mechanism. The specific steps are as follows:

[0058] 1) Image feature extraction: A ResNet50 convolutional neural network was used, and the extraction process was as follows: F img = CNN(X''' img ) =f ResNet50 (x''' img,1 , x''' img,2 ..., x''' img,T ), where X''' img For the preprocessed image sequence, f ResNet50 The feature extraction function for ResNet50 outputs a 256-dimensional image feature vector F. img ∈R 256 The vector elements correspond to the characteristics of pancreatic and bile duct wall thickness, morphology, and blood vessel distribution.

[0059] 2) Temporal feature extraction of stress: A temporal convolutional network is used, and the convolution operation formula is as follows:

[0060] ;

[0061] Among them, f TCN (·) represents the feature extraction function of the temporal convolutional network, K is the kernel size, d is the dilation coefficient, and w k The weights are the convolution kernel weights; through 3 layers of temporal convolution and activation function operations, a 128-dimensional stress temporal feature vector F is output. pre ∈R 128 The vector elements correspond to the characteristics of pressure change rate, peak value, and fluctuation amplitude.

[0062] 3) Operation timing feature encoding: Encode the preprocessed operation timing feature x''' seq, t The feature vector is mapped to a 64-dimensional feature vector through a fully connected layer, and the encoding formula is F. seq = W seq · X''' seq + b seq W seq ∈R 64×C Let b be the weight matrix. seq ∈R 64 For bias terms of fully connected layers;

[0063] 4) Attention Fusion: A scaled dot product attention mechanism is used to perform attention fusion on the image feature vector F. img Pressure time series eigenvector F pre and the operation timing feature vector F seqWeights are assigned to highlight features related to bleeding and perforation risk, and then fusion is performed to obtain the final fused feature vector. The specific formulas, symbol definitions, and steps are as follows:

[0064] ① Unify Feature Dimension: First, adjust the feature vectors of the three modalities to the same dimension to facilitate weight calculation. The adjustment method is a linear transformation. The adjusted feature vector is denoted as... , , ;

[0065] ② Correlation Score Calculation: Calculate the correlation score between the adjusted feature vector of each modality and the preset risk feature vector to measure the importance of each modality feature for risk warning. The formula is:

[0066] ;

[0067] ;

[0068] ;

[0069] in, The score represents the correlation between image features and risk features. This represents the correlation score between the time-series characteristics of stress and the characteristics of risk. This represents the correlation score between operational time-series characteristics and risk characteristics. This represents the image feature vector after dimensional adjustment; This represents the time-series feature vector of stress after dimensional adjustment; This represents the operational time-series feature vector after dimensional adjustment; Represents the query matrix; This represents the key matrix, used to convert risk feature vectors into key vectors; This represents a preset risk feature vector; This represents the matrix transpose operation;

[0070] ③ Weight Normalization: The relevance scores are normalized into weight coefficients using the softmax function. The formula is as follows:

[0071] ;

[0072] ;

[0073] ;

[0074] in Weighting coefficients representing image features; Weighting coefficients representing the time-series characteristics of pressure; Weighting coefficients representing the timing characteristics of operations;

[0075] and ; D represents the exponential function; D represents the dimension of the feature vector.

[0076] ④ Multimodal feature fusion: Multiply the adjusted feature vectors of each modality by their corresponding weight coefficients, then sum them to obtain the fused feature vector. The formula is:

[0077] ;

[0078] in: This represents the multimodal fusion feature vector.

[0079] Preferably, in step S4, the fused feature vectors are... Input a preset risk classification model to determine in real time whether there is a risk of bleeding or perforation during the current surgical procedure, and output the risk level; the specific steps are as follows:

[0080] Risk classification model training: Using a large amount of intraoperative multimodal data of ERCP labeled with bleeding and perforation risks, a Transformer binary classification model was trained to determine the model parameters and ensure the model's generalization ability. The specific steps are as follows:

[0081] ① Training samples: Collect multimodal data from N ERCP surgeries, with each sample corresponding to a fused feature vector. A risk label The sample set is denoted as:

[0082] ;

[0083] Where N represents the total number of training samples. This represents the fused feature vector of the i-th sample. ; This represents the risk label of the i-th sample; i represents the sample number.

[0084] ② Loss Function: The cross-entropy loss function is used to measure the deviation between the model's predicted values ​​and the true labels, and is used to optimize the model parameters. The formula is:

[0085] ;

[0086] Where L represents the cross-entropy loss value; This represents the probability that the model predicts the i-th sample has a risk.

[0087] ③ Optimizer: The Adam optimizer is used to update model parameters and minimize the loss function. The optimizer formula is:

[0088] ;

[0089] ;

[0090] ;

[0091] in: Denotes the first-order momentum in the t-th iteration; Denotes the first-order momentum in the (t-1)th iteration; This represents the first-order momentum decay coefficient; This represents the loss gradient in the t-th iteration; Let represent the second-order momentum of the t-th iteration; Let represent the second-order momentum of the (t-1)th iteration; This represents the second-order momentum decay coefficient; This represents the square of the loss gradient; —Model parameters in the t-th iteration; This represents the learning rate, used to adjust the step size for parameter updates. Describing second momentum The square root of; This represents a small constant to prevent the denominator from being zero; t represents the iteration number.

[0092] ④ Cross-validation: Five-fold cross-validation is used to ensure the model's generalization ability and avoid overfitting. The formula for the validation accuracy is:

[0093] ;

[0094] Among them, Accuracy k This represents the accuracy of the k-th fold validation set; k represents the fold number.

[0095] ⑤ Model Inference: The fused feature vectors generated in real time... The trained Transformer model is input and then subjected to layer normalization, multi-head attention, and feedforward network operations in sequence. Finally, the probability of bleeding or perforation at the current surgical moment is calculated using the sigmoid activation function. ;

[0096] ⑥ Risk level determination and early warning triggering: based on risk probability The size of the risk level is determined by setting a threshold, triggering corresponding early warning signals and intervention prompts. The determination formula is as follows:

[0097] ;

[0098] Here, Level represents the risk level at the current moment of surgery.

[0099] Preferably, in step S5, the early warning feedback and model iteration step involves medical staff inputting feedback information through the feedback iteration module. 0 represents a correct warning, 1 represents a missed warning, and 2 represents a false warning;

[0100] Feedback information and corresponding samples Store to the storage module; after accumulating a certain number of feedback samples, retrain the model, and update the parameters using the following formula:

[0101] ,

[0102] in, This refers to the amount of parameter updates during the new training process.

[0103] This invention also provides a multimodal fusion-based intraoperative risk warning system for ERCP, the system comprising:

[0104] The data acquisition module is used to simultaneously acquire pancreaticobiliary imaging data, pancreaticobiliary pressure monitoring data, and surgical operation sequence data during ERCP.

[0105] The data preprocessing module, electrically connected to the data acquisition module, is used to standardize the acquired multimodal data, remove noise and abnormal data, and achieve time-series alignment of the data of each modality.

[0106] The multimodal feature fusion module is electrically connected to the data preprocessing module and is used to extract features from each modality of data and fuse them through an attention mechanism to obtain a fused feature vector.

[0107] The risk warning module is electrically connected to the multimodal feature fusion module and has a built-in risk classification model for calculating risk probability, determining risk level, and triggering corresponding warning signals and intervention prompts.

[0108] The feedback iteration module, electrically connected to the risk warning module, is used to receive feedback information from medical staff, supplement training samples, and achieve iterative optimization of the model.

[0109] The storage module, electrically connected to each module, is used to store various types of data, model parameters, and warning-related records, facilitating subsequent queries, reviews, and model iterations.

[0110] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0111] 1. In this invention, multimodal feature fusion (image, pressure, operation sequence) and attention mechanism are used to allocate weights. Combined with algorithms such as CNN, TCN, and Transformer, risk-related features are accurately extracted. With a complete data preprocessing process, the rate of missed and false judgments is effectively reduced, the accuracy of bleeding and perforation risk warnings is improved, and a comprehensive and accurate risk assessment is achieved.

[0112] 2. This invention enables automated deployment without the need for manual operation by medical staff. Furthermore, through the lightweight design of the algorithm, namely the simplified ResNet50 structure and CUDA parallel acceleration, it provides medical staff with sufficient intervention time and avoids early warning delays caused by fatigue or judgment errors.

[0113] 3. This invention calculates the risk probability using the sigmoid activation function and establishes a risk grading mechanism. Each risk level corresponds to a clear warning signal and targeted intervention prompts, eliminating the need for medical staff to rely on experience to determine the treatment method. This reduces reliance on the experience of medical staff, making it particularly suitable for junior medical staff. It significantly improves the efficiency of risk response and treatment, and further reduces the incidence of complications such as bleeding and perforation during ERCP. Attached Figure Description

[0114] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0115] In the attached diagram:

[0116] Figure 1 This is a flowchart illustrating a multimodal fusion-based intraoperative risk warning method for ERCP according to the present invention.

[0117] Figure 2 This is a framework diagram of a multimodal fusion ERCP intraoperative risk early warning system according to the present invention. Detailed Implementation

[0118] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0119] Example: Figure 1 As shown, a multimodal fusion method for intraoperative risk warning in ERCP includes the following steps:

[0120] S1. Specific steps for data collection:

[0121] The ERCP endoscope's image acquisition unit (high-definition industrial camera, model IMX586) acquires pancreaticobiliary contrast imaging data in real time, with an acquisition frame rate of 15 frames / second and an image resolution of 1080P (1920×1080). An implanted fiber optic pressure sensor (model FOP-M260) acquires pancreaticobiliary pressure monitoring data, with an acquisition frequency of 5 frames / second. A timing recording unit (STM32F407 microcontroller) records the timing data of the surgical procedure.

[0122] Among them, the pancreatocholangiography imaging data is a sequence of pancreatocholangiography images captured in real time during the operation, denoted as:

[0123] ;

[0124] Where T is the total number of frames captured, x img，t Let represent the image data of frame t, where t = 1, 2, ..., T; the pressure monitoring data consists of continuous pressure values ​​from 1 to 10 frames per second, denoted as:

[0125] ;

[0126] Where x pre,t This represents the pressure value at time t; the operation timing data includes the start time, duration, and operation type of key operations such as instrument insertion, contrast agent injection, and sphincter incision, denoted as:

[0127] ;

[0128] x seq,t The operation type code at time t is represented by a one-hot coding method, such as [1,0,0,0] for instrument insertion and [0,1,0,0] for contrast agent injection.

[0129] S2. Data preprocessing steps:

[0130] Preprocessing of pancreaticobiliary contrast imaging data: Grayscale correction, image enhancement, and region of interest (ROI) extraction are performed sequentially, using the following formulas:

[0131] 1) Grayscale correction: A linear grayscale correction algorithm is used, and the correction formula is as follows:

[0132] ;

[0133] Where x' img,t (i,j) represents the original grayscale value of pixel (i,j) in the t-th frame of the image, x' img,t (i,j) represents the corrected grayscale value, a is the contrast adjustment coefficient with a value range of 1.0-1.5, and b is the brightness adjustment coefficient with a value range of 0-50. a=1.2 and b=20. The optimal values ​​of a and b are determined by least squares fitting to ensure that the grayscale value is distributed in the range of 0-255.

[0134] 2) Image Enhancement: A histogram equalization algorithm is used. First, the image grayscale histogram is calculated:

[0135] ;

[0136] n k Let k be the number of pixels with a grayscale value of k, and M×N be the image resolution; then calculate the cumulative distribution function. Finally, through x''img,t (i,j) = round(255·cdf (x' img,t (i,j))) achieves grayscale mapping, enhancing the contrast between the pancreatic and bile duct wall and surrounding tissues by more than 20%;

[0137] 3) ROI Extraction: The Otsu thresholding algorithm is used to calculate the optimal segmentation threshold.

[0138] ;

[0139] Where w0(T) and w1(T) are the pixel proportions of the background and target regions after threshold T segmentation, respectively; u0(T) and u1(T) are the average grayscale values ​​of the background and target regions, respectively; and u is the average grayscale value of the entire image; This can be achieved using the following formula:

[0140] ;

[0141] Then, the ROI region is extracted, preserving key features such as the pancreatic and bile duct walls and blood vessels.

[0142] Pressure monitoring data preprocessing: Kalman filtering is used to remove outlier data, and linear interpolation is used to supplement missing data. The specific formulas are as follows:

[0143] 1) Kalman filter noise reduction:

[0144] The state equation is: x pre,k = A·x pre,k-1 + w k-1 ;

[0145] The observation equation is: z pre,k = H·x pre,k + v k ;

[0146] Where, x pre,k Let A represent the true pressure value at time k, A be the state transition matrix (value 1), H be the observation matrix (value 1), and w be the value of the observation matrix. k-1 For process noise, variance Q = 0.01, v k For the observed noise, the variance R = 0.1; the filtering update steps are as follows:

[0147] ① Prediction:

[0148] pre, k|k- 1 =A· pre , k- 1|k- 1 ;

[0149] P k|k-1= A·P k-1|k-1 · A T +Q;

[0150] ② Update:

[0151] K k =P k|k-1 ·H T · (H·P k|k-1 · H T +R) - ¹;

[0152] pre ,k|k = pre,K | k-1 +K k ·(Z pre,k -H· pre,k|k-1 );

[0153] P k | k =(IK k ·H)·P k|k-1 ;

[0154] Where I is the identity matrix; filtering removes residual outliers; P k | k K represents the covariance of the optimal estimate after filtering at time k. k This represents the Kalman gain at time k.

[0155] After filtering, if the difference between the filtered values ​​at adjacent time points exceeds a set threshold, it is determined to be a residual outlier and replaced. The determination formula is as follows:

[0156] | pre,k | k pre ,k-1|k-1 |>10mmHg / s;

[0157] That is, a pressure change exceeding 10 mmHg per second is considered abnormal.

[0158] 2) Linear interpolation completion: For missing data points x pre, m (Missing marker is f) lagm =0), through adjacent valid data points x pre,i and x pre,j Interpolation calculation, the formula is:

[0159] ;

[0160] Where, x pre,m This represents the missing pressure data point at time m;

[0161] 3) Standardization: The min-max standardization algorithm is used, and the formula is:

[0162] ;

[0163] Where x''' pre, t X represents the standardized pressure value at time t; pre, filter This is the filtered pressure data set, and the standardized data is mapped to the 0-1 interval.

[0164] Operation timing data preprocessing: Operation types are standardized using one-hot encoding, with the following formula:

[0165] x''' seq,t = [0,...,1,...,0]

[0166] Where x''' seq,t This represents the operation time sequence vector after one-hot encoding at time t (the encoding length equals the total number of operation types, with the corresponding operation type position being 1 and the rest being 0); the data of each modality is synchronized using a timestamp alignment algorithm, with the formula: t sync = max(t img , t pre , t seq ), where t sync t represents the unified timestamp after synchronization. img The timestamp of the image data, t pre The timestamp representing the stress data, t seq Indicates the time of operation timing data; interpolates and aligns the data of each modality to the synchronization timestamp t. sync This ensures timing consistency.

[0167] S3, Multimodal Feature Fusion:

[0168] A multimodal feature fusion model is constructed, which extracts features from each preprocessed data and fuses them using an attention mechanism. The specific steps are as follows:

[0169] 1) Image Feature Extraction (CNN Algorithm): A ResNet50 convolutional neural network is used. The extraction process is as follows: F... img = CNN(X''' img ) = f ResNet50 (x''' img,1 , x''' img,2 ..., x''' img,T ), where X''' img For the preprocessed image sequence, f ResNet50 The feature extraction function for ResNet50 (including convolution, batch normalization, and residual connection operations) outputs a 256-dimensional image feature vector F. img ∈R 256 The vector elements correspond to the characteristics of pancreatic and bile duct wall thickness, morphology, and blood vessel distribution, where fResNet50 The kernel sizes are 7×7 and 3×3, respectively, and the strides are 2 and 1, respectively. The activation function is ReLU.

[0170] 2) Temporal feature extraction of stress (TCN algorithm): A temporal convolutional network is used, and the convolution operation formula is as follows:

[0171] ;

[0172] Among them, f TCN (·) represents the feature extraction function of the temporal convolutional network, K is the kernel size (value 5), d is the dilation coefficient (values ​​1, 2, 4), and w k The weights are the convolution kernel weights; through 3 layers of temporal convolution, activation function (ReLU(x) = max(0,x)) and dropout (dropout probability 0.3), a 128-dimensional stress temporal feature vector F is output. pre ∈R 128 The vector elements correspond to the characteristics of pressure change rate, peak value, and fluctuation amplitude.

[0173] 3) Operation timing feature encoding: Encode the preprocessed operation timing feature x''' seq, t The feature vector is mapped to a 64-dimensional feature vector through a fully connected layer, and the encoding formula is F. seq = W seq · X''' seq + b seq W seq ∈R 64×C Here is the weight matrix, where C is the total number of operation types, and b is the weight matrix. seq ∈R 64 For the bias term of the fully connected layer, the output F seq Belongs to R 64 ;

[0174] 4) Attention Fusion: A scaled dot product attention mechanism is used to perform attention fusion on the image feature vector F. img Pressure time series eigenvector F pre and the operation timing feature vector F seq Weights are assigned to highlight features related to bleeding and perforation risk, and then fusion is performed to obtain the final fused feature vector. The specific formulas, symbol definitions, and steps are as follows:

[0175] ① Unified Feature Dimension: First, adjust the feature vectors of the three modalities to the same dimension, i.e., 256 dimensions, to facilitate weight calculation. The adjustment method is linear transformation (image features). It has reached 256 dimensions, and the pressure characteristics are... Operation timing characteristics (After linear transformation to 256 dimensions via a fully connected layer), the adjusted feature vector is denoted as... , , (The symbol definition is the same as the original feature vector, only the dimension is adjusted).

[0176] ② Correlation Score Calculation: Calculate the correlation score between the adjusted feature vector of each modality and the preset risk feature vector to measure the importance of each modality feature for risk warning. The formula is:

[0177] ;

[0178] ;

[0179] ;

[0180] in, The score represents the correlation between image features and risk features. This represents the correlation score between the time-series characteristics of stress and the characteristics of risk. This represents the correlation score between operational time-series characteristics and risk characteristics. This represents the image feature vector after dimension adjustment. ; This represents the time-series feature vector of stress after dimensional adjustment. ; This represents the operational time-series feature vector after dimensional adjustment. ; Represents a query matrix with dimensions of This is used to convert feature vectors into query vectors; Represents the key matrix, with dimension . This is used to convert risk feature vectors into key vectors; This represents a preset risk feature vector. It is determined by the average feature vector of a large number of samples with bleeding and perforation risk; This represents the matrix transpose operation;

[0181] ③ Weight Normalization: The relevance scores are normalized into weight coefficients using the softmax function. The formula is as follows:

[0182] ;

[0183] ;

[0184] ;

[0185] in The weighting coefficient represents the image features, and its value ranges from 0 to 1; The weighting coefficients represent the time-series characteristics of pressure, with values ​​ranging from 0 to 1; The weighting coefficients represent the operational time-series characteristics, with values ​​ranging from 0 to 1; and ; D represents the exponential function; D represents the dimension of the feature vector.

[0186] ④ Multimodal feature fusion: Multiply the adjusted feature vectors of each modality by their corresponding weight coefficients, then sum them to obtain the fused feature vector. The formula is:

[0187] ;

[0188] in: This represents a multimodal fusion feature vector.

[0189] First, the feature vectors of the three modalities are adjusted to 256 dimensions; then, the correlation score between each modality's features and the preset risk features is calculated; next, the weight coefficients of each modality are obtained by normalization using the softmax function; finally, the weighted feature vectors are summed to output a 512-dimensional fused feature vector. This vector integrates the core risk-related features of the three modalities, providing comprehensive and accurate input for subsequent risk warnings.

[0190] S4. Risk Warning:

[0191] fuse feature vectors Input a preset risk classification model, employing a Transformer binary classification model to determine in real time whether there is a risk of bleeding or perforation during the current surgical procedure, and output the risk level; if the risk level is medium or high, an early warning signal is triggered, and corresponding intervention prompts are generated; simultaneously, the accuracy of the early warning is ensured through model training, with the specific steps as follows:

[0192] Risk classification model training: Using a large amount of intraoperative multimodal data of ERCP labeled with bleeding and perforation risks, a Transformer binary classification model was trained to determine the model parameters and ensure the model's generalization ability. The specific steps are as follows:

[0193] ① Training samples: Collect multimodal data from N ERCP surgeries, with each sample corresponding to a fused feature vector. A risk label The sample set is denoted as:

[0194] ;

[0195] Where N represents the total number of training samples. This represents the fused feature vector of the i-th sample. ; The risk label for the i-th sample is represented by a value of 1 or 0 (1 indicates a risk of bleeding / perforation, and 0 indicates no risk); i represents the sample number, which can be 1, 2, ..., N.

[0196] ② Loss Function: The cross-entropy loss function is used to measure the deviation between the model's predicted values ​​and the true labels, and is used to optimize the model parameters. The formula is:

[0197] ;

[0198] Where L represents the cross-entropy loss value, the smaller the loss value, the more accurate the model prediction; This represents the probability that the model predicts the i-th sample has a risk.

[0199] ③ Optimizer: The Adam optimizer is used to update model parameters and minimize the loss function. The optimizer formula is:

[0200] ;

[0201] ;

[0202] ;

[0203] in: Let represent the first-order momentum of the t-th iteration, used to accumulate the first-order moment of the gradient; Denotes the first-order momentum in the (t-1)th iteration; This represents the first-order momentum decay coefficient; This represents the loss gradient in the t-th iteration, which is the partial derivative of the loss function L with respect to the model parameters; Let represent the second-order momentum of the t-th iteration, used to accumulate the second-order moment of the gradient; Let represent the second-order momentum of the (t-1)th iteration; This represents the second-order momentum decay coefficient; This represents the square of the loss gradient; —Model parameters (weight matrix, bias term) for the t-th iteration; This represents the learning rate, which takes a value of 1e-4 and is used to adjust the step size for parameter updates. Describing second momentum The square root of; represents a small constant to prevent the denominator from being zero, with a value of 1e-8; t represents the iteration number;

[0204] ④ Cross-validation: Five-fold cross-validation is used to ensure the model's generalization ability and avoid overfitting. The formula for the validation accuracy is:

[0205] ;

[0206] Among them, Accuracy k This represents the accuracy of the k-th fold validation set; k represents the fold number.

[0207] Specifically, the training samples are divided into training, validation, and test sets in a 7:2:1 ratio; model parameters are initialized; the model is iterated for 100 rounds, with the loss value L calculated using the training set in each round, and the model parameters updated using the Adam optimizer; after each iteration, the accuracy is calculated using the validation set, and an early stopping strategy is adopted (training stops if the validation set loss does not decrease for 10 consecutive rounds); finally, the model parameters are optimized using 5-fold cross-validation to obtain a well-trained risk classification model with a test set accuracy ≥92%;

[0208] ⑤ Model Inference (Risk Probability Calculation): This involves applying the real-time generated fused feature vectors... The trained Transformer model is input and then subjected to layer normalization, multi-head attention, and feedforward network operations in sequence. Finally, the probability of bleeding or perforation at the current surgical moment is calculated using the sigmoid activation function. Inference delay ≤ 0.2 seconds to ensure timely warning;

[0209] ⑥ Risk level determination and early warning triggering: based on risk probability The size of the risk level is determined by setting a threshold, triggering corresponding early warning signals and intervention prompts. The determination formula is as follows:

[0210] ;

[0211] Among them, Level represents the risk level at the current surgical moment, which is determined through validation of a large number of clinical samples to ensure the accuracy of the classification;

[0212] Specifically, the calculated risk probability Substituting the values ​​into the judgment formula, the risk level is determined. If the risk is low, no warning is triggered. If the risk is medium, the warning display unit displays a yellow warning, the sound alarm unit emits a low-frequency alarm sound (500Hz, 1-second interval), and outputs an intervention prompt: "Please closely monitor pressure changes and imaging characteristics, and slow down the operation speed." If the risk is high, the warning display unit displays a red warning, the sound alarm unit emits a high-frequency alarm sound (800Hz, continuous sound), and outputs an intervention prompt: "Suspected bleeding / perforation risk, please stop the operation immediately and perform endoscopic examination and intervention," ensuring a rapid response from medical staff.

[0213] S5. Early Warning Feedback and Model Iteration:

[0214] Medical staff input feedback information through the feedback iteration module. 0 represents a correct warning, 1 represents a missed warning, and 2 represents a false warning;

[0215] Feedback information and corresponding samples Store to the storage module; retrain the model every 200 accumulated feedback samples, with the parameter update formula as follows:

[0216] ,

[0217] in, The parameter update amount in the new training process (learning rate adjusted to 5e-5) improves the model's warning accuracy by ≥5% after iteration, continuously optimizing the model's generalization ability.

[0218] like Figure 2 As shown, the present invention also provides a multimodal fusion-based intraoperative risk warning system for ERCP, the system comprising:

[0219] The data acquisition module employs a high-definition industrial camera (model: IMX586), connected to the eyepiece of the ERCP endoscope, capable of capturing real-time pancreaticobiliary contrast images and outputting 1080P resolution image data, which is transmitted to the data preprocessing module via a USB 3.0 interface. An implantable fiber optic pressure sensor (model: FOP-M260) is used, implanted into the pancreaticobiliary duct through the ERCP endoscope biopsy channel, with a measurement range of 0-100 mmHg and a measurement accuracy of ±0.5 mmHg. Pressure signals are transmitted via fiber optic cable, converted to digital signals by a signal converter, and then transmitted to the data preprocessing module. A microcontroller (model: STM32F407) is connected to the buttons on the surgical control console, recording the trigger time and operation type of each key operation, generating time-series data, and calibrating the timestamps via GPS timing to synchronize data from each unit. The synchronization algorithm uses the aforementioned timestamp alignment formula.

[0220] The data preprocessing module uses an FPGA chip (model: Xilinx Zynq-7000) and incorporates the aforementioned grayscale correction, histogram equalization, Otsu threshold segmentation, Kalman filtering, linear interpolation, normalization, and one-hot coding algorithms. The algorithms are hardware-accelerated and can quickly process multimodal data (processing latency ≤ 0.3 seconds), remove noise and outlier data, and output preprocessed image data, stress data, and time series data, which are then transmitted to the multimodal feature fusion module.

[0221] The multimodal feature fusion module uses a GPU (model: NVIDIA RTX 3090) and incorporates the multimodal feature fusion model (ResNet50 CNN, TCN timely attention fusion algorithm) from Example 1 above. It can process image feature extraction, stress time series feature extraction and feature fusion in parallel, output fused feature vector, and transmit it to the risk warning module. The model parameters are stored in the GPU cache for easy and quick access.

[0222] The risk warning module can display real-time risk levels, warning prompts, multimodal data waveforms (pressure change curves, image frames), and simultaneously display the risk probability P. risk The system includes the core parameter values ​​of each algorithm; it can emit alarm sounds of different frequencies to distinguish between medium and high risks; the control chip has a built-in risk classification model (Transformer binary classification model) and the above-mentioned risk probability calculation and risk level determination algorithms, receives fused feature vectors, judges the risk level in real time, and triggers early warning signals.

[0223] The feedback iteration module uses touch buttons and a data transmission interface. Medical staff input feedback information (correct warning, missed judgment, misjudgment) through touch buttons. The feedback information is transmitted to the storage module through the control chip, and at the same time, it triggers the model iteration command to control the multimodal feature fusion module and the risk warning module to iteratively optimize the model. The iteration algorithm uses the above parameter update formula, and the iteration process is completed automatically without manual intervention.

[0224] The storage module, electrically connected to each module, is used to store various types of data, model parameters, and warning-related records, facilitating subsequent queries, reviews, and model iterations.

[0225] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal fusion method for intraoperative risk warning in ERCP, characterized in that, Includes the following steps: S1. First, acquire multimodal data during ERCP, including cholangiography imaging data, intraductal pressure monitoring data, and surgical operation timing data. S2. Standardize the multimodal data to remove noise and abnormal data, and achieve temporal consistency alignment of each modality data. S3. Perform multimodal feature fusion, extract the features of each modal data after preprocessing, and fuse the features of each modality through an attention mechanism to obtain a fused feature vector; S4. Input the fused feature vector into a preset risk classification model, calculate the risk probability and determine the risk level in real time, and trigger the corresponding early warning signal and intervention prompt according to the risk level. S5. Receive early warning feedback information from medical staff, supplement training samples, and iteratively optimize the risk classification model and feature fusion related algorithms.

2. The multimodal fusion-based intraoperative risk warning method for ERCP according to claim 1, characterized in that: In step S1, the specific steps for data acquisition are as follows: Multimodal data is simultaneously acquired intraoperatively through the image acquisition module, pressure sensor module, and operation timing recording module of the ERCP surgical equipment. This multimodal data includes cholangiography image data, intraductal pressure monitoring data, and surgical operation timing data. The cholangiography image data is a sequence of cholangiography images captured in real-time during the procedure, denoted as: ， Where T is the total number of frames captured, x img，t Let represent the image data of frame t, where t = 1, 2, ..., T; the pressure monitoring data consists of continuous pressure values ​​from 1 to 10 frames per second, denoted as: ， Where x pre,t This represents the pressure value at time t; the operation timing data includes the start time, duration, and operation type of instrument insertion, contrast agent injection, and sphincter incision, denoted as: ， Where, x seq,t Indicates the operation type at time t.

3. The multimodal fusion method for intraoperative risk warning in ERCP according to claim 1, characterized in that: In step S2, the cholangiography image data preprocessing is performed sequentially: grayscale correction, image enhancement, and region of interest extraction are executed, with the specific formulas as follows: 1) Grayscale correction: A linear grayscale correction algorithm is used, and the correction formula is as follows: ， Where x img,t (i,j) represents the original grayscale value of pixel (i,j) in the t-th frame of the image, x' img,t (i,j) represents the corrected grayscale value, a represents the contrast adjustment coefficient, and b represents the brightness adjustment coefficient. The optimal values ​​of a and b are determined by least squares fitting. 2) Image Enhancement: A histogram equalization algorithm is used. First, the image grayscale histogram is calculated: ， n k Let k be the number of pixels with a grayscale value of k, and M×N be the image resolution; then calculate the cumulative distribution function. Finally, through x'' img,t (i,j) = round(255·cdf (x' img,t (i,j))) Implements grayscale value mapping to improve image contrast; 3) ROI Extraction: The Otsu thresholding algorithm is used to calculate the optimal segmentation threshold. ， Where w0(T) and w1(T) are the pixel proportions of the background and target regions after threshold T segmentation, respectively; u0(T) and u1(T) are the average grayscale values ​​of the background and target regions, respectively; and u is the average grayscale value of the entire image; This can be achieved using the following formula: ； Then, the ROI region is extracted.

4. The multimodal fusion method for intraoperative risk warning in ERCP according to claim 1, characterized in that: In step S2, pressure monitoring data preprocessing involves using Kalman filtering to remove outlier data and linear interpolation to supplement missing data. The specific formula is as follows: 1) Kalman filter noise reduction: The state equation is: x pre,k = A·x pre,k-1 + w k-1 ; The observation equation is: z pre,k = H·x pre,k + v k ; Where, x pre,k Let A represent the true pressure value at time k, H be the state transition matrix, and W be the observation matrix. k-1 For process noise, v k To observe noise; Filtering update steps: ① Prediction: pre, k|k- 1 =A· pre , k- 1|k- 1 ； P k|k-1= A·P k-1|k-1 · A T +Q; ② Update: K k =P k|k-1 ·H T · (H·P k|k-1 · H T +R) - ¹; pre ,k|k = pre,K | k-1 +K k ·(Z pre,k -H· pre,k|k-1 ); P k | k =(I-K k ·H)·P k|k-1 ; Where I is the identity matrix; filtering removes residual outliers; P k | k K represents the covariance of the optimal estimate after filtering at time k. k This represents the Kalman gain at time k. After filtering, if the difference between the filtered values ​​at adjacent time points exceeds a set threshold, it is determined to be a residual outlier and replaced. The determination formula is as follows: | pre,k | k pre ,k-1|k-1 |>10mmHg / s; That is, a pressure change exceeding 10 mmHg per second is considered abnormal. 2) Linear interpolation completion: For missing data points x pre, m Through adjacent valid data points x pre,i and x pre,j Interpolation calculation, the formula is: ； Where, x pre,m This represents the missing pressure data point at time m; 3) Standardization: The min-max standardization algorithm is used, and the formula is: ； Where x''' pre, t X represents the standardized pressure value at time t; pre, filter This is the filtered pressure data set, and the standardized data is mapped to the 0-1 interval.

5. The multimodal fusion method for intraoperative risk warning in ERCP according to claim 1, characterized in that: In step S2, the operation timing data is preprocessed: the operation type is standardized using one-hot encoding, and the formula is as follows: x''' seq,t = [0,...,1,...,0]； Where x''' seq, t This represents the operation timing vector after one-hot encoding at time t; the data of each modality is synchronized using a timestamp alignment algorithm, with the formula: t sync = max(t img , t pre , t seq ), where t sync t represents the unified timestamp after synchronization. img The timestamp of the image data, t pre The timestamp representing the stress data, t seq Indicates the time of operation timing data; interpolates and aligns the data of each modality to the synchronization timestamp t. sync This ensures timing consistency.

6. The multimodal fusion-based intraoperative risk warning method for ERCP according to claim 1, characterized in that: In step S3, a multimodal feature fusion model is constructed, extracting features from each preprocessed data and fusing them through an attention mechanism. The specific steps are as follows: 1) Image feature extraction: A ResNet50 convolutional neural network was used, and the extraction process was as follows: F img = CNN(X''' img ) =f ResNet50 (x''' img,1 , x''' img,2 ..., x''' img,T ), where X''' img For the preprocessed image sequence, f ResNet50 The feature extraction function for ResNet50 outputs a 256-dimensional image feature vector F. img ∈R 256 The vector elements correspond to the characteristics of bile duct wall thickness, morphology, and blood vessel distribution. 2) Temporal feature extraction of stress: A temporal convolutional network is used, and the convolution operation formula is as follows: ； Among them, f TCN (·) represents the feature extraction function of the temporal convolutional network, K is the kernel size, d is the dilation coefficient, and w k The weights are the convolution kernel weights; through 3 layers of temporal convolution and activation function operations, a 128-dimensional stress temporal feature vector F is output. pre ∈R 128 The vector elements correspond to the characteristics of pressure change rate, peak value, and fluctuation amplitude. 3) Operation timing feature encoding: Encode the preprocessed operation timing feature x''' seq, t The feature vector is mapped to a 64-dimensional feature vector through a fully connected layer, and the encoding formula is F. seq = W seq · X''' seq + b seq W seq ∈R 64×C Let b be the weight matrix. seq ∈R 64 For bias terms of fully connected layers; 4) Attention Fusion: A scaled dot product attention mechanism is used to perform attention fusion on the image feature vector F. img Pressure time series eigenvector F pre and the operation timing feature vector F seq Weights are assigned to highlight features related to bleeding and perforation risk, and then fusion is performed to obtain the final fused feature vector. The specific formulas, symbol definitions, and steps are as follows: ① Unify Feature Dimension: First, adjust the feature vectors of the three modalities to the same dimension to facilitate weight calculation. The adjustment method is a linear transformation. The adjusted feature vector is denoted as... , , ; ② Correlation Score Calculation: Calculate the correlation score between the adjusted feature vector of each modality and the preset risk feature vector to measure the importance of each modality feature for risk warning. The formula is: ； ； ； in, The score represents the correlation between image features and risk features. This represents the correlation score between the time-series characteristics of stress and the characteristics of risk. This represents the correlation score between operational time-series characteristics and risk characteristics. This represents the image feature vector after dimensional adjustment; This represents the time-series feature vector of stress after dimensional adjustment; This represents the operational time-series feature vector after dimensional adjustment; Represents the query matrix; This represents the key matrix, used to convert risk feature vectors into key vectors; This represents a preset risk feature vector; This represents the matrix transpose operation; ③ Weight Normalization: The relevance scores are normalized into weight coefficients using the softmax function. The formula is as follows: ； ； ； in Weighting coefficients representing image features; Weighting coefficients representing the time-series characteristics of pressure; Weighting coefficients representing the timing characteristics of operations; and ; D represents the exponential function; D represents the dimension of the feature vector. ④ Multimodal feature fusion: Multiply the adjusted feature vectors of each modality by their corresponding weight coefficients, then sum them to obtain the fused feature vector. The formula is: ； in: This represents the multimodal fusion feature vector.

7. The multimodal fusion-based intraoperative risk warning method for ERCP according to claim 1, characterized in that: In step S4, the fused feature vectors are... Input a preset risk classification model, determine in real time whether there is a risk of bleeding or perforation during the current operation, and output the risk level; The specific steps are as follows: Risk classification model training: Using a large amount of intraoperative multimodal data of ERCP labeled with bleeding and perforation risks, a Transformer binary classification model was trained to determine the model parameters and ensure the model's generalization ability. The specific steps are as follows: ① Training samples: Collect multimodal data from N ERCP surgeries, with each sample corresponding to a fused feature vector. A risk label The sample set is denoted as: ； Where N represents the total number of training samples. This represents the fused feature vector of the i-th sample. ; This represents the risk label of the i-th sample; i represents the sample number. ② Loss Function: The cross-entropy loss function is used to measure the deviation between the model's predicted values ​​and the true labels, and is used to optimize the model parameters. The formula is: ； Where L represents the cross-entropy loss value; This represents the probability that the model predicts the i-th sample has a risk. ③ Optimizer: The Adam optimizer is used to update model parameters and minimize the loss function. The optimizer formula is: ； ； ； in: Denotes the first-order momentum in the t-th iteration; Denotes the first-order momentum in the (t-1)th iteration; This represents the first-order momentum decay coefficient; This represents the loss gradient in the t-th iteration; Let represent the second-order momentum of the t-th iteration; Let represent the second-order momentum in the (t-1)th iteration; This represents the second-order momentum decay coefficient; This represents the square of the loss gradient; —Model parameters in the t-th iteration; This represents the learning rate, used to adjust the step size for parameter updates. Describing second momentum The square root of; This represents a small constant to prevent the denominator from being zero; t represents the iteration number. ④ Cross-validation: Five-fold cross-validation is used to ensure the model's generalization ability and avoid overfitting. The formula for the validation accuracy is: ； Among them, Accuracy k This represents the accuracy of the k-th fold validation set; k represents the fold number. ⑤ Model Inference: The fused feature vectors generated in real time... The trained Transformer model is input and subjected to layer normalization, multi-head attention, and feedforward network operations in sequence. Finally, the probability of bleeding and perforation at the current surgical moment is calculated using the sigmoid activation function. ; ⑥ Risk level determination and early warning triggering: based on risk probability The size of the risk level is determined by setting a threshold, triggering corresponding early warning signals and intervention prompts. The determination formula is as follows: ； Here, Level represents the risk level at the current moment of surgery.

8. The multimodal fusion method for intraoperative risk warning in ERCP according to claim 1, characterized in that: In step S5, the early warning feedback and model iteration steps are as follows: Medical staff input feedback information through the feedback iteration module. 0 represents a correct warning, 1 represents a missed warning, and 2 represents a false warning; Feedback information and corresponding samples Store to the storage module; after accumulating a certain number of feedback samples, retrain the model, and update the parameters using the following formula: ， in, This refers to the amount of parameter updates during the new training process.

9. A multimodal fusion-based intraoperative risk early warning system for ERCP, characterized in that, The system is used to implement the multimodal fusion-based intraoperative risk warning method for ERCP as described in any one of claims 1-8, the system comprising: The data acquisition module is used to simultaneously acquire cholangiography images, intra-biliary pressure monitoring data, and surgical procedure timing data during ERCP. The data preprocessing module, electrically connected to the data acquisition module, is used to standardize the acquired multimodal data, remove noise and abnormal data, and achieve time-series alignment of the data of each modality. The multimodal feature fusion module is electrically connected to the data preprocessing module and is used to extract features from each modality of data and fuse them through an attention mechanism to obtain a fused feature vector. The risk warning module is electrically connected to the multimodal feature fusion module and has a built-in risk classification model for calculating risk probability, determining risk level, and triggering corresponding warning signals and intervention prompts. The feedback iteration module, electrically connected to the risk warning module, is used to receive feedback information from medical staff, supplement training samples, and achieve iterative optimization of the model. The storage module, electrically connected to each module, is used to store various types of data, model parameters, and warning-related records, facilitating subsequent queries, reviews, and model iterations.

Images