TIPS postoperative hepatic encephalopathy prediction system based on multi-modal feature fusion
Through multimodal feature fusion technology, CT images and clinical data are integrated, and attention mechanism and residual connection are used to improve the prediction accuracy and stability of hepatic encephalopathy after TIPS, and solve the problem of single data characteristics in the existing technology. The problem of insufficient prediction has achieved efficient prediction results.
Patent Information
- Application Number
- CN202410869131.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2024-11-29
AI Technical Summary
The existing prediction methods for postoperative hepatic encephalopathy of TIPS rely on a single imaging data, neglecting multimodal information such as clinical data and biomarkers, resulting in insufficient prediction accuracy and stability.
A prediction system based on multimodal feature fusion was designed, including a data preprocessing module, a feature extraction module based on pretraining parameters, a feature fusion module based on attention mechanism and classifier module, which integrates abdominal CT images and clinical data. , using cross attention mechanism and residual connections to enhance feature expression and information flow, and improve prediction accuracy.
A high-precision prediction of hepatic encephalopathy after TIPS was achieved, with the AUC of the model reaching 91.4%, and the response time is less than or equal to 2 seconds, which reduces the diagnostic cost and improves the accuracy and stability of the prediction.
Smart Images

Figure CN119033394A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multimodal fusion algorithms in image processing, and in particular to a TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion. Background Art
[0002] Transjugular intrahepatic portal shunt (TIPS) is an interventional procedure that relieves portal hypertension through intrahepatic portosystemic shunt and is one of the important methods for treating liver diseases such as cirrhosis. However, TIPS is often accompanied by hepatic encephalopathy (HE), a neurological complication caused by abnormal liver function, which has a negative impact on the patient's quality of life and increases the patient's mortality rate. Therefore, a more accurate prediction of the risk of HE before surgery is of great significance in deciding whether to use TIPS.
[0003] At present, the prediction of HE after TIPS surgery mainly relies on the experience of clinicians and prediction models based on clinical indicators. However, this method has some problems, including: relying on the subjective judgment of experts and limited prediction accuracy. With the continuous in-depth research of computer vision theory and the development of modern medical technology, HE prediction technology based on computer vision is becoming an emerging method with broad development prospects. Traditional HE prediction algorithms based on computer vision mainly rely on the analysis of imaging features, extracting the patient's brain structure features through imaging data such as brain MRI, and combining machine learning methods to establish prediction models. However, this traditional algorithm only uses imaging data and ignores the importance of multimodal information such as clinical data and biomarkers, which leads to the simplification and limitation of information, affecting the comprehensiveness and accuracy of the prediction model. In order to improve the accuracy and stability of HE prediction, it is necessary to introduce methods such as multimodal fusion and optimization algorithm design for improvement.
[0004] The introduction of multimodal fusion of clinical data and image data is of great significance for HE prediction after TIPS. Clinical data can provide information on various physiological indicators of patients, while image data reflects the morphological characteristics of patients such as tissue structure and lesion conditions. Fusion of the above two data features can achieve a comprehensive assessment of the patient's condition and improve the accuracy of HE prediction. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion in view of the deficiencies in the prior art.
[0006] The technical solution of the present invention is as follows:
[0007] A prediction system for hepatic encephalopathy after TIPS surgery based on multimodal feature fusion, including a data preprocessing module, a feature extraction module based on pre-trained parameters, a feature fusion module based on an attention mechanism, and a classifier module;
[0008] Data preprocessing module: for abdominal CT images and clinical data of cirrhosis patients before TIPS surgery provided by the hospital, under the guidance of doctors, the CT image data, clinical data and different labels of the patients are sorted out; different labels include: with HE, without HE; the labeled sequence images and corresponding clinical data are processed into corresponding one-hot encodings according to the hierarchy, and the training set and validation set are divided;
[0009] The feature extraction module based on pre-trained parameters includes a fully connected operation unit: input clinical data into the network to extract clinical data features; and a convolution operation unit: input CT images into the convolution layer to perform convolution operations to extract image features;
[0010] The feature fusion module includes:
[0011] Pooling unit: used to compress the features extracted by the convolutional layer;
[0012] Cross-attention unit: It is used to fuse the effective features obtained from clinical data and CT images using the cross-attention mechanism, so that the modalities can pay attention to the correlation between each other and enhance the original feature expression;
[0013] Connection units: Using connections alleviates the problem of information being difficult to pass through in deep networks, helping to maintain a more direct flow of information;
[0014] The connected features are passed through the classifier module to obtain high-precision prediction results.
[0015] The TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion, the fully connected operation unit: clinical data input fully connected layer, assuming X = [x1, x2, ..., x i ] is the input data, W=[w1,w2,...,w i ] is the weight of the fully connected operation unit, b represents the bias, and Y represents the output result. The formula of the fully connected operation unit is as follows:
[0016] Y=X×W T +b
[0017] The TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion, the convolution operation unit: assuming v = [v1, v2, ..., v c] is the convolution kernel of Ftr operation, F represents the convolution operation and pooling operation before the last convolution layer, and Ftr represents the convolution operation of the last convolution layer; v c is the parameter of the cth convolution kernel, and the feature map output by the Ftr operation is represented as u=[u1,u2,…,u c ], then the feature map of the cth channel is expressed as follows:
[0018]
[0019] In the above formula, * represents the convolution operation, which is a two-dimensional spatial kernel, represented by v c The convolution kernel of a single channel acts on the corresponding channel of X, where c represents the number of channel feature maps input to the convolution layer; by c ∈R H×W Perform global average pooling and global maximum pooling operations to effectively learn the location information and object features of objects. The formal description is shown in the formula:
[0020]
[0021] z c2 =max(u c (i,j))(i∈[1,2,···,W])
[0022] The TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion, the pooling unit: extracts the effective features extracted from each layer of the ResNet network, uses the pooling operation to reduce the spatial dimension of the feature map, helps reduce the computational complexity and enhances the translation invariance of the model, and facilitates the subsequent cross-attention calculation with the effective features extracted by the MLP. Specifically, the size of the input feature map is I H ×I W ×C i n , where I H Indicates the input height, I W Indicates the width of the input, C in Represents the number of input channels. The output size after pooling is calculated as follows:
[0023]
[0024]
[0025] Here, K H and K W represents the height and width of the pooling kernel, and S represents the stride of the pooling kernel. H and O W Represent the height and width of the output feature map respectively.
[0026] The TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion, the cross attention unit: in order to obtain a deeper relationship between the two modal data, the pooling unit is used to reduce the dimension of the features obtained by the convolution layer, and is used for cross attention with the features of the same dimension obtained by the MLP layer. Specifically, first, we need to define three weight matrices W q (Query Matrix), W k (Key Matrix) and W v (Value Matrix), which are used to calculate queries, keys, and values.
[0027] For each input element x i and i , we calculate its query q, key k and value v by the following formula:
[0028]
[0029]
[0030] Next, we compute the attention score for each element x or y by taking the dot product of the query q and the key k and normalizing it with the softmax function:
[0031] score(x,x)=softmax(qk) or score(y,y)=softmax(qk)
[0032] We then compute the output for each element by adding the attention score for each element x or y to the corresponding value v i Multiplying and then summing gives:
[0033] output(x i )=∑j(score(x i ,x j )*v j ) or output(y i )=∑j(score(y i ,y j )*v j )
[0034] In the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion, the connection unit in the fusion module is a residual connection, and its formula can be expressed as y=f(x)+x, where x represents the input of the layer, f(x) represents the output of the layer, and y represents the output after the residual connection. During forward propagation, the sum of x and f(x) is used as the output of the layer and directly input to the next layer for processing; during reverse propagation, the residual connection makes it easier for the gradient to be passed back to the previous layer, accelerating the convergence of the model.
[0035] The TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion, the classifier module: the loss function used by the entire network is the cross entropy loss:
[0036] m is the number of images, Y is the known installment label, so the loss functions of A and B are
[0037] The gradient is found by the chain rule:
[0038]
[0039] The gradient descent algorithm is used to minimize the loss function, and the parameters in the network model are adjusted according to the direction of the gradient descent. After the model training is completed, the loss function is minimized as much as possible, and the type Y predicted by the network is output.
[0040] The beneficial effects achieved by the present invention are:
[0041] The method of the present invention constructs and trains a prediction model for hepatic encephalopathy after TIPS surgery. The model is trained using multimodal technology using preoperative CT image data and clinical data of patients collected from the hospital. The trained prediction model is compatible with the equipment of most hospitals currently, and can be flexibly applied to various auxiliary diagnosis systems to assist doctors in making faster and more accurate predictions. The results of the present invention will soon be applied to multiple medical fields: hospitals do not need to invest a lot of resources in purchasing additional diagnostic equipment, and existing medical equipment can meet the deployment requirements of the model, reducing the cost of accurate diagnosis of patients; this project will also cultivate a group of important technical backbones for computer-aided diagnosis. They will become the backbone of supporting the sustainable development of the project team, and their published papers, monographs and other scientific research results will also be applied to clinical treatment.
[0042] The AUC of the prediction results of the TIPS postoperative hepatic encephalopathy model was 91.4%, and the response time for completing a prediction task was less than or equal to 2 seconds. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1It is a schematic diagram of the overall process of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion of the present invention.
[0044] Figure 2 The present invention is a flowchart of the operation of the image feature extraction module of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion.
[0045] Figure 3 This is a flow chart of the operation of the feature fusion module of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion of the present invention.
[0046] Figure 4 These are images without HE labels during the training of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion of the present invention.
[0047] Figure 5 These are images with HE labels used when training the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion of the present invention.
[0048] Figure 6 This is a comparison chart of the detection rates of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion of the present invention and the other two methods.
[0049] Figure 7 This is a comparison chart of the false detection rates of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion of the present invention and the other two methods.
[0050] Figure 8 This is a comparison chart of the detection time of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion of the present invention and the other two methods.
[0051] Fig. 9 It is a ROC curve diagram of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion of the present invention and the other two methods. DETAILED DESCRIPTION
[0052] The present invention is described in detail below in conjunction with specific embodiments.
[0053] A method for predicting the stage of pneumonia caused by COVID-19 infection based on attention mechanism. The overall process is as follows: Figure 1 As shown, the image features of the two categories are as follows Figure 4 ,5, including the following:
[0054] A1 For the TIPS preoperative CT images and clinical data of patients provided by the hospital, select the CT image sequences with better quality and clinical data with higher integrity, unify the image size, adjust the input image to 512×512, use multiple machine learning algorithms to evaluate the importance of each feature in the clinical data, and select some features with greater correlation. According to the physician's label, the sequence images and clinical data with different labels (postoperative HE, postoperative no HE) are obtained;
[0055] A2 processes the labeled sequence images (postoperative HE, postoperative non-HE) and the corresponding clinical data into corresponding one-hot encodings and divides them into training sets and validation sets;
[0056] A3 CT images and clinical data were input into a neural network based on multimodal feature fusion for feature extraction;
[0057] Step A3: The neural network based on multimodal feature fusion is as follows Figure 1 As shown in the figure, the feature fusion module based on the attention mechanism is as follows Figure 3 As shown in the figure. The image feature extraction network is built with ResNet50, and the clinical data feature extraction network is composed of a four-layer fully connected network. The features extracted by the two feature extraction networks are extracted and cross-attention operations are performed on them according to the corresponding layers. The image features are first pooled to reduce the dimension to the same dimension as the features extracted from the clinical data, and the features obtained by the two cross-attention operations are connected. After four fusion processes, the fused features are connected and input into the classifier for classification.
[0058] The specific steps of step A3 are as follows:
[0059] A31. Input clinical data into the fully connected layer for clinical feature extraction
[0060] Clinical data is input into the fully connected layer, assuming that X = [x1, x2, ..., x i ] is the input data, W=[w1,w2,...,w i ] is the weight of the fully connected operation unit, b represents the bias, and Y represents the output result. The formula of the fully connected operation unit is as follows:
[0061] Y=X×W T +b
[0062] A32. Input the CT image into the convolution layer for convolution operation to extract image features
[0063] Image data is input into the convolutional layer, assuming v = [v1, v2, ..., v c] is the convolution kernel of Ftr operation, F represents the convolution operation and pooling operation before the last convolution layer, and Ftr represents the convolution operation of the last convolution layer; v c is the parameter of the cth convolution kernel, and the feature map output by the Ftr operation is represented as u=[u1,u2,…,u c ], then the feature map of the cth channel is expressed as follows:
[0064]
[0065] In the above formula, * represents the convolution operation, which is a two-dimensional spatial kernel, represented as the convolution kernel of a single channel of vc, acting on the corresponding channel of X', where c' represents the number of channel feature maps input to the convolution layer; by performing global average pooling and global maximum pooling operations on the feature map uc∈RH×W, the position information and object features of the object can be effectively learned. Its formal description is shown in the formula:
[0066]
[0067] z c2 =max(u c (i,j))(i∈[1,2,···,W])
[0068] A33. Feature fusion module based on attention mechanism
[0069] After the image data is extracted by the convolutional layer, it is input to the pooling unit, which extracts the effective features extracted by each layer of the ResNet network and uses the pooling operation to reduce the spatial dimension of the feature map, helping to reduce the computational complexity and enhance the translation invariance of the model, so as to facilitate the subsequent cross-attention calculation with the effective features extracted by the fully connected layer MLP. Specifically, the size of the input feature map is I H ×I W ×C i n , where I H Indicates the input height, I W Represents the width of the input, and Cin represents the number of channels of the input. The output size after pooling is calculated as follows:
[0070]
[0071]
[0072] Here, K H and K W represents the height and width of the pooling kernel, and S represents the stride of the pooling kernel. H and O W Represent the height and width of the output feature map respectively.
[0073] The pooled features and clinical data features extracted by the MLP layer are input into the cross attention unit. This unit is used to obtain a deeper relationship between the two modal data. The pooling unit is used to reduce the dimension of the features obtained by the convolution layer and to perform cross attention on the features of the same dimension obtained by the MLP layer. Specifically, first, we need to define three weight matrices W q (Query Matrix), W k (Key Matrix) and W v (Value Matrix), which are used to calculate queries, keys, and values.
[0074] For each input element x i and i , we calculate its query q, key k and value v by the following formula:
[0075]
[0076]
[0077] Next, we compute the attention score for each element x or y by taking the dot product of the query q and the key k and normalizing it with the softmax function:
[0078] score(x,x)=softmax(qk) and score(y,y)=softmax(qk)
[0079] We then compute the output for each element by adding the attention score for each element x or y to the corresponding value v i Multiplying and then summing gives:
[0080] output(x i )=∑j(score(x i ,x j )*v j ) or output(y i )=∑j(score(y i ,y j )*v j )
[0081] The TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion, the residual connection unit: its formula can be expressed as y=f(x)+x, where x represents the input of the layer, f(x) represents the output of the layer, and y represents the output after the residual connection. During forward propagation, the sum of x and f(x) is used as the output of the layer and directly input to the next layer for processing; during reverse propagation, the residual connection makes it easier for the gradient to be passed back to the previous layer, accelerating the convergence of the model.
[0082] A4 concatenates the features for input into the next layer of classifiers.
[0083] The features after A5 connection are passed through the classifier to obtain high-precision prediction results
[0084] During the training process, the image feature extraction network loads the ResNet50 pre-trained parameters, and the parameters of the clinical data network are initialized. The loss function of the entire network during training is defined as the cross entropy loss.
[0085]
[0086] m is the number of images, and Y is the known period label, so the gradient of the loss function of A and B is obtained by the chain rule:
[0087]
[0088] The gradient descent algorithm is used to minimize the loss function, and the parameters in the network model are adjusted according to the direction of the gradient descent. After the model training is completed, the loss function is minimized as much as possible, and the type Y predicted by the network is output.
[0089] Figure 6 This is a comparison chart of the prediction rates of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion and the other two systems. Method 1 uses a traditional machine learning model for classification prediction without adding a residual module and an attention mechanism, while method 2 uses a single ResNet-50 model for classification prediction without using the patient's preoperative clinical data. It can be seen that after adding the patient's preoperative clinical data and using the attention mechanism in the fusion process, the average prediction rate of the TIPS postoperative hepatic encephalopathy prediction system is the best. This is because adding the patient's clinical data increases the number of features, and the attention mechanism helps the system find more useful features in decision-making, thereby improving the prediction rate.
[0090] Figure 7 This is a comparison chart of the false detection rates of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion and the other two systems. Figure 6It can be seen that compared with other methods, the system proposed in the present invention can achieve the lowest false positive rate, but the average false positive rate is only 0.06.
[0091] Figure 8 This is a comparison chart of the detection time of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion and the other two systems. Method 1 and Method 2 describe the same Figure 6 It can be seen that the average decision-making time of the system of the present invention is only 4ms, which is 2.4ms and 7.8ms shorter than the average time of the other two methods, respectively, indicating that adding the attention mechanism can effectively reduce the running time of the system.
[0092] Fig. 9 The figure is the ROC curve of the TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion and the other two systems. Figure 6 It can be seen that the AUC value of the present invention reaches 0.914. Compared with the other two methods, the overall accuracy, sensitivity and specificity of the system are higher.
[0093] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.
Claims
1. A prediction system for hepatic encephalopathy after TIPS based on multimodal feature fusion, characterized in that: It includes data preprocessing module, feature extraction module based on pre-trained parameters, feature fusion module based on attention mechanism and classifier module; Data preprocessing module: for abdominal CT images and clinical data of cirrhosis patients before TIPS surgery provided by the hospital, under the guidance of doctors, the CT image data, clinical data and different labels of the patients are sorted out; different labels include: with HE, without HE; the labeled sequence images and corresponding clinical data are processed into corresponding one-hot encodings according to the hierarchy, and the training set and validation set are divided; The feature extraction module based on pre-trained parameters includes a fully connected operation unit: input clinical data into the network to extract clinical data features; and a convolution operation unit: input CT images into the convolution layer to perform convolution operations to extract image features; The feature fusion module includes: Pooling unit: used to compress the features extracted by the convolutional layer; Cross-attention unit: It is used to fuse the effective features obtained from clinical data and CT images using the cross-attention mechanism, so that the modalities can pay attention to the correlation between each other and enhance the original feature expression; Connection units: Using connections alleviates the problem of information being difficult to pass through in deep networks, helping to maintain a more direct flow of information; The connected features are passed through the classifier module to obtain high-precision prediction results.
2. According to claim 1, a TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion is characterized in that: The fully connected operation unit: clinical data input fully connected layer, assuming X = [x1, x2, ..., x i ] is the input data, W=[w1,w2,...,w i ] is the weight of the fully connected operation unit, b represents the bias, and Y represents the output result. The formula of the fully connected operation unit is as follows: Y=XⅹW T +b。 3. According to claim 1, a TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion is characterized in that: The convolution operation unit: Assume v = [v1, v2, ..., v c ] is the convolution kernel of Ftr operation, F represents the convolution operation and pooling operation before the last convolution layer, and Ftr represents the convolution operation of the last convolution layer; v c is the parameter of the cth convolution kernel, and the feature map output by the Ftr operation is represented as u=[u1,u2,…,u c ], then the feature map of the cth channel is expressed as follows: In the above formula, * represents the convolution operation, which is a two-dimensional spatial kernel, represented by v c The convolution kernel of a single channel acts on the corresponding channel of X, where c represents the number of channel feature maps input to the convolution layer; by c ∈R H×W Perform global average pooling and global maximum pooling operations to effectively learn the location information and object features of objects. The formal description is shown in the formula: z c2 =max(u c (i,j))(i∈[1,2,…,W])。 4. According to claim 1, a TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion is characterized in that: The pooling unit described above extracts the effective features extracted from each layer of the ResNet network, uses the pooling operation to reduce the spatial dimension of the feature map, helps reduce the computational complexity and enhances the translation invariance of the model, and facilitates the subsequent cross-attention calculation with the effective features extracted by the MLP. Specifically, the size of the input feature map is I H ×I W ×C in , where I H Indicates the input height, I W Indicates the width of the input, C in Represents the number of input channels; the output size after pooling is calculated as follows: Here, K H and K W represents the height and width of the pooling kernel, S represents the stride of the pooling kernel; O H and O W Represent the height and width of the output feature map respectively.
5. The TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion according to claim 1, characterized in that: The cross attention unit: In order to obtain a deeper relationship between the two modal data, the pooling unit is used to reduce the dimension of the features obtained by the convolution layer, and to perform cross attention on the features of the same dimension obtained by the MLP layer; Specifically, first, we need to define three weight matrices W q (Query Matrix), W k (KeyMatrix) and W v (Value Matrix), which are used to calculate query, keys and values; For each input element x i and i , we calculate its query q, key k and value v by the following formula: Next, we compute the attention score for each element x or y by taking the dot product of the query q and the key k and normalizing it with the softmax function: score(x, x)=softmax(qk) or score(y, y)=softmax(qk) We then compute the output for each element by adding the attention score for each element x or y to the corresponding value v i Multiplying and then summing gives: output(x i ) = ∑j(score(x i , x j ) * v j ) or output(y i ) = ∑j(score(y i , y j ) * v j ).
6. The TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion according to claim 1, characterized in that: The connection unit in the fusion module is a residual connection, and its formula can be expressed as y=f(x)+x, where x represents the input of the layer, f(x) represents the output of the layer, and y represents the output after the residual connection; during forward propagation, the sum of x and f(x) is used as the output of the layer and is directly input into the next layer for processing; during backward propagation, the residual connection makes it easier for the gradient to be passed back to the previous layer, accelerating the convergence of the model.
7. The TIPS postoperative hepatic encephalopathy prediction system based on multimodal feature fusion according to claim 1, characterized in that: The classifier module: The loss function used by the entire network is cross entropy loss: m is the number of images, and Y is the known period label, so the gradient of the loss function of A and B is obtained by the chain rule: The gradient descent algorithm is used to minimize the loss function, and the parameters in the network model are adjusted according to the direction of the gradient descent. After the model training is completed, the loss function is minimized as much as possible, and the type Y predicted by the network is output.
Citation Information
Cited By
Lung infectious disease prediction system based on multi-modal data fusion
CN119480124A
Axis type spondyloarthritis auxiliary analysis system based on multi-modal data fusion
CN119763807A
Digestive system disease-based risk prediction method and device
CN122050796A
