A liver and bile duct stone recurrence intelligent prediction method based on multi-source heterogeneous data fusion
By using multi-source heterogeneous data fusion and the FastBERT network prediction method, the problem of accuracy in predicting the recurrence of hepatobiliary stones was solved, enabling accurate prediction and personalized treatment, and reducing the risk of recurrence.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG UNIV OF TECH
- Filing Date
- 2023-01-12
- Publication Date
- 2026-05-15
AI Technical Summary
Current techniques for predicting the recurrence of hepatobiliary stones rely on the doctor's subjective judgment, which has a large margin of error and is difficult to predict accurately. Traditional surgery is also unable to effectively relieve bile duct stenosis, resulting in a high recurrence rate and affecting patients' health and quality of life.
A multi-source heterogeneous data fusion method is adopted. CT and MRI images are reconstructed through a diffusion model, and the fusion features of image depth and clinical data are extracted and input into the FastBERT network for prediction. Combined with transfer learning and optimization algorithms, accurate prediction is achieved.
It improves the accuracy of predicting the recurrence of hepatobiliary stones, reduces reliance on doctors' subjective judgment, lowers the risk of recurrence, and enables early detection and personalized treatment.
Smart Images
Figure CN115965618B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of predicting the recurrence of hepatobiliary stones, and more specifically, to an intelligent prediction method for the recurrence of hepatobiliary stones by fusing multi-source heterogeneous data. Background Technology
[0002] Hepatolithiasis is a common hepatobiliary disease caused by intrahepatic bile duct stones located above the confluence of the left and right hepatic ducts. It has an insidious onset, complex symptoms, and is often complicated by biliary obstruction, infection, and even cancer. Due to its prolonged course, the affected liver segment suffers parenchymal destruction and atrophy, and severe cases of hepatolithiasis require liver transplantation to prolong life.
[0003] In recent years, the development of minimally invasive surgery, especially percutaneous transhepatic cholangiostomy (PTOBF) for stone removal, combined with endoscopic techniques, has enabled direct visualization and fragmentation of stones. This allows for the complete removal of intrahepatic bile duct stones in one stage, significantly reducing surgical trauma and risks, and decreasing the rate of open laparotomy. This minimally invasive and precise stone removal has led to its widespread clinical application. However, due to the complexity and wide distribution of intrahepatic bile duct stones, current minimally invasive surgeries often employ a surgical approach that explores from the common hepatic duct or common bile duct towards the distal bile duct. While this approach allows for rapid and safe stone removal for patients with both hepatobiliary stones and bile duct strictures, it is difficult to resolve the stricture, inevitably leading to a risk of recurrence of hepatobiliary stones. This results in persistently high rates of residual stones and recurrence after hepatobiliary duct stone surgery: 60%-80% of patients require reoperation due to stone recurrence, with a recurrence rate as high as 18.8%. However, repeated surgeries can lead to secondary biliary cirrhosis, liver parenchymal damage, and intrahepatic cholangiocarcinoma in the late stages of the disease, severely impacting patients' health and quality of life. Furthermore, recurrent hepatobiliary duct stones often create a vicious cycle of "stenosis / obstruction – stone formation – restenosis / obstruction – stone re-formation." Therefore, treating hepatobiliary duct stones requires not only treating the stones as much as possible but also effectively relieving biliary strictures, removing infected and cancerous lesions, and reducing the risk of recurrence.
[0004] Bile duct stricture (HR=4.89) is the most critical factor for recurrence after gallstone surgery. Therefore, adequately relieving the stricture during surgery is key to preventing recurrence. Bile duct stricture is often caused by irritation from a history of biliary surgery, gallstones, or biliary inflammation, leading to fibrous tissue hyperplasia and thickening of the bile duct wall, resulting in narrowing of the bile duct lumen. Currently, it is mainly repaired surgically. However, because bile duct strictures often exist covertly after biliary surgery, they are often difficult to detect promptly postoperatively. Traditional surgery is technically challenging, has a lower safety profile, and is more prone to recurrence. Current methods for predicting recurrence of hepatobiliary stones mainly rely on periodic examinations by doctors. This not only depends heavily on the doctor's subjective judgment but also has significant margins of error between different doctors, making accurate prediction difficult and thus affecting patient treatment. Summary of the Invention
[0005] To overcome the shortcomings of existing technologies that do not accurately predict the recurrence of hepatobiliary stones, this invention provides an intelligent prediction method for the recurrence of hepatobiliary stones based on the fusion of multi-source heterogeneous data.
[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:
[0007] A method for intelligent prediction of recurrence of hepatobiliary stones by fusing multi-source heterogeneous data includes the following steps:
[0008] S1: Obtain 2D CT and 2D MRI images of the abdomen of patients with hepatobiliary stones. Use a diffusion model to reconstruct the 2D CT and 2D MRI images respectively, and extract image depth features from the reconstructed 3D CT and 3D MRI images.
[0009] The preoperative blood test indicators, bile duct stenosis / dilation degree data, and time series of the occurrence of bile duct diseases of the patients with hepatobiliary stones were obtained, and the fusion features of the clinical data were extracted.
[0010] S2: The feature fusion of image depth features and clinical data is then input into the pre-trained FastBERT network to obtain accurate prediction results for the recurrence of hepatobiliary stones.
[0011] In the above scheme, image reconstruction is performed using a diffusion model to obtain a more realistic and clearer 3D image. Then, the image depth features and the fusion features of clinical data are extracted and fused to organically integrate knowledge from multiple domains, thereby improving the prediction accuracy. Finally, the fused features are input into a pre-trained FastBERT network to obtain accurate prediction results for the recurrence of hepatobiliary stones. This avoids the problem that traditional recurrence prediction relies on the subjective judgment of doctors, and the judgment errors of different doctors are large, making it difficult to make accurate predictions.
[0012] Preferably, image reconstruction using a diffusion model includes both a diffusion process and a reverse process;
[0013] During the diffusion process, for a two-dimensional image X0, Gaussian noise is added to iteratively by the diffusion model. After J iterations, a pure noise image X is generated. J ;
[0014] In the reverse process, the pure noisy image X is combined with the MBIR optimization strategy. J Progressive denoising is performed using a U-Net neural network to estimate the true distribution q(X) at the j-th step of the denoising process. j-1 |X j Finally, the reconstructed three-dimensional image is obtained.
[0015] The preferred MBIR optimization strategy is as follows:
[0016] Let the linear forward model be:
[0017] y = Ax + n
[0018] Where, y∈U m For the measured value, x∈U m For the image to be reconstructed, A∈U m×n U is the discrete transformation matrix, n is the measurement noise, and U is the variable. m Let be the set of all m-dimensional vectors;
[0019] Since the problem is uncertain, the standard approach to estimating the inverse of x from the measured value y is to perform the following regularized reconstruction:
[0020]
[0021] Where, x * R is the inverse of x, and R(x) is the regularization of x.
[0022] Preferably, the reverse process further includes the following steps: using a total variational model prior applied only to the z-axis direction to enhance the data-driven prior, and then unifying the data by aggregating slices; wherein, the two-dimensional image is an axial slice, the axial plane is the xy plane, and the axis perpendicular to the axial plane is the z-axis.
[0023] Preferably, the Vision Transformer neural network is used to extract image depth features by capturing global and local features in CT and MRI 3D images.
[0024] Preferably, in the dual-modal feature fusion stage of the Vision Transformer neural network, a cross-attention feature fusion method is used to implement the intermediate layer fusion strategy, specifically as follows:
[0025] Suppose the Vision Transformer neural network has L layers. In the first L f layers, self-attention within a single modality is first performed, and then the feature vectors of the two modalities obtained are concatenated and input into the subsequent L - L f layers for fusion. By controlling 0 < L f < L, an intermediate layer fusion strategy is implemented.
[0026] Preferably, the specific steps for extracting the fusion features of clinical data are as follows: First, use a stacked autoencoder to sequentially compress the preoperative blood test indicators and the data on the degree of hepatobiliary duct stenosis / dilation into a low-dimensional first hepatobiliary duct feature vector, and then decompress and reconstruct it in sequence into the first hepatobiliary duct feature with the same input dimension; at the same time, use an RNN model based on LSTM to non-linearly transform the time series of the occurrence of hepatobiliary duct diseases into a second hepatobiliary duct feature vector and decode it to obtain the second hepatobiliary duct feature; fuse the first hepatobiliary duct feature and the second hepatobiliary duct feature to obtain the fusion features of clinical data.
[0027] Preferably, when training the FastBERT network, the Brier score is used to evaluate the performance of the FastBERT network. The Brier score is expressed as:
[0028]
[0029] where L is the number of samples, p i is the probability predicted by the naive Bayes, and o i is the true result corresponding to the sample; the value range of the Brier score is [0, 1]. The higher the score, the worse the prediction result and the worse the calibration degree. Therefore, the closer the Brier score is to 0, the better.
[0030] Preferably, it also includes performing transfer learning and optimization on the pre-trained FastBERT network:
[0031] The value of each state s of the hepatobiliary duct stone patient under the current diagnosis π is denoted as V π (s). For the infinite discount model in the time domain, V π (s) is expressed as
[0032]
[0033] where E π is the expected value under the policy, γ is the damping coefficient, t is the current moment, k is the number of iterations, r t+k is the reward obtained when transferring from state s t to s t+k , s t is the state at time t,
[0034] The corresponding strategy benefit Q obtained by taking action a thereafter is... π (s,a) is:
[0035]
[0036] Among them, a t For the action at time t;
[0037] The goal of any given Markov decision process is to find an optimal policy, i.e., the policy that maximizes the reward. This requires updating the policy to maximize the reward, and the optimal solution must satisfy the following equation:
[0038]
[0039] Among them, V * (s) is the optimal state value function, S is a finite set of states, T is the state transition function, R is the reward function, and s ′ V is the next state that state s reaches after action a. * (s ′ ) is the optimal state value function under state s′;
[0040] To select the optimal action for the best state value, a greedy rule is applied, employing the strategy that maximizes the reward:
[0041]
[0042] Where, π * (s) is the strategy with the highest return.
[0043] Preferably, it also includes an optimal policy search algorithm based on the FastBERT network after transfer learning and optimization:
[0044] For a fixed problem in a random domain, search for the lower bound of its sample complexity:
[0045]
[0046] Where o(·) is the lower bound of sample complexity, δ is the confidence level, N is the number of states, A is the number of actions, γ1 is the learning damping coefficient, and ε is the optimal policy under ideal conditions.
[0047] After each correction, the given strategy of the medical order is updated, and the output function is:
[0048] π i (s)=argmax Q i (s,a)
[0049] Where, π i(s) is the policy function, Q i (s,a) represents the corresponding strategy benefit obtained after taking action a.
[0050] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:
[0051] This invention provides an intelligent prediction method for recurrence of hepatobiliary stones based on multi-source heterogeneous data fusion. By using a diffusion model for image reconstruction, a more realistic and clearer three-dimensional image is obtained. Then, the image depth features and the fusion features of clinical data are extracted and fused together to organically integrate knowledge from multiple domains, thereby improving prediction accuracy. Finally, the fused features are input into a pre-trained FastBERT network to obtain accurate prediction results for recurrence of hepatobiliary stones. This avoids the problem that traditional recurrence prediction relies on the subjective judgment of doctors, and the judgment errors between different doctors are large, making accurate prediction difficult. Attached Figure Description
[0052] Figure 1 This is a schematic diagram of the overall process of the present invention;
[0053] Figure 2 This is a schematic diagram of the process after transfer learning and optimization in this invention;
[0054] Figure 3 This is a schematic diagram illustrating the construction and training process of the Markov decision process in this invention;
[0055] Figure 4 This is a schematic diagram illustrating the working process of the Markov decision process in this invention. Detailed Implementation
[0056] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.
[0057] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;
[0058] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.
[0059] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0060] Example 1
[0061] like Figure 1 As shown, a method for intelligent prediction of recurrence of hepatobiliary stones based on multi-source heterogeneous data fusion includes the following steps:
[0062] S1: Obtain 2D CT and 2D MRI images of the abdomen of patients with hepatobiliary stones. Use a diffusion model to reconstruct the 2D CT and 2D MRI images respectively, and extract image depth features from the reconstructed 3D CT and 3D MRI images.
[0063] The preoperative blood test indicators, bile duct stenosis / dilation degree data, and time series of the occurrence of bile duct diseases of the patients with hepatobiliary stones were obtained, and the fusion features of the clinical data were extracted.
[0064] S2: The feature fusion of image depth features and clinical data is then input into the pre-trained FastBERT network to obtain accurate prediction results for the recurrence of hepatobiliary stones.
[0065] In the specific implementation process, the diffusion model is used to reconstruct the image to obtain a more realistic and clearer three-dimensional image. Then, the image depth features and the fusion features of clinical data are extracted and fused to organically integrate knowledge from multiple fields, thereby improving the prediction accuracy. Finally, the fused features are input into the pre-trained FastBERT network to obtain accurate prediction results for the recurrence of hepatobiliary stones. This avoids the problem that traditional recurrence prediction relies on the subjective judgment of doctors, and the judgment error between different doctors is large, making it difficult to make accurate predictions.
[0066] Example 2
[0067] A method for intelligent prediction of recurrence of hepatobiliary stones by fusing multi-source heterogeneous data includes the following steps:
[0068] S1: Obtain 2D CT and 2D MRI images of the abdomen of patients with hepatobiliary stones. Use a diffusion model to reconstruct the 2D CT and 2D MRI images respectively, and extract image depth features from the reconstructed 3D CT and 3D MRI images.
[0069] More specifically, image reconstruction using a diffusion model includes both the diffusion process and the inverse process;
[0070] During the diffusion process, for a two-dimensional image X0, Gaussian noise is added to iteratively by the diffusion model. After J iterations, a pure noise image X is generated. J ;
[0071] In the reverse process, the pure noisy image X is combined with the MBIR optimization strategy. J Progressive denoising is performed using a U-Net neural network to estimate the true distribution q(X) at the j-th step of the denoising process. j-1 |X j Finally, the reconstructed three-dimensional image is obtained.
[0072] In the specific implementation process, a more realistic and clearer 3D image is obtained through diffusion model reconstruction.
[0073] More specifically, the MBIR optimization strategy is as follows:
[0074] Let the linear forward model of the imaging system (CT, MRI) be:
[0075] y = Ax + n
[0076] Where, y∈U m For the measured value, x∈U m For the image to be reconstructed, A∈U m×n U is the discrete transformation matrix, n is the measurement noise, and U is the variable. m Let be the set of all m-dimensional vectors;
[0077] Since the problem is uncertain, the standard approach to estimating the inverse of x from the measured value y is to perform the following regularized reconstruction:
[0078]
[0079] Where, x * R(x) is the inverse of x, and R(x) is the regularization of x, for example, the sparsity in a certain transform domain; minimization in the equation can be performed using a robust optimization algorithm, such as the Fast Iterative Soft Thresholding Algorithm (FISTA).
[0080] More specifically, the reverse process also includes the following steps: using a total variation model prior applied only to the z-axis direction to enhance the data-driven prior, and then unifying the data by aggregating slices; wherein, the two-dimensional image is an axial slice, the axial plane is the xy plane, and the axis perpendicular to the axial plane is the z-axis.
[0081] More specifically, the Vision Transformer neural network is used to extract image depth features by capturing global and local features in CT and MRI 3D images.
[0082] More specifically, in the dual-modal feature fusion stage of the Vision Transformer neural network, a cross-attention feature fusion method is used to implement the intermediate layer fusion strategy, specifically as follows:
[0083] Assuming the Vision Transformer neural network has L layers, in the first L... f The layer first performs self-attention within a single modality (CT, MRI), then concatenates the feature vectors from the two modalities and inputs them into the subsequent LL layer. f Layer fusion is performed by controlling 0 <Lf <L Implement the intermediate layer fusion strategy;
[0084] Based on the intermediate layer fusion strategy, during the attention flow stage, cross-modal attention is performed between tokens of different layers of different modalities, while self-attention is still performed within a single modality; during the cross-modal fusion stage, partial token information of each module is used for cross-attention.
[0085] In the specific implementation process, the information within a single modality is fully extracted through the intermediate layer fusion strategy, avoiding the problem of large differences in data structures and distributions in the bimodal modality, significantly reducing redundant computational amounts, and enabling attention to fully perceive complementary information between cross-modalities.
[0086] Obtain the preoperative blood test indexes, hepatobiliary duct stenosis / dilation degree data, and time series of the occurrence of hepatobiliary duct diseases of the hepatolithiasis patient, and extract the fusion features of the clinical data;
[0087] More specifically, the specific steps for extracting the fusion features of clinical data are as follows: First, use a stacked autoencoder to sequentially compress the preoperative blood test indexes and hepatobiliary duct stenosis / dilation degree data into a low-dimensional first hepatobiliary duct feature vector, and then decompress and reconstruct it in order into a first hepatobiliary duct feature with the same input dimension; at the same time, use an RNN model based on LSTM to nonlinearly transform the time series of the occurrence of hepatobiliary duct diseases into a second hepatobiliary duct feature vector and decode it to obtain a second hepatobiliary duct feature; fuse the first hepatobiliary duct feature and the second hepatobiliary duct feature to obtain the fusion features of the clinical data.
[0088] In the specific implementation process, by combining a stacked autoencoder and an RNN based on LSTM, while the stacked autoencoder reads data, the LSTM mechanism is introduced, referring to the time series of the patient's disease occurrence, discarding medical information irrelevant to the disease, and retaining more important medical information targeted at stone diseases, so as to better fuse with the deep image features and more accurately achieve prediction.
[0089] S2: After fusing the image deep features and the fusion features of the clinical data, input them into the pre-trained FastBERT network to obtain an accurate prediction result of the recurrence of hepatolithiasis.
[0090] In the specific implementation process, the fused features are randomly classified into a training set (80%) and a validation set (20%). During the training phase of the FastBERT (Fast Bidirectional Encoder Representations from Transformers) network, the training set is input into the FastBERT network. After the FastBERT network performs inference decisions, a weighted model is obtained. The advantage of FastBERT lies in its sample adaptive mechanism, which can adaptively adjust the computational cost of each sample. If a high confidence level is achieved in a shallow model, subsequent layers are unnecessary, greatly shortening the inference time. Finally, in the validation part, the validation set is used for verification. If the verification is correct, the data is moved from the validation set to the training set. This step is used to increase the robustness of the prediction model. During the process, the Brier Score, commonly used to measure the predictive ability of survival models, is used to evaluate the performance of each model. Finally, a model with high accuracy in predicting the recurrence risk of hepatobiliary stones is obtained. Based on the predicted recurrence risk, patients are classified into severe, moderate, and mild risk levels. According to the risk level, the follow-up plan for patients with hepatobiliary stones is adjusted and optimized in real time in combination with the new electronic health record (EHR) to achieve the prevention and control strategy of "early detection, early intervention, and early control".
[0091] More specifically, when training the FastBERT network, the Brillouin score is used to evaluate the performance of the FastBERT network. The Brillouin score is expressed as:
[0092]
[0093] Where L is the number of samples, p i The probability predicted by Naive Bayes, o i The true result corresponds to the sample; the Brill score ranges from [0,1]. The higher the score, the worse the prediction result and the worse the calibration. Therefore, the closer the Brill score is to 0, the better.
[0094] Example 3
[0095] A method for intelligent prediction of recurrence of hepatobiliary stones by fusing multi-source heterogeneous data includes the following steps:
[0096] S1: Obtain 2D CT and 2D MRI images of the abdomen of patients with hepatobiliary stones. Use a diffusion model to reconstruct the 2D CT and 2D MRI images respectively, and extract image depth features from the reconstructed 3D CT and 3D MRI images.
[0097] The preoperative blood test indicators, bile duct stenosis / dilation degree data, and time series of the occurrence of bile duct diseases of the patients with hepatobiliary stones were obtained, and the fusion features of the clinical data were extracted.
[0098] S2: The feature fusion of image depth features and clinical data is then input into the pre-trained FastBERT network to obtain accurate prediction results for the recurrence of hepatobiliary stones.
[0099] More specifically, such as Figure 2-4 As shown, it also includes transfer learning and optimization of the pre-trained FastBERT network:
[0100] The value of each state s in a patient with hepatobiliary duct stones under the current diagnosis π is represented by V. π (s), for the infinite discount model in the time domain (i.e., assuming the recovery process is long enough), then V π (s) is represented as
[0101]
[0102] Among them, E π Let γ be the expected value under the strategy, t be the current time, k be the iteration number, and r be the expected value. t+k For from state s t Transfer to s t+k The reward obtained at that time, s t Let be the state at time t.
[0103] The corresponding strategy benefit Q obtained by taking action a thereafter is... π (s,a) is:
[0104]
[0105] Among them, a t For the action at time t;
[0106] The goal of any given Markov decision process is to find an optimal strategy, i.e., the strategy that yields the maximum reward and also the strategy that leads to the fastest patient recovery. This requires updating the strategy to maximize the reward, and the optimal solution must satisfy the following equation:
[0107]
[0108] Among them, V * (s) is the optimal state value function, S is a finite set of states, T is the state transition function, R is the reward function, and s ′ V is the next state that state s reaches after action a. * (s ′ ) is the optimal state value function under state s′;
[0109] This policy expression is the Bellman optimality equation. Under an optimal policy, the state value must equal the expected value of the optimal action in that state. To select the optimal action with the optimal state value, the greedy rule is applied to adopt the policy that maximizes the payoff:
[0110]
[0111] Where, π * (s) is the strategy with the highest return.
[0112] In practice, patients' physical conditions may change over time, causing the FastBERT network to become ill-suited to these new circumstances. Therefore, based on patient follow-up data, a value iteration algorithm is used to identify the position of the prospective cohort within the patient probability distribution, measuring the similarity between patients and between two follow-up visits. This data is then used to study the transfer learning and optimization of the FastBERT network. This approach ensures that the assessment of patient recovery status based on Markov decision processes, where differences from the original recovery plan exist, becomes more accurate with updated follow-up data.
[0113] More specifically, it also includes introducing an optimal policy search algorithm based on the FastBERT network after transfer learning and optimization:
[0114] For a fixed problem in a random domain, search for the lower bound of its sample complexity:
[0115]
[0116] Where o(·) is the lower bound of sample complexity, δ is the confidence level, N is the number of states (immediately need to be re-examined, relatively serious and need to be re-examined as soon as possible, general, not important), A is the number of actions (change in the degree of recurrence risk, with an upper bound but generally set as the upper bound), γ1 is the learning damping coefficient, and ε is the optimal strategy under ideal conditions (interventions include seeking medical treatment as soon as possible, drug intervention as soon as possible, etc.).
[0117] After each correction, the given strategy of the medical order is updated, and the output function is:
[0118] π i (s)=argmax Q i (s,a)
[0119] Where, π i (s) is the policy function, Q i (s,a) represents the corresponding strategy benefit obtained after taking action a.
[0120] In the specific implementation process, by introducing the optimal strategy search algorithm to re-evaluate and optimize the patient prevention and treatment strategy, hidden recurrence risks can be detected and intervened in early, so as to achieve full-process, precise and controlled management of hepatobiliary duct stones.
[0121] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A method for intelligent prediction of recurrence of hepatobiliary stones based on multi-source heterogeneous data fusion, characterized in that, Includes the following steps: S1: Obtain 2D CT and 2D MRI images of the abdomen of patients with hepatobiliary stones. Use a diffusion model to reconstruct the 2D CT and 2D MRI images respectively, and extract image depth features from the reconstructed 3D CT and 3D MRI images. The preoperative blood test indicators, bile duct stenosis / dilation degree data, and time series of the occurrence of bile duct diseases of the patients with hepatobiliary stones were obtained, and the fusion features of the clinical data were extracted. S2: After fusing the image depth features and the fusion features of clinical data, the data is input into the pre-trained FastBERT network to obtain accurate prediction results of recurrence of hepatobiliary stones. The Vision Transformer neural network is used to extract image depth features by capturing global and local features in CT and MRI 3D images. In the dual-modal feature fusion stage of the Vision Transformer neural network, a cross-attention feature fusion method is used to implement the intermediate layer fusion strategy, specifically as follows: Vision Transformer neural network has Number of blocks, in front The layer first performs self-attention within a single modality, then concatenates the feature vectors from the two modalities before inputting them into the subsequent layers. Layer fusion is performed by controlling Implement a middleware fusion strategy; The specific steps for extracting the fusion features of clinical data are as follows: First, a stacked autoencoder is used to sequentially compress the preoperative blood test indicators and the degree of stenosis / dilation of the hepatobiliary ducts into low-dimensional first hepatobiliary feature vectors. Then, these vectors are decompressed and reconstructed sequentially into first hepatobiliary features with the same dimension as the input. Simultaneously, an LSTM-based RNN model is used to nonlinearly convert the time series of hepatobiliary disease occurrences into second hepatobiliary feature vectors and decode them to obtain the second hepatobiliary features. Finally, the first and second hepatobiliary features are fused to obtain the fusion features of the clinical data. When training the FastBERT network, the Brillouin score is used to evaluate its performance. The Brillouin score is expressed as: in, L It is the sample size. The probability predicted by Naive Bayes. The true result corresponds to the sample; the Brill score ranges from [0,1]. The higher the score, the worse the prediction result and the worse the calibration. Therefore, the closer the Brill score is to 0, the better.
2. The intelligent prediction method for recurrence of hepatobiliary stones based on multi-source heterogeneous data fusion according to claim 1, characterized in that, Image reconstruction using a diffusion model includes both the diffusion process and the inverse process. During the diffusion process, for two-dimensional images Gaussian noise is added iteratively to the diffusion model step by step, after... After one iteration, a pure noise image is generated. ; In the reverse process, the MBIR optimization strategy is combined to transform the pure noise image. Progressive denoising is performed using a U-Net neural network to estimate the first step in the denoising process. j The true distribution of steps Finally, the reconstructed three-dimensional image is obtained.
3. The intelligent prediction method for recurrence of hepatobiliary stones based on multi-source heterogeneous data fusion according to claim 2, characterized in that, The MBIR optimization strategy is as follows: Let the linear forward model be: in, For measured values, For the image to be reconstructed, For discrete transformation matrix, To measure noise, For all A set of dimensional vectors; Since the problem is uncertain, based on the measured values estimate The standard approach to the inverse problem is to perform the following regularization reconstruction: in, for The reverse, for Regularization.
4. The intelligent prediction method for recurrence of hepatobiliary stones based on multi-source heterogeneous data fusion according to claim 2, characterized in that, The reverse process also includes the following steps: using only applied to A total variational model prior along the axial direction is used to enhance the data-driven prior, followed by data unification through aggregate slicing; where the two-dimensional image is an axial slice, and the axial plane is denoted as... The plane, the axis perpendicular to the axial plane is axis.
5. The intelligent prediction method for recurrence of hepatobiliary stones based on multi-source heterogeneous data fusion according to claim 1, characterized in that, This also includes transfer learning and optimization of the pre-trained FastBERT network: In the current diagnosis The following describes each state of patients with hepatobiliary duct stones. Value is expressed as For the infinite discount model in the time domain, then Represented as in, The expected value under the strategy, The damping coefficient is... t For the current moment, k For the number of iterations, From state Transferred to The reward received at that time In order to be in The state at any given moment, Then take action The corresponding strategy benefit obtained for: in, In order to be in Actions at any given moment; The goal of any given Markov decision process is to find an optimal policy, i.e., the policy that maximizes the reward. This requires updating the policy to maximize the reward, and the optimal solution must satisfy the following equation: in, The optimal state value function. For a finite set of states, This is the state transition function. For the reward function, For state After the action The next state to be reached, For state The optimal state value function under the given conditions; To select the optimal action for the best state value, a greedy rule is applied, employing the strategy that maximizes the reward: in, The strategy that yields the greatest profit.
6. The intelligent prediction method for recurrence of hepatobiliary stones based on multi-source heterogeneous data fusion according to claim 5, characterized in that, It also includes an optimal policy search algorithm based on the FastBERT network after transfer learning and optimization: For a fixed problem in a random domain, search for the lower bound of its sample complexity: in, This is a lower bound for the sample complexity. For confidence level, The number of states. The number of actions. To learn the damping coefficient, This is the optimal strategy under ideal conditions; After each correction, the given strategy of the medical order is updated, and the output function is: in, For the policy function, To adopt actions The corresponding strategy benefits obtained afterward.