Atrial fibrillation recurrence prediction method, system, electronic device and storage medium

Abstract

The present disclosure provides an atrial fibrillation recurrence prediction method, system, electronic device and storage medium, the prediction method comprising: training an atrial fibrillation recurrence prediction model based on a plurality of sets of multimodal first sample training data; each set of first sample training data comprises a first number of sample intracavity ultrasound images and at least one other modal data, calculating a first contribution value of each frame of sample intracavity ultrasound image to the training of the atrial fibrillation recurrence prediction model, the matching degree between each frame of sample intracavity ultrasound image and each other modal data, and the cross-modal credibility factor of each other modal data, to determine the target contribution value of each frame of sample intracavity ultrasound image to the training of the atrial fibrillation recurrence prediction model, and obtain different target intracavity ultrasound image types; obtaining a plurality of frames of actual intracavity ultrasound images of target data and inputting them into the atrial fibrillation recurrence prediction model to obtain a target atrial fibrillation recurrence prediction result, so as to ensure the prediction efficiency and accuracy of atrial fibrillation recurrence.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of medical data processing, in particular to a method and system for predicting recurrence of atrial fibrillation, an electronic device and a storage medium. BACKGROUND

[0002] Currently, the surgical treatment of arrhythmia (especially atrial fibrillation ablation) in clinical practice still mainly relies on the preoperative experience judgment of the doctor, real-time image observation during operation and postoperative follow-up data for diagnosis and treatment decision. However, this traditional process has the following limitations: first, the patient's multi-source examination data (such as medical history, electrocardiogram, ultrasound image, intracardiac echocardiography (ICE) image, etc.) are scattered and have low utilization efficiency, making it difficult to achieve systematic integration and in-depth analysis of information; second, the preoperative prediction of atrial fibrillation recurrence is often limited to a single modality or relies on subjective experience, lacking objective quantitative tools based on multi-modal fusion; third, the identification of abnormal activation sites and surgical risks (such as thrombus, puncture deviation, etc.) of real-time images such as intracardiac echocardiography during operation mostly relies on manual interpretation by doctors, which has the risk of delayed reaction, misjudgment and omission; in addition, some patients have missing data in some modalities before, during or after the operation, which further affects the completeness and accuracy of risk assessment. SUMMARY

[0003] The technical problem to be solved by the present disclosure is to overcome the above-mentioned defects in the prior art, and to provide a method and system for predicting recurrence of atrial fibrillation, an electronic device and a storage medium.

[0004] The present disclosure solves the above technical problems by the following technical solutions:

[0005] In a first aspect, the present disclosure provides a method for predicting recurrence of atrial fibrillation, the method comprising:

[0006] training a pre-set network based on a plurality of groups of multi-modal first sample training data to obtain the atrial fibrillation recurrence prediction model;

[0007] wherein each group of the first sample training data comprises a first number of sample intracardiac echocardiography images and at least one other modality data, and the other modality data includes case text data, preoperative electrocardiogram data, preoperative transthoracic echocardiogram data or postoperative review electrocardiogram data;

[0008] calculating a first contribution value of each frame of the sample intracardiac echocardiography image to the training of the atrial fibrillation recurrence prediction model, a matching degree between each frame of the sample intracardiac echocardiography image and each of the other modality data, and a cross-modality credibility factor of each of the other modality data;

[0009] determine a target contribution value of each frame of the sample intracardiac ultrasound image to the atrial fibrillation recurrence prediction model training based on the first contribution value, the matching degree and the cross-modal confidence factor;

[0010] select a second number of the sample intracardiac ultrasound images from the first number of the sample intracardiac ultrasound images based on the size of the different target contribution values, and obtain different target intracardiac ultrasound image types;

[0011] obtain target data to be processed;

[0012] extract a plurality of frames of actual intracardiac ultrasound images belonging to the target intracardiac ultrasound image type from the target data, and input them into the atrial fibrillation recurrence prediction model to obtain a target atrial fibrillation recurrence prediction result.

[0013] Optionally, the first number of the sample intracardiac ultrasound images correspond to different image groups under different preset surgical procedure stages, and each image group corresponds to a plurality of frames of the sample intracardiac ultrasound images;

[0014] The step of calculating the first contribution value of each frame of the sample intracardiac ultrasound image to the atrial fibrillation recurrence prediction model training comprises:

[0015] a first contribution value calculation function constructed based on an output function of the currently trained atrial fibrillation recurrence prediction model;

[0016] calculating a group contribution value of each image group to the atrial fibrillation recurrence prediction model training by using the first contribution value calculation function;

[0017] a second contribution value calculation function constructed based on an attention mechanism of the currently trained atrial fibrillation recurrence prediction model;

[0018] adopting the second contribution value calculation function to calculate an attention score corresponding to each frame of the sample intracardiac ultrasound image in each image group as a contribution weight value for the atrial fibrillation recurrence prediction model training;

[0019] obtaining the first contribution value corresponding to each frame of the sample intracardiac ultrasound image based on the product of the contribution weight value and the group contribution value.

[0020] Optionally, the step of calculating the matching degree between each frame of the sample intracardiac ultrasound image and each other modality data comprises:

[0021] a matching degree calculation function constructed based on a feature vector of the currently trained atrial fibrillation recurrence prediction model;

[0022] The matching degree between each frame of the sample intracardiac ultrasound image and each of the other modal data is calculated by using the matching degree calculation function.

[0023] Optionally, the step of calculating the cross-modal confidence factor of each of the other modal data comprises:

[0024] The mean similarity and similarity standard deviation of the other modal data on the training set are obtained to construct a confidence calculation function;

[0025] The cross-modal confidence factor of each of the other modal data is calculated based on the confidence calculation function.

[0026] Optionally, the step of determining the target contribution value of each frame of the sample intracardiac ultrasound image to model training based on the first contribution value, the matching degree and the cross-modal confidence factor comprises:

[0027] The confidence mean corresponding to the cross-modal confidence factor corresponding to different other modal data is calculated;

[0028] The target contribution value of each frame of the sample intracardiac ultrasound image to model training is calculated based on the first contribution value, the matching degree and the confidence mean;

[0029] The step of screening a second number of sample intracardiac ultrasound images from the first number of sample intracardiac ultrasound images based on the size of different target contribution values comprises:

[0030] The different target contribution values are sorted and the second number of sample intracardiac ultrasound images with higher target contribution values are selected.

[0031] Optionally, after the step of obtaining a plurality of groups of multimodal first sample training data, and before the step of training a pre-set network based on the plurality of groups of multimodal first sample training data to obtain the atrial fibrillation recurrence prediction model, the prediction method further comprises:

[0032] A first feature vector corresponding to each modality data is extracted;

[0033] A multimodal feature correlation matrix is formed based on different first feature vectors;

[0034] In response to the presence of missing at least one modality data in the first feature vector, a cross-modal generation network is constructed based on the multimodal feature correlation matrix;

[0035] estimate an uncertainty score of the completed feature based on the cross-modal generation network, to obtain a fusion feature of the completed modality;

[0036] fuse the fusion feature and the features in the first feature vector to obtain a target feature vector, so that the preset network is trained based on the target feature vector.

[0037] Optionally, the step of estimating an uncertainty score of the completed feature based on the cross-modal generation network, to obtain a fusion feature of the completed modality, comprises:

[0038] using a preset estimation algorithm to estimate the uncertainty score of the completed feature of the cross-modal generation network;

[0039] obtaining a credibility weight of the completed modality based on the uncertainty score;

[0040] calculating the fusion feature of the completed modality according to the credibility weight;

[0041] and / or,

[0042] introducing a modality reconstruction loss in the preset network, so that the difference between the fusion feature and the real modality feature in the feature distribution space is less than a preset condition.

[0043] Optionally, the step of extracting a first feature vector corresponding to each modality data comprises:

[0044] using a feature extraction method matched with each modality data to extract features to correspond to the first feature vector respectively;

[0045] and / or,

[0046] After the step of obtaining a target feature vector, further comprising:

[0047] inputting the features corresponding to the target feature vector into a fully connected layer classification network in the preset network for training, to obtain the atrial fibrillation recurrence prediction model.

[0048] The second aspect of the present disclosure also provides an atrial fibrillation recurrence prediction system, the prediction system comprising:

[0049] a prediction model acquisition module, configured to train a preset network based on a plurality of groups of first sample training data of multi-modal, to obtain the atrial fibrillation recurrence prediction model;

[0050] wherein each group of the first sample training data comprises a first number of sample intracardiac echocardiogram images and at least one other modality data, and the other modality data comprises case text data, preoperative electrocardiogram data, preoperative transthoracic echocardiogram image data or postoperative review electrocardiogram data.

[0051] a first contribution value calculation module configured to calculate a first contribution value of each frame of the sample intracardiac ultrasound image to training of the atrial fibrillation recurrence prediction model;

[0052] a matching degree calculation module configured to calculate a matching degree between each frame of the sample intracardiac ultrasound image and each other modality data;

[0053] a credibility factor calculation module configured to calculate a cross-modality credibility factor of each of the other modality data;

[0054] a target contribution value calculation module configured to determine a target contribution value of each frame of the sample intracardiac ultrasound image to training of the atrial fibrillation recurrence prediction model based on the first contribution value, the matching degree and the cross-modality credibility factor;

[0055] a screening module configured to screen a second number of the sample intracardiac ultrasound images from the first number of the sample intracardiac ultrasound images based on sizes of different target contribution values, to obtain different target intracardiac ultrasound image types;

[0056] a target data acquisition module configured to acquire target data to be processed;

[0057] a prediction module configured to extract actual intracardiac ultrasound images belonging to a target intracardiac ultrasound image type from the target data, and input the actual intracardiac ultrasound images to the atrial fibrillation recurrence prediction model to obtain a target atrial fibrillation recurrence prediction result.

[0058] Optionally, the first number of the sample intracardiac ultrasound images correspond to different image groups under different preset surgical procedure stages, and each of the image groups corresponds to a number of frames of the sample intracardiac ultrasound images.

[0059] The first contribution value calculation module comprises:

[0060] a first function construction unit configured to construct a first contribution value calculation function based on an output function of the atrial fibrillation recurrence prediction model currently trained;

[0061] a group contribution calculation unit configured to calculate a group contribution value of each of the image groups to training of the atrial fibrillation recurrence prediction model by using the first contribution value calculation function;

[0062] a second function construction unit configured to construct a second contribution value calculation function based on an attention mechanism of the atrial fibrillation recurrence prediction model currently trained;

[0063] The contribution weight determination unit is configured to calculate an attention score corresponding to each frame of the sample intracardiac ultrasound image in each of the image groups by using the second contribution value calculation function, and use the attention score as a contribution weight value for training of the atrial fibrillation recurrence prediction model.

[0064] The first contribution value calculation unit is configured to obtain the first contribution value corresponding to each frame of the sample intracardiac ultrasound image by multiplying the contribution weight value and the group contribution value.

[0065] Optionally, the matching degree calculation module comprises:

[0066] The third function construction unit is configured to construct a matching degree calculation function based on a feature vector of the atrial fibrillation recurrence prediction model that is currently trained.

[0067] The matching degree calculation unit is configured to calculate the matching degree between each frame of the sample intracardiac ultrasound image and each type of the other modal data by using the matching degree calculation function.

[0068] Optionally, the credibility factor calculation module comprises:

[0069] The fourth function construction unit is configured to obtain a mean similarity and a similarity standard deviation of the other modal data on a training set, and construct a credibility calculation function.

[0070] The credibility factor calculation unit is configured to calculate the cross-modal credibility factor of each type of the other modal data based on the credibility calculation function.

[0071] Optionally, the target contribution value calculation module comprises:

[0072] The credibility mean calculation unit is configured to calculate a credibility mean corresponding to the cross-modal credibility factor of different types of the other modal data.

[0073] The target contribution value calculation unit is configured to calculate the target contribution value for training of the model by each frame of the sample intracardiac ultrasound image based on the first contribution value, the matching degree, and the credibility mean.

[0074] The screening module is further configured to sort different target contribution values, and select the second number of sample intracardiac ultrasound images with higher target contribution values.

[0075] Optionally, the prediction system further comprises:

[0076] The first feature vector extraction module is configured to extract a first feature vector corresponding to each type of modal data.

[0077] The correlation matrix acquisition module is configured to form a multi-modal feature correlation matrix based on the different first feature vectors being spliced.

[0078] The generation network construction module is configured to construct a cross-modal generation network based on the multi-modal feature correlation matrix in response to there being missing at least one modal data in the first feature vector.

[0079] The fusion feature acquisition module is configured to estimate an uncertainty score of the completed feature based on the cross-modal generation network to obtain a fusion feature of the completed modal.

[0080] The target feature vector acquisition module is configured to fuse the fusion feature and the feature in the first feature vector to obtain a target feature vector, so that the preset network is trained based on the target feature vector.

[0081] Optionally, the fusion feature acquisition module is further configured to estimate the uncertainty score of the completed feature of the cross-modal generation network by using a preset estimation algorithm, obtain a credibility weight of the completed modal based on the uncertainty score, and calculate the fusion feature of the completed modal according to the credibility weight.

[0082] And / or,

[0083] A modal reconstruction loss is introduced into the preset network, so that a difference between the fusion feature and a real modal feature in a feature distribution space is less than a preset condition.

[0084] In a third aspect, the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and configured to run on the processor, wherein the processor executes the computer program to implement the atrial fibrillation recurrence prediction method.

[0085] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the atrial fibrillation recurrence prediction method.

[0086] In a fifth aspect, the present disclosure provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the atrial fibrillation recurrence prediction method.

[0087] On the basis of common sense in the art, the above-mentioned preferred conditions can be combined arbitrarily to obtain each preferred example of the present disclosure.

[0088] The positive progress effect of the present disclosure is that:

[0089] In the present disclosure, an intraoperative atrial fibrillation recurrence risk real-time evaluation and precise guidance scheme based on a multi-modal artificial intelligence model is proposed, which can specifically combine case text data, preoperative electrocardiogram data, preoperative transthoracic echocardiogram image data, postoperative review electrocardiogram data and other modal data for model training, thereby ensuring the accuracy of the atrial fibrillation recurrence prediction model training.

[0090] Meanwhile, the influence degree of each frame of ICE image in each group of sample training data on the prediction result of the model is calculated quantitatively to select a certain number of ICE images with a high influence degree; then, the intersection of the certain number of ICE images selected based on a plurality of groups of sample training data is taken as the ICE image finally selected, and different target intracardiac ultrasound image ICE types corresponding to the ICE image are obtained; that is, the contribution of each frame of sector image to the prediction result of the prediction model is realized through the explainability analysis tool. Thus, in an actual prediction scene, a first number of ICE images do not need to be acquired completely, but only a small number of ICE images meeting the requirements need to be collected and input into the atrial fibrillation recurrence prediction model to obtain a high-precision prediction result. For example, when it is detected that the input is the top 10 important sector images in the aforementioned analysis, the ICE image is automatically saved and the atrial fibrillation recurrence prediction model is immediately run. This not only provides explainable prediction basis for doctors and helps to select the ICE image frame with the highest diagnostic value in the operation, but also provides the operator with an operation effect prompt and dynamically feeds back the key ICE sector type required by the current atrial fibrillation recurrence prediction model. That is, by collecting and automatically classifying the intracardiac ultrasound ICE image in real time during the operation, the key view with the highest value for atrial fibrillation recurrence prediction can be dynamically identified and saved, thereby guiding the clinical operation and data collection. The image collection is reduced, and the efficiency and accuracy of the overall atrial fibrillation recurrence prediction processing are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0091] Figure 1 A flowchart of the atrial fibrillation recurrence prediction method of the present disclosure embodiment 1;

[0092] Figure 2 A schematic diagram of the patient information registration interface of the present disclosure embodiment 1;

[0093] Figure 3 A first flowchart of the atrial fibrillation recurrence prediction method of the present disclosure embodiment 2;

[0094] Figure 4 A second flowchart of the atrial fibrillation recurrence prediction method of the present disclosure embodiment 2;

[0095] Figure 5 A schematic diagram of the ICE image acquisition guidance of the present disclosure embodiment 3;

[0096] Figure 6This is the third flowchart of the atrial fibrillation recurrence prediction method of Embodiment 2 of this disclosure;

[0097] Figure 7 This is a schematic diagram of the training process of the atrial fibrillation recurrence prediction system according to Embodiment 2 of this disclosure;

[0098] Figure 8 This is a schematic diagram of the atrial fibrillation recurrence prediction system according to Embodiment 3 of this disclosure;

[0099] Figure 9 This is a schematic diagram of the atrial fibrillation recurrence prediction system according to Embodiment 4 of this disclosure;

[0100] Figure 10 This is a schematic diagram of the structure of the electronic device according to Embodiment 5 of this disclosure. Detailed Implementation

[0101] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0102] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not be construed as an unnecessary limitation. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0103] In this embodiment of the disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good morals.

[0104] This disclosure recognizes the urgent clinical need for a solution that can integrate and fuse multimodal medical data, possess robustness to missing modalities, and support intelligent auxiliary assessment throughout the entire process from preoperative to postoperative. The aim is to improve the objectivity, accuracy, and safety of cardiac surgery diagnosis and treatment, and to promote the realization of precision medicine and intelligent decision-making in the field of arrhythmia surgery. Specifically, it proposes a method, system, electronic device, and storage medium for predicting atrial fibrillation recurrence, including:

[0105] Example 1

[0106] like Figure 1 As shown, the atrial fibrillation recurrence prediction method in this embodiment includes:

[0107] S101. Based on several sets of first sample training data of multimodal modes, train the preset network to obtain the atrial fibrillation recurrence prediction model.

[0108] wherein each set of first sample training data comprises preoperative input patient partial or whole examination data (including case text, electrocardiogram data, transthoracic echocardiogram images, etc.), i.e., can cover automatic processing of various medical data (including text, electrocardiogram, image or video, etc.) of a patient at different stages of preoperative, intraoperative and postoperative. Figure 2 As shown, the registration interface of the patient medical data can record all information of the case of any patient, preoperative, intraoperative and postoperative at different stages.

[0109] Optionally, the multi-modal data included in each set of first sample training data comprises a first number of sample intracardiac echocardiogram images and at least one other modal data; the other modal data comprises case text data, preoperative electrocardiogram data, preoperative transthoracic echocardiogram image data, postoperative review electrocardiogram data, etc.; meanwhile, the model is trained with the recurrence of atrial fibrillation in the review result at 12 months after the operation as the gold standard label. In addition, the preset network includes but is not limited to an artificial intelligence network model, a large model, etc.

[0110] Specifically, the sample intracardiac echocardiogram image (i.e., ICE image) comprises several (such as 25) 2D (two-dimensional) images, covering key views of the ICE (intracardiac echocardiogram) catheter at different sectors of the left atrium during the operation; the case text data comprises text or structured form, reflecting the basic information, medical history, medication, etc. of the patient; the preoperative electrocardiogram data comprises 1D (one-dimensional) signal data, usually time-series ECG (electrocardiogram) waveform, etc.; the preoperative transthoracic echocardiogram image data comprises 3D (three-dimensional) video data, reflecting the structure and function of the heart, etc.; the postoperative review electrocardiogram data comprises 1D signal data, used to reflect the postoperative heart rhythm change, etc.

[0111] S102, calculate a first contribution value of each frame of sample intracardiac echocardiogram image to the training of the atrial fibrillation recurrence prediction model, a matching degree between each frame of sample intracardiac echocardiogram image and each other modal data, and a cross-modal credibility factor of each other modal data;

[0112] S103, determine a target contribution value of each frame of sample intracardiac echocardiogram image to the training of the atrial fibrillation recurrence prediction model based on the first contribution value, the matching degree and the cross-modal credibility factor;

[0113] S104, select a second number of sample intracardiac echocardiogram images from the first number of sample intracardiac echocardiogram images based on the size of the different target contribution values, and obtain different target intracardiac echocardiogram image types;

[0114] Wherein, the influence degree of each frame ICE image participating in model training in any set of sample training data on the model prediction result is calculated by quantization, and then a certain number of ICE images with the top influence degree are selected;

[0115] Specifically, for example, for 25 ICE images in each set of sample training data for model training, the SHAP (a feature importance decomposition technique based on cooperative game theory) method is used to quantitatively calculate the influence (target contribution value) of each ICE image on the prediction result. Based on the score ranking of different target contribution values, the sector types of the top 10 important intracardiac echocardiography (ICE) images are recorded, which provides a basis for subsequent intraoperative real-time prediction.

[0116] S105, acquiring target data to be processed;

[0117] S106, extracting a plurality of frames of actual intracardiac echocardiogram images belonging to the target intracardiac echocardiogram image type from the target data, and inputting them into the atrial fibrillation recurrence prediction model to obtain a target atrial fibrillation recurrence prediction result.

[0118] Wherein, the target atrial fibrillation recurrence prediction result includes the probability of atrial fibrillation recurrence, the abnormal excitation position in the left atrium of the patient (i.e. the intracardiac position requiring ablation), etc.

[0119] In the present disclosure, an intraoperative atrial fibrillation recurrence risk real-time evaluation and precise guidance scheme based on a multi-modal artificial intelligence model is proposed, which can combine case text data, preoperative electrocardiogram data, preoperative transthoracic echocardiogram image data, postoperative review electrocardiogram data and other modal data for model training, thereby ensuring the accuracy of the atrial fibrillation recurrence prediction model training.

[0120] Meanwhile, the influence degree of each frame of ICE image in each group of sample training data on the prediction result of the model is calculated quantitatively to select a certain number of ICE images with high influence degree; and the intersection of the certain number of ICE images selected based on several groups of sample training data is taken as the ICE image finally selected, and different target intracardiac ultrasound images ICE types corresponding to the ICE image are obtained; that is, the contribution of each frame of sector image to the prediction result of the prediction model is realized by the explainability analysis tool. Therefore, in the actual prediction scene, the first number of ICE images do not need to be acquired completely, but only a small number of ICE images meeting the requirements need to be collected and input into the atrial fibrillation recurrence prediction model to obtain a high-precision prediction result. For example, when it is detected that the input is the top 10 important sector images in the above analysis, the ICE image is automatically saved and the atrial fibrillation recurrence prediction model is immediately run. Not only can the explainable prediction basis be provided for the doctor to help select the ICE image frame with the highest diagnostic value in the operation, but also the operation effect can be prompted for the operator, and the key ICE sector type required by the atrial fibrillation recurrence prediction model can be dynamically fed back. That is, by collecting and automatically classifying the intracardiac ultrasound ICE image in the operation in real time, the key view with the highest value for atrial fibrillation recurrence prediction can be dynamically identified and saved, thereby guiding the clinical operation and data collection, reducing the image acquisition, and effectively improving the efficiency and precision of the whole atrial fibrillation recurrence prediction processing.

[0121] It should be noted that after the contribution of each frame of sector image to the prediction result of the prediction model is realized by the explainability analysis tool (such as SHAP), the prediction result can be optimized in the actual prediction scene to guide the clinical operation, and the explainability analysis tool can also be applied in the iterative training and optimization process of the atrial fibrillation recurrence prediction model. That is, in the iterative training process, the sample intracardiac ultrasound image in the first sample training data input can be determined based on the top 10 important sector images selected above. Only in the model training stage can the collection of intraoperative data be further reduced, and the model training complexity can be simplified, and the training effect of the atrial fibrillation recurrence prediction model with higher precision can be realized with the least intraoperative data.

[0122] Embodiment 2

[0123] The atrial fibrillation recurrence prediction method of this embodiment is a further improvement of the atrial fibrillation recurrence prediction method of embodiment 1. Specifically:

[0124] Existing intracardiac ultrasound (ICE) can provide high-resolution images of intracardiac structures during interventional procedures such as atrial fibrillation ablation. Combined with artificial intelligence (AI) prediction models, it can provide intraoperative risk warnings and predict postoperative recurrence. However, the prediction process of AI models often exhibits "black box" characteristics, lacking transparent explanation of the model's decision-making basis, which hinders physicians' trust in the prediction results and their ability to make rapid decisions during the procedure. This is especially true in scenarios involving multi-frame ICE images and multimodal data fusion, where it is difficult to determine which specific image frames or image types play a crucial role in the final prediction result.

[0125] Based on this, the inventors proposed a technical solution that can quantify the impact of each ICE image participating in model training on the model prediction results in each set of sample training data. Specifically:

[0126] In one feasible scheme, a first number of sample intracardiac ultrasound images correspond to different image groups under different preset surgical procedure stages, and each image group corresponds to several frames of sample intracardiac ultrasound images.

[0127] Specifically, for example, the first number of intracardiac ultrasound images in each group of sample training data is 25 frames of ICE images, which are divided into three image groups according to the surgical procedure: (1) First stage (observation of the heart chamber before ablation): containing 10 frames of images; (2) Second stage (observation of the heart chamber after ablation): containing 10 frames of images; (3) Third stage (observation of the ablation location after ablation): containing 5 frames of images.

[0128] like Figure 3 As shown, the specific steps in step S102 for calculating the first contribution value of each frame of intracardiac ultrasound image to the training of the atrial fibrillation recurrence prediction model include:

[0129] S10211, The first contribution value calculation function is constructed based on the output function of the currently trained atrial fibrillation recurrence prediction model;

[0130] S10212. The group contribution value of each image group to the training of the atrial fibrillation recurrence prediction model is calculated using the first contribution value calculation function.

[0131] S10213, The second contribution value calculation function is constructed based on the attention mechanism of the currently trained atrial fibrillation recurrence prediction model;

[0132] S10214. The attention score corresponding to the intracardiac ultrasound image of each frame sample in each image group is calculated using the second contribution value calculation function, and used as the contribution weight value for training the atrial fibrillation recurrence prediction model.

[0133] S10215. Based on the product of the contribution weight value and the group contribution value, the first contribution value corresponding to the intracardiac ultrasound image of each frame is obtained.

[0134] Specifically: (1) Group-level SHAP calculation

[0135] The first contribution value calculation function is constructed as follows, where each stage of image group is regarded as a feature, and the first contribution value of each image group is calculated:

[0136]

[0137] Wherein, represents the average contribution value of any target image group to the prediction model, G represents the set of image groups G = {image group of the first stage, image group of the second stage, image group of the third stage}, g represents the target image group, which is a subset of G; represents the stage subset not containing g, |G|, |S| represents the number of elements of the set,! represents factorial, f(·) represents the output function of the trained atrial fibrillation recurrence model; x represents the original multi-modal input of the current case, x S represents the input that only enables the image groups in the set S, and the rest of the image groups are completely masked; x S∪{g} represents the simultaneous enablement of S and g; by grouping, the number of combinations of features is reduced from 2 25 to 2 3 , thereby achieving the effect of greatly reducing the calculation amount.

[0138] (2) Frame-level contribution value calculation within the image group

[0139] After obtaining the contribution value of the image group g , the contribution value of the image group needs to be allocated to each frame ICE image within the group;

[0140] The second contribution value calculation function is constructed, and according to the attention mechanism in the prediction model, the attention score corresponding to each frame ICE image is directly extracted as the contribution weight value:

[0141]

[0142] Wherein, the contribution weight value of the i-th frame image, q i is the Query vector (query vector) of the i-th frame image in the attention layer, k i is the Key vector (key vector) of the i-th frame image in the attention layer, d is the dimension of the feature vector, n g is the number of frames in the group g, and then the contribution value of each frame ICE image is represented as:

[0143]

[0144] Wherein, The first contribution value of the prediction model for the i-th frame of image pair.

[0145] In an implementable solution, as shown in Figure 3 the specific steps of calculating the matching degree between each frame of sample intracardiac ultrasound image and each other modality data in step S102 include:

[0146] S10221, the matching degree calculation function constructed based on the feature vector of the currently trained atrial fibrillation recurrence prediction model;

[0147] S10222, the matching degree between each frame of sample intracardiac ultrasound image and each other modality data is calculated by using the matching degree calculation function.

[0148] Specifically, the input of the prediction model as a whole can be represented as:

[0149] M={T,E pre ,U,I g,i ,E post}

[0150] Wherein, M represents all input sets, T represents text input of the case, E pre represents preoperative electrocardiogram input, U represents preoperative transthoracic ultrasound input, I g,i represents intracardiac ultrasound image input, i represents frame number, E post represents postoperative electrocardiogram input.

[0151] The matching degree calculation function is constructed as follows, specifically, the matching degree between each frame of ICE image in intracardiac ultrasound image and other modality input is:

[0152]

[0153] Wherein, r g,i represents the matching degree value of the i-th frame and other modality input, ranging from 0 to 1; M \I represents the input set of other modalities except intracardiac ultrasound, F g,i represents the feature vector of the i-th frame in the atrial fibrillation recurrence prediction model, F m represents the feature vector of input m in the atrial fibrillation recurrence prediction model; sim(·) represents the cosine similarity of two feature vectors.

[0154] In an implementable solution, as shown in Figure 3 the specific steps of calculating the cross-modality credibility factor of each other modality data in step S102 include:

[0155] S10231, the mean similarity and similarity standard deviation of other modality data on the training set are obtained to construct the credibility calculation function;

[0156] S10232, calculate the cross-modal credibility factor of each other modal data based on the credibility calculation function.

[0157] The credibility calculation function is constructed as follows. Specifically, in order to reflect the credibility of the modal m in the case, the cross-modal credibility factor of each modal input data is expressed as:

[0158]

[0159] wherein μ m represents the mean similarity of the modal m feature on the training set, σ m represents the similarity standard deviation of the modal m feature on the training set, ∈ represents a small number to prevent the divisor from being 0, and if σ m is large, β m is reduced, indicating that the credibility is reduced, and vice versa.

[0160] In the present scheme, the consistency information between the ICE image and other modal data (case text, electrocardiogram, transthoracic ultrasound, etc.) is considered to ensure the cross-modal credibility of the interpretation result.

[0161] In an implementable scheme, as shown in Figure 4 , step S103 comprises:

[0162] S1031, calculate the credibility mean value corresponding to the cross-modal credibility factor corresponding to different other modal data respectively;

[0163] S1032, based on the first contribution value, the matching degree and the credibility mean value, calculate the target contribution value of each frame of sample intracardiac ultrasound image to the model training;

[0164] Specifically, the contribution value of each frame of ICE image in the image group, the matching degree value of the i-th frame and other modal input, and the cross-modal credibility adjustment factor are combined to obtain the final weighted frame contribution value, which is expressed as:

[0165]

[0166] wherein, represents the final target contribution value of each frame of ICE image after weighting, r g,i is the matching degree of the i-th frame and other modal input, represents the credibility mean value of all other modalities.

[0167] In the scheme, the SHAP quantification is used to calculate the contribution of each frame of ICE image in the sample training data to the prediction accuracy of the prediction model, and the exponential complexity of the frame-by-frame SHAP is reduced to the constant complexity of the image group, which can greatly reduce the calculation amount, is suitable for real-time application, can quickly respond in the actual prediction scene, ensures that the explanation result is more stable, and fuses the multi-modal consistency weight to make the selected key ICE image highly matched with other modal information, which is more consistent with the clinical diagnosis logic and has higher clinical credibility. The scheme can adapt to different numbers of surgical stages and multi-modal input types, has strong universality, is easy to expand, and has other advantages.

[0168] In an implementable scheme, the step of selecting the second number of sample ICE images from the first number of sample ICE images based on the size of the different target contribution values comprises:

[0169] The different target contribution values are sorted, and the second number of sample ICE images with high target contribution values are selected.

[0170] Specifically, for the first number of ICE images in any group of sample training data, the different target contribution values of each frame of ICE image are obtained based on the above calculation process, and then the target contribution values are sorted from large to small or from small to large according to the size of the target contribution values. The greater the target contribution value, the greater the degree of influence on the prediction accuracy of the prediction model, and a certain number of ICE images with the highest target contribution value are selected therefrom. For a plurality of groups of sample training data, a certain number of ICE images corresponding to the sample training data can be selected. The intersection of the certain number of ICE images of the plurality of groups of sample training data is taken to obtain the second number of sample ICE images with high target contribution values.

[0171] For example, the target contribution values of 25 frames of ICE images in each group of sample training data are sorted from high to low to finally select the top 10 important ICE images.

[0172] In an actual prediction scene, the atrial fibrillation recurrence prediction model can perform real-time discrimination on the collected ICE image fan types. When it is detected that the input is the top 10 important fan of the SHAP analysis, the ICE image is automatically saved and the atrial fibrillation recurrence risk prediction is immediately run to provide the operator with a surgical effect prompt. In addition, the key ICE fan type required by the current prediction model can be dynamically fed back, such as Figure 5 As shown, the ICE image corresponding to the required fan is obtained by guiding the ICE to the required fan during the operation, that is, the image acquisition is guided, so as to further accelerate and accurately collect all required ICE images, thereby further improving the efficiency of atrial fibrillation recurrence prediction.

[0173] In an implementable solution, after the step of obtaining a plurality of sets of multimodal first sample training data, the step of training the preset network based on the plurality of sets of multimodal first sample training data to obtain the atrial fibrillation recurrence prediction model is preceded by, as shown in Figure 6 The prediction method further comprises:

[0174] S10101, a first feature vector corresponding to each modality data is extracted;

[0175] As shown in Figure 7 To further ensure the accuracy of model training, different types of modality data are processed by matching independent feature encoders, specifically: 1D Encoder (one-dimensional encoder): time series and text feature extraction are performed on electrocardiogram and medical history data to obtain latent feature vectors; 3D Encoder (three-dimensional encoder): for preoperative ultrasound video data, a 3D (three-dimensional) convolutional network or a spatiotemporal feature extraction module is used to obtain heart structure and motion information features; 2D*25Encoder: for 25 sector images of intraoperative intracardiac ultrasound, 2D convolutional networks are used to extract features to obtain image-level deep features of each sector.

[0176] S10102, a multimodal feature correlation matrix is formed based on different first feature vectors;

[0177] S10103, in response to the presence of missing at least one modality data in the first feature vector, a cross-modality generation network is constructed based on the multimodal feature correlation matrix;

[0178] S10104, the uncertainty score of the completed feature is estimated based on the cross-modality generation network to obtain the fusion feature of the completed modality;

[0179] S10105, the fusion feature and the feature in the first feature vector are fused to obtain a target feature vector, so that the preset network is trained based on the target feature vector.

[0180] As shown in Figure 7 The training process of the atrial fibrillation recurrence prediction system in the present disclosure.

[0181] In the present solution, a set of multimodal deep learning network structure and training paradigm supporting modality missing is innovatively designed, which can efficiently fuse medical history, electrocardiogram, ultrasound and other heterogeneous medical data. In view of the common modality missing problem of actual clinical data, the network introduces a random mask mechanism and a feature generation type Auto-Encoder fusion module in the training stage, which significantly enhances the adaptability and prediction robustness to missing data.

[0182] Specifically, the feature vectors of all modalities are concatenated to form a multi-modal feature set as the input of the Auto-Encoder; the Auto-Encoder module further fuses and reconstructs the multi-modal features, which can enhance the correlation expression between features, help improve the robustness of the model to missing modalities and the effectiveness of feature fusion, and finally obtain a new high-level fusion feature vector representing the comprehensive diagnostic information of the current case.

[0183] In the training phase, the network uses a random mask mechanism to replace part of the input modalities with null or specific features (such as all-1 vectors), which significantly improves the robustness and generalization ability of the model to missing clinical data; and the Auto-Encoder method generates features to compensate for the impact of missing data on feature content.

[0184] In this scheme, missing modality data can be identified in a timely manner. For incomplete data, the fusion features of the completed modalities are quickly determined based on the cross-modal generation network to obtain the target feature vector, ensuring the feasibility of the preset network model training, and supporting effective handling of the missing modality problem that may exist in multi-modal atrial fibrillation recurrence prediction.

[0185] In an implementable scheme, step S10104 includes:

[0186] The preset estimation algorithm is used to estimate the uncertainty score of the cross-modal generation network completion feature;

[0187] The uncertainty score is used to obtain the credibility weight of the completed modality;

[0188] The credibility weight is used to calculate the fusion feature of the completed modality;

[0189] In an implementable scheme, a modality reconstruction loss is introduced into the preset network to make the difference between the fusion feature and the real modality feature in the feature distribution space less than a preset condition.

[0190] In an implementable scheme, step S10101 includes:

[0191] The feature extraction method matched with each modality data is used for feature extraction to obtain the corresponding first feature vector;

[0192] In an implementable scheme, after the step of obtaining the target feature vector, the method further includes:

[0193] The target feature vector is input into the full connection layer classification network of the preset network for training to obtain an atrial fibrillation recurrence prediction model.

[0194] Among them, the new fusion feature is input to the three fully connected layer classification network, through nonlinear transformation and feature interaction, the final prediction output of whether the atrial fibrillation recurs is realized; the output is the binary classification result of recurrence / non-recurrence, and the Softmax (normalized exponential function) is used for output.

[0195] After the training is completed, the SHAP interpretive analysis method is used to attribute the contribution degree of each type of ICE image input, to determine which view has the greatest influence on the model decision, and the top 10 important ICE image sector types are recorded to guide the intraoperative real-time acquisition, so as to ensure the atrial fibrillation recurrence prediction efficiency and accuracy of the atrial fibrillation recurrence prediction model, and closed-loop optimization of clinical operation, etc.

[0196] In the scheme, the features of each modality data are extracted by an independent encoder, and then are deeply fused, and the contribution of each modality and each ICE sector in the prediction model is quantitatively evaluated by the explainable analysis tool SHAP, so that the obtained atrial fibrillation recurrence prediction model not only improves the depth and effectiveness of multi-source information fusion, but also provides more explainable intelligent auxiliary support for clinical decision-making.

[0197] Specifically, the implementation process of the multi-modal neural network supporting modality missing in the embodiment is as follows:

[0198] (1) Construct a multi-modal feature correlation matrix

[0199] Based on the multi-modal data input by the model, the corresponding feature vectors are extracted:

[0200] z m =Encoder m (x m ),m∈{T,E pre ,U,I g,i ,E post}

[0201] Wherein, T represents the text input of the case, E pre represents the preoperative electrocardiogram input, U represents the preoperative transthoracic ultrasound input, I g,i represents the intracardiac ultrasound image input, i represents the frame number, E post represents the postoperative electrocardiogram input, and then the multi-modal feature correlation matrix is calculated by using the historical complete sample as:

[0202]

[0203] Wherein, R a,b represents the correlation coefficient of modality a and modality b, Cov represents the covariance operation, and σ represents the standard deviation.

[0204] (2) Cross-modality missing modality prediction network

[0205] When a certain modality m * When missing, select the most relevant k modality (k = 3) according to the constructed multi-modal feature correlation matrix, and construct a cross-modal generation network:

[0206]

[0207] wherein, represents the modality set with the highest correlation with m * The CMGN adopts a cross-modal attention mechanism to realize the weighted fusion of different source modalities.

[0208] (3) Uncertainty weighted fusion

[0209] The dropout method (a regularization technique that prevents overfitting by randomly discarding neurons during deep learning training) is used to estimate the uncertainty of the completed features:

[0210]

[0211] wherein, represents the uncertainty score, and S represents the number of sampling times. Further, in the final fusion, the confidence weight of the completed modality is calculated:

[0212]

[0213] (4) Final input feature fusion

[0214] The original modality features and the completed modality features are fused according to the weights:

[0215] z fusion = Concat({w m ,z m})

[0216] wherein, z fusion is the fusion feature of the fused original modality and completed modality, and Concat is the concatenation operation of the feature vector.

[0217] (5) Add modality reconstruction loss

[0218] For the completion process of the missing modality, a modality reconstruction loss is introduced in the atrial fibrillation recurrence prediction network to ensure that the completed modality features are as close as possible to the real modality features in the feature distribution space, thereby improving the semantic consistency and discriminability of the generated results. The modality reconstruction loss can be represented as:

[0219]

[0220] wherein, is a real modality m * characteristic vector of the feature vector, represents the vector Euclidean distance square.

[0221] In this embodiment, (1) in the actual prediction scene, the atrial fibrillation recurrence prediction model can distinguish the type of the collected ICE image fan in real time, and when the input is detected as the top 10 important fans in the SHAP analysis, the ICE image will be automatically saved and the atrial fibrillation recurrence risk prediction will be run immediately to provide the operator with operation effect prompts and dynamic feedback of the key ICE fan types required by the current prediction model, thereby guiding the clinical operation and data collection. (2) Based on the prediction results of the left atrial abnormal excitation site by the atrial fibrillation recurrence prediction model based on multi-modal data before operation, the system can indicate and visualize the high-risk excitation area in real time during operation; the operator can quickly locate and mark the abnormal excitation site on the intracardiac echocardiogram, and perform targeted examination and ablation in actual operation; this closed-loop guidance mechanism not only improves the accuracy and efficiency of ablation operation, but also helps to minimize the risk of atrial fibrillation recurrence. (3) It can combine preoperative examination data, intraoperative ICE images and postoperative electrocardiogram review results, etc. to realize personalized re-evaluation of the risk of atrial fibrillation recurrence after operation, optimize follow-up and treatment decisions, and improve the precision and intelligent level of patient management.

[0222] That is, this mechanism breaks through the closed loop of "preoperative-intraoperative-postoperative" risk assessment, realizes individualized, intelligent and dynamic adjustment of surgical auxiliary decision-making, and effectively improves the safety and efficacy of atrial fibrillation ablation surgery.

[0223] Embodiment 3

[0224] As shown in Figure 8 , the atrial fibrillation recurrence prediction system of the embodiment comprises:

[0225] The prediction model acquisition module 1 is configured to train a preset network based on a plurality of groups of multi-modal first sample training data to obtain an atrial fibrillation recurrence prediction model.

[0226] Each group of first sample training data comprises a first number of sample intracardiac echocardiogram images and at least one other modality data, and the other modality data comprises case text data, preoperative electrocardiogram data, preoperative transthoracic echocardiogram image data or postoperative review electrocardiogram data.

[0227] The first contribution value calculation module 2 is configured to calculate a first contribution value of each frame of sample intracardiac echocardiogram image to the training of the atrial fibrillation recurrence prediction model.

[0228] The matching degree calculation module 3 is configured to calculate the matching degree between each frame of sample intracardiac echocardiogram image and each kind of other modality data.

[0229] a credibility factor calculation module 4 configured to calculate a cross-modal credibility factor of each other modality data;

[0230] a target contribution value calculation module 5 configured to determine a target contribution value of each frame of sample intracardiac ultrasound image to the training of the atrial fibrillation recurrence prediction model based on the first contribution value, the matching degree and the cross-modal credibility factor;

[0231] a screening module 6 configured to screen a second number of sample intracardiac ultrasound images from the first number of sample intracardiac ultrasound images based on the size of different target contribution values, and obtain different target intracardiac ultrasound image types;

[0232] a target data acquisition module 7 configured to acquire target data to be processed;

[0233] a prediction module 8 configured to extract actual intracardiac ultrasound images belonging to the target intracardiac ultrasound image type from the target data, and input the actual intracardiac ultrasound images to the atrial fibrillation recurrence prediction model to obtain a target atrial fibrillation recurrence prediction result.

[0234] In the present disclosure, an intraoperative atrial fibrillation recurrence risk real-time evaluation and precise guidance scheme based on a multi-modal artificial intelligence model is proposed, which can combine case text data, preoperative electrocardiogram data, preoperative transthoracic echocardiogram data, postoperative review electrocardiogram data and other modality data for model training, thereby ensuring the accuracy of the training of the atrial fibrillation recurrence prediction model;

[0235] Meanwhile, the influence degree of each frame ICE image in each group of sample training data on the prediction result of the model is calculated quantitatively to select a certain number of ICE images with a high influence degree; and the intersection of the certain number of ICE images selected based on several groups of sample training data is taken as the ICE image finally selected, and different target ICE images are obtained. That is, the contribution of each frame of sector image to the prediction result of the prediction model is realized by the explainability analysis tool. Thus, in the actual prediction scene, the first number of ICE images does not need to be acquired completely, but only a small number of ICE images meeting the requirements need to be collected and input into the atrial fibrillation recurrence prediction model to obtain a high-precision prediction result. For example, when it is detected that the input is the top 10 important sector images in the above analysis, the ICE image is automatically saved and the atrial fibrillation recurrence prediction model is immediately run. Not only can the doctor be provided with an interpretable prediction basis to help select the ICE image frame with the highest diagnostic value in the operation, but also the operator can be provided with an operation effect prompt and dynamic feedback of the key ICE sector type required by the current atrial fibrillation recurrence prediction model. That is, by collecting and automatically classifying the ICE image in the operation in real time, the key view with the highest value for atrial fibrillation recurrence prediction can be dynamically identified and saved, thereby guiding the clinical operation and data collection, reducing the image collection, and effectively improving the efficiency and precision of the overall atrial fibrillation recurrence prediction processing.

[0236] It should be noted that after the contribution of each frame of sector image to the prediction result of the prediction model is realized by the explainability analysis tool, the prediction result can be optimized in the actual prediction scene to guide the clinical operation, and the explainability analysis tool can also be applied in the iterative training and optimization process of the atrial fibrillation recurrence prediction model. That is, in the iterative training process, the sample ICE image in the first sample training data input can be determined based on the top 10 important sector images selected above. Only in the model training stage can the intraoperative data collection be further reduced, and in the meanwhile, the model training complexity can be simplified, and the training effect of the atrial fibrillation recurrence prediction model with higher precision can be realized with the least intraoperative data.

[0237] For the system embodiment, since it basically corresponds to the method embodiment, the related parts can be referred to the part of the method embodiment. The system embodiment described above is only illustrative, and the units described as separate components can or can not be physically separated, and the components of the unit can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. According to the actual needs, some or all of the modules can be selected to achieve the purpose of the present disclosure.

[0238] The scheme of the embodiment belongs to the cross field of artificial intelligence and medical information processing, and relates to a multi-modal data fusion and intelligent analysis system for intra-cardiac surgery auxiliary decision-making. The system integrates advanced technologies such as natural language processing, medical image recognition, time series signal processing and deep learning models, and can efficiently fuse and model multi-source heterogeneous medical data such as patient medical history texts, electrocardiogram data, transthoracic and intracardiac ultrasound images. The system is mainly applied to preoperative recurrence risk prediction of arrhythmia such as atrial fibrillation (AF), real-time identification of abnormal excitation sites and risk signs during operation, and postoperative recurrence probability re-evaluation, and corresponds to technical scenes such as medical artificial intelligence, clinical intelligent decision support and digital medical software system.

[0239] Embodiment 4

[0240] As shown in Figure 9 The AF recurrence prediction system of the embodiment is a further improvement of the embodiment 3. Specifically:

[0241] In an implementable scheme, the first number of sample intracardiac ultrasound images correspond to different image groups under different preset operation process stages, and each image group corresponds to a plurality of frames of sample intracardiac ultrasound images.

[0242] The first contribution value calculation module 2 comprises:

[0243] The first function construction unit is configured to construct a first contribution value calculation function based on an output function of the currently trained AF recurrence prediction model;

[0244] The group contribution calculation unit is configured to calculate a group contribution value of each image group to the training of the AF recurrence prediction model by using the first contribution value calculation function;

[0245] The second function construction unit is configured to construct a second contribution value calculation function based on an attention mechanism of the currently trained AF recurrence prediction model;

[0246] The contribution weight determination unit is configured to calculate an attention score corresponding to each frame of sample intracardiac ultrasound image in each image group by using the second contribution value calculation function, so as to serve as a contribution weight value of the AF recurrence prediction model training;

[0247] The first contribution value calculation unit is configured to obtain a first contribution value corresponding to each frame of sample intracardiac ultrasound image based on the product of the contribution weight value and the group contribution value.

[0248] In an implementable scheme, the matching degree calculation module 3 comprises:

[0249] The third function construction unit is configured to construct a matching degree calculation function based on a feature vector of the currently trained AF recurrence prediction model;

[0250] The matching degree calculation unit is configured to calculate the matching degree between each frame of the sample intracardiac ultrasound image and each other modality data by using a matching degree calculation function.

[0251] In an implementable scheme, the credibility factor calculation module 4 comprises:

[0252] The fourth function construction unit is configured to obtain the mean similarity and the similarity standard deviation of the other modality data on the training set, so as to construct the credibility calculation function.

[0253] The credibility factor calculation unit is configured to calculate the cross-modality credibility factor of each other modality data based on the credibility calculation function.

[0254] In an implementable scheme, the target contribution value calculation module 5 comprises:

[0255] The credibility mean calculation unit is configured to calculate the credibility mean corresponding to the cross-modality credibility factor of different other modality data.

[0256] The target contribution value calculation unit is configured to calculate the target contribution value of each frame of the sample intracardiac ultrasound image to the model training based on the first contribution value, the matching degree and the credibility mean.

[0257] The screening module 6 is further configured to sort the different target contribution values and select the second number of sample intracardiac ultrasound images with higher target contribution values.

[0258] In an implementable scheme, the prediction system further comprises:

[0259] The first feature vector extraction module 9 is configured to extract the first feature vector corresponding to each modality data.

[0260] The correlation matrix acquisition module 10 is configured to splice the different first feature vectors to form a multi-modality feature correlation matrix.

[0261] The generation network construction module 11 is configured to, in response to the absence of at least one modality data in the first feature vector, construct a cross-modality generation network based on the multi-modality feature correlation matrix.

[0262] The fusion feature acquisition module 12 is configured to estimate the uncertainty score of the completed feature based on the cross-modality generation network, so as to obtain the fusion feature of the completed modality.

[0263] The target feature vector acquisition module 13 is configured to fuse the fusion feature and the feature in the first feature vector to obtain a target feature vector, so that the preset network is trained based on the target feature vector.

[0264] In an implementable solution, the fusion feature acquisition module 12 is further configured to estimate the uncertainty score of the cross-modal generative network completion feature by using a preset estimation algorithm; obtain the confidence weight of the completion modal based on the uncertainty score; and calculate the fusion feature of the completion modal according to the confidence weight.

[0265] and / or,

[0266] The modal reconstruction loss is introduced in the preset network, so that the difference between the fusion feature and the real modal feature in the feature distribution space is less than a preset condition.

[0267] In this embodiment, (1) in the actual prediction scene, the atrial fibrillation recurrence prediction model can distinguish the type of the collected ICE image fan in real time. When it is detected that the input is the top 10 important fan in the SHAP analysis, the ICE image will be automatically saved and the atrial fibrillation recurrence risk prediction will be run immediately, providing the operator with a surgical effect prompt and dynamically feeding back the key ICE fan type required by the current prediction model, thereby guiding the clinical operation and data collection. (2) Based on the prediction result of the left atrial abnormal excitation position by the atrial fibrillation recurrence prediction model based on multi-modal data, the system can indicate and visualize the high-risk excitation area in real time during the operation; the operator can quickly locate and mark the abnormal excitation site on the intracardiac echocardiogram, and in actual operation, the operator can perform targeted examination and ablation; this closed-loop guidance mechanism not only improves the accuracy and efficiency of ablation operation, but also helps to minimize the risk of atrial fibrillation recurrence. (3) It can combine preoperative examination data, intraoperative ICE images and postoperative electrocardiogram review results, etc., to realize personalized re-evaluation of the postoperative atrial fibrillation recurrence risk of the patient, optimize subsequent follow-up and treatment decisions, and improve the precision and intelligent level of patient management.

[0268] That is, this mechanism breaks through the closed loop of "preoperative-intraoperative-postoperative" risk assessment, realizes individualized, intelligent and dynamically adjusted surgical decision-making, and effectively improves the safety and efficacy of atrial fibrillation ablation surgery.

[0269] For the system embodiment, since it basically corresponds to the method embodiment, the related parts can be referred to the part of the method embodiment. The system embodiment described above is only illustrative, and the units described as separate components can or can not be physically separated, and the components of the unit can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. According to actual needs, some or all of the modules can be selected to achieve the purpose of the present disclosure.

[0270] Embodiment 5

[0271] Figure 10This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the atrial fibrillation recurrence prediction method described in any of the above embodiments. Figure 10 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0272] like Figure 10 As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).

[0273] Bus 93 includes a data bus, an address bus, and a control bus.

[0274] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.

[0275] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 924, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0276] The processor 91 executes various functional applications and data processing by running computer programs stored in the memory 92, such as the atrial fibrillation recurrence prediction method provided in any of the above embodiments.

[0277] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 96. As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0278] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into units / modules embodied by multiple units / modules.

[0279] Embodiment 6

[0280] The embodiments of the present disclosure further provide a computer readable storage medium, which has stored thereon a computer program. The program, when executed by a processor, implements the atrial fibrillation recurrence prediction method provided by any of the above embodiments.

[0281] More specifically, the readable storage medium can include, but is not limited to, a portable disc, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0282] Embodiment 7

[0283] The embodiments of the present disclosure further provide a computer program product, which comprises a computer program. The computer program, when executed by a processor, implements the atrial fibrillation recurrence prediction method according to any of the above embodiments.

[0284] The program code for executing the computer program product of the present disclosure can be written in any combination of one or more programming languages, and can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0285] Although the specific embodiments of the present disclosure are described above, those skilled in the art should understand that this is only an illustration, and the protection scope of the present disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present disclosure, and such changes and modifications all fall within the protection scope of the present disclosure.

Claims

1. A method of predicting recurrence of atrial fibrillation, characterized by, The prediction method comprises: training a preset network based on a plurality of sets of multimodal first sample training data to obtain an atrial fibrillation recurrence prediction model; each set of the first sample training data comprises a first number of sample intracardiac ultrasound images and at least one other modality data, and the other modality data comprises case text data, preoperative electrocardiogram data, preoperative transthoracic ultrasound image data or postoperative review electrocardiogram data; a first contribution value of each frame of the sample intracardiac ultrasound image to the training of the atrial fibrillation recurrence prediction model, a matching degree between each frame of the sample intracardiac ultrasound image and each of the other modality data, and a cross-modality credibility factor of each of the other modality data are calculated; based on the first contribution value, the matching degree and the cross-modality credibility factor, a target contribution value of each frame of the sample intracardiac ultrasound image to the training of the atrial fibrillation recurrence prediction model is determined; based on the size of different target contribution values, a second number of sample intracardiac ultrasound images are screened from the first number of sample intracardiac ultrasound images, and different target intracardiac ultrasound image types are obtained; target data to be processed is obtained; a plurality of frames of actual intracardiac ultrasound images belonging to the target intracardiac ultrasound image type are extracted from the target data and input into the atrial fibrillation recurrence prediction model to obtain a target atrial fibrillation recurrence prediction result; the first number of sample intracardiac ultrasound images correspond to different image groups under different preset surgical procedure stages, and each image group corresponds to a plurality of frames of sample intracardiac ultrasound images; the step of calculating the first contribution value of each frame of the sample intracardiac ultrasound image to the training of the atrial fibrillation recurrence prediction model comprises: a first contribution value calculation function constructed based on an output function of the currently trained atrial fibrillation recurrence prediction model; a group contribution value of each image group to the training of the atrial fibrillation recurrence prediction model is calculated by using the first contribution value calculation function; a second contribution value calculation function constructed based on an attention mechanism of the currently trained atrial fibrillation recurrence prediction model; an attention score corresponding to each frame of the sample intracardiac ultrasound image in each image group is calculated by using the second contribution value calculation function, so as to serve as a contribution weight value for the training of the atrial fibrillation recurrence prediction model; the first contribution value corresponding to each frame of the sample intracardiac ultrasound image is obtained based on the product of the contribution weight value and the group contribution value; the step of calculating the cross-modality credibility factor of each of the other modality data comprises: an average similarity and a similarity standard deviation of the other modality data on a training set are obtained to construct a credibility calculation function; the cross-modality credibility factor of each of the other modality data is calculated based on the credibility calculation function.

2. The method of predicting recurrence of atrial fibrillation according to claim 1, wherein the step of calculating the matching degree between each frame of the sample intracardiac ultrasound image and each of the other modality data comprises: a matching degree calculation function constructed based on a feature vector of the currently trained atrial fibrillation recurrence prediction model; The matching degree between each frame of the sample intracardiac ultrasound image and each of the other modal data is calculated by using the matching degree calculation function.

3. The method of predicting recurrence of atrial fibrillation according to claim 1 or 2, characterized in that, The step of determining the target contribution value of each frame of the sample intracardiac ultrasound image to model training based on the first contribution value, the matching degree and the cross-modal confidence factor comprises: The confidence mean value corresponding to the cross-modal confidence factor corresponding to different other modal data is calculated. The target contribution value of each frame of the sample intracardiac ultrasound image to model training is calculated based on the first contribution value, the matching degree and the confidence mean value. The step of screening a second number of sample intracardiac ultrasound images from the first number of sample intracardiac ultrasound images based on the size of different target contribution values comprises: The target contribution values are sorted, and the second number of sample intracardiac ultrasound images with higher target contribution values are selected.

4. The method of predicting recurrence of atrial fibrillation according to claim 1 or 2, characterized in that, After the step of obtaining a plurality of groups of multimodal first sample training data, before the step of training a preset network based on the plurality of groups of multimodal first sample training data to obtain the atrial fibrillation recurrence prediction model, the prediction method further comprises: A first feature vector corresponding to each modal data is extracted. A multimodal feature correlation matrix is formed based on different first feature vectors. In response to the existence of missing at least one modal data in the first feature vector, a cross-modal generation network is constructed based on the multimodal feature correlation matrix. The uncertainty score of the completed feature is estimated based on the cross-modal generation network to obtain a fusion feature of the completed modal. The fusion feature and the feature in the first feature vector are fused to obtain a target feature vector, so that the preset network is trained based on the target feature vector.

5. The method of predicting recurrence of atrial fibrillation according to claim 4, wherein The step of estimating the uncertainty score of the completed feature based on the cross-modal generation network to obtain a fusion feature of the completed modal comprises: The uncertainty score of the completed feature of the cross-modal generation network is estimated by using a preset estimation algorithm; A confidence weight of the completed modal is obtained based on the uncertainty score; The fusion feature of the completed modal is calculated based on the confidence weight; And / or, A modal reconstruction loss is introduced into the preset network, so that the difference between the fusion feature and the real modal feature in the feature distribution space is less than a preset condition.

6. The method of predicting recurrence of atrial fibrillation according to claim 4, wherein The step of extracting a first feature vector corresponding to each modal data comprises: Feature extraction is performed by using a feature extraction method matched with each modal data to correspond to the first feature vector respectively; And / or, After the step of obtaining a target feature vector, the method further comprises: The feature corresponding to the target feature vector is input into a fully connected layer classification network in the preset network for training to obtain the atrial fibrillation recurrence prediction model.

7. A system for predicting recurrence of atrial fibrillation, characterized by The prediction system comprises: A prediction model acquisition module is configured to train a preset network based on a plurality of groups of multimodal first sample training data to obtain an atrial fibrillation recurrence prediction model. Each of the first sample training data sets comprises a first number of sample intracardiac echocardiogram images and at least one other modality data, and the other modality data comprises case text data, preoperative electrocardiogram data, preoperative transthoracic echocardiogram image data, or postoperative review electrocardiogram data; a first contribution value calculation module configured to calculate a first contribution value of each of the sample intracardiac echocardiogram images to training of the atrial fibrillation recurrence prediction model; a matching degree calculation module configured to calculate a matching degree between each of the sample intracardiac echocardiogram images and each of the other modality data; a credibility factor calculation module configured to calculate a cross-modality credibility factor of each of the other modality data; a target contribution value calculation module configured to determine, based on the first contribution value, the matching degree, and the cross-modality credibility factor, a target contribution value of each of the sample intracardiac echocardiogram images to training of the atrial fibrillation recurrence prediction model; a screening module configured to screen, based on sizes of different target contribution values, a second number of the sample intracardiac echocardiogram images from the first number of the sample intracardiac echocardiogram images, and obtain different target intracardiac echocardiogram image types; a target data acquisition module configured to acquire target data to be processed; a prediction module configured to extract, from the target data, actual intracardiac echocardiogram images belonging to a target intracardiac echocardiogram image type, and input the actual intracardiac echocardiogram images to the atrial fibrillation recurrence prediction model to obtain a target atrial fibrillation recurrence prediction result; the first number of the sample intracardiac echocardiogram images correspond to different image groups under different preset surgical procedure stages, and each of the image groups corresponds to a number of the sample intracardiac echocardiogram images; the first contribution value calculation module comprises: a first function construction unit configured to construct a first contribution value calculation function based on an output function of the atrial fibrillation recurrence prediction model currently trained; a group contribution calculation unit configured to calculate a group contribution value of each of the image groups to training of the atrial fibrillation recurrence prediction model by using the first contribution value calculation function; a second function construction unit configured to construct a second contribution value calculation function based on an attention mechanism of the atrial fibrillation recurrence prediction model currently trained; a contribution weight determination unit configured to calculate an attention score corresponding to each of the sample intracardiac echocardiogram images in each of the image groups as a contribution weight value for training of the atrial fibrillation recurrence prediction model by using the second contribution value calculation function; a first contribution value calculation unit configured to obtain the first contribution value corresponding to each of the sample intracardiac echocardiogram images based on a product of the contribution weight value and the group contribution value; the credibility factor calculation module comprises: a fourth function construction unit configured to obtain a mean similarity and a similarity standard deviation of the other modality data on a training set to construct a credibility calculation function; a credibility factor calculation unit configured to calculate the cross-modality credibility factor of each of the other modality data based on the credibility calculation function.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory for running on the processor, characterized in that, The processor executes the computer program to implement the atrial fibrillation recurrence prediction method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the atrial fibrillation recurrence prediction method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the atrial fibrillation recurrence prediction method of any one of claims 1 to 6.

Images