A cardiac surgery data processing method and device based on multi-modal AI

Abstract

This invention provides a method and apparatus for processing cardiac surgery data based on multimodal AI, relating to the field of medical imaging and data processing technology. The method includes: generating a preoperative three-dimensional cardiac model based on the target patient's MRI and / or CT images; collecting intraoperative multimodal data and combining it with the preoperative three-dimensional cardiac model to generate a pathological three-dimensional cardiac model; automatically filtering important surgical data points from the intraoperative multimodal data using an LLM auditing engine to generate a postoperative report; when receiving a question from the target patient via voice or text, identifying the question's intent based on a medical visual language model, matching it with the pathological three-dimensional cardiac model, displaying the lesion type and location information related to the pathological three-dimensional cardiac model, and providing an answer generated by the medical visual language model or triggering a physician intervention mechanism according to a preset decision-making mechanism. This addresses problems such as patients' insufficient understanding of cardiac-related information, lack of understanding of the surgical procedure, and difficulty in reviewing postoperative information.

Description

Technical Field

[0001] This invention relates to the field of medical imaging and data processing technology, and in particular to a method and apparatus for processing cardiac surgery data based on multimodal AI. Background Technology

[0002] In cardiac intubation-related medical procedures, patients often lack a comprehensive understanding of their own heart structure, potential lesions, and the intubation path and parameters during surgery, leading to insufficient comprehension of their condition and poor quality of preoperative communication. Furthermore, during the procedure, surgeons struggle to accurately match intraoperative data with preoperative models in real time, and postoperatively, patients cannot directly review key surgical steps and parameter changes, hindering informed understanding and postoperative recovery.

[0003] Currently, existing cardiac modeling and intubation technologies have many shortcomings. In cardiac modeling, the generated models are often not accurate enough to accurately reflect the patient's specific cardiac structure and lesion condition. In pathological simulation, it is impossible to generate diverse, high-probability lesion models, which cannot effectively help patients understand the progression of the disease. In intraoperative data processing, the integration and analysis of multi-source data is not efficient enough, and the accuracy of scenario matching needs to be improved. Postoperative reports are not presented intuitively, making it difficult for patients to understand key information, and there is a lack of effective mechanisms for continuous optimization suggestions.

[0004] Therefore, a cardiac surgery data processing method based on multimodal AI is needed to solve the problems existing in the above-mentioned prior art. Summary of the Invention

[0005] This invention provides a method and apparatus for processing cardiac surgery data based on multimodal AI, which solves the problems in the prior art such as patients' insufficient understanding of cardiac-related information, lack of understanding of the surgical procedure, and difficulty in reviewing postoperative information.

[0006] This invention provides a method for processing cardiac surgery data based on multimodal AI, comprising: Based on the MRI and / or CT images of the target patient, a preoperative three-dimensional cardiac model of the target patient is generated by using a preset DMCVR morphological guided diffusion model. Intraoperative multimodal data of the target patient is collected, and combined with the preoperative three-dimensional cardiac model, a pathological three-dimensional cardiac model of the target patient is generated through the preset DMCVR model. The LLM audit engine automatically filters important surgical data points from the intraoperative multimodal data, generates a postoperative report, and makes visual annotations in the postoperative report, which is used to visually present the data to the target patient. When the target patient asks a question via voice or text, the intent of the question is identified based on the medical visual language model and matched with the three-dimensional heart model of the pathological body and the postoperative report. The lesion type and lesion location information related to the three-dimensional heart model of the pathological body and the relevant report content in the postoperative report are displayed. The answer generated by the medical visual language model is provided or the doctor intervention mechanism is triggered according to the preset decision mechanism.

[0007] According to the present invention, a method for processing cardiac surgery data based on multimodal AI, wherein a preoperative three-dimensional cardiac model of the target patient is generated based on the MRI and / or CT images of the target patient using a preset DMCVR morphological guided diffusion model, comprising: Acquire MRI and / or CT images of the target patient and perform preprocessing; The processed images were segmented using MRSegmentator multimodal segmentation to generate segmentation masks to label the main anatomical structures. The segmentation mask, the MRI image and / or the CT image are input into the preset DMCVR model to generate a preoperative three-dimensional cardiac model of the target patient. Based on the lesion information input by the doctor, the lesion area is marked in the preoperative three-dimensional cardiac model.

[0008] According to a method for processing cardiac surgery data based on multimodal AI provided by the present invention, the method involves collecting intraoperative multimodal data of the target patient, combining it with the preoperative three-dimensional cardiac model, and generating a pathological three-dimensional cardiac model of the target patient using the preset DMCVR model, comprising: The preoperative 3D heart model and the segmentation mask are encoded into a dictionary structure that can be used by the generator and the retrieval machine. The dictionary structure includes: a global feature vector describing the overall morphology of the heart, a local feature vector describing the internal structure of the heart, a segmentation mask, and a skeleton diagram. Real-time acquisition of bone conduction voice data of doctors' communication, endoscopic or X-ray catheter image data, and vital sign data of the target patient throughout the entire surgical process; The bone conduction speech data is transcribed and intent recognition is performed to generate speech intent information, and the catheter trajectory features of the endoscope or X-ray catheter image data are extracted. The voice intent information, the catheter trajectory features, and the vital signs data are merged according to timestamps to generate intraoperative text data; The text instruction parser enables intent recognition and condition mapping of the intraoperative text data and clinical reports, mapping them into fixed-dimensional condition vectors. The dictionary structure and the fixed-dimensional condition vector are fused into a condition vector through a cross-modal multi-attention mechanism; The conditional vector is input into the preset DMCVR model to generate a three-dimensional cardiac model of the pathological body.

[0009] According to the present invention, a method for processing cardiac surgery data based on multimodal AI includes automatically filtering important surgical data points from the intraoperative multimodal data using an LLM auditing engine, generating a postoperative report, and making visual annotations in the postoperative report, including: The intraoperative multimodal data is input into the artificial intelligence speech model in the LLM audit engine to extract key surgical data related to the target patient's condition; The key surgical data is input into the ReportGenChain model in the LLM audit engine to generate a postoperative report, and the key surgical values ​​in the postoperative report are highlighted.

[0010] According to a multimodal AI-based cardiac surgery data processing method provided by the present invention, when a target patient asks a question via voice or text, the method identifies the intent of the question based on a medical visual language model and matches it with the three-dimensional cardiac model of the pathological body and the postoperative report. It then displays information on the lesion type and location related to the three-dimensional cardiac model of the pathological body and relevant report content from the postoperative report. Based on a preset decision-making mechanism, it provides an answer generated by the medical visual language model or triggers a doctor's intervention mechanism, including: The medical visual language model is invoked to identify the question intent based on the target patient's voice or text question and to extract key medical entities; The problem intent is matched with the key medical entity, the three-dimensional cardiac model of the pathological body, and the postoperative report to determine the lesion type, the lesion location information, and the relevant report content; The risk score and regional uncertainty of the lesion location information are determined, wherein the risk score is determined based on the voxel confidence, global uncertainty, multimodal evidence consistency score, and patient factor of the lesion location information, and the regional uncertainty is the proportion of voxel confidence of the target region; When the risk score is less than a first preset threshold, the medical visual language model generates and outputs the corresponding answer based on the question intent, the lesion type, the lesion location information, and the relevant report content; When the risk score is not less than the first preset threshold or the regional uncertainty is not less than the second preset threshold, the preset doctor intervention text information is embedded in the answer, and a review item and explanation of the triggering reason and priority are generated for the doctor in the background.

[0011] According to a multimodal AI-based cardiac surgery data processing method provided by the present invention, after calling the medical visual language model to identify the question intent based on the target patient's voice or text question and extract key medical entities, the method further includes: A natural language understanding model fine-tuned based on a medical corpus is used to perform entity recognition and relation extraction, and output structured query objects, which include: anatomical entities, pathological entities, and spatial modifiers. The spatial-semantic mapping table is used to retrieve the structured query object and output the spatial range data of the target area of ​​the three-dimensional heart model of the pathological body. The spatial-semantic mapping table records the anatomical units and their geometric representations of the three-dimensional heart model of the pathological body. A post-processing algorithm is used to extract and thicken the spatial extent data of the target area to improve visibility; The target area is displayed in a semi-transparent manner to ensure the visibility of the underlying structure; Multiple pathological interpretations of the target area are highlighted side-by-side, and a brief comparative description is given next to the view.

[0012] A cardiac surgery data processing method based on multimodal AI according to the present invention further includes: Interactive prompts are displayed on the target area, along with triggering conditions, button behaviors, and dialogue templates, to guide the target patient to ask questions or select the next step.

[0013] The present invention also provides a cardiac surgery data processing device based on multimodal AI, comprising: The preoperative three-dimensional cardiac modeling module is configured to generate a preoperative three-dimensional cardiac model of the target patient based on the target patient's MRI and / or CT images using a preset DMCVR morphological guided diffusion model. The pathological body three-dimensional heart modeling module is configured to collect intraoperative multimodal data of the target patient, combine it with the preoperative three-dimensional heart model, and generate a pathological body three-dimensional heart model of the target patient through the preset DMCVR model. The postoperative report highlighting module is configured to automatically filter important surgical data points in the intraoperative multimodal data through the LLM auditing engine, generate a postoperative report, and make visual annotations in the postoperative report; The postoperative VQA driver module is configured to, when receiving a question from the target patient via voice or text, identify the intent of the question based on a medical visual language model, match it with the three-dimensional heart model of the pathological body, display the lesion type and lesion location information related to the three-dimensional heart model of the pathological body, and provide the answer generated by the medical visual language model or trigger the doctor's intervention mechanism according to a preset decision mechanism.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cardiac surgery data processing method based on multimodal AI as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cardiac surgery data processing method based on multimodal AI as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cardiac surgery data processing method based on multimodal AI as described above. This invention provides a method and apparatus for processing cardiac surgery data based on multimodal AI. It utilizes AI technology to generate a preoperative 3D heart model from a 2D model of the patient's heart, helping patients intuitively understand their heart structure and pathological conditions. By collecting intraoperative multimodal data throughout the surgery, the data is converted into structured features suitable for scene matching and fused with the preoperative 3D heart model to generate a pathological 3D heart model (equivalent to a postoperative heart model), helping patients understand postoperative changes in their heart. An LLM auditing engine automatically filters out important data points during the surgery, allowing patients to intuitively review core events and parameter changes. Highlighted prompts highlight the significance and potential impact of each key point, enhancing patient understanding of their surgery and providing a clear and visual tool for doctor-patient communication, thus increasing transparency and trust. Based on the patient-specific heart model, visual question answering (VQA) helps patients gain a deeper understanding of possible postoperative changes and coping strategies, improving awareness and communication efficiency, and effectively providing relevant nursing advice. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the cardiac surgery data processing method based on multimodal AI provided by the present invention. Figure 2 This is a schematic diagram of the cardiac surgery data processing device based on multimodal AI provided by the present invention; Figure 3This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] It should be noted that in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediate medium; or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0021] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0022] The following is combined Figures 1 to 3 This invention describes a method and apparatus for processing cardiac surgery data based on multimodal AI.

[0023] like Figure 1 As shown, this invention provides a method for processing cardiac surgery data based on multimodal AI, which specifically includes: Step 110: Based on the MRI and / or CT images of the target patient, generate a preoperative three-dimensional cardiac model of the target patient using a preset DMCVR morphological guided diffusion model. Step 120: Collect intraoperative multimodal data of the target patient, combine it with the preoperative three-dimensional cardiac model, and generate a pathological three-dimensional cardiac model of the target patient through the preset DMCVR model; Step 130: Automatically filter important surgical data points in the intraoperative multimodal data using the LLM audit engine, generate a postoperative report, and make visual annotations in the postoperative report, wherein the postoperative report is used to visually display to the target patient; Step 140: When the target patient asks a question via voice or text, the intent of the question is identified based on the medical visual language model and matched with the three-dimensional heart model of the pathological body and the postoperative report. The lesion type and lesion location information related to the three-dimensional heart model of the pathological body and the relevant report content in the postoperative report are displayed. The answer generated by the medical visual language model is provided or the doctor intervention mechanism is triggered according to the preset decision mechanism.

[0024] 1. Utilize AI technology to generate a preoperative three-dimensional cardiac model from the patient's MRI and / or CT images.

[0025] Specifically, step 110 may include the following steps: Acquire MRI and / or CT images of the target patient and perform preprocessing; The processed images were segmented using MRSegmentator multimodal segmentation to generate segmentation masks to label the main anatomical structures. The segmentation mask, the MRI image and / or the CT image are input into the preset DMCVR model to generate a preoperative three-dimensional cardiac model of the target patient. Based on the lesion information input by the doctor, the lesion area is marked in the preoperative three-dimensional cardiac model.

[0026] 1.1 U-Ne data acquisition and preprocessing: Acquire standardized, high-contrast multi-view two-dimensional cardiac images to facilitate subsequent 3D cardiac model reconstruction.

[0027] Data source: Sparse multi-view two-dimensional cardiac MRI / CT slices of the target patients, with a typical slice spacing of 5–8 mm.

[0028] Data preprocessing: Registration and standardization, using a rigid + non-rigid registration algorithm based on mutual information, with a registration error of <1mm. Image intensity is normalized to the 0–1 range.

[0029] 1.2 MRSegmentator Multimodal Segmentation: MRSegmentator (an open-source medical image segmentation project) was used to construct a complete 3D heart volume from the processed 2D image, generating and labeling the main anatomical structures to provide necessary anatomical information for subsequent 3D heart model reconstruction. A multi-label segmentation mask with a resolution of 512×512×256 voxels was generated by combining 3D convolution with the Transformer module.

[0030] 1.3 Morphology-Guided Diffusion Model (DMCVR) reconstruction: The structural mask and the original images (MRI and / or CT) are input into the DMCVR conditional diffusion model to generate a high-resolution preoperative three-dimensional cardiac model.

[0031] Conditional inputs include the MRSegmentator segmentation mask as a morphological prior and the original sparse slice features. Together, these serve as conditional inputs, enabling the diffusion model to maintain anatomical realism while synthesizing high-resolution image details during image generation, thus improving the medical effectiveness of the 3D heart model.

[0032] Network structure: A conditional UNet based on DDPM, containing 12 residual blocks, with the number of channels increasing from 64 to 512. Morphological conditions are injected into the noise prediction network using 3D residual blocks and cross-attention.

[0033] Sampling steps: 1000-step linear noise scheduling, 50-step DDIM-accelerated sampling. The super-resolution target is 256 mm³ voxels.

[0034] Training details: L1 and perceptual loss weighted (α=0.9 L1 + 0.1 VGG perception) were used. Performance verification: PSNR>30dB, SSIM>0.85.

[0035] 1.4 Lesion Area Marking: Based on the lesion information entered by the doctor, lesion areas such as embolism and dissection are marked and highlighted to show the patient the abnormal heart condition.

[0036] Lesion detection: A pre-trained 3D-CNN lesion detection network (ResNet-3D-18 backbone) was used to scan the cardiac chambers and coronary artery regions on the reconstructed 3D volume. Based on the lesion information input by the doctor, lesions such as embolism and dissection were identified and marked as highlighted areas.

[0037] Probability heatmap: Outputs the lesion probability (range 0–1) for each voxel, with a threshold of 0.5 to mark candidate regions for embolism or dissection.

[0038] Example scenario: The system automatically generates multiple typical lesion severity scenarios (normal / mild / moderate), and patients can switch between them to view the differences and understand which state they are in.

[0039] Thus, high-resolution preoperative three-dimensional cardiac reconstruction can be achieved through sophisticated multimodal segmentation and morphology-guided diffusion technology.

[0040] 2. Collect intraoperative multimodal data throughout the patient's surgery, convert the raw multi-source data into structured features that can be used for scene matching, and combine the intraoperative multimodal data with the preoperative 3D heart model to generate a pathological 3D heart model of the patient.

[0041] Specifically, step 120 may include the following steps: The preoperative 3D heart model and the segmentation mask are encoded into a dictionary structure that can be used by the generator and the retrieval machine. The dictionary structure includes: a global feature vector describing the overall morphology of the heart, a local feature vector describing the internal structure of the heart, a segmentation mask, and a skeleton diagram. Real-time acquisition of bone conduction voice data of doctors' communication, endoscopic or X-ray catheter image data, and vital sign data of the target patient throughout the entire surgical process; The bone conduction speech data is transcribed and intent recognition is performed to generate speech intent information, and the catheter trajectory features of the endoscope or X-ray catheter image data are extracted. The voice intent information, the catheter trajectory features, and the vital signs data are merged according to timestamps to generate intraoperative text data; The text instruction parser enables intent recognition and condition mapping of the intraoperative text data and clinical reports, mapping them into fixed-dimensional condition vectors. The dictionary structure and the fixed-dimensional condition vector are fused into a condition vector through a cross-modal multi-attention mechanism; The conditional vector is input into the preset DMCVR model to generate a three-dimensional cardiac model of the pathological body.

[0042] 2.1 Feature extraction of preoperative 3D cardiac model and segmentation mask: The high-resolution preoperative 3D cardiac model and segmentation mask specific to the target patient are encoded into a semantic representation that can be used by the generator and retrieval machine.

[0043] Data preprocessing: The preoperative 3D cardiac model and segmentation mask were standardized, and the coordinates were normalized to the center of the [−1,1] cube using zero mean and unit variance.

[0044] Resolution strategy: The original high-resolution preoperative 3D heart model is divided into patches (e.g., 128³ patches) and the globally downsampled version is retained (e.g., 256³ downsampled to 64³).

[0045] Feature Extraction: To obtain the overall 3D morphology of the heart, a 3D-UNet encoder (e.g., conv3d-InstanceNorm-LeakyReLU) with 4 layers of downsampling and 64³ downsampled voxels (or 128³ patches) is used to output a global feature vector z_global, which describes the overall morphology of the entire heart. Dimensionality example: C=512, spatial=4×4×4 → flatten → z_global ∈ R^512 To capture the internal structures of the heart and preserve cardiac details (such as valves and focal lesion boundaries), PointNet++ with multi-scale grouping is used to output local feature vectors z_local, which describe local cardiac structural features (e.g., the anterior wall of the left ventricle). For each Region of Interest (ROI), a 128D Descriptor is obtained.

[0046] Then, the outputs of the two parts are concatenated, and after passing through a linear layer and LayerNorm, the output z_3D = [z_global, {z_local}] is obtained.

[0047] Another 3D-UNet encoder is used for chamber / valve segmentation and outputs voxel-level labels (trained using Diceloss). Structured attribute extraction is then used to extract additional output skeleton maps, boundary point sets, and major vessel centerlines (using distance transform + A* extraction).

[0048] This 3D-UNet decoder is used to extract structured attributes such as skeleton maps, boundary point sets, and major blood vessel centerlines. Finally, the above outputs are fused to output a dictionary structure Z_3D, which includes: z_global (global feature vectors describing the overall morphology of the heart), z_local (local feature vectors describing local structural features of the heart), seg_masks (segmentation masks), and skeletons (skeletons).

[0049] 2.2 Surgical process data acquisition and extraction: Intraoperative data is collected as input to the text / instruction parser. After cross-modal multi-attention fusion, the output condition vector and source information are used as condition input to the downstream 3D generator to guide the generation or updating of the pathological body 3D heart model.

[0050] Data sources: bone conduction voice data, endoscopic or X-ray catheter image data, and vital sign data of the target patient monitored by a monitor.

[0051] Embedded ASR (Automatic Speech Recognition) module: Deploy speech recognition models such as WhisperMed on a local server in the operating room, and collect doctors' conversations in real time through bone conduction microphones, with a recognition rate of >95% (signal-to-noise ratio of 20dB).

[0052] Visual key point tracking: Integrating DeepCatheterTrack (deep learning-driven catheter tracking technology), based on YOLOv8 backbone (COCO pre-trained, mAP@0.5=0.78) + Kalman filtering, it continuously tracks the catheter tip in endoscopic / X-ray images with a positional accuracy of <0.5mm and a direction vector recording frequency of 30Hz.

[0053] Vital signs synchronization: Using the FHIRStream (Fast Healthcare Interoperability Resources) framework, the monitors (sampling frequency of 1Hz, including heart rate, systolic blood pressure, diastolic blood pressure, and SpO2) are stored in the database according to the HL7FHIR standard, and all data are timestamped (UTC ± 0.1s accuracy).

[0054] Time-series database: It uses TimescaleDB (an open-source time-series database based on PostgreSQL) to store streaming data, supporting 1000 writes per second and query latency <10ms.

[0055] Output data: Raw surgical event flow E_raw={(t, text, x,y,z,θ, HR, BP, SpO2)}.

[0056] 2.3 Text and Structured Parsing Module: Converts the raw surgical event stream into controlled textual records.

[0057] Speech transcription and intent recognition: Based on an intent classifier fine-tuned by ClinicalBERT, the text of the ASR module is labeled to identify intent categories (such as "assess the degree of stenosis" and "adjust balloon pressure"), with F1>0.88.

[0058] Catheter trajectory feature extraction: Calculate catheter advancement speed v_t (mm / s), distance from centerline d_t (mm), and instantaneous curvature κ_t at a frequency of 30Hz.

[0059] Text Construction: The above multimodal data streams (voice intent, trajectory features, vital signs) are merged according to timestamps to form intraoperative text data x_cur∈R. 8 .

[0060] 2.4 Text / Instruction Parser: Intent recognition and condition mapping are implemented using an LLM (such as a local fine-tuned LLaMA). The intraoperative text data and clinical test reports are input into the LLM, which outputs JSON format. One-Hot encoding is used to map the structured JSON into a fixed-dimensional condition vector c_text. This fixed-dimensional condition vector describes the lesion type mapped to the Z_3D ROI index or spatial coordinates.

[0061] 2.5 Cross-modal multi-attention fusion: The dictionary structure Z_3D and the fixed-dimensional condition vector c_text are fused into a condition vector C through multimodal processing. This C serves as the conditional input for the 3D generator, guiding the generation of a 3D heart model of a pathological body.

[0062] Model: Employs a Cross-Attention Transformer module for cross-modal fusion. It uses one modality as a query to "focus" on the features (key / value) of another modality, thereby achieving selective information fusion.

[0063] Query: Local feature representation from Z_3D, i.e., z_local. These features correspond to different spatial regions in the 3D heart latent space (which can be regarded as a set of spatial tokens), and each token represents the geometric and structural information of a local region of the heart.

[0064] Key and Value: These are derived from the text modality, specifically the clinical report embedding processed by a text encoder (such as BERT or BiLSTM). These text tokens encode the pathological features described intraoperatively.

[0065] Query = Local tokens of the preoperative 3D cardiac model (representing different spatial regions).

[0066] Key / Value = Text token (instruction embedding from the large language model).

[0067] In Cross-Attention, 3D features selectively focus based on text semantics, forming "semantic localization." Introducing 3D spatial location encoding (such as 3D relative location encoding or Fourier features) ensures the model can identify local locations.

[0068] Tokenization: The Z_3D is decomposed into T spatial tokens, for example, each 16³ patch is a token (approximately 64 tokens). Each token contains local anatomical information (such as a region with a wall thickness). The text is encoded into vector tokens using a Large Language Model (LLM).

[0069] Cross-modal attention: Uses 2–4 layers of Cross-Attention (multiple heads per layer, typically 8 heads), with 3D tokens as queries, to "query" relevant semantics in the text tokens, such as the spatial region corresponding to "left ventricular anterior wall".

[0070] Output condition vector: A unified condition vector c = fuse(z_3D, c_text, meta). The fuse() function merges z_3D, c_text, and meta into a 512-d global and per-token positional condition vector C. This condition vector C contains the overall cardiac state (approximately 512 dimensions) and multiple local condition vectors with spatial positions. Meta consists of structured clinical data (numerical + categorical) including age, gender, EF, blood pressure, NYHA classification, etc., serving to provide objective, quantitative priors on cardiac state, enhancing the accuracy and clinical credibility of the generative model.

[0071] 2.6 Generation of the 3D Heart Model of the Pathological Body: The output condition vector C above is directly used as the input of the preset DMCVR model to guide the generation of the 3D heart model of the pathological body.

[0072] Cross-Attention: A cross-modal attention layer is embedded in the encoder path of UNet, using the spatial token of the 3D feature map as the query and the conditional vector C as the key / value to achieve semantically guided spatial localization.

[0073] FiLM Global Modulation: Feature linear modulation is used in each residual block, scaling and bias parameters are generated based on the condition vector C, and global semantic modulation is performed on the convolutional features.

[0074] Multi-scale conditional fusion: Conditional information is injected into different resolution levels of UNet, with lower levels focusing on local details and higher levels focusing on the overall shape.

[0075] Model output: The reconstruction output head generates a high-resolution three-dimensional heart model of the pathological body, and the confidence output head outputs a confidence map U(v)∈[0,1] of the same resolution in parallel. Each voxel has a value range of [0,1] and is activated by sigmoid.

[0076] In some embodiments, the method further includes embedding intraoperative and postoperative image data into a preset DMCVR model to perform local deformation and update of the three-dimensional cardiac model of the pathological body.

[0077] The LLM audit engine automatically filters out important data points during the surgical process, generates a postoperative report, and adds visual annotations to the report.

[0078] Specifically, step 130 may include the following steps: The intraoperative multimodal data is input into the artificial intelligence speech model in the LLM audit engine to extract key surgical data related to the target patient's condition; The key surgical data is input into the ReportGenChain model in the LLM audit engine to generate a postoperative report, and the key surgical values ​​in the postoperative report are highlighted.

[0079] In some embodiments, step 130 may further include: summarizing key surgical procedure paragraphs in the postoperative report and highlighting the key surgical procedure paragraphs.

[0080] In some embodiments, step 130 may further include: automatically filtering out important data points during the surgical process through the LLM audit engine and visually annotating them in the report, which further includes: The operation summary and timestamp of the key surgical procedure segment are hashed, and an audit certificate is generated through the blockchain system; The surgical event log and the postoperative report are stored off-chain, and the integrity is verified using the audit credentials.

[0081] 3.1 Rule base and knowledge graph construction (GuidelineKG & FHIR-RuleKit): The industry guidelines and operational specifications are structured into an easy-to-search knowledge base, supporting the filtering of key information.

[0082] GuidelineKG Ontology and Rule Set: Key points from ASA / ACC / AHA guidelines and the institute's SOPs are transformed into conceptual nodes such as "Operation Name," "Parameter Range," and "Risk Warning," forming a knowledge graph. The ontology and business rules are stored in the graph database in the form of nodes and relationships, such as "Step X requires parameter Y to not exceed Z," and rule retrieval using natural language or predefined templates is supported.

[0083] FHIR-RuleKit assertion extension: Models surgical events in FHIR resource format, defines the compliance constraints that each event node should meet, and describes the compliance conditions in JSON semantics. Through a general event parsing framework, event logs are mapped to corresponding FHIR resources, and then pre-configured assertion templates are used to check whether threshold or order requirements are met.

[0084] 3.2 LLM Audit Engine (AuditGPT & ReportGenChain): Automatically filters out important data points during the surgical process and visually annotates them in the report.

[0085] Large model review: Input the surgical event log (operation name, parameter value, timestamp) into the fine-tuning GPT-4 to automatically extract key data related to guideline thresholds or patient conditions.

[0086] Report generation and highlighting: ReportGenChain automatically summarizes the non-compliant paragraphs based on the model output and highlights them in the report using color or markers, along with the violation type and guiding correction suggestions. The final report can be exported as PDF or HTML format and linked to the institute's document management system for archiving.

[0087] The postoperative report interface highlights key values ​​(such as "balloon pressure reached 4.5 atm") in red or orange, along with brief explanations: how this value compares to the general range (3.5–4.0 atm), and the potential impact on the heart. The postoperative report serves as a visual presentation to the target patient, reviewing the surgical procedure and showcasing key values.

[0088] Specific examples: System Role: Postoperative Information Review Assistant enter: - Operation Log: Balloon inflation; Time: 2025-07-19T10:23:45; Pressure: 4.5 atm - Operation Log: Catheter Advancement; Time: 2025-07-19T10:24:00; Depth: 45 mm - Guideline thresholds: Balloon pressure ≤ 4.0 atm; Catheter depth 20–40 mm Task: Based on the above records and guidelines, identify operation records that exceed the recommended scope and briefly explain the reasons. AuditGPT Output Example: 【2025-07-19T10:23:45】Balloon inflation pressure is 4.5 atm, exceeding the upper limit of 4.0 atm; 【2025-07-19T10:24:00】Catheter depth 45mm, exceeding the recommended range of 40mm.

[0089] 3.3 Blockchain-based Evidence Storage (Hyperledger Fabric) Key records are uploaded to the blockchain: The operation summary and timestamp of the AuditGPT audit are hashed and the transaction is submitted through the Hyperledger Fabric network to generate an immutable audit credential.

[0090] Audit traceability: Complete event logs and audit reports are stored off-chain, and blockchain credentials are used for rapid integrity verification to ensure that the authenticity of records can be verified during subsequent compliance reviews.

[0091] As a result, patients can intuitively review the core events and parameter changes during the surgery, and understand the significance and potential impact of each key point through highlighted prompts. This not only improves patients' informed understanding of their own surgery, but also provides a clear and visual tool for doctor-patient communication, enhancing surgical transparency and trust.

[0092] Fourth, visual question-and-answer interaction with patients is achieved through medical visual language models and three-dimensional heart models of pathological bodies.

[0093] Specifically, step 140 may include the following steps: The medical visual language model is invoked to identify the question intent based on the target patient's voice or text question and to extract key medical entities; The problem intent is matched with the key medical entity, the three-dimensional cardiac model of the pathological body, and the postoperative report to determine the lesion type, the lesion location information, and the relevant report content; The risk score and regional uncertainty of the lesion location information are determined, wherein the risk score is determined based on the voxel confidence, global uncertainty, multimodal evidence consistency score, and patient factor of the lesion location information, and the regional uncertainty is the proportion of voxel confidence of the target region; When the risk score is less than a first preset threshold, the medical visual language model generates and outputs the corresponding answer based on the question intent, the lesion type, the lesion location information, and the relevant report content; When the risk score is not less than the first preset threshold or the regional uncertainty is not less than the second preset threshold, the preset doctor intervention text information is embedded in the answer, and a review item and explanation of the triggering reason and priority are generated for the doctor in the background.

[0094] Specifically, VQA (Visual Question Answering) uses dedicated medical visual language models such as SurgVLM or MedVisualBERT to intelligently display and interact with a 3D heart model of the pathological body when the patient asks a question via VQA (voice / text) or when triggered by the system. It clarifies uncertainties, provides parallel decision-making information, and, when necessary, forces or suggests that doctors intervene to achieve cross-modal understanding and answering of 3D scene diagrams and text questions.

[0095] Intent recognition and model matching: Inputting a patient's natural language question (text or speech-to-text), a fine-tuned clinical BERT or dedicated LLM is used to identify the question's intent (e.g., asking about diagnosis, prognosis, lifestyle, or risk), and key medical entities are extracted (e.g., coronary artery calcification, left atrial appendage thrombosis). The intent and entities are matched with a 3D cardiac model of the pathological body to identify the most relevant lesion type and location information, and matched with postoperative reports to filter out the most relevant report content.

[0096] Decision and Triggering Layer: Determines the risk score (risk_score) based on the lesion location information. In one example, risk_score = clamp( p_model * (1 + 0.1*risk_factor), 0,1 ).

[0097] Where p_model represents the model reliability, p_model = w1 × (1 - P(i)) + w2 × U_global + w3 × (1 - C_evidence).

[0098] P(i): Prior probability / confidence of the generated 3D cardiac model of the pathological body (from step 120), U_global: Global uncertainty = low confidence of voxels (from step 120), C_evidence: Multimodal evidence consistency score (from the comparison of intraoperative text data x_cur and conditional vector C in step 120, similarity measure).

[0099] The risk_factor is a patient factor (age, comorbidities, etc.), and the risk score is multiplied by a patient factor for minor adjustment and simple calibration.

[0100] Regional uncertainty U: U = the proportion of low-confidence voxels in the target region (model output U⁽) 1 (v) The proportion of regions with low voxel confidence is higher than the second preset threshold. The standard for low confidence can be determined according to the actual situation.

[0101] 1. Model predicts low-risk situations: When the risk score is less than the first preset threshold, the medical visual language model generates and outputs the corresponding answer based on the question intent, lesion type and lesion location information.

[0102] 2. Model predicts high-risk situations: When the risk score is not less than the first preset threshold or the regional uncertainty is not less than the second preset threshold, the preset doctor intervention text information is embedded in the generated answer, and a review item and explanation of the triggering reason and priority are generated for the doctor in the background.

[0103] In some embodiments, high-risk situations also trigger proactive information collection (e.g., short questionnaires, system-initiated suggestions for supplementary imaging), and the main alternative explanations are displayed side-by-side in the responses. The system can calculate and return the "expected rate of decrease in uncertainty of supplementary imaging (%)".

[0104] Prompt retrieval and generation: Combined with the LangChain RAG pipeline, it automatically retrieves relevant cases, guideline clauses, and simulation results to generate high-quality answers.

[0105] In some embodiments, after invoking the medical visual language model to identify the question intent based on the target patient's voice or text question and extract key medical entities, the method further includes: A natural language understanding model fine-tuned based on a medical corpus is used to perform entity recognition and relation extraction, and output structured query objects, which include: anatomical entities, pathological entities, and spatial modifiers. The spatial-semantic mapping table is used to retrieve the structured query object and output the spatial range data of the target area of ​​the three-dimensional heart model of the pathological body. The spatial-semantic mapping table records the anatomical units and their geometric representations of the three-dimensional heart model of the pathological body. A post-processing algorithm is used to extract and thicken the spatial extent data of the target area to improve visibility; The target area is displayed in a semi-transparent manner to ensure the visibility of the underlying structure; Multiple pathological interpretations of the target area are highlighted side-by-side, and a brief comparative description is given next to the view.

[0106] In this embodiment, a semantic parsing-driven 3D heart model is used for association and display. Specifically, a natural language understanding (NLP) pipeline fine-tuned based on a medical corpus is used to perform entity recognition and relation extraction on the target patient's speech or text (i.e., VQA data). This identifies anatomical entities (such as "left anterior descending artery" and "aortic valve"), pathological entities (such as "calcification" and "stenosis"), and spatial modifiers (such as "proximal" and "terminal"), outputting a structured query object {anatomical structure: ..., pathological features: ..., spatial modifiers: ...}. A spatial-semantic index matching module is used to search and match the structured query object with the spatial-semantic mapping table, outputting the spatial range data of the target region (focus region).

[0107] The spatial-semantic mapping table is constructed during the generation of the three-dimensional heart model of the pathological body, and records the anatomical units and their geometric representations (mesh, bounding box, center line segment).

[0108] After determining the target area, the following rendering process is performed on the target area: 1. Contour Enhancement: Post-processing algorithms are used to extract and thicken the spatial extent data (focus region) of the target area matched by spatial-semantic indexing to improve visibility.

[0109] 2. Area filling: Use a semi-transparent material to cover the area, ensuring the visibility of the underlying structure.

[0110] 3. Explanation: Multiple explanations for the same anatomical area are highlighted side by side (distinguished by color / legend) and a brief comparison explanation is given next to the view.

[0111] In some embodiments, the method further includes: displaying interactive prompts on the target area and providing triggering conditions, button behaviors, and dialogue templates to guide the target patient to ask questions or select the next step.

[0112] In this embodiment, existing "highlighted areas" are directly mapped to interactive prompts that can be displayed instantly on the interface, and triggering conditions, button behaviors, and dialogue templates are provided. For example, each highlight will automatically generate 1-3 interactive prompts, the purpose of which is to use simple guiding sentences to encourage patients to ask questions or choose the next step (understand / ask the doctor / schedule a follow-up visit).

[0113] For example, prompt templates can be divided into: 1. Functional impact (when the LLM audit engine outputs a highlighted section and mentions "function / systole / endurance" in the impact section). For example, prompt the patient with, "This area may affect cardiac function. Would you like to know how this area affects walking, work, or exercise?" 2. The highlighted parts of the LLM audit engine output provide suggested follow-up examinations / clearly defined actions (recommended_action specifies the examination / timeframe). For example, "It is recommended to have a follow-up ultrasound examination within 4 weeks. Would you like us to help you understand the precautions for this area's follow-up examination or schedule an appointment?" 3. High risk with high confidence (risk_score ≥ 0.8, U ≥ 0.6). The system may prompt you to ask, "The system has marked this area as high-risk (potentially affecting recovery). Are you concerned that this area may affect your daily activities or require prompt medical review?" 4. Low confidence level but moderate risk (U < 0.5, risk_score ≥ 0.5). For example, "The model is not certain about this conclusion. Would you like to submit this result to a doctor for review or understand the significance of such a review?" In some embodiments, the method further includes: Based on the confidence heatmap of the three-dimensional heart model of the pathological body, a three-dimensional region growing algorithm or connected component analysis is used to perform connected segmentation on voxels with confidence scores less than a third preset threshold. Calculate the three-dimensional bounding box and centroid coordinates for each of the uncertain regions; Based on the three-dimensional bounding box and the centroid coordinates, a three-dimensional bounding box and a simplified surface mesh are generated for each of the uncertain regions in the three-dimensional cardiac model of the pathological body.

[0114] In this embodiment, the confidence heatmap of the 3D heart model of the pathological body is generated simultaneously during model generation, identifying the confidence level (i.e., confidence) of each voxel location in the model in the form of a 3D array. Using a 3D region growing algorithm or connected component analysis, voxels with confidence levels below a threshold (Example 0.7, configurable) are segmented for connectivity. The 3D bounding box and centroid coordinates of each uncertain region are calculated, generating the bounding box of the uncertain region and a simplified surface mesh (used to render the "red mesh"). This mesh will serve as the basis for the "red mesh" overlay layer of the subsequent 3D heart model of the pathological body, which can be visually displayed to the target patient.

[0115] In some embodiments, the method further includes: Based on the vital signs data of the target patient and the three-dimensional cardiac model of the pathological body, a potential subsequent pathological evolution scenario is generated, and the local blood flow distribution and mesh deformation field are output.

[0116] In this embodiment, based on a hybrid architecture of conditional variational autoencoder (CVAE) and 3D-GAN, a three-dimensional cardiac model of the pathological body and changes in key vital signs of the target patient are input to generate 3-5 potential subsequent pathological evolution scenarios, outputting local blood flow distribution and mesh deformation fields. This helps patients intuitively understand the possible changes in cardiac structure and functional trends in the short term after surgery. Specifically, the training data comes from historical complication cases within the hospital and publicly available arterial simulation databases.

[0117] The following exemplary embodiments demonstrate the application scenarios of this pathological 3D heart model in the visual question-and-answer assisted interaction process.

[0118] Scene 1: (1) Scene description: The system message reads: "A possible blood clot (or suspected embolus) has been detected in this area. If you have recently experienced dizziness, speech difficulties, or limb weakness, please seek immediate medical attention or call emergency services. Would you like to see an explanation of the blood clot location and associated risks?" Example of a patient's question: "Where is this blood clot located? Could it travel to my brain and cause a stroke?" (2) Matching the intent with the location of the lesion module in the model: NLP intent identification = [lesion location + concurrent consequences] Entity extraction: {Anatomical structure: left atrial appendage, Pathological features: thrombus} Match the location of the thrombus lesion module in the left atrial appendage of the 3D heart model Vpath.

[0119] (3) Decision and Triggering Layer: If risk_score ≥ 0.7 and the confidence level of the Vpath(A : thrombus) module is below the threshold → it is automatically marked and needs to be reviewed by a doctor; Otherwise, directly display the model and provide a visual simulation.

[0120] (4) 3D model display: Model Focus: Magnify and focus on the attachment point of the left atrial appendage; Dynamic pathway: Thrombus detachment → Arterial flow → Blockage of local blood vessels in the brain (showing local stroke area in the brain).

[0121] Side-by-side display: Risk comparison explanation (small thrombus vs. large thrombus).

[0122] (5) Decision output and its reflection in real life: Recommendation: Take anticoagulant medication regularly and avoid strenuous exercise; Note: If neurological symptoms (dizziness / speech impairment) occur, seek medical attention immediately. The system provides a "Life Impact Card" on the interface and marks it as high-risk.

[0123] Scene 2: (1) Scene description: The system message reads: "This calcification may be related to the surgical procedure or long-term medication. Would you like to know the specific impact of this on surgery, medication, or exercise?" Example of a patient's question: "The system shows that I have calcification in my blood vessels. Will this mean I will need more complicated surgery or long-term medication? Can I still run?" (2) Matching the intent with the location of the lesion module in the model: NLP model identifies intent as [understanding of lesions + life impact]. Solid extraction: {Anatomical structure: coronary artery, Pathological features: calcification, Spatial modification: focal / diffuse} Match 3D model → Output model Vpath internal lesion module (A: focal, B: diffuse) (3) Decision and Triggering Layer: If module A < 0.6 and confidence level B < 0.6, then the default display will show the focal type. If the confidence scores of A and B are greater than 0.6, and the patient's response explicitly includes a suggestion to consult a doctor, a review entry will be generated for the doctor in the background, along with an explanation of the triggering reason and priority.

[0124] (4) 3D model display: The default rendering renders a localized (low complexity) area and adds a diffuse overlay of a semi-transparent red grid to the suspected area B to indicate "uncertainty". After the user clicks, the comparison model module will be displayed: localized vs. diffuse, distinguished by color (blue = mild, red = diffuse).

[0125] (5) Decision output and its reflection in real life: Localized symptoms: Short-term recommendation → Resume light to moderate exercise in March; Diffuse: Requires complex equipment → longer hospital stay → delayed sports recovery; Rule engine: If risk_score ≥ 0.65 → force pop-up doctor's explanation.

[0126] Scene 3: (1) Scene description: "Will I need to take medication for life? Will I need to have frequent heart CT scans?" (2) Matching the intent with the location of the lesion module in the model: NLP intent identification = [follow-up + medication adherence] Physical sample extraction: {Anatomical structure: coronary artery / stent, Pathological features: postoperative recovery, Modification: long-term follow-up} Matching model: Vpath (postoperative recovery model, long-term follow-up curve).

[0127] (3) Decision and Triggering Layer: If risk_score < 0.5 and patient compliance is high → provide a standard follow-up pathway (annual imaging); If the patient is elderly or has multiple comorbidities, the risk score will increase, and more frequent follow-up is recommended.

[0128] (4) 3D model display: Demonstrates stent placement and blood flow stability; The interface provides an interactive "follow-up timeline" feature, which patients can click to view 3D status predictions for each stage.

[0129] (5) Decision output and its reflection in real life: Long-term medication adherence (antiplatelet / anticoagulant); Reminder: Do not stop taking medication arbitrarily; have regular check-ups. The system provides a "compliance score" and prompts doctors to pay attention to it.

[0130] This invention provides a cardiac surgery data processing method based on multimodal AI. It utilizes AI technology to generate a preoperative 3D heart model from a 2D model of the patient's heart, helping patients intuitively understand their heart structure and pathological conditions. By collecting intraoperative multimodal data throughout the surgery, the method converts this data into structured features suitable for scene matching, which are then fused with the preoperative 3D heart model to generate a pathological 3D heart model (equivalent to a postoperative heart model), helping patients understand postoperative changes in their heart. Finally, an LLM auditing engine automatically filters out key data points during the surgery, allowing patients to intuitively review core events and parameters during the procedure. The system displays changes in the heart and highlights key points to understand their significance and potential impact. This not only enhances patients' informed understanding of their surgery but also provides a clear and visual tool for doctor-patient communication, increasing transparency and trust. Based on a patient-specific cardiac model, patients can use Visual Question Answering (VQA) to gain a deeper understanding of possible postoperative changes in their condition and corresponding coping strategies, improving awareness and communication efficiency. This effectively provides patients with relevant nursing advice.

[0131] The cardiac surgery data processing device based on multimodal AI provided by the present invention will be described below. The cardiac surgery data processing device based on multimodal AI described below and the cardiac surgery data processing method based on multimodal AI described above can be referred to in correspondence with each other.

[0132] like Figure 2 As shown, the present invention also provides a cardiac surgery data processing device based on multimodal AI, comprising: The preoperative three-dimensional cardiac modeling module 210 is configured to generate a preoperative three-dimensional cardiac model of the target patient based on the MRI and / or CT images of the target patient by using a preset DMCVR morphological guided diffusion model. The pathological body three-dimensional heart modeling module 220 is configured to collect intraoperative multimodal data of the target patient, combine it with the preoperative three-dimensional heart model, and generate a pathological body three-dimensional heart model of the target patient through the preset DMCVR model. The postoperative report highlighting module 230 is configured to automatically filter important surgical data points in the intraoperative multimodal data through the LLM auditing engine, generate a postoperative report, and make visual annotations in the postoperative report; The postoperative VQA driver module 240 is configured to, when receiving a question from the target patient via voice or text, identify the intent of the question based on a medical visual language model, match it with the three-dimensional heart model of the pathological body, display the lesion type and lesion location information related to the three-dimensional heart model of the pathological body, and provide the answer generated by the medical visual language model or trigger the doctor's intervention mechanism according to a preset decision mechanism.

[0133] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a multimodal AI-based cardiac surgery data processing method, which includes: Based on the MRI and / or CT images of the target patient, a preoperative three-dimensional cardiac model of the target patient is generated by using a preset DMCVR morphological guided diffusion model. Intraoperative multimodal data of the target patient is collected, and combined with the preoperative three-dimensional cardiac model, a pathological three-dimensional cardiac model of the target patient is generated through the preset DMCVR model. The LLM audit engine automatically filters important surgical data points from the intraoperative multimodal data, generates a postoperative report, and makes visual annotations in the postoperative report; When the target patient asks a question via voice or text, the intent of the question is identified based on the medical visual language model and matched with the three-dimensional heart model of the pathological body. The lesion type and lesion location information related to the three-dimensional heart model of the pathological body are displayed. The answer generated by the medical visual language model is provided or the doctor intervention mechanism is triggered according to the preset decision mechanism.

[0134] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the cardiac surgery data processing method based on multimodal AI provided by the above methods, the method comprising: Based on the MRI and / or CT images of the target patient, a preoperative three-dimensional cardiac model of the target patient is generated by using a preset DMCVR morphological guided diffusion model. Intraoperative multimodal data of the target patient is collected, and combined with the preoperative three-dimensional cardiac model, a pathological three-dimensional cardiac model of the target patient is generated through the preset DMCVR model. The LLM audit engine automatically filters important surgical data points from the intraoperative multimodal data, generates a postoperative report, and makes visual annotations in the postoperative report; When the target patient asks a question via voice or text, the intent of the question is identified based on the medical visual language model and matched with the three-dimensional heart model of the pathological body. The lesion type and lesion location information related to the three-dimensional heart model of the pathological body are displayed. The answer generated by the medical visual language model is provided or the doctor intervention mechanism is triggered according to the preset decision mechanism.

[0136] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the multimodal AI-based cardiac surgery data processing method provided by the methods described above, the method comprising: Based on the MRI and / or CT images of the target patient, a preoperative three-dimensional cardiac model of the target patient is generated by using a preset DMCVR morphological guided diffusion model. Intraoperative multimodal data of the target patient is collected, and combined with the preoperative three-dimensional cardiac model, a pathological three-dimensional cardiac model of the target patient is generated through the preset DMCVR model. The LLM audit engine automatically filters important surgical data points from the intraoperative multimodal data, generates a postoperative report, and makes visual annotations in the postoperative report; When the target patient asks a question via voice or text, the intent of the question is identified based on the medical visual language model and matched with the three-dimensional heart model of the pathological body. The lesion type and lesion location information related to the three-dimensional heart model of the pathological body are displayed. The answer generated by the medical visual language model is provided or the doctor intervention mechanism is triggered according to the preset decision mechanism.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing cardiac surgery data based on multimodal AI, characterized in that, include: Based on the MRI and / or CT images of the target patient, a preoperative three-dimensional cardiac model of the target patient is generated by using a preset DMCVR morphological guided diffusion model. Intraoperative multimodal data of the target patient is collected, and combined with the preoperative three-dimensional cardiac model, a pathological three-dimensional cardiac model of the target patient is generated through the preset DMCVR model. The LLM audit engine automatically filters important surgical data points from the intraoperative multimodal data, generates a postoperative report, and makes visual annotations in the postoperative report, which is used to visually present the data to the target patient. When the target patient asks a question via voice or text, the intent of the question is identified based on the medical visual language model and matched with the three-dimensional heart model of the pathological body and the postoperative report. The system displays the lesion type and location information related to the three-dimensional heart model of the pathological body and the relevant report content in the postoperative report. Based on the preset decision mechanism, the system provides the answer generated by the medical visual language model or triggers the doctor's intervention mechanism.

2. The cardiac surgery data processing method based on multimodal AI according to claim 1, characterized in that, The process of generating a preoperative three-dimensional cardiac model of the target patient based on MRI and / or CT images using a pre-defined DMCVR morphological guided diffusion model includes: Acquire MRI and / or CT images of the target patient and perform preprocessing; The processed images were segmented using MRSegmentator multimodal segmentation to generate segmentation masks to label the main anatomical structures. The segmentation mask, the MRI image and / or the CT image are input into the preset DMCVR model to generate a preoperative three-dimensional cardiac model of the target patient. Based on the lesion information input by the doctor, the lesion area is marked in the preoperative three-dimensional cardiac model.

3. The cardiac surgery data processing method based on multimodal AI according to claim 2, characterized in that, The process of collecting intraoperative multimodal data from the target patient, combining it with the preoperative three-dimensional cardiac model, and generating a pathological three-dimensional cardiac model of the target patient using the preset DMCVR model includes: The preoperative 3D heart model and the segmentation mask are encoded into a dictionary structure that can be used by the generator and the retrieval machine. The dictionary structure includes: a global feature vector describing the overall morphology of the heart, a local feature vector describing the internal structure of the heart, a segmentation mask, and a skeleton diagram. Real-time acquisition of bone conduction voice data of doctors' communication, endoscopic or X-ray catheter image data, and vital sign data of the target patient throughout the entire surgical process; The bone conduction speech data is transcribed and intent recognition is performed to generate speech intent information, and the catheter trajectory features of the endoscope or X-ray catheter image data are extracted. The voice intent information, the catheter trajectory features, and the vital signs data are merged according to timestamps to generate intraoperative text data; The text instruction parser enables intent recognition and condition mapping of the intraoperative text data and clinical reports, mapping them into fixed-dimensional condition vectors. The dictionary structure and the fixed-dimensional condition vector are fused into a condition vector through a cross-modal multi-attention mechanism; The conditional vector is input into the preset DMCVR model to generate a three-dimensional cardiac model of the pathological body.

4. The cardiac surgery data processing method based on multimodal AI according to claim 1, characterized in that, The process involves automatically filtering important surgical data points from the intraoperative multimodal data using an LLM auditing engine, generating a postoperative report, and adding visual annotations to the postoperative report, including: The intraoperative multimodal data is input into the artificial intelligence speech model in the LLM audit engine to extract key surgical data related to the target patient's condition; The key surgical data is input into the ReportGenChain model in the LLM audit engine to generate a postoperative report, and the key surgical values ​​in the postoperative report are highlighted.

5. The cardiac surgery data processing method based on multimodal AI according to claim 1, characterized in that, When the system receives a question from the target patient via voice or text, it identifies the question's intent based on a medical visual language model and matches it with the 3D heart model of the pathological body and the postoperative report. It then displays information about the lesion type and location related to the 3D heart model of the pathological body and relevant report content from the postoperative report. Based on a preset decision-making mechanism, it provides the answer generated by the medical visual language model or triggers a doctor's intervention mechanism, including: The medical visual language model is invoked to identify the question intent based on the target patient's voice or text question and to extract key medical entities; The problem intent is matched with the key medical entity, the three-dimensional cardiac model of the pathological body, and the postoperative report to determine the lesion type, the lesion location information, and the relevant report content; The risk score and regional uncertainty of the lesion location information are determined, wherein the risk score is determined based on the voxel confidence, global uncertainty, multimodal evidence consistency score, and patient factor of the lesion location information, and the regional uncertainty is the proportion of voxel confidence of the target region; When the risk score is less than a first preset threshold, the medical visual language model generates and outputs the corresponding answer based on the question intent, the lesion type, the lesion location information, and the relevant report content; When the risk score is not less than the first preset threshold or the regional uncertainty is not less than the second preset threshold, the preset doctor intervention text information is embedded in the answer, and a review item and explanation of the triggering reason and priority are generated for the doctor in the background.

6. The cardiac surgery data processing method based on multimodal AI according to claim 5, characterized in that, After invoking the medical visual language model to identify the question intent based on the target patient's voice or text question and extract key medical entities, the process further includes: A natural language understanding model fine-tuned based on a medical corpus is used to perform entity recognition and relation extraction, and output structured query objects, which include: anatomical entities, pathological entities, and spatial modifiers. The spatial-semantic mapping table is used to retrieve the structured query object and output the spatial range data of the target area of ​​the three-dimensional heart model of the pathological body. The spatial-semantic mapping table records the anatomical units and their geometric representations of the three-dimensional heart model of the pathological body. A post-processing algorithm is used to extract and thicken the spatial extent data of the target area to improve visibility; The target area is displayed in a semi-transparent manner to ensure the visibility of the underlying structure; Multiple pathological interpretations of the target area are highlighted side-by-side, and a brief comparative description is given next to the view.

7. The cardiac surgery data processing method based on multimodal AI according to claim 5, characterized in that, Also includes: Interactive prompts are displayed on the target area, along with triggering conditions, button behaviors, and dialogue templates, to guide the target patient to ask questions or select the next step.

8. A cardiac surgery data processing device based on multimodal AI, characterized in that, include: The preoperative three-dimensional cardiac modeling module is configured to generate a preoperative three-dimensional cardiac model of the target patient based on the target patient's MRI and / or CT images using a preset DMCVR morphological guided diffusion model. The pathological body three-dimensional heart modeling module is configured to collect intraoperative multimodal data of the target patient, combine it with the preoperative three-dimensional heart model, and generate a pathological body three-dimensional heart model of the target patient through the preset DMCVR model. The postoperative report highlighting module is configured to automatically filter important surgical data points in the intraoperative multimodal data through the LLM auditing engine, generate a postoperative report, and make visual annotations in the postoperative report; The postoperative VQA driver module is configured to, when receiving a question from the target patient via voice or text, identify the intent of the question based on a medical visual language model, match it with the three-dimensional heart model of the pathological body, display the lesion type and lesion location information related to the three-dimensional heart model of the pathological body, and provide the answer generated by the medical visual language model or trigger the doctor's intervention mechanism according to a preset decision mechanism.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cardiac surgery data processing method based on multimodal AI as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cardiac surgery data processing method based on multimodal AI as described in any one of claims 1 to 7.

Images