Method, device and program product for processing 4D cardiovascular magnetic resonance image

Abstract

The invention relates to the field of intelligent medical treatment, in particular to a method, equipment and a program product for processing 4D cardiovascular magnetic resonance images. Comprising the following steps: acquiring a 4D cardiovascular magnetic resonance image sequence; carrying out Patch division on the image sequence, and marking a token for each sequence Patch to obtain a sequence embedding tensor; layering the sequence embedded tensor and then outputting the layered sequence embedded tensor to N parallel neural network models for feature learning to obtain N output features, wherein N is a natural number greater than 1; the layering is that a sequence embedding tensor is divided based on different dimensions to obtain a layering tensor, the layering tensor is transmitted to N parallel neural network models, and the different dimensions comprise any one or more of the following dimensions: time * height * width dimension, time * slice * height * width dimension and slice dimension. The application has good clinical value.

Description

Technical Field

[0001] This application relates to the field of intelligent healthcare, specifically to a method, apparatus, program product, and computer-readable storage medium for processing 4D cardiovascular magnetic resonance images. Background Technology

[0002] Cardiovascular magnetic resonance imaging (CMR) is the non-invasive gold standard for assessing cardiac structure, function, and tissue characteristics, playing a crucial role in the diagnosis of cardiovascular diseases (CVDs). However, CMR interpretation is extremely complex, requiring highly trained experts to spend significant time analyzing its multi-sequence 4D dynamic information. While foundational models (especially visual-language models, VLMs) are reshaping the field of medical image analysis, their impact remains limited to static 2D imaging and is difficult to extend to multi-sequence 4D CMR data. Progress is hampered by two key obstacles: the lack of an architecture capable of modeling the spatiotemporal dynamics of CMRs and the absence of large-scale paired image-report CMR datasets to train a general foundational model. More fundamentally, current VLMs do not reflect the actual CMR interpretation process used by clinicians—who follow a structured diagnostic path from anatomical localization → functional quantification → anomaly detection → diagnostic synthesis. This gap highlights the need for a unified foundational model that "fits clinical reasoning and integrates CMR spatiotemporal information." Summary of the Invention

[0003] To address the above problems, the present invention provides a method for processing 4D cardiovascular magnetic resonance images, specifically including: Acquire 4D cardiovascular magnetic resonance imaging sequences; The image sequence is divided into patches and each sequence patch is marked with a token to obtain a sequence embedding tensor. After the sequence embedding tensor is layered, it is fed into N parallel neural network models for feature learning to obtain N output features, where N is a natural number greater than 1; the layering is based on dividing the sequence embedding tensor into layered tensors according to different dimensions, and the layered tensors are fed into N parallel neural network models, wherein the different dimensions include any one or more of the following: time × height × width dimension, time × slice × height × width dimension, slice dimension.

[0004] Optionally, when the neural network model performs feature learning on the hierarchical tensor of time × slice × height × width dimensions, it first performs feature aggregation on the inter-layer and in-plane cardiac spatial structure of the slice × height × width dimensions to obtain a cardiac spatial aggregated representation, and then performs time aggregation on the time dimension to obtain the dynamic features of the cardiac spatial aggregated representation.

[0005] Optionally, when the neural network model performs feature learning on the hierarchical tensor of time × height × width dimensions, it captures spatial features in the plane while capturing long-term dependent and periodic motion features, thereby generating multi-scale temporal features.

[0006] Optionally, when the neural network model performs feature learning on the hierarchical tensor of the slice dimension, it performs scar delineation and subtle intensity change detection on the slice to obtain slice-level features; the neural network model includes any one or more of the following: Swin Transformer, CSwin Transformer, SG-Former, DiNAT, gSwin.

[0007] The purpose of this invention is to provide a multi-task processing method based on 4D cardiovascular magnetic resonance imaging, including: Acquire 4D cardiovascular magnetic resonance imaging sequences; The image sequence is input into the above-described method for processing 4D cardiovascular magnetic resonance images to obtain N output features; The N output features are fused to obtain the fused visual features; The fused visual features are input into a multi-task neural network model to obtain the results of the disease classification task and the anatomical localization task.

[0008] Optionally, the multi-task neural network model is a model constructed by combining a single model with a multi-task head. The fused features are input into the multi-task neural network model, and the fused visual features are learned by the single model to obtain the features of the hidden state. Then, the features of the hidden state are input into the disease classification task head and the anatomical localization task head for processing to obtain the disease classification task results and the anatomical localization task results. The single model includes any one or more of the following: MiniCPM-4B, mixed model, Qwen2.5 series model, and Phi-3.5-Mini model.

[0009] Optionally, the multi-task further includes a text generation task and / or a segmentation task. Visual features are fused and input into a multi-task neural network model to obtain disease classification task results, anatomical localization task results, text generation task results, and / or segmentation task results. Radiological report text data corresponding to 4D cardiovascular magnetic resonance imaging sequences are acquired. The obtained text data is fused with the fused visual features to obtain a second fused feature. The second fused feature is input into a multi-task neural network model and fused feature learning is performed through a single model to obtain features of a second hidden state. The features of the second hidden state are then input into the disease classification task head and anatomical localization task head, the text generation task head, and / or segmentation task head, respectively, for processing to obtain disease classification task results, anatomical localization task results, text generation task results, and / or segmentation task results.

[0010] The purpose of this invention is to provide a computer program product that includes a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the above-described method for processing 4D cardiovascular magnetic resonance images, or to implement the above-described multi-task processing method based on 4D cardiovascular magnetic resonance images.

[0011] The purpose of this invention is to provide a computer device comprising a memory, a processor, and a computer program or instructions stored in the memory, wherein the computer program or instructions are executed by the processor to implement the above-described method for processing 4D cardiovascular magnetic resonance images, or to implement the above-described multi-task processing method based on 4D cardiovascular magnetic resonance images.

[0012] The purpose of this invention is to provide a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions are executed by a processor to implement the above-described method for processing 4D cardiovascular magnetic resonance images, or to implement the above-described multi-task processing method based on 4D cardiovascular magnetic resonance images.

[0013] Advantages of this invention: 1. To address the lack of architectures capable of modeling the spatiotemporal dynamics of CMR, this invention proposes a method for spatiotemporal data processing of 4D CMR data. By performing 2D, 3D, and 4D decomposition on the 4D CMR data (slice dimension, time × height × width dimension, time × slice × height × width dimension), the spatiotemporal features of the 4D CMR data are extracted from the tensors of different dimensions obtained from the layering, thus providing a processing scheme for the spatiotemporal features of 4D CMR data.

[0014] 2. To address the issue that current VLMs do not reflect the actual CMR interpretation process of clinicians, this invention achieves multi-task processing results simultaneously through a single model combined with multiple task heads. The processing results of these multiple tasks are consistent with the actual CMR interpretation process of clinicians, including the entire process from anatomical localization, functional and structural assessment to abnormality detection and diagnostic report generation. End-to-end simulation of the clinical reasoning process: CMR-VLM, as an integrated cognitive assistant, strictly follows the clinical CMR interpretation path from anatomical localization to diagnostic report, without the need for fragmented AI tools, and realistically restores the reasoning process of clinicians, thus having great clinical application value.

[0015] 3. Flexible Instruction-Driven and Robust Inference: CMR-VLM supports flexible instruction-driven interpretation beyond fixed task frameworks; it can still perform robust inference and accurately complete screening even in modality-deficient situations (such as using only movie sequence input); and when data is available, it can seamlessly integrate late gadolinium enhancement (LGE) to achieve tissue-level diagnosis. This invention, by introducing this universal and efficient basic model that aligns with clinical logic and follows the CMR inference path, provides a new paradigm for the widespread adoption of high-quality CMR interpretation. It is expected to: reduce diagnostic delays, alleviate expert shortages, and enable rapid contrast-free screening in resource-constrained or emergency scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic flowchart of a method for processing 4D cardiovascular magnetic resonance images provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a system for processing 4D cardiovascular magnetic resonance images provided in an embodiment of the present invention; Figure 3 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a multi-task processing method based on 4D cardiovascular magnetic resonance imaging provided in an embodiment of the present invention; Figure 5 The heterogeneous data processing flow and spatiotemporal vision-language basic model architecture provided in the embodiments of the present invention; Figure 6 An overview of the CMR-VLM framework for multi-sequence 4D CMR interpretation provided in embodiments of the present invention; Figure 7 An evaluation of the effectiveness and robustness of CMR-VLM in the classification of heart disease provided in this embodiment of the invention; Figure 8 Performance evaluation of CMR-VLM in flexible command-driven anatomical localization tasks provided in embodiments of the present invention; Figure 9 A comprehensive assessment of the advanced reasoning capabilities of CMR-VLM provided for embodiments of the present invention (covering open-ended visual question answering and report generation). Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0019] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as S101, S102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0020] Figure 1 A schematic diagram of a method for processing 4D cardiovascular magnetic resonance images provided in this embodiment of the invention specifically includes: S1: Acquire 4D cardiovascular magnetic resonance imaging sequences; In one embodiment, the 4D cardiovascular magnetic resonance imaging sequence includes LGE SAX, 4CH cine, and SAX cine.

[0021] In one specific embodiment, the internal data—a large-scale paired image-text dataset—is required for developing a visual-language-based model of cardiovascular magnetic resonance (CMR). To this end, this invention integrates multicenter CMR data from four independent medical centers (Centers A through D) from 2014 to 2024, ultimately forming a dataset containing 4058 paired image-report cases. This dataset consists of 3.1 million 2D images derived from 109,000 3D volumetric reconstructions, providing rich material for spatiotemporal analysis of the heart. Specifically, centers A through D (contributing 2067, 998, and 425 cases respectively) constitute the internal cohort, while 568 cases from center D serve as an independent external validation set to evaluate the model's out-of-distribution generalization ability. All studies were approved by the ethics committees of the participating hospitals, and written informed consent from patients was obtained where necessary.

[0022] Public datasets: To enhance data diversity and model robustness, this invention additionally includes four public datasets: Automated Cardiac Diagnosis Challenge (ACDC, 150 cases); Multicenter, multi-vendor, multi-disease cardiac segmentation (M&Ms, 720 cases) and its extension (M&Ms2, 2000 cases); National Heart Structured Data (NCSD, 1000 cases).

[0023] These datasets are primarily used for segmentation tasks and functional evaluation, but do not include paired radiological reports.

[0024] Heterogeneous data processing workflow: Each case was paired with the original radiology report to ensure a strong correlation between visual and verbal signals. To mitigate the linguistic heterogeneity of multicenter reports (such as differences in expression style and terminology), this invention designed a standardized processing procedure (e.g., Figure 5 As shown in a): First, the search agent (suggested to be a senior cardiac radiologist) retrieves matching report templates and terminology mappings; then, a generalized large language model generates a structured draft; finally, manual post-processing calibrates medical accuracy and completeness, ultimately producing a standardized report—the system covers key findings in five core areas: cardiac structure, function, late gadolinium enhancement (LGE) patterns, other abnormalities, and final imaging impression. Specifically: Prior guidance for unification: Utilizing established clinical guidelines (ESC / RSNA) and a standardized medical terminology database (RadLex) as constraints, large language models (LLMs) are guided to transform unstructured reports into five unified structured formats: (1) Heart structure (such as chamber size and wall thickness); (2) Cardiac motion and function (e.g., ejection fraction, abnormal wall motion); (3) Late gadolinium enhancement (LGE) patterns (e.g., location / range of fibrosis); (4) Other relevant findings (such as pericardial and valvular lesions); (5) Final image impression.

[0025] Human-machine loop calibration: The accuracy and integrity of medical data are verified through manual post-processing steps, ultimately generating 4,058 standardized reports.

[0026] VQA Pair Generation: 46,000 Visual Question Answering (VQA) pairs were extracted from standardized reports, covering closed-ended questions (such as "Is the left ventricular ejection fraction decreased?") and open-ended queries (such as "Describe the distribution pattern of LGE"), used to train the model's interactive reasoning capabilities. Process: Rule templates + NER (Named Entity Recognition) + manual review generated a total of 46,000 VQA pairs.

[0027] Image acquisition and preprocessing: Scanning equipment and sequences: CMR scans were performed using 1.5T (Siemens Aera) and 3.0T (Philips Achieva / Elition) scanners. Cine sequences were performed using balanced steady-state free precession (bSSFP) to cover both short-axis (SAX) and long-axis (4CH) views, providing a complete record of the cardiac cycle; LGE sequences were acquired 10–15 minutes after contrast agent injection, capturing SAX view images.

[0028] Anonymization and preprocessing: All patient identifiers and protected health information are removed; before model input, images are resampled to a uniform resolution of 224×224 pixels and intensity is normalized to eliminate device differences.

[0029] S2: Divide the image sequence into patches and mark each sequence patch with a token to obtain a sequence embedding tensor; In one specific embodiment, unified patch embedding: All input CMR sequences are first divided into fixed-size spatiotemporal patches (e.g., 4×4×4), and then mapped to a high-dimensional embedding space through linear projection. The embedding output formats for different sequences are as follows: 4CH movie sequence: Tensor shape is (T×C×H×W) (T: time frame; C: channel dimension; H / W: spatial height / width); SAX movie sequence: (T×C×D×H×W) (D: number of slices); LGE SAX sequence: (C×D×H×W).

[0030] Each sequence is therefore represented as a set of embedded Patch tokens.

[0031] S3: After the sequence embedding tensor is layered, it is input into N parallel neural network models for feature learning to obtain N output features, where N is a natural number greater than 1; the layering is based on dividing the sequence embedding tensor into layered tensors according to different dimensions, and the layered tensors are input into N parallel neural network models, wherein the different dimensions include any one or more of the following: time × height × width dimension, time × slice × height × width dimension, slice dimension.

[0032] In one embodiment, when the neural network model performs feature learning on the hierarchical tensor of time × slice × height × width dimensions, it first performs feature aggregation on the inter-layer and in-plane cardiac spatial structure of the slice × height × width dimensions to obtain a cardiac spatial aggregated representation, and then performs time aggregation on the time dimension to obtain the dynamic features of the cardiac spatial aggregated representation.

[0033] In one embodiment, the dynamic features of the cardiac spatial aggregation representation include two aggregation representations: a temporal aggregation representation of the anatomical structure at each phase, and a cardiac spatial aggregation representation of the cardiac anatomy including interlayer and planar structures. The time dimension is a cardiac cycle. By performing temporal aggregation on the time dimension, we obtain a cardiac spatial aggregation representation of the dynamic features within a cardiac cycle.

[0034] In one embodiment, when the neural network model performs feature learning on a hierarchical tensor with time × height × width dimensions, it captures spatial features in the plane while capturing long-term dependent and periodic motion features, thereby generating multi-scale temporal features.

[0035] In one embodiment, multi-scale time features include short-term change features and global change features. Short-term change features refer to continuous time changes and planar structural spatial feature changes over a short period or one cycle. Global change features refer to continuous time changes and planar structural spatial feature changes over a long period.

[0036] In one embodiment, when the neural network model performs feature learning on the hierarchical tensor of the slice dimension, it obtains slice-level features by delineating scars and detecting subtle intensity changes in the slice.

[0037] In one embodiment, the layering is based on dividing the sequence embedding tensor into layered tensors according to different dimensions to obtain layered tensors, and the layered tensors are input into N parallel neural network models, wherein the different dimensions include: time × height × width dimension, time × slice × height × width dimension, and slice dimension.

[0038] In one embodiment, the hierarchical division of the sequence embedding tensor based on different dimensions yields N hierarchical tensors, and the N hierarchical tensors are input into N parallel neural network models for feature learning to obtain N output features.

[0039] In one embodiment, when the parallel neural network model includes extracting slice dimensions to obtain slice-level features, the slice-level features are projected onto the shared hidden dimension.

[0040] In one embodiment, the neural network model includes any one or more of the following: Swing Transformer, CSwin Transformer, SG-Former, DiNAT, and gSwin; Optionally, the neural network model is a Swing Transformer; Optionally, the neural network model includes a 2D Swin Transformer, a 3D Swin Transformer, and a decomposed 4D Swin Transformer; the decomposed 4D Swin Transformer is a Swin Transformer with processing time × slice × height × width dimensions.

[0041] In one embodiment, the 4CH cine sequence performs 3D spatiotemporal modeling to assess overall cardiac function, valvular regurgitation, and atrial size, requiring modeling of long-term periodic motion (e.g., end-diastole → end-systole); the SAX cine sequence models by decomposing 4D (combining interlaminar anatomy (D-dimensional) and temporal dynamics (T-dimensional)) to accurately quantify ventricular volume, ejection fraction, and wall motion; the LGE SAX sequence does not require temporal modeling but requires high spatial resolution to identify minute scars, therefore using 2D modeling is more stable and avoids interpolation artifacts, detecting myocardial fibrosis / infarction (static enhancement).

[0042] The advantages of the layered visual coding process of this invention are: Accuracy improvement: Dice improved by 1–3.8% (vs. Unified 3D processing); Robustness enhancement: When the number of LGE layers is inconsistent, 2D processing avoids the failure of 3D convolution; Computationally efficient: Decomposing 4D saves >60% of attention compared to full 4D; Clinical alignment: Matching the radiologist's reading process of "look at the structure first (LGE) → look at the function (cine)".

[0043] In one specific embodiment, a hierarchical spatiotemporal visual encoder: CMR sequences differ in dimensionality, coverage, and spatiotemporal dynamics. Therefore, this invention designs a hierarchical spatiotemporal encoder based on SwinTransformer for 2D / 3D / decomposed 4D inputs. All encoders follow the common principle of "hierarchical patch embedding → shifted window local self-attention → patch merging progressive token aggregation" to ensure efficient modeling of local anatomical details and long-term contextual dependencies across space, time, and slices. The specific sequence encoder design is as follows: SAX Movie Sequence (4D Spacetime): A factorized encoder is employed—first, a 3D Swin Transformer is used to process each time frame, modeling interlayer and in-plane structures (slices × height × width); then, a temporal attention module is used to integrate cardiac cycle information. The encoder outputs two representations: ① a temporal aggregation representation (summarizing the anatomical structure of each phase); ② a spatial aggregation representation (capturing temporal dynamics), supporting the joint encoding of downstream structural and functional information.

[0044] 4CH Cinema Sequence (3D Time): The 3D Swin Transformer is used to process the "time × height × width" dimension, capturing long-term temporal dependencies and periodic motion while preserving intra-frame spatial details. Layered patches are merged to generate multi-scale temporal features, taking into account both short-term variations and global rhythms.

[0045] LGE SAX sequence (3D space): A 2D Swin Transformer is applied independently to each slice—local window self-attention supports accurate scar delineation and detection of subtle intensity changes; independent slice processing avoids failures caused by missing or irregular spacing. Slice-level features are projected onto a shared hidden dimension for compatibility with other sequences.

[0046] Visual token integration and LLM input: The encoder outputs a Patch token sequence for each sequence, which is then projected onto the shared embedding dimension and normalized using RMSnorm before being concatenated into a unified token sequence. Special marker tokens (such as...) are added to the sequence. vision , vsax_sep This unified token sequence is used to indicate modal and sequence boundaries. As visual input to the LLM backbone, it enables attention across heterogeneous spatiotemporal contexts and joint reasoning of anatomical, functional, and pathological information.

[0047] In one specific embodiment, the model structure of the hierarchical visual encoder is as follows: Figure 5 As shown in b, multi-sequence 4D CMR data (4CH film, SAX film, LGE SAX) are deeply fused with text input using a dedicated visual encoder: Each sequence is processed into tokenized features using a custom hierarchical Swing Transformer based on its own spatiotemporal characteristics: 4CH Cinema (3D Spacetime) → 3D Swin Transformer; SAX Cinema (Deconstructing 4D Spacetime) → Deconstructing 4D Swin Transformer; LGE SAX (3D space) → 2D Swin Transformer.

[0048] These visual tokens, after being fused with text embeddings, are input into a pre-trained language model (MiniCPM-4B). Finally, through a task-specific header, the system generates outputs including reference-guided segmentation, visual question answering (VQA), report generation, and disease classification.

[0049] In one specific embodiment, the structural differences, processing flow, and data interaction between 2D / 3D Swin Transformer and decomposed 4D Swin Transformer are as follows: LGE SAX (3D space = multi-layer 2D image) uses a 2D Swin Transformer, with each slice processed independently (no time dimension): Input: A single-layer image with shape (C, H, W).

[0050] Processing: 2D window attention (e.g., window_size=(7,7)), modeling local-global context only in spatial dimensions.

[0051] Advantages: LGE typically has fewer layers and no dynamics, making it more robust to 2D processing and avoiding 3D modeling failures due to inconsistent layer counts.

[0052] 4CH Cine (3D Spatiotemporal = Time Frame × H × W) uses 3D Swin Transformer: Input: (T, H, W), where T is the number of time frames (usually 25–50).

[0053] Processing: 3D window attention (e.g., window_size=(2,7,7)), while modeling inter-frame motion (temporal) and heart chamber morphology (spatial).

[0054] Output: Spatiotemporal fusion features, preserving long-period contraction / diastole rhythms.

[0055] SAX cine (4D spacetime = T × D × H × W) adopts a decompositional 4D architecture (Spatial–Temporal Decomposition): Phase 1 (Spatial Modeling): For each time frame, use 3D Swin Transformer to process (D, H, W) and model the interlayer structure of the myocardium (e.g., from endocardium to epicardium).

[0056] Phase 2 (Temporal Modeling): After spatial encoding, the Temporal Attention module is used to aggregate dynamic information (such as changes in left ventricular volume) along the T dimension.

[0057] Advantages: Avoids the explosion of direct 4D attention computation (O(T·D·H·W)²) while preserving spatiotemporal coupling semantics.

[0058] This invention is the first to propose a “space-time decoupled 4D CMR modeling architecture” that takes into account both anatomical structure (SAX layers) and functional dynamics (ventricular systolic cycle).

[0059] This invention provides a multi-task processing method based on 4D cardiovascular magnetic resonance imaging, including: Acquire 4D cardiovascular magnetic resonance imaging sequences; The image sequence is input into the above-described method for processing 4D cardiovascular magnetic resonance images to obtain N output features; The N output features are fused to obtain the fused visual features; The fused visual features are input into a multi-task neural network model to obtain the results of the disease classification task and the anatomical localization task.

[0060] In one embodiment, the multi-task neural network model is a model constructed by combining a single model with a multi-task head. The fused visual features are input into the multi-task neural network model, and the fused visual features are learned by the single model to obtain the features of the hidden state. The features of the hidden state are then input into the disease classification task head and the anatomical localization task head for processing to obtain the disease classification task results and the anatomical localization task results. The single model includes any one or more of the following: MiniCPM-4B, mixed model, Qwen2.5 series model, and Phi-3.5-Mini model.

[0061] In one embodiment, the multi-task further includes a text generation task and / or a segmentation task. Visual features are fused and input into a multi-task neural network model to obtain disease classification task results, anatomical localization task results, text generation task results, and / or segmentation task results. Radiological report text data corresponding to 4D cardiovascular magnetic resonance imaging sequences are acquired. The obtained text data is fused with fused visual features to obtain a second fused feature. The second fused feature is input into a multi-task neural network model and fused feature learning is performed through a single model to obtain features of a second hidden state. The features of the second hidden state are then input into the disease classification task head and anatomical localization task head, the text generation task head, and / or segmentation task head, respectively, for processing to obtain disease classification task results, anatomical localization task results, text generation task results, and / or segmentation task results.

[0062] Optionally, the second hidden state features are routed to the corresponding task heads for different task processing according to the token; the second hidden state features are routed to the text generation task head to fuse visual features and text features (features of the LLM single model, text prompt features) according to the token; the second hidden state features are routed to the disease classification task head to fuse dynamic features, slice-level features, and global change features according to the token; the second hidden state features are routed to the segmentation task head to fuse visual features and text features (features of the LLM single model, text conditional embedding features) according to the token; and the second hidden state features are routed to the anatomical localization task head to short-term change features according to the token.

[0063] The fusion of visual features includes: dynamic features, slice-level features, global change features, and short-term change features.

[0064] In one embodiment, radiological report text data corresponding to a 4D cardiovascular magnetic resonance imaging sequence is acquired. The obtained text data is fused with fusion features to obtain a second fusion feature. The fusion feature and the second fusion feature are input into a multi-task neural network model to obtain a hidden state and a second hidden state. The hidden state is input into a disease classification task head and an anatomical localization task head to obtain disease classification task results and anatomical localization task results. The second hidden state is input into a text generation task head and / or a segmentation task head to obtain text generation task results and / or segmentation task results.

[0065] In one embodiment, the classification task head includes two layers of MLP and softmax, and the hidden state or the second hidden state obtains the disease category probability through the two layers of MLP and softmax; the anatomical localization task head is composed of a lightweight MLP, and the hidden state or the second hidden state obtains the sequence / layer / phase classification recognition result through the lightweight MLP; the text generation task head includes a linear projection layer and an LM head, and the hidden state or the second hidden state obtains autoregressive text through linear projection + LM Head; the segmentation task head is composed of multiple alternating decoders and upsampling layers, and the hidden state or the second hidden state obtains pixel-level mask segmentation result through multiple alternating decoders and upsampling layers.

[0066] In one specific embodiment, the LLM backbone and multi-task header: This invention selects MiniCPM-4B as the language backbone because it balances accuracy and efficiency in large-scale clinical scenarios. MiniCPM-4B adopts a streaming Transformer design, supports inter-layer parameter sharing, long sequence sparse attention, and mixed-precision inference, reducing memory usage and latency while maintaining strong generation / understanding capabilities.

[0067] To support multi-task operation within a single model, this invention appends a multi-task header to the shared hidden state and routes the output based on the supervision signal or instruction token: Text generation head: Performs linear projection on the LLM hidden state to generate vocabulary logits, which are used for sequence recognition, phase recognition, VQA and structured report generation. It is trained using autoregressive cross-entropy loss.

[0068] Classification Head: A two-layer MLP maps pooled / averaged token features to class logits for binary / multi-class diagnostics. For multi-class scenarios, class cross-entropy is used; for class imbalance, focus loss or BCE loss can be selected.

[0069] Segmentation Head: A multimodal decoder aggregates hierarchical visual features and intermediate LLM layer features, employing a U-Net-style encoder-decoder (with skip connections) to generate pixel-level masks for the left ventricle (LV), right ventricle (RV), and myocardium (MYO). The optimization objective is Dice loss + BCE loss, with optional addition of focus loss or IoU term.

[0070] Anatomical localization head: Lightweight MLP for localization and classification of sequences / slices / phases.

[0071] In one specific embodiment, the core architecture of CMR-VLM consists of three parts: a sequence-specific visual encoder, a cross-sequence fusion module, and a large language model (LLM) backbone with a task-specific head. The sequence-specific visual encoder is the 4D CMR sequence processing step in the method described above for processing 4D cardiovascular magnetic resonance images, outputting different visual features.

[0072] The task header settings are shown in the table below:

[0073] Key Design: 1. The segmentation head incorporates text-based conditional guidance (instruction-aware) to achieve "natural language-driven segmentation"; 2. The classification head integrates complementary features of multiple sequences (such as LGE scars + SAX ejection fraction); 3. All heads share the underlying visual encoder, avoiding redundant calculations.

[0074] In one specific embodiment, text generation includes VQA and structured reports, image summaries, and VQA including closed-ended visual question answering VQA and open-ended VQA across clinical question types, as well as abnormality-specific VQA.

[0075] In one specific implementation, all models were implemented using PyTorch 2.5.1 + CUDA 11.8 and trained on a server equipped with four NVIDIA A100 (40GB) GPUs. The dataset was divided into training and test sets in an 8:2 ratio, and data augmentation techniques such as random rotation, scaling, flipping, and intensity normalization were used to improve robustness.

[0076] Pre-trained weights: The LLM backbone and language encoder are initialized with pre-trained weights from MiniCPM-4B (which was pre-trained on approximately 1.2 trillion tokens of general English-Chinese mixed text).

[0077] Mixed precision training: Uses bfloat16 format for storage and computation, reducing memory and computational overhead.

[0078] Optimizer and Hyperparameters: Using the Adafactor optimizer with an initial learning rate of 1e 5. Weight decay of 0.005, batch size of 1 (gradient accumulation in 16 steps). The learning rate adopts a cosine annealing strategy (warmup ratio of 0.03), and the total number of training epochs is 30.

[0079] Fine-tuning strategy: Freeze only the text encoder, and update all parameters of the LLM backbone and the visual encoder.

[0080] In one specific embodiment, CMR-VLM is evaluated on four types of tasks (a total of 15 sub-tasks) covering the core clinical CMR interpretation workflow: Task classification and definition: (i) Anatomical localization: building basic perception and localization capabilities; Sequence identification (4CH / SAX / LGE); Phase identification (end-diastolic / end-systolic / mid-cycle); Layered localization (basal segment / intermediate ventricular segment / apical segment); Reference-guided segmentation: Segmenting the left ventricle (LV), right ventricle (RV), and myocardium (MYO) using natural language commands.

[0081] (ii) Abnormal Visual Question Answering (VQA): Supports interactive clinical reasoning; Closed-ended VQA (multiple choice questions); Open VQA (Free Text); Abnormal VQA (such as wall motion, LGE pattern).

[0082] (iii) Diagnostic classification: Supports comprehensive disease assessment; Binary anomaly classification; Common classifications of heart disease (5 categories); Multiple disease classifications (16 categories).

[0083] (iv) Text generation: generating clinically meaningful narratives; Image summary (a concise description of a single sequence); Structured radiology report generation (integrating measurements with clinical findings).

[0084] In one specific embodiment, all tasks employ standardized metrics that conform to clinical and technical practices to ensure interpretable results: Classification tasks: accuracy, precision, recall, F1 score; Segmentation task: Dice similarity coefficients of LV / RV / MYO (consistent with medical imaging benchmarks); Text generation task: Classic NLP metrics (BLEU, METEOR, ROUGE) + modern LLM evaluators (BERTScore, Qwen, DeepSeek, GPT-4).

[0085] Baseline models (M3D, BiomedGPT, RadFM) are evaluated under the same data partitioning and metrics to ensure fair comparison.

[0086] In one specific embodiment, the multi-sequence 4D CMR interpretation CMR-VLM framework proposed in this invention is as follows: Figure 6 As shown, Figure 6 The 'a' indicates multi-sequence 4D CMR data processing: the model processes four-chamber (4CH) cinema sequences, short-axis (SAX) cinema sequences, late gadolinium enhancement (LGE) sequences, and text information, simultaneously capturing dynamic cardiac motion and tissue features.

[0087] Figure 6 The 'b' indicates the dataset composition and disease stratification: Our large-scale dataset contains 4,058 paired image-report cases from 4 medical centers from 2014 to 2024, encompassing 101,000 3D volumes and 46,000 VQA pairs. The pie chart shows the distribution of 16 cardiac diseases, with dilated cardiomyopathy (29%) and hypertrophic cardiomyopathy (15%) being the predominant types.

[0088] Figure 6 The 'c' represents the task-oriented capability of CMR-VLM: the model unifies spatiotemporal understanding (sequence, phase, and layer identification) with four core clinical CMR interpretation pathways out of 15 tasks into a single instruction-driven architecture. (i) Anatomical localization (reference-guided segmentation of the left ventricle (LV), right ventricle (RV), and myocardium (MYO)); (ii) Visual Question Answering (VQA) (closed and open queries); (iii) Diagnosis (abnormality detection, classification of common cardiovascular diseases, classification of multiple cardiovascular diseases, inference of missing modalities); (iv) Report generation (image summary and structured report generation).

[0089] In one specific embodiment, achieving accurate cardiac classification from CMR is extremely challenging due to the diverse phenotypes of cardiac diseases and the need for multi-sequence imaging. This invention evaluates CMR-VLM across a range of tasks, including binary anomaly detection, common disease classification, and comprehensive 16-category differential diagnosis.

[0090] Binary anomaly detection: CMR-VLM achieved an AUC of 0.933 on the internal test set and 0.881 on the external test set. Figure 7 (as shown in a). Common Disease Classification: The model's AUC performance for five common heart diseases is as follows: dilated cardiomyopathy 0.94 (95% confidence interval: 0.92–0.96), hypertensive heart disease 0.89 (0.82–0.96), hypertrophic cardiomyopathy 0.84 (0.79–0.89), myocardial infarction 0.82 (0.76–0.88), myocarditis 0.82 (0.75–0.89). Figure 7 (as shown in b). 16 Differential Diagnoses: CMR-VLM maintains high accuracy for common conditions such as dilated cardiomyopathy, while also possessing significant distinguishing ability for rare conditions such as cardiac amyloidosis and stroke-cardiac syndrome. Figure 7 (as shown in c).

[0091] We compared CMR-VLM with existing methods (M3D, BiomedCLIP, 3DTransformer)—all methods were fine-tuned on the same training set and evaluated on the same internal and external test sets. CMR-VLM outperformed in all tasks: in the anomaly detection task, the AUC on the internal / external test sets was improved by 2.7–7.9% / 2.8–16.4%, respectively; in the common disease classification task, the internal / external AUC reached 0.937 / 0.860, exceeding competitors by 1.6–18.4%; in the 16-class diagnostic task, the internal / external AUC reached 0.931 / 0.819, exceeding existing methods by 1.9–9.3%. Figure 7 (as shown in g).

[0092] These results demonstrate that CMR-VLM can accurately capture complex multimodal cardiac phenotypes across a range of diagnostic tasks. Ablation studies highlight the crucial role of the spatiotemporal visual encoder: removal of the visual encoder leads to a 6.2–10.8% decrease in AUC, while ablation of the language model results in only a small decrease of 2.1–2.8%. Figure 7 (as shown in h). This indicates that the visual encoder is the main driver of diagnostic performance, while the language model primarily supports the generation of structured output.

[0093] In one specific embodiment, in clinical practice, CMR imaging protocols are often incomplete (similar to modality loss) due to contraindications (such as the need for contrast agent injection in late gadolinium enhancement (LGE)). To evaluate the suitability of CMR-VLM, this invention conducted a modal ablation study, analyzing how diagnostic performance changes with available sequences.

[0094] For binary anomaly detection, the AUC (0.927 / 0.877) of the standard cinema sequence (short axis (SAX) + four-chamber (4CH)) is almost identical to the performance of the full modality (SAX + 4CH + LGE) (0.937 / 0.888). Figure 7(As shown in d). For comprehensive differential diagnosis, tissue characterization is crucial: the mean AUC for multiple tasks across the entire modality setting reached 0.857, superior to all double sequence variants. LGE showed the greatest disease-specific gain in myocarditis (+0.08 AUC) and myocardial infarction (+0.07 AUC), highlighting its importance in detecting inflammatory and ischemic lesion patterns. Figure 7 (e–f).

[0095] This invention further evaluates the model's robustness to cross-center distribution shifts. Despite differences in imaging protocols, scanner manufacturers, and patient demographics across different centers, the performance degradation from the internal to the external test set was minimal—from 0.042 for binary anomaly detection to 0.077 for common disease classification. Figure 7 (As shown in g). This demonstrates that CMR-VLM maintains diagnostic accuracy in real-world multi-institutional settings. In summary, these results indicate that CMR-VLM not only boasts high accuracy but also maintains stable performance even with incomplete image data or data from different clinical centers, supporting its reliable application in various diagnostic scenarios.

[0096] The effectiveness and robustness assessment of CMR-VLM in cardiac classification, such as Figure 7 As shown, a represents the AUC comparison of abnormal diagnoses between internal and external test sets; b represents the AUC comparison of common heart disease classifications; c represents the confusion matrix of multiple heart disease diagnoses (DCM: dilated cardiomyopathy; HCM: hypertrophic cardiomyopathy; M: myocarditis; MI: myocardial infarction; HHD: hypertensive heart disease; N: normal; CA: cardiac amyloidosis; CSD: conduction system disease; NCM: non-compacting cardiomyopathy; TCM: stress cardiomyopathy; ACM: arrhythmogenic cardiomyopathy; P: pericarditis; PH: pulmonary hypertension; VHD: Valvular heart disease; CHD: congenital heart disease; TRHD: tumor-associated heart disease; PCM: perinatal cardiomyopathy; SDRHD: systemic disease-associated heart disease; S: stroke); d represents the comparison of AUC for abnormal diagnosis under different modalities; e represents the comparison of AUC for common heart disease classification under different modalities; f represents the AUC for multi-class diagnosis under different modalities; g represents the comparison of AUC between CMR-VLM and the state-of-the-art (SOTA) method; h represents the ablation study with removal of the visual encoder (VisEncoder) and LLM components.

[0097] In one specific embodiment, accurate anatomical understanding of CMR requires both basic spatiotemporal awareness and flexible interaction with user-specified queries. CMR-VLM achieves this through two core capabilities: (1) spatiotemporal awareness of 4D CMR data: recognizing sequences (four-chamber heart (4CH), short axis (SAX), late gadolinium enhancement (LGE)), cardiac cycle phase (systole, diastole), and slice location (basal segment, mid-segment, apical segment); (2) instruction-driven visual localization: segmenting specified anatomical structures through natural language instructions (such as "segment the left ventricle") without modifying the network architecture for a specific task.

[0098] First, we assess its instruction-driven spatiotemporal awareness: in terms of sequence classification accuracy, 4CH is 95.3%, SAX is 96.5%, and LGE is 90.9%—the main confusion (8.8%) occurs between SAX and LGE, as both are short-axis orientations ( Figure 8 (As shown in a). Phase recognition accuracy: 90.2% during systole and 88.7% during diastole. Figure 8 As shown in b). Layered localization demonstrates precise spatial perception: apical segment 82.7%, intermediate ventricular segment 74.2%, basal segment 84.6%, with errors concentrated near the anatomical transition zone ( Figure 8 As shown in c), this reflects the variability among clinical observers.

[0099] Based on this, CMR-VLM has achieved instruction-driven anatomical segmentation on multiple datasets (ACDC, M&Ms, M&Ms2). Unlike traditional models (such as UNet or nnUNet), which require pre-setting output channels for each anatomical structure and cannot handle new user queries without modifying the network architecture, CMR-VLM requires no task-specific architectural modifications and can directly perform segmentation using natural language instructions. Importantly, all state-of-the-art (SOTA) method comparisons are conducted on the same dataset using a consistent training and fine-tuning protocol to ensure fairness in the evaluation.

[0100] CMR-VLM achieves Dice scores on ACDC that approach specialist levels: left ventricle (LV) 0.939, right ventricle (RV) 0.878, and myocardium (MYO) 0.858—only within 0.01 of the state-of-the-art fixed-task model. Compared to another instruction-driven method, M3D, CMR-VLM consistently outperforms all datasets: particularly on M&Ms2, with left ventricle (LV) +3.1%, right ventricle (RV) +3.8%, and myocardium (MYO) +1.0%; similar gains are observed on ACDC and M&Ms (2.4–3.3% improvement across structures).

[0101] These results demonstrate that CMR-VLM unifies instruction-driven spatiotemporal awareness with flexible user-guided anatomical segmentation, enabling highly interactive and clinically adaptable CMR analysis without modifying the network architecture.

[0102] Performance evaluation results of CMR-VLM in flexible command-driven anatomical localization tasks are as follows: Figure 8 As shown, a–c represent perceptual accuracy: The heatmap demonstrates the basic spatiotemporal understanding capabilities of CMR-VLM. a: Sequence recognition (4CH / short axis / LGE); b: Phase recognition (systole / diastole / mid-cycle); c: Layer location (basal segment / middle segment / apical segment).

[0103] The accuracy ranged from 74.2% to 96.5%, with the main confusion occurring between SAX and LGE sequences (as both are short-axis orientations), reflecting inter-observer variability.

[0104] d represents the segmentation quality: The bar chart compares the Dice similarity coefficients of CMR-VLM with conventional state-of-the-art (SOTA) models (UNet, nnUNet, TransUNet) and reference-guided methods (M3D-Ref, CMR-VLM-Ref) in left ventricle (LV), right ventricle (RV), and myocardial fossa (MYO) segmentation. CMR-VLM achieves near-specialist-level accuracy, outperforming M3D by 3.8% on the M&Ms2 dataset, validating the effectiveness of its instruction-driven paradigm.

[0105] e–g represents closed-ended visual question answering (VQA): e: The bar chart shows that CMR-VLM maintains high performance on both the internal test set (AUC 0.842) and the external test set (AUC 0.825), with LGE feature description results being the best (AUC 0.867).

[0106] g: On the external test set, CMR-VLM significantly outperforms the existing best medical VLMs (RadFM, BiomedGPT, M3D), with an accuracy of 0.745, representing a 136% relative improvement over RadFM (0.315).

[0107] In one specific embodiment, in clinical practice, radiologists often query CMR data with targeted questions (ranging from chamber size to scar pattern) to guide diagnosis. To simulate this interactive reasoning process, we constructed a VQA benchmark directly derived from expert-generated CMR reports. The questions cover the entire CMR interpretation process: cardiac morphology (e.g., chamber size, wall thickness), cardiac motion (e.g., overall / local contractility), and late gadolinium enhancement (LGE) feature descriptions (e.g., scar pattern, enhancement site).

[0108] Closed-ended VQA: Stable performance in factual recall; In closed-loop VQA (i.e., testing factual recall with limited options), CMR-VLM maintains high performance across multiple evaluation metrics on both internal and external test sets: Internal test set: accuracy 0.764, precision 0.763, recall 0.763, F1 score 0.761, AUROC 0.842; External test set: The corresponding metrics are 0.745, 0.739, 0.735, 0.727, and 0.825, respectively, indicating that the performance degradation under distribution shift is minimal.

[0109] Across different question types, CMR-VLM maintains a stable advantage: overall accuracy of 0.764 on the internal test set and 0.745 on the external set. Among them, LGE feature description performs best (0.867 internal, 0.855 external), followed by cardiac motion (0.753 internal, 0.780 external) and cardiac morphology (0.688 internal, 0.784 external).

[0110] Open-ended VQA: The dual advantages of meticulous clinical practice and superior quantitative analysis; In open-ended VQA requiring coherent free text generation, CMR-VLM demonstrates performance that combines clinical detail with quantitative superiority: When assessed by clinical problem type, LGE feature description and cardiac morphology performance were the best—BLEU-4 scores of 0.33 and 0.40, respectively, and GPT-4 large language model score of over 8.1; Cardiac motion issues showed slightly lower but still robust performance, reflecting the inherent challenges of describing temporal dynamics in text.

[0111] VQA (Various Quality Analysis): Multi-dimensional reasoning that matches clinical interpretation logic.

[0112] In the abnormality-focused VQA task, CMR-VLM performed consistently and conformed to established patterns of clinical CMR interpretation. Figure 9 (as shown in c) Structural assessment: The four-chamber sequence (4CH) showed the highest accuracy—robust quantification of chamber size (right ventricular end-diastolic diameter (RVEDD) 0.87, left atrium (LA) 0.79), and reliable detection of ventricular wall motion (0.71), consistent with the high geometric resolution of the 4CH sequence; Functional / volume parameters: Left ventricular end-diastolic volume (LVEDV) 0.72, right ventricular end-systolic volume (RVESV) 0.80, and right ventricular ejection fraction (RVEF) 0.83 in short-axis cine sequence, showing stable performance; Tissue characteristics: The contrast enhancement abnormalities of LGE (left ventricular LGE (LV-LGE) 0.81, pericardial enhancement 0.78) were well observed; Remaining limitations: primarily in the assessment of subtle left ventricular fibrosis (0.61) and critical wall thickness – reflecting known challenges even for expert readers.

[0113] In summary, these results demonstrate that CMR-VLM provides reliable multi-sequence anomaly inference in structural, functional, and organizational tasks, and is a powerful tool for interpreting broad-spectrum CMRs.

[0114] Comparison with SOTA medical VLM: significant performance advantages; Crucially, CMR-VLM significantly outperforms the current state-of-the-art (SOTA) medical VLM in both closed and open VQA models. Figure 9 b, Figure 9 (as shown in e) Closed-loop VQA: External test set accuracy of 0.745, which is 136%, 88%, and 110% higher than RadFM (0.315), BiomedGPT (0.397), and M3D (0.354), respectively. Open-ended VQA: BLEU-4 score for all aggregation types was 0.33, 83% higher than M3D (0.18) and 38% higher than BiomedGPT (0.24); METEOR score was 49% higher than M3D and 29% higher than BiomedGPT. LLM Assessment: DeepSeek gave CMR-VLM a score of 7.76, which is 34% higher than M3D and 14% higher than BiomedGPT.

[0115] These advantages stem from the report-oriented, modality-specific design of CMR-VLM—the model incorporates specialized knowledge of CMR semantics and reporting conventions; in contrast, general medical vision-language models lack such domain expertise and are unable to generate responses that combine linguistic fluency and clinical depth (especially for complex pathological descriptions such as late gadolinium enhancement).

[0116] A comprehensive assessment of CMR-VLM advanced reasoning capabilities (covering open-ended visual question answering and report generation) such as Figure 9 As shown.

[0117] a. Open-ended VQA across clinical problem types: The radar chart shows the performance of CMR-VLM in the multivariate VQA categories (cardiac morphology, motion, late gadolinium enhancement (LGE) feature description) – LGE feature description and cardiac morphology have the highest BLEU-4 scores.

[0118] b. Comparison of open-ended VQA methods: CMR-VLM significantly outperforms existing state-of-the-art (SOTA) medical VLMs (RadFM, M3D, BiomedGPT): its BLEU-4 score is 0.33, an 83% improvement over M3D and a 38% higher score than BiomedGPT.

[0119] c. Exception-specific VQA: The pie chart summarizes the performance of CMR-VLM in specific queries of anomalies (such as wall motion, LGE pattern, and valve function).

[0120] d. Image summary: CMR-VLM outperforms state-of-the-art (SOTA) methods (RadFM, M3D, BiomedGPT). LLM-based assessments further confirm its superior clinical consistency in output.

[0121] e. Structured report generation: CMR-VLM once again demonstrated outstanding performance: its BLEU-1 score was 31.9% higher than RadFM's; the LLM-based automated assessment (GPT-4 score: 6.34) further highlighted the structured features and clinical information value of its output.

[0122] f. Reasoning efficiency: CMR-VLM can generate a diagnostic report for each case in just 4.2 seconds—about 400 times faster than human experts (27.8 minutes) and 2-3 times faster than SOTA VLM, supporting near real-time reporting for high-throughput screening.

[0123] g. Text consistency analysis: A side-by-side comparison of the reference report and the CMR-VLM output shows that the two have a phrase-level semantic consistency of 78.4%.

[0124] In one specific implementation, automatically generating accurate and clinically relevant text from CMR data is key to enabling scalable reporting workflows. CMR-VLM integrates this capability into our instruction-driven framework, supporting the generation of two types of output: brief image summaries (similar to image captioning) and complete structured CMR reports. In both tasks, CMR-VLM consistently outperforms state-of-the-art (SOTA) medical VLMs, including RadFM, M3D, and BiomedGPT—an advantage validated by both traditional NLP metrics and LLM-based clinical assessments.

[0125] Imaging summary: Excellent in both language fluency and clinical semantics; In the task of image summarization ( Figure 9 As shown in d), the BLEU-1 score of CMR-VLM was 0.69, which was 35.3% higher than RadFM (0.51), 30.2% higher than BiomedGPT (0.53), and 25.5% higher than M3D (0.55). On more stringent metrics, the advantage expanded further: the BLEU-4 score was 47.2% higher than RadFM, METEOR improved by 53.3%, and ROUGE-L improved by 27.9%. LLM-based assessment further validated its clinical relevance: the GPT-4 score for the CMR-VLM summary was 6.44, which was 43.1% higher than RadFM (4.50), 32.8% higher than BiomedGPT (4.85), and 57.1% higher than M3D (4.10). These results indicate that the summaries generated by CMR-VLM are not only linguistically fluent but also more clinically relevant.

[0126] Structured reports: comprehensive, logical, and clinically valuable; In the task of generating a complete structured report ( Figure 9As shown in e), CMR-VLM maintains its lead—the BLEU-1 score is 0.62, 31.9% higher than RadFM (0.47), 34.8% higher than BiomedGPT (0.46), and 29.2% higher than M3D (0.48). The advantage in the METEOR metric is most significant: CMR-VLM is 40.0% higher than RadFM and 26.0% higher than M3D; ROUGE-L is 31.8% better than RadFM. Automated assessment based on LLM further reinforces this conclusion: GPT-4 scores CMR-VLM reports at 6.34, 6.4% higher than RadFM (5.95) and M3D (6.00), and 11.1% higher than BiomedGPT (5.80); DeepSeek scores show that CMR-VLM reports 28.3% better than BiomedGPT and 22.1% better than RadFM. Overall, these metrics demonstrate that the reports generated by CMR-VLM are more comprehensive, logically clear, and clinically informative than previous models—which is precisely the core requirement for real-world radiology integration.

[0127] Aligning with expert patterns: room for improvement under high consistency; Apart from standard metrics, the performance of CMR-VLM is highly consistent with expert writing patterns. Figure 9 As shown in g): its phrase-level consistency reached 78.4%, reflecting a high degree of clinical coherence. Nevertheless, two error patterns were observed: ① Quantitative discrepancies occurred in 12.6% of cases, primarily concerning precise measurements (such as ventricular size and ejection fraction); ② Semantic changes in clinical terminology occurred in 9% of cases (such as replacing "normal" with "acceptable"). Despite these biases, the high consistency rate supports the practical potential of CMR-VLM for automated reporting systems, and the observed errors provide clear directions for future optimization.

[0128] Efficiency Revolution: Generates reports in 4 seconds, far surpassing human capabilities and existing models; Crucially, CMR-VLM possesses unprecedented efficiency ( Figure 9 As shown in f): Based on a single NVIDIA A100 40GB GPU, CMR-VLM generates a diagnostic report per case in just 4.2 seconds—approximately 400 times faster than the average human expert (27.8 minutes, calculated based on 20 reports annotated by 3 senior radiologists) and 2-3 times faster than the state-of-the-art VLM. This efficiency enables CMR-VLM to achieve near-real-time reporting, potentially reducing wait times for emergency patients, supporting high-throughput screening processes, and thus alleviating the workload of specialists.

[0129] The present invention also discloses a computer program product or system, including a computer program that, when executed by a processor, implements the above-described method for processing 4D cardiovascular magnetic resonance images, or implements the steps of the above-described multi-task processing method based on 4D cardiovascular magnetic resonance images.

[0130] Figure 2 A schematic diagram of a system for processing 4D cardiovascular magnetic resonance images provided in this embodiment of the invention specifically includes: Acquisition Unit: Acquires 4D cardiovascular magnetic resonance imaging sequences; Embedding unit: Divides the image sequence into patches and marks each sequence patch with a token to obtain a sequence embedding tensor; Feature unit: After the sequence embedding tensor is layered, it is input into N parallel neural network models for feature learning to obtain N output features, where N is a natural number greater than 1; the layering is based on dividing the sequence embedding tensor into layered tensors according to different dimensions, and the layered tensors are input into N parallel neural network models, wherein the different dimensions include any one or more of the following: time × height × width dimension, time × slice × height × width dimension, slice dimension.

[0131] This invention provides a multi-task processing system based on 4D cardiovascular magnetic resonance imaging, comprising: Acquisition module: Acquires 4D cardiovascular magnetic resonance imaging sequences; Feature module: The image sequence is input into the above-described method for processing 4D cardiovascular magnetic resonance images to obtain N output features; Fusion module: fuses N output features to obtain a fused feature; Task module: Input the fused features into a multi-task neural network model to obtain the results of the disease classification task and the anatomical localization task.

[0132] Figure 3 An embodiment of the present invention provides a schematic diagram of a computer device, specifically including: The system includes a memory and a processor; the memory is used to store program instructions; the processor is used to invoke the program instructions, when the program instructions are executed to perform any of the above-described methods for processing 4D cardiovascular magnetic resonance images, or to execute the above-described multi-task processing method based on 4D cardiovascular magnetic resonance images.

[0133] The present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, performs any of the above-described methods for processing 4D cardiovascular magnetic resonance images, or performs the above-described multi-task processing method based on 4D cardiovascular magnetic resonance images.

[0134] The verification results of this verification embodiment show that assigning inherent weights to indications can improve the performance of this method compared to the default settings. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separated; the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of this embodiment. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0135] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0136] The computer device provided by the present invention has been described in detail above. For those skilled in the art, there will be changes in the specific implementation and application scope based on the ideas of the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for processing 4D cardiovascular magnetic resonance images, characterized in that, include: Acquire 4D cardiovascular magnetic resonance imaging sequences; The image sequence is divided into patches and each sequence patch is marked with a token to obtain a sequence embedding tensor. After the sequence embedding tensor is layered, it is fed into N parallel neural network models for feature learning to obtain N output features, where N is a natural number greater than 1; The layering is based on dividing the sequence embedding tensor into layered tensors according to different dimensions. The layered tensors are then input into N parallel neural network models. The different dimensions include any one or more of the following: time × height × width dimension, time × slice × height × width dimension, and slice dimension.

2. The method for processing 4D cardiovascular magnetic resonance images according to claim 1, characterized in that, When the neural network model performs feature learning on the hierarchical tensor of time × slice × height × width dimensions, it first performs feature aggregation on the inter-layer and in-plane cardiac spatial structure of the slice × height × width dimensions to obtain the cardiac spatial aggregate representation, and then performs time aggregation on the time dimension to obtain the dynamic features of the cardiac spatial aggregate representation.

3. The method for processing 4D cardiovascular magnetic resonance images according to claim 1, characterized in that, When the neural network model performs feature learning on the hierarchical tensor with time × height × width dimensions, it captures spatial features in the plane while capturing long-term dependent and periodic motion features, thus generating multi-scale temporal features.

4. The method for processing 4D cardiovascular magnetic resonance images according to claim 1, characterized in that, When the neural network model performs feature learning on the hierarchical tensor of the slice dimension, it obtains slice-level features by delineating scars and detecting subtle intensity changes in the slice; the neural network model includes any one or more of the following: Swin Transformer, CSwin Transformer, SG-Former, DiNAT, gSwin.

5. A multi-task processing method based on 4D cardiovascular magnetic resonance imaging, characterized in that, include: Acquire 4D cardiovascular magnetic resonance imaging sequences; The image sequence is input into the method for processing 4D cardiovascular magnetic resonance images according to any one of claims 1-4 to obtain N output features; The N output features are fused to obtain the fused visual features; The fused visual features are input into a multi-task neural network model to obtain the results of the disease classification task and the anatomical localization task.

6. The multi-task processing method based on 4D cardiovascular magnetic resonance imaging according to claim 5, characterized in that, The multi-task neural network model is a model constructed by combining a single model with a multi-task head. The fused visual features are input into the multi-task neural network model, and the fused features are learned by the single model to obtain the features of the hidden state. The features of the hidden state are then input into the disease classification task head and the anatomical localization task head for processing to obtain the disease classification task results and the anatomical localization task results. The single model includes any one or more of the following: MiniCPM-4B, mixed model, Qwen2.5 series model, and Phi-3.5-Mini model.

7. The multi-task processing method based on 4D cardiovascular magnetic resonance imaging according to claim 5, characterized in that, The multi-task also includes text generation and / or segmentation tasks, and the fused visual features are input into the multi-task neural network model to obtain the results of disease classification task, anatomical localization task, text generation task, and / or segmentation task. The radiological report text data corresponding to the 4D cardiovascular magnetic resonance imaging sequence is obtained. The obtained text data is fused with the fusion visual features to obtain the second fusion feature. The second fusion feature is input into a multi-task neural network model to learn the fusion feature through a single model to obtain the features of the second hidden state. The features of the second hidden state are then input into the disease classification task head and the anatomical localization task head, the text generation task head and / or the segmentation task head respectively for processing to obtain the disease classification task results, the anatomical localization task results, the text generation task results and / or the segmentation task results.

8. A computer program product comprising a computer program or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the method for processing 4D cardiovascular magnetic resonance images as described in any one of claims 1-4, or to implement the multi-task processing method based on 4D cardiovascular magnetic resonance images as described in any one of claims 5-7.

9. A computer device comprising a memory, a processor, and a computer program or instructions stored in the memory, characterized in that, The computer program or instructions are executed by the processor to implement the method for processing 4D cardiovascular magnetic resonance images as described in any one of claims 1-4, or to implement the multi-task processing method based on 4D cardiovascular magnetic resonance images as described in any one of claims 5-7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, The computer program or instructions are executed by the processor to implement the method for processing 4D cardiovascular magnetic resonance images as described in any one of claims 1-4, or to implement the multi-task processing method based on 4D cardiovascular magnetic resonance images as described in any one of claims 5-7.

Images