A medical basis model for chest x-ray image understanding and generation

Abstract

The present application relates to the technical field of medical image understanding and generation, and provides a unified medical basic model for chest X-ray image understanding and generation. The present application decouples two types of tasks into an autoregressive branch responsible for understanding tasks and a diffusion branch responsible for high-fidelity generation tasks. The present application introduces a cross-modal self-attention mechanism to dynamically guide the generation process through the features output by the understanding branch. Combined with a strict data cleaning pipeline and a progressive optimization strategy, the network architecture can fully utilize the advantages of diffusion models to achieve excellent generation results, while realizing the coordination between the two types of tasks. In two representative benchmark tests, the present application uses only one quarter of the parameter amount of the LLM-CXR model, and achieves a 46.1% improvement in understanding task performance and a 24.2% optimization in generation quality. The model performance of the present application can be comparable to that of a special single-task model, providing an extensible technical paradigm for the collaborative optimization of medical image understanding and generation.

Description

Technical Field

[0001] This invention relates to the field of medical image understanding and generation technology, and in particular to a medical basic model for understanding and generating chest X-ray images. Background Technology

[0002] In recent years, medical foundational models based on vision-language pre-training have demonstrated significant effectiveness in both understanding and generation tasks. As research progresses, medical image understanding and generation are increasingly seen as interconnected tasks, with semantic reasoning and visual synthesis mutually reinforcing each other. This understanding has driven the development of a unified medical foundational model, aiming to integrate these two capabilities into a single framework.

[0003] However, unifying the modeling of these two capabilities is inherently challenging due to the fundamental difference between the goals of semantic abstraction and pixel-level reconstruction. Existing research (such as LLM-CXR) often employs parameter sharing and joint multi-task heads for ensemble learning. However, this approach is prone to task competition and feature interference, leading to a decline in both understanding and generation performance. HealthGPT alleviates this problem through a task-specific H-LoRA module, providing a structured compromise but failing to fundamentally solve the issue. Furthermore, most current unified medical foundation models still rely on discretized generation paradigms, whose outputs are limited by lexical granularity, making it difficult to reproduce the fine structural details in medical images. A direct alternative is to attach a diffusion model to a pre-trained vision-language model. While this improves generation quality to some extent, it fails to fully exploit understanding features to guide the generation process, thus failing to fully realize the potential of a unified architecture.

[0004] Through systematic analysis, we identified two inherent limitations in existing unified medical foundation models. First, the tasks of understanding and generation have conflicting objectives: understanding requires semantic abstraction, while generation relies on pixel-level reconstruction techniques. Simultaneously learning these two opposing objectives in a shared feature space leads to information interference. Second, there is a paradigm mismatch between discrete autoregressive models and continuous imaging. Discrete methods are inherently difficult to capture the fine structural details of medical images; therefore, previous studies often employed surface stacking or cascading strategies to integrate the tasks. This strategy only achieves formal unification and fails to fully explore the deep architectural synergies between the two capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide a fundamental medical model for understanding and generating chest X-ray images. To address the aforementioned problems, this invention provides a fundamental medical model for understanding and generating chest X-ray images. This architecture fundamentally resolves the contradiction between semantic understanding and visual synthesis. This invention employs a decoupled dual-branch structure to eliminate the inherent conflict between the understanding and generation tasks: the autoregressive branch focuses on semantic abstraction, while the independent generation branch is responsible for pixel-level image reconstruction. To bridge the paradigm differences between the two tasks, a diffusion model is introduced into the generation branch. This design aligns with the continuous characteristics of medical images and avoids the loss of detail granularity inherent in discrete word segmentation. Finally, this invention introduces a cross-modal self-attention mechanism to achieve efficient collaboration between the two branches. Unlike shallow module stacking, this module can dynamically inject the semantic features of the understanding branch into the diffusion generation process, thereby effectively achieving deep linkage between semantic reasoning and high-fidelity generation.

[0006] The above-mentioned technical objective of this invention is achieved through the following technical solution: a medical basic model for understanding and generating chest X-ray images, comprising...

[0007] The autoregressive understanding branch, built upon a visual language model, is responsible for handling semantic abstraction and reasoning reports;

[0008] Diffusion-based generative branches, directly based on the language model backbone inherited from the autoregressive understanding branches, specialize in synthesizing high-fidelity chest X-ray images;

[0009] The cross-modal self-attention module connects the autoregressive understanding branch and the generation branch, enabling dynamic feature exchange and semantic conditionalization during the generation process.

[0010] A further provision of the present invention is that, in the autoregressive understanding branch, a multimodal lexical sequence S=[V,T] is defined. in ,T out ], where V and T in and T out Let m represent visual words, input text words, and output text words, respectively, and let T be a given string of words. out The starting index in sequence S, n is the ending index of S, and the cross-entropy loss is obtained through T. out The autoregressive prediction values ​​for all tokens in the text are calculated: Where n represents the index of the last word in sequence S, ω u This represents the parameters for understanding the branch. The word "token" is the English word for the token.

[0011] A further feature of this invention is that, in the diffusion-based generative branch, a latent diffusion framework is employed to reconstruct the medical image by understanding the high-level semantic information extracted from the branch; the diffusion process is not performed in the pixel space, but is completed in the latent space encoded by VAE, thereby significantly improving efficiency and stability; given a noise distribution p at time t... t The latent variable x sampled in (x) t The model learns to estimate the target velocity field u by minimizing the mean square error. t (x),

[0012] , where ω g The parameters representing the generated branches, and the semantic embedding of branch generation as conditional input, can achieve disease-specific synthesis and improve the accuracy of lesion localization.

[0013] A further feature of this invention is that the cross-modal self-attention module can establish a bidirectional information flow between the understanding branch and the generation branch. By performing joint self-attention processing on a unified multimodal lexical sequence, the semantic representation generated by the understanding branch can directly control the generation trajectory. At the same time, the generation state can also be fed back to the semantic space for optimization. This design not only realizes the bidirectional interaction between semantic representation and generation process, but also ensures the dynamic adaptation of generation results to input semantics.

[0014] A further provision of the present invention is that, in the cross-modal self-attention module, the unified sequence is set as S=[T in [N], where T in S represents the text lexicon generated by the comprehension branch, and N represents the noisy conditional latent embedding of the generated branch. For each lexicon S i We compute the modality-specific projections of the query, key, and value: ,

[0015] The modality selector is defined as follows: This formalized method, while maintaining shared attention operations across a unified sequence, provides two distinct parameter spaces for understanding and generating lexical units. The resulting attention map is computed in standard form: However, all cross-modal interactions are achieved through implicit learning of joint attention scores, rather than through explicit conditioning.

[0016] The cross-modal self-attention module improves the fidelity and clinical consistency of generated images by synchronizing the autoregressive branch for understanding with the diffusion branch for high-fidelity generation.

[0017] A further provision of the present invention includes a progressive optimization strategy, including standardized medical training and multi-center unified teaching optimization, which ensures that the basic medical model can acquire robust basic capabilities before adapting to heterogeneous clinical environments.

[0018] A further provision of the present invention is that standardized basic medical training includes:

[0019] In the first stage, the medical understanding model is fine-tuned. During this stage, the generation branch remains frozen. By pairing medical images with clinical report data, the visual encoder, visual connector, and language model backbone in the understanding branch are optimized. This process constructs robust semantic anchors, enabling the model to master the ability to interpret medical images. Crucially, this transforms the understanding branch into a high-level semantic feature generator, providing direction for subsequent generation branches.

[0020] In the second stage, medical image generation pre-training is performed. In this stage, the understanding branch is frozen and the generation branch is pre-trained using text-low resolution images. To accelerate the convergence process and ensure structural consistency, we adopt a representation alignment technique to align the hidden state of the generation branch with RadDino image features. This design enables the generation branch to effectively utilize high-level semantic information, thereby achieving coarse-grained medical image synthesis.

[0021] In the third stage, based on the freezing strategy of the second stage, the text-high resolution image generation branch was fine-tuned; by expanding the positional encoding to support higher resolution and removing feature-level supervision to focus on texture refinement, the final model achieved high-fidelity synthesis effect: the lesion site was accurately presented and the alignment between the reported image and the actual image was significantly improved.

[0022] A further feature of this invention is: multi-center unified instruction tuning, in addition to VAE, all components of the understanding branch and the generation branch are unfrozen and jointly tuned using a clinical instruction hybrid scheme. This scheme integrates the tasks centered on understanding and the tasks centered on generation. This unified optimization strategy keeps the feature spaces of the two branches consistent, bridging the gap between standardized pre-training and real-world clinical diversity, thereby significantly improving the model's cross-domain generalization ability and realizing multifunctional interaction, ensuring consistent performance in heterogeneous diagnostic schemes.

[0023] The invention is further configured such that both the understanding branch and the generation branch are partially initialized from Janus-Pro; for the understanding branch, we use siglip-large-patch16-384 as the visual encoder, whose 1024-dimensional embedding vector is projected onto a 2048-dimensional large language model space through a two-layer MLP; the language backbone network contains 24 Transformer layers with QK normalization and QKV bias.

[0024] A further feature of this invention is that the branch generation adopts 16× downsampling and 16-channel VAE, and achieves bidirectional projection between the 16-dimensional latent space and the 2048-dimensional language space through two single-layer MLPs.

[0025] The beneficial effects of this invention are:

[0026] This invention proposes a new generation of unified medical foundation model. Structurally, this model decouples the understanding and generation tasks while simultaneously achieving deep collaboration between the two types of tasks. To the best of the applicant's knowledge, this solution is the first in the field of medical imaging to deeply integrate autoregressive and diffusion paradigms.

[0027] This invention proposes a cross-modal self-attention mechanism to bridge the understanding branch and the generation branch. This module seamlessly integrates the features extracted from the understanding branch as contextual conditions, providing dynamic and content-aware guidance throughout the generation process. Attached Figure Description

[0028] Figure 1 These are the quantitative and qualitative results of the medical foundation model of this invention. The quantitative results show that, in medical understanding and generation tasks, the performance of this invention is superior to existing unified medical foundation models and single-task medical foundation models. Qualitatively, this invention can achieve multi-disease X-ray image interpretation and high-fidelity medical image generation.

[0029] Figure 2 Schematic diagram of model architecture, and schematic diagram of data processing and standardized medical basic training process.

[0030] Model Architecture: This invention comprises two decoupled and collaborative branches: an autoregressive understanding branch for semantic encoding and a diffusion-based generative branch for visual synthesis. To achieve efficient collaboration between the two branches, we introduce a cross-modal self-attention mechanism, enabling semantic features to dynamically guide the generation process.

[0031] Data Processing and Standardized Medical Fundamentals Training: To fully explore the performance potential of this architecture, we designed a rigorous data cleaning pipeline and a three-stage training strategy. This strategy progressively freezes branches at different training stages, ensuring efficient knowledge transfer and a stable training process.

[0032] Figure 3 The data processing and report generation performance verification demonstrates that the application of large language models can clean and purify raw data by removing redundant and irrelevant information. This process ensures that the model prioritizes extracting core and critical information relevant to disease diagnosis.

[0033] Figure 4The qualitative examples of the model in this invention (A)–(C) demonstrate the model's ability to precisely control the attributes of generated signs, covering the severity and location of the lesion. In Figure (D), the model can efficiently synthesize complex radiological image scenes containing multiple abnormal signs, and the output results are highly consistent with the complete clinical report, highlighting its ability to process and integrate massive amounts of contextual information. Detailed Implementation

[0034] Example: A medical foundational model for chest X-ray image understanding and generation, aiming to achieve decoupled but collaborative learning between chest X-ray image understanding and generation.

[0035] like Figure 2 As shown, the medical foundation model in this embodiment comprises two core components: an autoregressive understanding branch and a diffusion-based generative branch. The understanding branch, built upon a visual language model, is responsible for handling semantic abstraction and reasoning reports. The generative branch, directly based on the language model backbone inherited from the understanding branch, specializes in synthesizing high-fidelity chest X-ray images. A cross-modal self-attention module connects these two branches, enabling dynamic feature exchange and semantic conditionalization during the generation process.

[0036] Understanding through autoregression

[0037] The understanding branch models multimodal understanding as an autoregressive sequence modeling problem. This modeling approach is highly suitable for medical report generation tasks because the model needs to reason about visual and textual contexts in a causal manner.

[0038] Specifically, this embodiment defines a multimodal lexical sequence S=[V,T] in ,T out ], where V and T in and T out Let m represent visual words, input text words, and output text words, respectively, and let T be a given string of words. out The starting index in sequence S, n is the ending index of S, and the cross-entropy loss is obtained through T. out The autoregressive prediction values ​​for all tokens in the text are calculated:

[0039] Where n represents the index of the last word in sequence S, ω u The parameters represent the understanding branches. This design enables the model to simultaneously capture visual semantics and linguistic reasoning within a unified space.

[0040] Generation through latent diffusion

[0041] The branch generation employs a latent diffusion framework, reconstructing medical images by understanding the high-level semantic information extracted from the branches. The diffusion process does not occur in the pixel space, but rather in the latent space encoded by VAE, thus significantly improving efficiency and stability.

[0042] Given a noise distribution p at time t t The latent variable x sampled in (x) t The model learns to estimate the target velocity field u by minimizing the mean square error. t (x), , where ω g The parameters representing the generated branches, and the semantic embedding of branch generation as conditional input, can achieve disease-specific synthesis and improve the accuracy of lesion localization.

[0043] Cross-modal self-attention mechanism

[0044] To achieve semantic-driven visual generation, we propose a cross-modal self-attention mechanism that establishes a bidirectional information flow between the understanding and generation branches. Unlike traditional cross-attention mechanisms that rely on static contextual conditions, our framework performs joint self-attention processing on a unified multimodal lexical sequence. This allows the semantic representation generated in the understanding branch to directly regulate the generation trajectory, while the generation state is also fed back into the semantic space for optimization. This design achieves both bidirectional interaction between semantic representation and the generation process, and ensures dynamic adaptation between the generated result and the input semantics.

[0045] Let the unified sequence be S=[T in [N], where T in S represents the text lexicon generated by the comprehension branch, and N represents the noisy conditional latent embedding of the generated branch. For each lexicon S i We compute the modality-specific projections of the query, key, and value: ,

[0046] The modality selector is defined as follows: This formalized method, while maintaining shared attention operations across a unified sequence, provides two distinct parameter spaces for understanding lexical units and generating lexical symbols. The resulting attention map is computed in standard form: However, all cross-modal interactions are achieved through implicit learning of joint attention scores, rather than through explicit conditioning.

[0047] This mechanism improves the fidelity and clinical consistency of the generated images by synchronizing the autoregressive branch used for comprehension with the diffusion branch used for high-fidelity generation.

[0048] Incremental optimization strategy

[0049] like Figure 2As shown, we designed a progressive optimization strategy comprising two macro-stages: standardized basic medical training (stages 1-3) and multi-center unified teaching tuning (stage 4). This strategy ensures that the model acquires robust basic capabilities before adapting to heterogeneous clinical environments.

[0050] Standardized basic medical training, this stage separates the learning process of semantic understanding from that of visual generation to avoid task interference, and includes three progressive stages:

[0051] Phase 1: Fine-tuning the Medical Understanding Model. During this phase, the generation branch remains frozen. We optimize the visual encoder, visual connectors, and language model backbone within the understanding branch by pairing medical images with clinical report data. This process constructs robust semantic anchors, enabling the model to master the interpretation of medical images. Crucially, this transforms the understanding branch into a high-level semantic feature generator, providing direction for subsequent generation branches.

[0052] Phase 2: Medical Image Generation Pre-training. In this phase, we freeze the understanding branch and pre-train the generation branch using text-low-resolution image pairs. To accelerate the convergence process and ensure structural consistency, we employ a representation alignment technique, aligning the hidden states of the generation branch with RadDino image features. This design enables the generation branch to effectively utilize high-level semantic information, thereby achieving coarse-grained medical image synthesis.

[0053] Phase 3: Fine-tuning the medical generative model. Building upon the freezing strategy from Phase 2, we fine-tuned the text-high-resolution image generation branch. By expanding positional encoding to support higher resolutions and removing feature-level supervision to focus on texture refinement, the final model achieved high-fidelity synthesis results: lesion sites were accurately presented, and the alignment between the reported image and the actual image was significantly improved.

[0054] Multi-center unified instruction tuning

[0055] Building upon the foundational capabilities established in the standardized medical basic training phase, we implemented a unified medical optimization phase to maximize the model's synergistic efficacy and clinical robustness. In this phase, all components of both branches, except for the VAE, were unfrozen. We employed a carefully designed hybrid clinical instruction scheme for joint optimization, integrating: 1) an understanding-centric task (integrating VQA and report generation from multi-center datasets); and 2) a generation-centric task (context-aware high-fidelity synthesis). This unified optimization strategy ensured consistency in the feature spaces of both branches, bridging the gap between standardized pre-training and real-world clinical diversity. This significantly improved the model's cross-domain generalization ability and enabled multifunctional interactions, ensuring consistent performance across heterogeneous diagnostic protocols.

[0056] experiment

[0057] This experiment demonstrates how a basic medical model can fully utilize its decoupled autoregressive-diffusion bi-branch design through data processing, model configuration, and incremental optimization strategies.

[0058] A. Implementation Details

[0059] Understanding the data. We constructed dedicated datasets for two training phases: (1) Phase 1: We used the mimic-CXR dataset (AE Johnson, TJ Pollard, SJ Berkowitz, NR Greenbaum, MPLungren, C.-y. Deng, RG Mark, and S. Horng, “Mimic-cxr, a deidentified publicly available database of chest radiographs with free-text reports,” Scientific data, vol. 6, no. 1, p. 317, 2019.) (frontal view). Following the official splitting criteria, the training set used 163,344 image-report pairs, and the test set used 2,365 reports, optimized using the DeepSeek-V3 pipeline. (2) Phase 2: We expanded the training pool to a multi-center scale by integrating three additional datasets. For CheXpertPlus, we selected frontal view images, obtaining 190,753 training pairs; since the official test set was unavailable, we used the official validation set (202 pairs) for testing. For CXR-27 (Y. Bai, R. Zhang, Y. Lei, J. Yao, S. Ju, C. Wang, W. Yao, Y. Guo, G. Zhang, C. Wan et al., “From bench to bedside: Adeepseek-powered ai system for automated chest radiograph interpretation inclinical practice,”arXiv preprint arXiv:2507.19493,2025), a random 9:1 split ratio was used to generate 9,227 training pairs and 1,026 test pairs.For mimic-CXR-VQA (S. Bae, D. Kyung, J. Ryu, E. Cho, G. Lee, S. Kweon, J. Oh, L. Ji, E. Chang, T. Kim et al., “Mimic-ext-mimic-cxr-vqa: A complex, diverse, and large-scale visual question answering dataset for chest x-ray images,” 2024.), following the official guidelines but merging the validation set into the training set to maximize coverage, we ultimately obtained 322,516 training samples and 11,309 test samples. All images were resized to 384×384 pixels and white padding was added.

[0060] Data Generation. Consistent with both Phase 1 and Phase 2, we followed the processing and segmentation protocol of ChexGenBench [R. Dutt, P. Sanchez, Y. Yao, S. McDonald, S. A. Saftaris, and T. Hospedales, “Chexgenbench: Aunified benchmark for fidelity, privacy and utility of synthetic chestradiographs,” arXiv preprint arXiv:2505.10496, 2025.], using a simulated CXR dataset. This yielded 237,387 training pairs and 4,352 test pairs. Images were resized to 256×256 pixels during the pre-training phase and to 512×512 pixels during the fine-tuning phase, maintaining aspect ratio through padding.

[0061] Model Details. Both branches are partially initialized from Janus-Pro (X. Chen, Z. Wu, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, and C. Ruan, “Janus-pro: Unified multimodal understanding and generation with data and model scaling,” arXiv preprint arXiv:2501.17811, 2025.). For the understanding branch, we use siglip-large-patch16-384 (X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” 2023.) as the visual encoder, whose 1024-dimensional embedding vectors are projected onto a 2048-dimensional large language model space through a two-layer MLP. The language backbone network consists of 24 Transformer layers with QK normalization and QKV bias. The branch generation uses 16× downsampling and 16-channel VAE, and achieves bidirectional projection between the 16-dimensional latent space and the 2048-dimensional language space through two single-layer MLPs.

[0062] Training details. All models were trained with full parameter fine-tuning on eight NVIDIA L20 GPUs. See Table I for complete hyperparameter configurations.

[0063] Table I

[0064] Hyperparameter settings for the incremental optimization strategy. Note that we have introduced weights for the multiple loss function to improve its accessibility. Specifically, "REPA loss weight" refers to the weight ratio between the MSE loss and the REPA loss.

[0065] B Evaluation Indicators

[0066] To comprehensively evaluate the performance of the medical basic model in this embodiment, we adopted a diverse set of evaluation metrics covering clinical semantic accuracy and visual fidelity.

[0067] Medical understanding metrics: In the understanding task, our primary focus is on the clinical reliability of the generated reports.

[0068] Clinical efficacy assessment (CheXbert F1). We used the CheXbert annotation tool to extract the presence status of 14 common chest examination indicators from the generated report and the actual report. The consistency of clinical findings was assessed by calculating the F1 score, which serves as the core evaluation criterion for diagnostic accuracy.

[0069] Natural Language Generation Metrics. To assess the language quality and fluency of the reports, we employed standard natural language processing metrics, including BLEU and rouge-L. Additionally, we utilized RadGraph, a specialized metric that provides a more structure-aware assessment of medical texts by measuring the overlap between clinical entities and relationships.

[0070] Medical image generation metrics: For the generation task, we evaluate the synthesized images from three dimensions: fidelity, diversity, and semantic consistency.

[0071] Visual fidelity (FD-RadDino, KD-RadDino) measures the distributional distance between synthetic and real images; this paper uses Fraser distance (FD) and kernel distance (KD). Unlike the general evaluation metrics based on Inception-V3, this study uses the RadDino feature space for evaluation. RadDino is a visual Transformer model pre-trained specifically for large-scale chest X-ray data. This approach ensures that the evaluation focuses on medically relevant visual features, rather than the general statistical features of ordinary natural images.

[0072] Image-Text Consistency (Alignment Score). To quantify the accuracy with which the generated image reflects the input medical prompt, we compute an alignment score. Specifically, we utilize a pre-trained BioViL-T image and text encoder to calculate the cosine similarity between the synthesized image and its corresponding text prompt embedding vector.

[0073] Diversity and Coverage (PRDC). To ensure that the model generates diverse samples rather than getting stuck in a single pattern, we report precision, recall, density, and coverage metrics for a few modes. These metrics provide a detailed analysis of the manifold overlap between the generated data distribution and the real data distribution.

[0074] Understanding: See the supplementary material for details on the training dynamics of the understanding branch. As training progresses, the cross-entropy loss steadily decreases, and the model's report generation ability initially improves rapidly before gradually stabilizing. Although continuous fine-tuning can further reduce the loss, we observe that key performance indicators begin to decline, suggesting that the model tends to overfit specific patterns rather than learn general principles.

[0075] Table II summarizes the understanding performance of various basic medical models. The basic medical model of this invention demonstrates superior performance compared to a unified model with a significantly reduced number of parameters. Compared to single-task models, its performance not only surpasses models of the same scale but also approaches that of larger-scale models. It should be noted that this paper does not cover research on medical intelligent agent systems, as these are typically built upon existing basic medical models and rely on multi-model collaborative mechanisms. A comparative analysis of this invention with such intelligent agent systems can be found in the supplementary materials.

[0076] This invention conducts extensive comparative experiments on purely interpretive, purely generative, and unified medical basic models. Figure 3 , Figure 4 This paper demonstrates the qualitative effects of the present invention on both understanding and generation tasks. The results show that the present invention can generate accurate medical reports and high-fidelity images. Detailed experimental results for the understanding and generation tasks will be described below.

[0077] Generation Phase: The study found that in both training phases, the mean squared error of the generated branches decreased rapidly in the early stage, and the rate of decrease gradually slowed down thereafter. During the medical generation pre-training process, key indicators showed limited improvement in the 25K to 50K training step range, but a significant leap occurred in the 50K to 75K training step range.

[0078] Table III reports the generation results. In comparison, in the LLM-CXR domain, the medical foundation model of this invention shows significant improvements in both image quality and image-text alignment. Its performance consistently outperforms single-task models, fully demonstrating the advantages of its decoupled collaborative architecture. Notably, the performance of the medical foundation model of this invention is comparable to the robust benchmark model Sana. Although Sana is also fine-tuned based on the target dataset, the medical foundation model of this invention not only maintains performance parity but even slightly surpasses it in some metrics. This indicates that our unified method perfectly balances top-tier generation quality with semantic consistency.

[0079] We further evaluated the quality of pathological feature generation in Table IV. The medical model of this invention consistently demonstrates higher generation fidelity across various lesion types, accurately capturing subtle pathological clues and preserving clinically relevant details. UniX and Sana remain highly competitive in these sophisticated tasks. Notably, UniX employs a unified multi-task objective framework, while Sana focuses on specific generation tasks. Experimental results show that UniX not only possesses powerful sophisticated visual synthesis capabilities but also maintains robust performance across diverse diagnostic scenarios.

[0080] Verification of the C unified instruction tuning method

[0081] To assess the necessity of multi-center unified instruction tuning (Phase 4), we analyzed the performance evolution starting from the standardized training baseline (Phase 3). Table V shows a comparative analysis between the source domain (mimic-CXR) and two unseen target domains (CheXpert and CXR-27).

[0082] Intra-domain trade-offs and synergies: In the source domain (mimic-CXR), we observe a subtle trade-off. After uniform tuning, the understanding performance shows a slight decrease (e.g., a 2.2% reduction in the Micro F1 score). This marginal decrease is due to the regularization effect introduced by the heterogeneous multicenter data distribution, which effectively prevents the model from overfitting to the mimic-CXR-specific reporting style. However, this diversity has a positive impact on the generation task: visual fidelity (FD-RadDino) improves by 0.9%, and image-text alignment improves by 2.4%. This suggests that while expanding the domain scope slightly weakens the specificity of intra-domain text fitting, it can improve the robustness of the synthesis results by enriching the visual feature space.

[0083] Target domain generalization ability: Unlike the minor fluctuations in the source domain, the model exhibits significant improvements on the outer datasets. Without the fourth-stage training, the model's transferability is limited. However, after post-tuning, its performance on the CheXpert and CXR-27 datasets improves dramatically, with micro-F1 scores increasing by 15.6% and 24.3%, respectively. These results confirm that the fourth-stage training effectively bridges domain differences, transforming the model from a single-center expert into a robust general medical foundation model.

[0084] D Comparison with medical intelligent agent systems

[0085] To further explore the practical application potential of the medical foundation model in this embodiment, we designed a collaborative workflow: the EVA-X chest X-ray expert model first identifies imaging features, and then the medical foundation model in this embodiment generates a medical report based on multimodal input. We compared this collaborative model of small and large models with state-of-the-art medical intelligence systems, and the results are summarized in Table VI. This design fully leverages the complementary advantages of both—the lightweight model excels in rapid and accurate diagnosis, while the large model possesses superior report generation capabilities.

[0086] The following is an example of the input prompts used in this stage:

[0087] Expert analysis: Myocardial hypertrophy is visible; edema and pneumonia are uncertain. Please review the X-ray and provide detailed examination results and diagnostic impressions.

[0088] Table II

[0089] Performance comparison of different medical foundation models in X-ray interpretation tasks. Data shows that the medical foundation model of this invention achieves a significant improvement in interpretation ability compared to a unified medical foundation model. Notably, despite having fewer parameters, its performance is comparable to larger-scale single-task medical foundation models. Missing values ​​are indicated by "-".

[0090] Table III

[0091] Performance comparison of different medical image generation models in X-ray generation tasks. In standardized benchmark tests, the UNIX model is comparable to single-task medical models in output quality. Furthermore, this model demonstrates superior performance in both accuracy and diversity. HEALTHGPT was not included in this test due to the lack of publicly available text-to-image generation code. * indicates that all generative models from the natural image domain have been finely tuned based on the same X-ray dataset.

[0092] Table IV

[0093] Performance evaluation of generative models based on pathology. Within the framework of a unified medical foundation model, the system of this invention outperforms in comparative analysis, achieving the best performance in 13 out of 14 categories. *Indicates that all these generative models derived from the natural image domain are fine-tuned based on the same X-ray dataset.

[0094] Table V

[0095] We compared the model's performance before (Phase 3) and after (Phase 4) the unified instruction fine-tuning phase to assess its effectiveness. (Left figure) On the source domain (MIMIC-CXR dataset), the model achieved better generative fidelity at the cost of a slight loss in comprehension accuracy. (Middle and Right figures) Crucially, on the target domain (CheXpert and CXR-27 datasets), the model's performance was significantly improved, validating that unified fine-tuning can significantly enhance the model's clinical generalization ability. Missing values ​​are marked with "-".

[0096] Table VI

[0097] This invention is compared with the medical reagent model when coordinated with EVA-X. Missing values ​​are indicated by "-". Table VII

[0098] We compared the model performance of two schemes—freezing the understanding branch (generation only) and joint optimization (understanding + generation dual branches)—under different data mixing ratios (understanding task: generation task). Note: The calculation of understanding-related evaluation metrics (micro F1 score / macro F1 score) followed the rule of treating "uncertain" labels as positive.

[0099] ablation experiment

[0100] The impact of data cleaning on comprehension ability

[0101] Figure 3 This study demonstrates the crucial role of DeepSeek data cleaning in alleviating hallucination symptoms. Raw hospital reports often contain significant noise, such as underlines and technical terminology. The presence of metadata and dialogue filler complicates the correspondence between visual features and textual descriptions. By employing targeted cues to remove these non-diagnostic elements, we construct a more concise and semantically dense target dataset for the model. This preprocessing step is critical because it forces the model to rigorously focus on clinically relevant patterns during training, resulting in fact-based, unstructured hallucination reports.

[0102] B. The impact of unfreezing the visual encoder

[0103] To explore the necessity of fine-tuning the visual encoder, we conducted ablation experiments in the first phase. In these experiments, we froze the visual encoder while keeping all parameters consistent with the baseline model. The results confirm that, despite the increased computational cost, unfreezing the encoder remains essential for the understanding task. Specifically, keeping the encoder frozen leads to a significant performance drop, with the micro-F1-14 score decreasing from 52.6 to 43.2. This gap stems from the fact that encoders pre-trained on natural images often struggle to capture the uniqueness of medical features, such as subtle pathological patterns and fine grayscale textures.

[0104] To investigate the impact of joint fine-tuning in the first and third stages, we designed multiple medical image generation experiments, each with 2K fine-tuning steps. The results in Table VIIlead reveal several key findings: First, the fine-tuned understanding branch should not participate in the subsequent training of the generation branch. Completely freezing the understanding branch can rapidly improve generation performance without compromising understanding accuracy; conversely, unfreezing the branch without providing understanding data has a negative impact—severely weakening understanding performance and offering no benefit to generation capability. Adjusting the ratio of understanding data to generation data can partially alleviate the decline in understanding performance, as semantic supervision effectively stabilizes the updated parameters. However, this setting forces the model to balance two competing objectives within the same update cycle, thus delaying the acquisition of strong generation capabilities.

[0105] in conclusion

[0106] We introduce a new generation of unified medical foundation model, achieving architectural decoupling and synergy between chest X-ray image understanding and generation tasks. Existing unified medical foundation models fail to resolve inherent contradictions or fully leverage the advantages of different modeling paradigms. To overcome these limitations, we designed a dual-branch architecture that combines autoregressive understanding with diffusion-based generation. This structure achieves task decoupling and avoids mutual interference, while introducing a cross-modal self-attention mechanism to align the two branches, enabling the understood features to dynamically guide the generation process. Through these designs, our medical foundation model achieves superior understanding and generation performance compared to traditional unified medical foundation models with fewer parameters, reaching a level of competitiveness with dedicated single-task medical foundation models. We expect our medical foundation model to provide a new perspective for the development of medical foundation models.

Claims

1. A medical foundational model for understanding and generating chest X-ray images, characterized in that: include The autoregressive understanding branch, built upon a visual language model, is responsible for handling semantic abstraction and reasoning reports; Diffusion-based generative branches, directly based on the language model backbone inherited from the autoregressive understanding branches, specialize in synthesizing high-fidelity chest X-ray images; The cross-modal self-attention module connects the autoregressive understanding branch and the generation branch, enabling dynamic feature exchange and semantic conditionalization during the generation process.

2. The medical basic model for understanding and generating chest X-ray images according to claim 1, characterized in that: In the autoregressive understanding branch, a multimodal lexical sequence S=[V,T] is defined. in ,T out ], where V and T in and T out Let m represent visual words, input text words, and output text words, respectively, and let T be a given string of words. out The starting index in sequence S, n is the ending index of S, and the cross-entropy loss is obtained through T. out The autoregressive prediction values ​​for all tokens in the text are calculated: Where n represents the index of the last word in sequence S, ω u This indicates the parameters for understanding the branch.

3. The medical basic model for understanding and generating chest X-ray images according to claim 1, characterized in that: In the diffusion-based generative branch, a latent diffusion framework is employed to reconstruct medical images by understanding the high-level semantic information extracted from the branches. The diffusion process does not occur in the pixel space but is completed in the latent space encoded by VAE, thus significantly improving efficiency and stability. Given a noise distribution p at time t... t The latent variable x sampled in (x) t The model learns to estimate the target velocity field u by minimizing the mean square error. t (x), , where ω g The parameters representing the generated branches, and the semantic embedding of branch generation as a conditional input, can achieve disease-specific synthesis and improve the accuracy of lesion localization.

4. The medical basic model for understanding and generating chest X-ray images according to claim 1, characterized in that: The cross-modal self-attention module can establish a bidirectional information flow between the understanding branch and the generation branch. By performing joint self-attention processing on a unified multimodal lexical sequence, the semantic representation generated by the understanding branch can directly control the generation trajectory. At the same time, the generation state can also be fed back to the semantic space for optimization. This design not only realizes the bidirectional interaction between semantic representation and generation process, but also ensures the dynamic adaptation of generation results to input semantics.

5. The medical basic model for understanding and generating chest X-ray images according to claim 4, characterized in that: In a cross-modal self-attention module, let the unified sequence be S=[T in [N], where T in S represents the text lexicon generated by the comprehension branch, and N represents the noisy conditional latent embedding of the generated branch. For each lexicon S i We compute the modality-specific projections of the query, key, and value: , The modality selector is defined as follows: This formalized method, while maintaining shared attention operations across a unified sequence, provides two distinct parameter spaces for understanding lexical units and generating lexical symbols. The resulting attention map is computed in standard form: However, all cross-modal interactions are achieved through implicit learning of joint attention scores, rather than through explicit conditioning. The cross-modal self-attention module improves the fidelity and clinical consistency of generated images by synchronizing the autoregressive branch for understanding with the diffusion branch for high-fidelity generation.

6. The medical basic model for understanding and generating chest X-ray images according to claim 1, characterized in that: It also includes a progressive optimization strategy, including standardized medical training and multi-center unified teaching optimization, which ensures that the basic medical model can first master robust basic capabilities before adapting to heterogeneous clinical environments.

7. The medical basic model for understanding and generating chest X-ray images according to claim 6, characterized in that: Standardized basic medical training includes: In the first stage, the medical understanding model is fine-tuned. During this stage, the generation branch remains frozen. By pairing medical images with clinical report data, the visual encoder, visual connector, and language model backbone in the understanding branch are optimized. This process constructs robust semantic anchors, enabling the model to master the ability to interpret medical images. Crucially, this transforms the understanding branch into a high-level semantic feature generator, providing direction for subsequent generation branches. In the second stage, medical image generation pre-training is performed. In this stage, the understanding branch is frozen and the generation branch is pre-trained using text-low resolution images. To accelerate the convergence process and ensure structural consistency, we adopt a representation alignment technique to align the hidden state of the generation branch with RadDino image features. This design enables the generation branch to effectively utilize high-level semantic information, thereby achieving coarse-grained medical image synthesis. In the third stage, based on the freezing strategy of the second stage, the text-high resolution image generation branch was fine-tuned; by expanding the positional encoding to support higher resolution and removing feature-level supervision to focus on texture refinement, the final model achieved high-fidelity synthesis effect: the lesion site was accurately presented and the alignment between the reported image and the actual image was significantly improved.

8. A medical basic model for understanding and generating chest X-ray images according to claim 6 or 7, characterized in that: Multi-center unified instruction tuning involves unfreezing all components of the understanding and generation branches, except for the VAE, and jointly tuning them using a clinical instruction hybrid scheme. This scheme integrates the understanding-centric task and the generation-centric task. This unified optimization strategy keeps the feature spaces of the two branches consistent, bridging the gap between standardized pre-training and real-world clinical diversity. This significantly improves the model's cross-domain generalization ability and enables multi-functional interactions, ensuring consistent performance across heterogeneous diagnostic schemes.

9. The medical basic model for understanding and generating chest X-ray images according to claim 1, characterized in that: Both the understanding branch and the generation branch are partially initialized from Janus-Pro. For the understanding branch, we use siglip-large-patch16-384 as the visual encoder, and its 1024-dimensional embedding vector is projected onto the 2048-dimensional large language model space through a two-layer MLP. The language backbone network contains 24 Transformer layers with QK normalization and QKV bias.

10. The medical basic model for understanding and generating chest X-ray images according to claim 9, characterized in that: The branch generation uses 16× downsampling and 16-channel VAE, and achieves bidirectional projection between the 16-dimensional latent space and the 2048-dimensional language space through two single-layer MLPs.

Images