Medical report generation method based on double-path multi-level semantic alignment

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a medical report generation method based on dual-path multi-level semantic alignment, which comprises the following steps: constructing a dual-path multi-modal feature extraction network, fusing static-dynamic collaborative paths of MedCLIP and a context sensing module and universal visual representation of ViT, and generating a medical report based on multi-level semantic alignment. Specific feature extraction and wide visual semantic understanding in the medical field are coordinated and balanced. Through a CLIP-driven hierarchical cross-modal alignment mechanism, global coarse-grained mapping based on comparative learning and local fine-grained interaction based on bidirectional attention are cooperatively executed, and heterogeneous visual features and text semantics are accurately projected to a unified metric space. The problems that feature representation is deficient and cross-modal alignment granularity is single in the prior art are solved. And multi-level alignment loss and autoregression generation loss are integrated in an optimization target, so that the generated report is ensured to have high clinical fidelity and keep the continuity of language logic at the same time.

Description

Technical Field

[0001] This invention belongs to the fields of image processing and artificial intelligence technology, and particularly relates to a medical report generation method based on dual-path multi-level semantic alignment. Background Technology

[0002] Medical imaging, due to its non-invasive, intuitive, and efficient characteristics, has become an important tool for disease diagnosis, treatment decisions, and postoperative evaluation. To fully utilize the diagnostic value of medical images, generating comprehensive and accurate medical reports is crucial. The core task of medical report generation is to accurately extract complex pathological information from medical images and transform it into a structured report. In clinical practice, radiologists face heavy workloads; interpreting images and writing reports is time-consuming and error-prone, given limited medical resources and an increasing number of patients. Therefore, automated medical report generation technology has received widespread attention, with the expectation of reducing physician workload, improving diagnostic efficiency, and minimizing human error.

[0003] Current medical report generation primarily employs an encoder-decoder paradigm, using deep neural networks to convert medical images into diagnostic text. With the widespread adoption of the Transformer architecture, the mainstream approach has evolved from the traditional CNN-RNN combination to a fusion framework of Vision Transformer and pre-trained models. Recent research has mainly focused on three aspects: visual feature extraction, cross-modal feature fusion, and explicit modeling of medical knowledge.

[0004] Despite the initial progress achieved by the aforementioned methods, the following core challenges remain: First, existing single-path feature extraction methods struggle to simultaneously model domain-specific medical features and general visual semantics. This leads to incomplete understanding of complex pathological patterns and may result in the omission of crucial diagnostic information. Furthermore, current alignment strategies mostly operate at a single granular level, typically establishing only global or local associations, lacking the ability to collaboratively integrate multi-level semantics. Global coarse-grained alignment can capture the macroscopic correspondence between the overall image content and report structure, but it ignores the fine-grained association between specific lesion features and corresponding clinical terms; while focusing solely on fine-grained alignment may lack the coherence of the global context.

[0005] Therefore, this invention aims to provide a medical report generation method based on dual-path multi-level semantic alignment, in order to solve the problem that existing single-path feature extraction methods are unable to simultaneously model domain-specific medical features and general visual semantics. Most current alignment strategies operate at a single granularity level, usually only establishing global or local associations, lacking the ability to collaboratively integrate multi-level semantics. Furthermore, while global coarse-grained alignment can capture the macroscopic correspondence between the overall image content and report structure, it ignores the fine-grained association between specific lesion features and corresponding clinical terms. On the other hand, focusing only on fine-grained alignment may lack the technical problem of coherence of the global context. Summary of the Invention

[0006] The purpose of this invention is to provide a medical report generation method based on dual-path multi-level semantic alignment, in order to solve the technical problems existing in the prior art.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A medical report generation method based on dual-path multi-level semantic alignment includes the following steps: S1: Acquire the medical image to be diagnosed, input it into the general visual pathway and the medical-specific pathway respectively, and extract the general visual features and the medical-specific features containing fine-grained pathological information respectively. The general visual features include image block features. S2: Obtain the diagnostic report text sequence corresponding to the medical image, and extract text word embedding features, including deep text semantic features and cross-modal text features; S3: Construct a unified mapping mechanism, receive visual features and text word embedding features, project them onto a unified CLIP semantic space, and eliminate heterogeneous distribution differences through a feature stabilization module to obtain stabilized features respectively; S4: Based on stabilization features, a multi-head cross-modal attention mechanism is used to establish the semantic correspondence between images and text at the global level, generating globally aligned visual features and globally aligned text features. S5: Extract the image block features from step S1 and the text word embedding features from step S2. After processing by the mapping strategy in step S3, use the bidirectional cross-modal attention mechanism to capture the fine-grained correspondence between local pathological regions and text descriptions to generate local enhancement features. S6: Fuse the global features obtained in step S4 with the local enhancement features obtained in step S5 to construct a multimodal joint representation. Input the representation into the Transformer decoder to generate a prediction report. Then, train the model end-to-end using a total objective function that includes global alignment loss, fine-grained alignment loss, and generation loss.

[0008] Preferably, the specific process of step S1 is as follows: S11: General Visual Feature Extraction: For a given medical image The Vision Transformer encoder, pre-trained on natural images, utilizes the global receptive field of its Transformer structure to extract general visual features. V vit ; S12: Basic Medical Feature Extraction: Extracting the same medical image I Input the MedCLIP visual encoder, which has been pre-trained using large-scale medical image-text contrastive learning, to extract basic features sensitive to medical semantics. V med ; S13: Context-Aware Feature Enhancement: Introduces a context-aware attention module and processes data through parallel branches. V med Static branching utilizes grouped convolution to extract local features. V static Dynamic branching utilizes convolution to generate value features V value ; S14: Medical-Specific Feature Aggregation: This method fuses static features with original medical features and constructs an enhanced contextual representation through spatial attention learning. V context Then, multi-scale attention weights are calculated. A And aggregated to obtain the final medical-specific features V cot .

[0009] Preferably, step S2 specifically includes: S21: Constructing the sequence modeling encoder channel: Obtaining the original medical report sequence Perform word embedding mapping on it and position encoding The data is then fed into a BERT encoder for deep sequence modeling to capture the complex semantic dependencies within medical reports. S22: Constructing a cross-modal alignment encoder channel: The same medical report sequence $R$ is input into the CLIP text encoder in parallel, leveraging its alignment capabilities learned on large-scale text-image pairs to extract text features. R Perform preprocessing operations to adapt the CLIP model. The cross-modal text features are then input into the encoder. T clip .

[0010] Preferably, the specific process of step S3 is as follows: S31: Constructing a unified mapping layer for heterogeneous features: targeting medical-specific features Vcot General visual features V vit and text semantic features T bert Medical-specific features are mapped to the CLIP semantic space through linear projection and regularization, respectively. V cot Projection yields features V m Using independent parameter matrices V vit Mapped to V v ,Will T bert Mapped to T b ; S32: Constructing a Feature Stabilization Module: Introducing a two-layer feedforward neural network (MLP) with residual connections to process the projected features and calculate stabilized features. V s : Repeat this step to obtain the stabilized general visual features. V v Medical-specific characteristics V s BERT text features T s CLIP text features T c This completes the preparation for unifying and aligning heterogeneous feature spaces.

[0011] Preferably, the specific process of step S4 is as follows: S41: Define semantic anchors A img If the CLIP feature exists, it is used first; otherwise, the stabilized feature obtained in step S32 is used. V s As anchor points, and to define semantic anchor points for text modalities. A txt ; S42: Cross-modal attention alignment: Utilizing a multi-head cross-modal attention mechanism, medical-specific features are mapped to a shared semantic space; Taking the image side as an example, the stabilized features obtained in step S32 V s For queries, using semantic anchors A img As keys and values, calculate alignment features. V align ; S43: Adaptive Global Fusion: Integrating Original Stabilized Features Vs Alignment features V align The data are stitched together and then generated using an adaptive fusion network to produce the final globally aligned visual features. V global Similarly, the globally aligned text features are calculated. T global ; S44: Global Consistency Constraints: Constructing a Global Alignment Loss Based on the InfoNCE Framework L global and image-to-text loss .

[0012] Preferably, step S5 specifically includes: S51: Intramodal context enhancement: Based on the image block feature sequence extracted in step S1 ,Include After the region features are mapped and stabilized as described in step S3, spatial dependencies are captured using a self-attention mechanism to obtain a context-enhanced visual representation. ; Context-enhanced text representation is obtained by processing the text word embedding sequence. ; S52: Bidirectional cross-modal attention alignment: Constructing an attention mechanism to establish patch-to-token level correspondences; calculating queries for the image-to-text direction. ,key Sum : Calculate the attention weight matrix Based on this, textual information is aggregated to obtain local enhancement features. ; S53: Adaptive fine-grained fusion: Utilizing learnable balancing coefficients to fuse the contextual features obtained in step S51 V s The local enhancement features obtained in step S52 V local The features are then fused to obtain the final fine-grained enhancement features. V enh And calculate fine-grained enhanced text features. T enh ; S54: Fine-grained consistency constraint: utilizing global average pooling The sequence features are aggregated into a compact vector, and the contrastive loss is calculated.

[0013] Preferably, step S6 specifically includes: S61: Construction of Joint Representation of Multimodal Features: The globally aligned features obtained in step S4 are then used to construct the joint representation. V global, T global After global average pooling, the data are concatenated to construct a unified multimodal joint semantic representation. Z joint ; S62: Autoregressive Generation: [This will...] Z joint As a conditional input to the Transformer decoder, at time step t Based on the generated word sequence Calculate the current word y t The generation probability; S63: Calculate the generation loss and the overall objective function: The generation loss is defined as the negative log-likelihood loss over the true reported sequence, and the overall objective function is constructed. L total .

[0014] The beneficial effects of this invention include: 1. By integrating the static-dynamic collaborative path of MedCLIP and the context-aware module (CoT) with the general visual representation of ViT, we can collaboratively balance the extraction of specific features in the medical field with a wide range of visual semantic understanding, thus making up for the shortcomings of traditional single-path encoders in capturing domain-specific pathological features.

[0015] 2. By designing a CLIP-driven hierarchical cross-modal alignment mechanism, we can collaboratively execute global coarse-grained mapping based on contrastive learning and local fine-grained interaction based on bidirectional attention. This accurately projects heterogeneous visual features and textual semantics into a unified metric space, thereby solving the core problems of insufficient feature representation and single granularity of cross-modal alignment in existing technologies.

[0016] 3. Multi-level alignment loss and autoregressive generation loss are integrated into the optimization objective to ensure that the generated report maintains the coherence of language logic while having high clinical fidelity, without relying on complex external knowledge bases.

[0017] 4. Compared with the most advanced report generation methods currently available, such as M2KT and R2Gen, the method of this invention achieves better performance. Significant improvements are particularly seen in natural language generation metrics (BLEU, METEOR) and clinical efficacy metrics (F1-score) for measuring disease diagnostic accuracy. Furthermore, it effectively suppresses the hallucinations and over-inferences in pathological descriptions commonly found in traditional methods. This can significantly reduce the workload of radiologists, accelerate the decision-making process, improve diagnostic efficiency, and minimize misdiagnosis caused by human factors. Attached Figure Description

[0018] Figure 1 This is a system architecture diagram provided in an embodiment of the present invention; Figure 2 These are the subjective experimental results on the IU Xray, MIMIC-CXR, and LGK datasets provided in this embodiment of the invention. Figure 3 This is a visualization result of attention mapping on the MIMIC-CXR and LGK datasets provided in an embodiment of the present invention. Detailed Implementation

[0019] The following is in conjunction with the appendix Figures 1-3 The present invention will be further described in detail below: Example 1 See appendix Figure 1 As shown, the medical report generation method based on dual-path multi-level semantic alignment includes the following steps: S1: Dual-path visual feature extraction: acquiring medical images for diagnosis. I These features are then input into the general visual pathway and the medical-specific pathway, respectively; the general visual features are extracted using the Vision Transformer. V vit Simultaneously, the MedCLIP encoder, combined with a context-aware attention module, is used to extract medical-specific features containing fine-grained pathological information. V cot This provides a multi-level visual representation for subsequent cross-modal alignment; S2: Dual-path text feature extraction: Obtaining the diagnostic report text sequence corresponding to the medical image. R These are then fed into a sequence modeling encoder and a cross-modal alignment encoder, respectively; deep textual semantic features are extracted using the BERT model. T bert Simultaneously, the CLIP text encoder is used to extract cross-modal text features. T clip To balance the semantic dependence of medical terminology with cross-modal alignment capabilities; S3: Unified mapping and stabilization of heterogeneous feature space: Construct a unified mapping mechanism, receive the visual features output from step S1 and the text features output from step S2, project them onto the unified CLIP semantic space, and eliminate heterogeneous distribution differences through the feature stabilization module to obtain the stabilized feature representations respectively. S4: CLIP-driven global cross-modal alignment: Based on the stabilized features output from step S3, a multi-head cross-modal attention mechanism is used to establish the semantic correspondence between the image and text at the global level, generating global alignment visual features. V global and globally aligned text features T global And semantic consistency is constrained by a global contrastive loss function; S5: Fine-grained local semantic alignment: Extract image block features from step S1 and text word embedding features from step S2. After processing by the mapping strategy in step S3, a bidirectional cross-modal attention mechanism is used to capture the fine-grained correspondence between local pathological regions and text descriptions, generating local enhancement features. V enh and T enh ; S6: Autoregressive Report Generation and Joint Optimization: The global features obtained in step S4 and the local features obtained in step S5 are fused to construct a multimodal joint representation. Z joint The model is then fed into the Transformer decoder to generate a prediction report, and the model is trained end-to-end using a total objective function that includes global alignment loss, fine-grained alignment loss, and generation loss.

[0020] In this embodiment, the specific process of step S1 is as follows: S11: General Visual Feature Extraction: For a given medical image The Vision Transformer (ViT) encoder, pre-trained on natural images, leverages the global receptive field of its Transformer structure to extract general visual features. V vit The calculation formula is: V vit = ViT ( I ); in, V vit This represents a feature vector that contains general visual semantics; S12: Medical Basic Feature Extraction: Input the same medical image into the MedCLIP visual encoder, which has been pre-trained using large-scale medical image-text contrast learning, to extract basic features sensitive to medical semantics. V med : ; S13: Context-Aware Feature Enhancement: To compensate for the limitations of MedCLIP's local receptive field, a context-aware attention module (CoT) is introduced. First, it processes data through parallel branches. V med Static branching utilizes grouped convolution to extract local features. V static Dynamic branching utilizes convolution to generate value features V value The formula is as follows: ; ; in, This represents the Sigmoid activation function. BN This indicates a batch normalization operation. Conv group This indicates a grouped convolution operation. Conv This represents the standard convolution operation; S14: Medical-Specific Feature Aggregation. This method fuses static features with original medical features and constructs an enhanced contextual representation through spatial attention learning. V context Then, multi-scale attention weights are calculated. A And aggregated to obtain the final medical-specific features V cot : ; ; in, This indicates element-wise addition. This indicates a tensor reorganization operation. Indicates in Average pooling operation in the spatial dimension, A The calculated spatial attention weight matrix is ​​used to weight the dynamic branch features.

[0021] Example 2 Based on Example 1, step S2 specifically includes: S21: Constructing the sequence modeling encoder channel: Obtaining the original medical report sequence First, perform word embedding mapping. and position encoding The data is then fed into the BERT encoder for deep sequence modeling to capture the complex semantic dependencies within the medical report. The calculation formula is as follows: ; in, R This represents the input text sequence. L For sequence length, T bert This represents the deep semantic sequence features of the output; further, the first CLS token vector of this sequence is selected as the global text representation. T global ,Right now ; S22: Constructing a cross-modal alignment encoder channel: The same medical report sequence $R$ is input into the CLIP text encoder in parallel, leveraging its alignment capabilities learned on large-scale text-image pairs to extract text features. R Perform preprocessing operations to adapt the CLIP model. The cross-modal text features are then input into the encoder. T clip : ; in, This includes word segmentation and standardization operations to ensure that the input format meets the requirements of the CLIP model.

[0022] The specific process of step S3 is as follows: S31: Construct a unified mapping layer for heterogeneous features: targeting the medical-specific features output from step S1. V cot General visual features V vit and the text semantic features output in step S21 T bert Due to their varying dimensions and semantic distributions, they are mapped to the CLIP semantic space through linear projection and regularization, respectively. (Using medical-specific features...) V cot For example, the projected features V m The calculation is as follows: ; in, W m and b m Let represent the learnable projective weight matrix and bias vector, respectively. LayerNorm Presentation layer normalization operation, Dropout This represents a random deactivation operation; similarly, using independent parameter matrices to... V vit Mapped to V v ,Will T bert Mapped to T b For the output of step S22 T clip Since it is already in the target space, only the execution layer normalization is obtained. ; S32: Constructing a feature stabilization module. To enhance the discriminative power of the projected features and prevent gradient vanishing, a two-layer feedforward neural network (MLP) with residual connections is introduced to process the projected features obtained in step S31. The projected medical features are then used as an example. V m For example, calculate the stabilization characteristics. V s : ; in, This represents a two-layer fully connected network containing a non-linear activation function; repeating this step yields the stabilized general visual features. V v Medical-specific characteristics V s BERT text features T s CLIP text features T c This completes the unification and alignment preparation of the heterogeneous feature space.

[0023] Example 3 Based on Example 1 or Example 2, step S4 specifically includes: S41: Semantic Anchor Definition: To establish a baseline for cross-modal alignment, semantic anchors are defined. A img If the CLIP feature exists, it is used first; otherwise, the stabilized feature obtained in step S32 is used. V s As an anchor point, the formula is expressed as: ; Similarly, define the semantic anchor point of the text modality. A txt ; S42: Cross-modal attention alignment: Utilizing multi-head cross-modal attention mechanism (MCA) to map medical-specific features to a shared semantic space; Taking the image side as an example, the stabilized features obtained in step S32 V s For queries, using semantic anchors A img Calculate alignment features as keys and values. V align : ; S43: Adaptive Global Fusion: Integrating Original Stabilized Features V s Alignment features V align The data are stitched together and then generated using an adaptive fusion network to produce the final globally aligned visual features. V global : ; in, For the learnable parameter matrix of the fusion network, This indicates a feature concatenation operation. GELU This is the activation function; similarly, the globally aligned text features are calculated.T global ; S44: Global Consistency Constraint. Constructing a global alignment loss based on the InfoNCE framework. L global Image-to-text loss The calculation is as follows: ; in, B For batch size, Represents the cosine similarity function. The temperature parameter is learnable; the total global loss consists of bidirectional losses: .

[0024] Preferably, step S5 specifically includes: S51: Intramodal context enhancement. Based on the image block feature sequence extracted in step S1. (Include After the mapping and stabilization processes described in step S3, the spatial dependencies of each region feature are captured using a self-attention mechanism (MHA) to obtain a context-enhanced visual representation. : ; in, Here, it refers to the processed local feature sequence; similarly, processing the text word embedding sequence yields a context-enhanced text representation. ; S52: Bidirectional cross-modal attention alignment. An attention mechanism is constructed to establish a patch-token level mapping. Taking the image-to-text direction as an example, the query is first calculated... ,key Sum : ; ; ; in, , , The projection matrix is ​​learnable; the attention weight matrix is ​​calculated. Based on this, textual information is aggregated to obtain local enhancement features. : ; ; S53: Adaptive fine-grained fusion. The contextual features obtained in step S51 are fused using learnable balancing coefficients. V sThe local enhancement features obtained in step S52 V local The features are then fused to obtain the final fine-grained enhancement features. V enh : ; in, The weight parameters for the contributions of the original features and the enhanced features are dynamically adjusted; similarly, fine-grained enhanced text features are calculated. T enh ; S54: Fine-grained consistency constraint. Utilizes global average pooling. Aggregate the sequence features into a compact vector and calculate the contrastive loss: ; in, Temperature parameters are used to control the sharpness of the distribution and to emphasize subtle local correspondences.

[0025] Preferably, step S6 specifically includes: S61: Construction of Joint Representation of Multimodal Features: The globally aligned features obtained in step S4 are then used to construct the joint representation. V global , T global After global average pooling, the data are concatenated to construct a unified multimodal joint semantic representation. Z joint : ; in, Concat This represents a vector concatenation operation. GAP Indicates global average pooling; S62: Autoregressive Generation: [This will...] Z joint As a conditional input to the Transformer decoder, at time step t Based on the generated word sequence Calculate the current word y t The generation probability: in, Indicates the preceding t- One generated token, These are the learnable parameters of the decoder; S63: Calculate the generation loss and the overall objective function: The generation loss is defined as the negative log-likelihood loss over the true reported sequence. ; To collaboratively optimize the quality of feature representation and generation, a general objective function is constructed.L total : ; in, , , These are hyperparameters used to balance the contributions of global alignment loss, fine-grained alignment loss, and generation loss to model optimization.

[0026] This invention organically embeds a dual-path multi-level semantic alignment mechanism into the core optimization objective of a medical report generation model, realizing a complete technical path from heterogeneous multimodal feature fusion theory to specific neural network implementation.

[0027] In the specific implementation process, given the input medical image data, these images are first transmitted to a complementary feature representation space through a constructed heterogeneous dual-path encoder. The goal is to train an autoregressive generator to predict the probability of generating the current clinical vocabulary. This probability is conditioned on the joint multimodal features after multi-level alignment and the preceding vocabulary, i.e., maximizing the generation probability distribution. ; in, This is the joint feature representation after global and local collaborative alignment. This represents the previously generated word sequence. These are the model parameters. Specifically, the probability distribution function aims to maximize the posterior probability of the generated sequence and the real clinical report in the semantic space. In actual training, this objective is transformed into minimizing the negative log-likelihood loss.

[0028] To ensure accurate topological mapping between heterogeneous visual features and textual semantics at multiple granularities, thereby addressing the issues of insufficient feature representation and single alignment granularity in traditional methods, this invention introduces a CLIP-driven hierarchical semantic alignment mechanism as theoretical guidance and defines a dual constraint space: let V and T define the feature distributions on the visual and textual manifolds, respectively. This invention constructs a joint constraint that includes global coarse-grained mapping and local fine-grained interaction. The global constraint aims to establish a macroscopic homeomorphism between overall image features and overall text embedding, using contrastive learning to project domain-specific medical representations onto a unified CLIP semantic space; the fine-grained local constraint, through a bidirectional attention mechanism, establishes an accurate correspondence between local image patches and discrete text tags.

[0029] The core technological contribution of this invention lies in establishing a systematic mapping mechanism from an abstract heterogeneous feature space to a unified semantic manifold. Through a carefully designed technical solution, the medical-specific paths and general visual paths within the theoretical framework, as well as the sequence modeling and cross-modal bridges on the text side, are respectively mapped to specific feature sets in the actual training process. Specifically, this invention utilizes the static and dynamic feature flows extracted by MedCLIP, and performs collaborative fusion through the contextual attention module CoT to construct a feature distribution with high pathological sensitivity. Simultaneously, it uses CLIP pre-training knowledge as semantic anchors to guide the projection and normalization of heterogeneous features, thereby mathematically ensuring the measurability of different modal features on the same manifold. To enable the model to possess accurate cross-modal semantic recognition capabilities in subsequent training stages and to ensure that the generated medical reports reside on the data manifold constituted by the real clinical context, this invention employs a hierarchical joint optimization strategy for end-to-end supervision of the network.

[0030] Based on the above-mentioned mapping relationship from theory to implementation, this invention fully embeds the multi-level alignment mechanism into the optimization objective of the Transformer generative network. Through mathematical transformation and engineering implementation, the theoretical formula is transformed into the following practically executable training objective function: ; in, , , These are hyperparameter weights that control the balance between different alignment granularities and the generation task. Each component of this objective function has a strict one-to-one correspondence with the theoretical definition of hierarchical alignment: Corresponding to the macroscopic mapping of the global semantic space, the InfoNCE loss is used to shorten the distance between matching image-text pairs in the feature space; Corresponding to local fine-grained interactions, a bidirectional attention mechanism is used to enhance the precise association between pathological regions and clinical terms; and This is the maximum likelihood estimate generated by autoregression.

[0031] Unlike existing technologies that rely on a single feature extraction path or single-granularity alignment, this invention achieves feature complementarity and deep semantic fusion through an innovative design of dual-path collaboration and multi-level alignment. This design not only utilizes the MedCLIP path to capture subtle pathological features specific to the domain, but also maintains general visual understanding capabilities through the ViT path, and forces the model to perform semantic calibration simultaneously in both global and local dimensions. This multi-level alignment approach theoretically ensures that the optimization process effectively avoids common technical challenges in traditional methods, such as semantic alignment misalignment and pathological description illusions, significantly improving the model's ability to capture complex pathological patterns and the clinical fidelity of generated reports.

[0032] Through the complete technical implementation path described above, this invention successfully transforms dual-path multi-level semantic alignment from a structural design concept to a practically deployable neural network training algorithm, providing a technical solution with high clinical accuracy and practical engineering feasibility for the automatic generation of medical reports.

[0033] Figures 2 and 3 respectively show the qualitative comparison and attention mapping visualization results of report generation on the MIMIC-CXR, IU X-ray, and LGK medical image datasets. Figure 2 The first part of the paper demonstrates report generation examples of this invention on the MIMIC-CXR and IUX-ray datasets. The highlighted text in green, blue, orange, red, and purple corresponds to the bone lesions, pleural effusion, lung abnormalities, pneumothorax, and heart-related lesions correctly identified by the model, respectively. Compared with three mainstream methods—R2Gen (published by Chen et al. at EMNLP 2020), CMN (published by Chen et al. at ACL 2021), and M2KT (published by Yang et al. at MIA 2023)—the reports generated by this method cover more accurate pathological details, are closer to real reports in clinical semantics, and effectively avoid pathological omissions and erroneous descriptions common in existing methods. Figure 3 shows the attention mapping visualization results of this invention on the MIMIC-CXR and LGK ultrasound datasets. This figure intuitively shows how the model accurately maps the generated text descriptions back to the visual cue regions in the image. In the diagram, the color gradient from blue to red represents a progression from low to high weight values. Specifically, warm-toned (red) areas indicate areas where the model gives high priority when generating corresponding pathological descriptions, while cool-toned (blue) areas indicate areas with lower priority. This visualization verifies that the present invention can accurately capture and focus on key lesion areas such as cardiomegaly, pneumonia, and gallstones through a dual-path multi-level alignment mechanism, thereby generating highly interpretable medical reports.

Claims

1. A medical report generation method based on dual-path multi-level semantic alignment, characterized in that, Includes the following steps: S1: Acquire the medical image to be diagnosed, input it into the general visual pathway and the medical-specific pathway respectively, and extract the general visual features and the medical-specific features containing fine-grained pathological information respectively. The general visual features include image block features. S2: Obtain the diagnostic report text sequence corresponding to the medical image, and extract text word embedding features, including deep text semantic features and cross-modal text features; S3: Construct a unified mapping mechanism, receive visual features and text word embedding features, project them onto a unified CLIP semantic space, and eliminate heterogeneous distribution differences through a feature stabilization module to obtain stabilized features respectively; S4: Based on stabilization features, a multi-head cross-modal attention mechanism is used to establish the semantic correspondence between images and text at the global level, generating globally aligned visual features and globally aligned text features. S5: Extract the image block features from step S1 and the text word embedding features from step S2. After processing by the mapping strategy in step S3, use the bidirectional cross-modal attention mechanism to capture the fine-grained correspondence between local pathological regions and text descriptions to generate local enhancement features. S6: Fuse the global features obtained in step S4 with the local enhancement features obtained in step S5 to construct a multimodal joint representation. Input the representation into the Transformer decoder to generate a prediction report. Then, train the model end-to-end using a total objective function that includes global alignment loss, fine-grained alignment loss, and generation loss.

2. The medical report generation method based on dual-path multi-level semantic alignment according to claim 1, characterized in that, The specific process of step S1 is as follows: S11: General Visual Feature Extraction: For a given medical image The Vision Transformer encoder, pre-trained on natural images, utilizes the global receptive field of its Transformer structure to extract general visual features. V vit ; S12: Basic Medical Feature Extraction: Extracting the same medical image I Input the MedCLIP visual encoder, which has been pre-trained using large-scale medical image-text contrastive learning, to extract basic features sensitive to medical semantics. V med ; S13: Context-Aware Feature Enhancement: Introduces a context-aware attention module and processes data through parallel branches. V med Static branching utilizes grouped convolution to extract local features. V static Dynamic branching utilizes convolution to generate value features V value ; S14: Medical-Specific Feature Aggregation: This method fuses static features with original medical features and constructs an enhanced contextual representation through spatial attention learning. V context Then, multi-scale attention weights are calculated. A And aggregated to obtain the final medical-specific features V cot .

3. The medical report generation method based on dual-path multi-level semantic alignment according to claim 1, characterized in that, Step S2 specifically includes: S21: Constructing the sequence modeling encoder channel: Obtaining the original medical report sequence Perform word embedding mapping on it and position encoding The data is then fed into a BERT encoder for deep sequence modeling to capture the complex semantic dependencies within medical reports. S22: Constructing a cross-modal alignment encoder channel: The same medical report sequence $R$ is input into the CLIP text encoder in parallel, leveraging its alignment capabilities learned on large-scale text-image pairs to extract text features. R Perform preprocessing operations to adapt the CLIP model. The cross-modal text features are then input into the encoder. T clip .

4. The medical report generation method based on dual-path multi-level semantic alignment according to claim 1, characterized in that, The specific process of step S3 is as follows: S31: Constructing a unified mapping layer for heterogeneous features: targeting medical-specific features V cot General visual features V vit and text semantic features T bert Medical-specific features are mapped to the CLIP semantic space through linear projection and regularization, respectively. V cot Projection yields features V m Using independent parameter matrices V vit Mapped to V v ,Will T bert Mapped to T b ; S32: Constructing a Feature Stabilization Module: Introducing a two-layer feedforward neural network (MLP) with residual connections to process the projected features and calculate stabilized features. V s : Repeat this step to obtain the stabilized general visual features. V v Medical-specific characteristics V s BERT text features T s CLIP text features T c This completes the preparation for unifying and aligning heterogeneous feature spaces.

5. The medical report generation method based on dual-path multi-level semantic alignment according to claim 4, characterized in that, The specific process of step S4 is as follows: S41: Define semantic anchors A img If the CLIP feature exists, it is used first; otherwise, the stabilized feature obtained in step S32 is used. V s As anchor points, and to define semantic anchor points for text modalities. A txt ; S42: Cross-modal attention alignment: Utilizing a multi-head cross-modal attention mechanism, medical-specific features are mapped to a shared semantic space; On the image side, the stabilized features obtained in step S32 V s For queries, using semantic anchors A img As keys and values, calculate alignment features. V align ; S43: Adaptive Global Fusion: Integrating Original Stabilized Features V s Alignment features V align The data is then stitched together, and an adaptive fusion network is used to generate the final globally aligned visual features. V global And calculate the global aligned text features. T global ; S44: Global Consistency Constraints: Constructing a Global Alignment Loss Based on the InfoNCE Framework L global and image-to-text loss .

6. The medical report generation method based on dual-path multi-level semantic alignment according to claim 1, characterized in that, Step S5 specifically includes: S51: Intramodal context enhancement: Based on the image block feature sequence extracted in step S1 ,Include After the region features are mapped and stabilized as described in step S3, spatial dependencies are captured using a self-attention mechanism to obtain a context-enhanced visual representation. ; Context-enhanced text representation is obtained by processing the text word embedding sequence. ; S52: Bidirectional cross-modal attention alignment: Constructing an attention mechanism to establish patch-to-token level correspondences; calculating queries for the image-to-text direction. ,key Sum : Calculate the attention weight matrix Based on this, textual information is aggregated to obtain local enhancement features. ; S53: Adaptive fine-grained fusion: Utilizing learnable balancing coefficients to fuse the contextual features obtained in step S51 V s The local enhancement features obtained in step S52 V local The features are then fused to obtain the final fine-grained enhancement features. V enh And calculate fine-grained enhanced text features. T enh ; S54: Fine-grained consistency constraint: utilizing global average pooling The sequence features are aggregated into a compact vector, and the contrastive loss is calculated.

7. The medical report generation method based on dual-path multi-level semantic alignment according to claim 1, characterized in that, Step S6 specifically includes: S61: Construction of Joint Representation of Multimodal Features: The globally aligned features obtained in step S4 are then used to construct the joint representation. V global , T global After global average pooling, the data are concatenated to construct a unified multimodal joint semantic representation. Z joint ; S62: Autoregressive Generation: [This will...] Z joint As a conditional input to the Transformer decoder, at time step t Based on the generated word sequence Calculate the current word y t The generation probability; S63: Calculate the generation loss and the overall objective function: The generation loss is defined as the negative log-likelihood loss over the true reported sequence, and the overall objective function is constructed. L total .

Images