A medical image report generation method based on a double graph encoder

Abstract

This invention relates to a medical image report generation method based on a dual-image encoder, comprising: acquiring a current chest X-ray image and historical chest X-ray images; inputting the current chest X-ray image and historical chest X-ray images into a medical image report generation model to obtain radiology report text corresponding to the current chest X-ray image; wherein, the medical image report generation model includes: a single-modal encoding module for extracting image features of the current chest X-ray image and historical chest X-ray images, as well as text features of the radiology report text; a cross-modal encoding module for performing cross-modal fusion of the image features and text features to obtain multimodal features incorporating visual difference information; and a text decoding module for generating the radiology report text corresponding to the current chest X-ray image based on the multimodal features incorporating visual difference information. This invention enables the modeling of image difference information and the generation of corresponding radiology reports.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and medical artificial intelligence, and in particular to a method for generating medical image reports based on a dual-image encoder. Background Technology

[0002] Longitudinal Radiological Report Generation (LRRG) is a comprehensive task integrating medical image understanding, temporal modeling, and natural language generation. Its goal is to generate radiological reports reflecting the evolution of a patient's condition based on multi-temporal medical images. Compared to static report generation tasks, longitudinal tasks require models not only to accurately understand individual medical images but also to effectively characterize the subtle changes between images at different time points.

[0003] Early related studies mostly drew on general visual change detection methods, combining them with traditional CNN visual feature extraction, RNN or LSTM temporal modeling techniques. These methods provided initial ideas for longitudinal medical image analysis, but due to the characteristics of medical images, such as highly similar backgrounds and localized and subtle lesion changes, general methods are difficult to directly adapt to complex medical scenarios.

[0004] Some studies have attempted to introduce explicit visual-text alignment mechanisms, such as relying on object detection or anatomical structure segmentation, to map image regions to disease labels or report sentences. CheXRelNet utilizes local anatomical region information between and within images, learning dependencies between region attributes through a graph attention network to track longitudinal pathological changes in chest images; RECAP constructs a change graph containing historical and current image observations, disease progression information, and related attributes, and designs a dynamic inference mechanism to generate reports. Such methods typically rely on dense manual annotation or external detection models, and their errors are easily amplified in multi-stage processing. Furthermore, the introduction of explicit bounding boxes and graph structures limits the model's ability to model global visual context and pathological relationships to some extent.

[0005] To reduce reliance on manual annotation and external models, recent research has increasingly shifted towards longitudinal modeling within a vision-language pre-training framework, achieving implicit alignment between images and text through self-supervised methods. BioViL-T employs a CNN-Transformer hybrid multi-image coding structure, fusing static structural features with dynamic change features at the image level to improve the consistency between image and text representations. TempA-VLP and Med-ST further enhance the model's ability to characterize the evolution of disease processes through time-aware contrastive learning and cross-modal cyclic consistency objectives.

[0006] Overall, research on longitudinal radiological image report generation is still relatively limited, and existing technologies still face several key challenges: First, spatial and semantic misalignment is common between images across time periods, necessitating fine-grained alignment without relying on explicit registration or detection; second, how to effectively distinguish and integrate static image findings with dynamic disease changes during report generation still requires in-depth research. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a medical image report generation method based on a dual-image encoder, which can realize the modeling of image difference information and the generation of corresponding radiological reports.

[0008] The technical solution adopted by this invention to solve its technical problem is: to provide a method for generating medical image reports based on a dual-image encoder, comprising the following steps:

[0009] Obtain current and historical chest X-ray images;

[0010] The current chest X-ray image and historical chest X-ray images are input into a medical image report generation model to obtain the radiology report text corresponding to the current chest X-ray image; wherein, the medical image report generation model includes:

[0011] A single-modal coding module is used to extract image features of the current chest X-ray image and historical chest X-ray images, as well as text features of the radiology report text;

[0012] A cross-modal coding module is used to fuse the image features and text features across modalities to obtain multimodal features that fuse visual difference information;

[0013] The text decoding module is used to generate the radiology report text corresponding to the current chest X-ray image based on the multimodal features that fuse visual difference information;

[0014] The image features of the current chest X-ray image and historical chest X-ray images, the text features of the radiology report text, and the multimodal features that integrate visual difference information are used to adjust the parameters of the medical image report generation model.

[0015] The single-modal coding module includes:

[0016] A single-image encoder is used to extract features from the current chest X-ray image and the historical chest X-ray image respectively to obtain current visual features and historical visual features;

[0017] A dual-image encoder is used to add positional and temporal encoding to the current visual features and historical visual features, and then achieve implicit alignment between the current visual features and historical visual features through a cross-attention mechanism to obtain difference-aware visual features.

[0018] A text encoder is used to semantically encode radiological report text to obtain text features.

[0019] The cross-modal coding module includes:

[0020] A text-image cross-modal encoder is used to inject visual information related to image changes into the text features through a cross-attention mechanism, using the text features as queries and the difference-aware visual features as keys and values, to obtain fused text features.

[0021] A cross-modal encoder is used to use the difference-aware visual features as queries and text features as keys and values, so that the difference-aware visual features are constrained by linguistic semantics during the update process, and visual noise unrelated to clinical description is suppressed.

[0022] The dual-image encoder and the image-text cross-modal encoder share self-attention parameters; the image-text cross-modal encoder and the text encoder share self-attention parameters; the image-text cross-modal encoder and the text decoding module share cross-modal cross-attention parameters.

[0023] The medical image report generation model is trained using a multi-task joint training method, which includes:

[0024] The masked report modeling training task is used to learn the contextual semantic structure of radiology reports on the text side and to constrain the medical image report generation model to the differentially perceived visual features when predicting masked terms.

[0025] The masked image difference modeling training task is used to set the masked target in the difference-aware visual feature space, so that the medical image report generation model learns the difference features of the current chest X-ray image relative to historical chest X-ray images.

[0026] An image-report contrastive learning training task is used to constrain the difference-aware visual features and the corresponding report representations to be in the same embedding space to maintain global semantic consistency.

[0027] Image-report matching prediction training task, used to determine at the instance level whether a given image pair strictly matches a report text;

[0028] A report generation training task is used to generate radiological report text based on the multimodal features of the fused visual difference information.

[0029] The masked report modeling training task adopts a domain-aware masking strategy, which includes: pre-constructing a change vocabulary related to changes in the disease condition; for each report text, prioritizing the high-probability masking of words in the change vocabulary; and performing hierarchical masking of the remaining words according to their semantic importance, setting different masking probabilities based on the importance interval.

[0030] The mask image difference modeling training task includes:

[0031] Perform random region masking on the current chest X-ray image to obtain the masked current chest X-ray image;

[0032] The masked current chest X-ray image and historical chest X-ray images are input into a single image encoder to obtain the corresponding difference-perceived visual features.

[0033] The difference-perceived visual features and the corresponding text features of the radiology report are input into the image-text cross-modal encoder to obtain the prediction results;

[0034] Using the unmasked input to obtain the prediction result as the target, the reconstruction error between the predicted features and the target features is calculated only at the masked location, and the parameters of the medical image report generation model are updated accordingly.

[0035] The image-report matching prediction training task includes:

[0036] Use real paired image pairs and their corresponding reports as positive samples;

[0037] Within a mini-batch, image pairs are randomly recombined with non-corresponding reports to obtain negative samples;

[0038] A matching marker is added before the text sequence, and the text features with the matching marker and the difference-aware visual features are input into the cross-modal coding module for joint modeling;

[0039] The multimodal features corresponding to the matching markers, which incorporate visual difference information, are used as the representation of the image-text matching state. The binary classification prediction head is then input to output the matching probability.

[0040] The loss is calculated based on the matching probability and the true matching label, and the parameters of the medical image report generation model are updated.

[0041] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned medical image report generation method based on dual-image encoder.

[0042] The technical solution adopted by the present invention to solve its technical problem is: to provide a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned medical image report generation method based on a dual-image encoder.

[0043] Beneficial effects

[0044] By employing the above-mentioned technical solutions, this invention has the following advantages and positive effects compared with existing technologies: This invention uses a dual-image differential coding mechanism to jointly model the current image and historical reference images, enabling the generated report to more accurately reflect the improvement, stabilization, or deterioration trend of the disease, thus improving the reliability of the report at the disease semantic level. Through structural differential modeling, it effectively utilizes longitudinal information, achieving stable results under normal input scale conditions, which helps reduce computational resource consumption and system deployment costs. This invention introduces cross-modal alignment and matching constraints through multi-task joint training, enabling the generated report to maintain linguistic fluency while better aligning with image evidence, reducing semantic mismatches or omissions of changes. Attached Figure Description

[0045] Figure 1 This is a flowchart of the medical image report generation method based on a dual-image encoder according to the first embodiment of the present invention;

[0046] Figure 2 This is a framework diagram of the medical image report generation model in the first embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of multi-task joint training in the first embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0049] The first embodiment of the present invention relates to a method for generating medical image reports based on a dual-image encoder, such as... Figure 1 As shown, it includes the following steps:

[0050] Step 1: Obtain the current chest X-ray image and historical chest X-ray images; the current chest X-ray image can be referred to as the current image, and the historical chest X-ray image can be referred to as the reference image.

[0051] Step 2: Input the current chest X-ray image and historical chest X-ray images into the medical image report generation model to obtain the radiology report text corresponding to the current chest X-ray image.

[0052] like Figure 2 As shown, the medical image report generation model in this embodiment includes:

[0053] A single-modal coding module is used to extract image features of the current chest X-ray image and historical chest X-ray images, as well as text features of the radiology report text;

[0054] A cross-modal coding module is used to fuse the image features and text features across modalities to obtain multimodal features that fuse visual difference information;

[0055] The text decoding module is used to generate the radiology report text corresponding to the current chest X-ray image based on the image features of the current chest X-ray image and historical chest X-ray images;

[0056] The textual features of the radiology report text and the multimodal features that integrate visual difference information are used to adjust the parameters of the medical image report generation model.

[0057] The single-modal encoding module in this embodiment includes a text encoder, a single-image encoder, and a dual-image encoder.

[0058] A text encoder is used to semantically encode radiology report text to obtain text features consistent with visual feature dimensions. The output of the text encoder includes global text features representing the semantics of the entire report and local text features representing the semantics of individual terms. The text encoder allows its parameters to be updated during training, thereby enhancing its ability to model expressions related to changes in the patient's condition in radiology reports and providing a semantic basis for subsequent cross-modal interactions.

[0059] A single-image encoder is used to extract visual features from the current image and the reference image separately to obtain current visual features and historical visual features that contain global structural information and local region semantics. In specific implementations, the current image and the reference image are input into the same single-image encoder for processing to ensure that they are represented within the same feature space.

[0060] The output of the single-image encoder is a sequence of visual features composed of multiple image feature units, including both global feature units representing the entire image and local feature units representing different image regions. This sequence of visual features serves as the basic input for subsequent difference modeling modules. In a preferred embodiment, the single-image encoder maintains its parameters unchanged during training to leverage its pre-learned stable visual semantic representations on large-scale medical image data, while simultaneously reducing overall training computational overhead.

[0061] The dual-image encoder is used to model the differences between the current image and the reference image. In its implementation, the dual-image encoder takes the current visual features and historical visual features output by the single-image encoder as input, and achieves implicit alignment between the two images through a cross-attention mechanism without relying on explicit lesion annotation or region-level matching.

[0062] To incorporate spatial and temporal prior information, this implementation adds positional and temporal codes to the current visual features and historical visual features respectively before difference modeling begins. The temporal code is used to indicate the temporal differences between the current image and the reference image, thereby establishing a clear temporal relationship at the feature level.

[0063] In the difference modeling process, historical visual features remain unchanged as a stable reference, while current visual features are updated layer by layer in a multi-layered cross-attention structure. Each layer of the cross-attention module uses the current visual features as the query and historical visual features as the key and value, thereby gradually fusing the change information generated compared to the reference image into the current image features. Through the above asymmetric modeling method, the dual-image encoder can simultaneously express the medical observation information of the current image and the longitudinal change information relative to the reference image within a unified feature space, effectively reducing the interference introduced by differences in imaging conditions.

[0064] The cross-modal coding module in this embodiment is used to realize bidirectional interaction between visual features and text features, and it includes a text-image cross-modal encoder and a text-image cross-modal encoder.

[0065] The text-image cross-modal encoder uses the text features as queries and the difference-aware visual features as keys and values. Through a cross-attention mechanism, visual information related to image changes is injected into the text features to obtain fused text features. This enables the text representation to focus on visual areas related to changes in the condition.

[0066] The image-text cross-modal encoder uses the difference-aware visual features as queries and text features as keys and values, so that the difference-aware visual features are constrained by linguistic semantics during the update process, suppressing visual noise that is irrelevant to the clinical description.

[0067] Both the text-image cross-modal encoder and the image-text cross-modal encoder mentioned above adopt a multi-layer stacked structure and are selectively called in different training tasks to support operations such as image-text alignment, matching prediction, and mask modeling.

[0068] In this embodiment, the text decoding module is used to generate radiological report text based on multimodal features that fuse visual difference information. During the generation process, the text decoding module takes the already generated text content as input and gradually predicts subsequent words through a causal attention mechanism. At the same time, it continuously utilizes the visual features of difference perception through cross-attention to guide the generated content to focus on the clinical observation results of the current image and its changes relative to the reference image.

[0069] To enable the medical image report generation model to stably learn disease progression information from longitudinal chest X-ray images and achieve cross-modal reasoning and generation with radiological report text, this implementation adopts a multi-task joint training method. Multiple complementary training objectives are set within the same model framework to collaboratively constrain the model's differential modeling ability, image-text semantic alignment ability, and report generation ability. By simultaneously applying training signals at three levels—visual representation learning, cross-modal semantic consistency constraints, and text generation—the problems of insufficient supervision and convergence instability caused by relying on a single training objective can be reduced.

[0070] The multi-task joint training includes at least the following five types of training tasks: 1) Masked Report Modeling (MRM); 2) Masked Image Difference Modeling (MIDM); 3) Image-Report Contrastive Learning (IRC); 4) Image-Report Matching Prediction (IRM); and 5) Report Generation (RG).

[0071] like Figure 3 As shown, the above tasks can be functionally divided into three layers: representation modeling layer, cross-modal alignment layer, and generation layer. The representation modeling layer includes MRM and MIDM, used to enhance the learning of text semantic structure and differential visual representations; the cross-modal alignment layer includes IRC and IRM, used to constrain the global consistency and instance-level matching of images and text in a shared semantic space; and the generation layer includes RG, used to guide the model to generate report text containing clinical observations and longitudinal comparison information.

[0072] MRM is used to learn the contextual semantic structure of radiology reports on the text side and to constrain the medical image report generation model to the differentially perceived visual features when predicting masked words. Specifically, in a training sample, a "masked text sequence" is constructed based on the target report text. The masked text sequence, along with the current image and a reference image, are input into the model, causing the model to output the prediction results of the masked words. The model parameters are then updated based on the consistency between the prediction results and the true words.

[0073] To avoid the problem that uniform random masking leads to models relying solely on textual context to recover lexical units, thus weakening visual constraints, this implementation adopts a domain-aware masking strategy: prioritizing the masking of medical keywords with diagnostic value, thereby forming a contextual structure that relies more heavily on image evidence, forcing the model to focus on differentially perceived visual representations. This domain-aware masking strategy includes: pre-constructing a change vocabulary related to disease progression, including descriptive words representing improvement, stabilization, and deterioration; for each report text, prioritizing high-probability masking of lexical units that match the change vocabulary; and hierarchically masking the remaining lexical units according to semantic importance, where semantic importance can be calculated using term frequency-inverse document frequency (TF-IDF) based on corpus statistics, with different masking probabilities set according to importance intervals.

[0074] The above approach enhances the MRM task's reliance on visual evidence, thereby facilitating fine-grained alignment between textual representations and image change cues.

[0075] MIDM is used to enhance the model's ability to complete and infer longitudinal variations within a difference representation space. Unlike masked image modeling, which aims at pixel reconstruction, this implementation sets the mask target in a difference-aware feature space, enabling the model to learn the difference features of the current image relative to a reference image, rather than reconstructing image pixels. The MIDM includes the following steps:

[0076] S1) Perform random region masking only on the current image to obtain the masked current image;

[0077] S2) Input the masked current image and the reference image into the single image encoder to obtain the corresponding visual features;

[0078] S3) Input the above visual features and corresponding report text features into the image-text cross-modal coding module so that the model outputs the prediction results of the difference perception visual features;

[0079] S4) Using the differentially perceived visual features obtained from the unmasked input as the target, the reconstruction error between the predicted features and the target features is calculated only at the masked location, and the model parameters are updated accordingly.

[0080] To prevent representational instability during model reconstruction, a stop-gradient approach can be used on the target branch to maintain a stable reference for the target features. Through MIDM, the model is explicitly constrained to complete the masked regions in the differential representation space, thereby enhancing the dual-image differential coding module's ability to model real pathological changes.

[0081] IRC is used to constrain the difference-aware visual features and their corresponding report representations to be in the same embedding space to maintain global semantic consistency. Specifically, the global features on the image side and the global features on the text side are respectively input into the projection module and mapped to the same contrast space. Positive and negative sample pairs are constructed through similarity measurement, so that the similarity of matching image-text pairs is higher than that of non-matching image-text pairs, and the model parameters are updated accordingly.

[0082] Negative samples can be reorganized within mini-batches or constructed using a momentum queue mechanism to improve negative sample diversity and training stability. The contrastive learning loss can adopt a symmetrical contrastive objective, optimizing in both the "image-to-text" and "text-to-image" directions.

[0083] IRM is used to determine at the instance level whether a given image pair strictly matches the reported text, thus overcoming the limitation of relying solely on contrastive learning in characterizing "semantically similar but mismatched" samples. This IRM includes:

[0084] S1) Construct positive samples: real paired image pairs and their corresponding reports;

[0085] S2) Constructing negative samples: Randomly recombining image pairs with non-corresponding reports within a mini-batch;

[0086] S3) Add a matching marker (e.g., [MATCH]) before the text sequence, and input the text features with the matching marker and the difference-aware visual features into the cross-modal coding module for joint modeling;

[0087] S4) Take the fusion feature corresponding to the matching tag as the image-text matching state representation, input the binary classification prediction head and output the matching probability;

[0088] S5) Calculate the loss and update the model parameters based on the matching probability and the true matching label.

[0089] This task enhances the modeling of the correspondence between local changes and text descriptions through token-level interaction, thereby improving the model's ability to distinguish mismatched samples.

[0090] RG is used to transform multimodal features incorporating visual difference information into radiological report text. During the training phase, a start marker (e.g., [START]) is added before the target report sequence. The text decoding module generates subsequent text word by word in an autoregressive manner, based on the generated lexical units and difference-perceived visual features, and updates the model parameters according to the consistency between the generated distribution and the real report. Through RG, the model integrates the multimodal information obtained from the aforementioned representation modeling and alignment learning into natural language output, achieving a unified expression of the current clinical observation of images and the longitudinal changes relative to reference images.

[0091] To improve parameter utilization efficiency and enhance synergy among different functional modules in multi-task joint training scenarios, this implementation introduces a weight-sharing mechanism among multiple modules with similar structures and functions. In this implementation, the dual-image encoder and the image-text cross-modal encoder share self-attention parameters; the image-text cross-modal encoder and the text encoder share self-attention parameters; and the image-text cross-modal encoder and the text decoding module share cross-modal cross-attention parameters.

[0092] Through the aforementioned weight sharing method, different training tasks can jointly optimize the model parameters on the shared attention structure, thereby continuously strengthening the core feature modeling capability under various task constraints, thus improving the stability and generalization ability of this invention in the process of medical image difference modeling and report generation.

[0093] Because the different training tasks mentioned above vary in learning difficulty, convergence speed, and impact on the generated target, if all tasks are jointly optimized with fixed weights throughout the training process, the model may prematurely rely on language priors and ignore the real differences between paired images. Therefore, this implementation adopts a curriculum learning strategy. By dividing the training process into multiple stages and adjusting the participation or weights of each task at stage transitions, the model learns the "representation-alignment-generation" capability from shallow to deep. This training process is divided into four consecutive stages:

[0094] Phase 1: Characterization and Alignment Warm-up Phase

[0095] Enable MRM, MIDM, and IRC to enable the model to prioritize the formation of differentially perceived visual representations and preliminary cross-modal alignment relationships.

[0096] Phase Two: Fine-grained discrimination enhancement phase

[0097] While maintaining IRC, IRM is introduced to enhance the ability to distinguish mismatched samples; at the same time, the influence of MRM and MIDM is reduced so that they can serve as regularization constraints to prevent forgetting the ability to model differences; and RG is introduced with a small weight so that the model can gradually adapt to the generated target.

[0098] Phase 3: Generation-led Joint Fine-tuning Phase

[0099] Significantly increasing the involvement of RG enables the model to transform stable differential representations and alignment capabilities into coherent and clinically logical report descriptions, while other tasks serve as weak constraints to participate in training and maintain representation stability.

[0100] Phase Four: Downstream Fine-tuning Phase for Report Generation

[0101] Only RG is retained as the optimization target, and the decoding module and its coupled cross-modal module are fine-tuned in a targeted manner to further improve the clinical consistency and expression integrity of the generated report.

[0102] During the aforementioned phased training process, the task configuration remains unchanged within each phase. When switching phases, the task combination or weight parameters are adjusted according to preset rules, thereby achieving a gradual transfer of the model's learning focus.

[0103] It is easy to see that this invention, through a dual-image differential coding mechanism, jointly models the current image and historical reference images, enabling the generated report to more accurately reflect the improvement, stabilization, or deterioration trend of the disease, thus improving the reliability of the report at the disease semantic level. By effectively utilizing longitudinal information through structural differential modeling, stable results can be achieved under normal input scale conditions, which helps reduce computational resource consumption and system deployment costs. This invention introduces cross-modal alignment and matching constraints through multi-task joint training, enabling the generated report to maintain linguistic fluency while better aligning with image evidence, reducing semantic mismatches or omissions of changes.

[0104] The second embodiment of the present invention relates to an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the medical image report generation method based on a dual-image encoder of the first embodiment.

[0105] The third embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the medical image report generation method based on a dual-image encoder of the first embodiment.

[0106] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0107] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction methods implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating medical image reports based on a dual-image encoder, characterized in that, Includes the following steps: Obtain current and historical chest X-ray images; The current chest X-ray image and historical chest X-ray images are input into a medical image report generation model to obtain the radiology report text corresponding to the current chest X-ray image; wherein, the medical image report generation model includes: A single-modal coding module is used to extract image features of the current chest X-ray image and historical chest X-ray images, as well as text features of the radiology report text; A cross-modal coding module is used to fuse the image features and text features across modalities to obtain multimodal features that fuse visual difference information; The text decoding module is used to generate the radiology report text corresponding to the current chest X-ray image based on the multimodal features that fuse visual difference information; The image features of the current chest X-ray image and historical chest X-ray images, the text features of the radiology report text, and the multimodal features that integrate visual difference information are used to adjust the parameters of the medical image report generation model.

2. The medical image report generation method based on a dual-image encoder according to claim 1, characterized in that, The single-modal coding module includes: A single-image encoder is used to extract features from the current chest X-ray image and the historical chest X-ray image respectively to obtain current visual features and historical visual features; A dual-image encoder is used to add positional and temporal encoding to the current visual features and historical visual features, and then achieve implicit alignment between the current visual features and historical visual features through a cross-attention mechanism to obtain difference-aware visual features. A text encoder is used to semantically encode radiological report text to obtain text features.

3. The medical image report generation method based on a dual-image encoder according to claim 2, characterized in that, The cross-modal coding module includes: A text-image cross-modal encoder is used to inject visual information related to image changes into the text features as queries and the differentially perceived visual features as keys and values ​​through a cross-attention mechanism, thereby obtaining fused text features. A cross-modal encoder is used to use the difference-aware visual features as queries and text features as keys and values, so that the difference-aware visual features are constrained by linguistic semantics during the update process, and visual noise unrelated to clinical description is suppressed.

4. The medical image report generation method based on a dual-image encoder according to claim 3, characterized in that, The dual-image encoder and the image-text cross-modal encoder share self-attention parameters; the image-text cross-modal encoder and the text encoder share self-attention parameters; the image-text cross-modal encoder and the text decoding module share cross-modal cross-attention parameters.

5. The medical image report generation method based on a dual-image encoder according to claim 3, characterized in that, The medical image report generation model is trained using a multi-task joint training method, which includes: The masked report modeling training task is used to learn the contextual semantic structure of radiology reports on the text side and to constrain the medical image report generation model to the differentially perceived visual features when predicting masked terms. The masked image difference modeling training task is used to set the masked target in the difference-aware visual feature space, so that the medical image report generation model learns the difference features of the current chest X-ray image relative to historical chest X-ray images. An image-report contrastive learning training task is used to constrain the difference-aware visual features and the corresponding report representations to be in the same embedding space to maintain global semantic consistency. Image-report matching prediction training task, used to determine at the instance level whether a given image pair strictly matches a report text; A report generation training task is used to generate radiological report text based on the multimodal features of the fused visual difference information.

6. The medical image report generation method based on a dual-image encoder according to claim 5, characterized in that, The masked report modeling training task adopts a domain-aware masking strategy, which includes: pre-constructing a change vocabulary related to changes in the disease condition; for each report text, prioritizing the high-probability masking of words in the change vocabulary; and performing hierarchical masking of the remaining words according to their semantic importance, setting different masking probabilities based on the importance interval.

7. The medical image report generation method based on a dual-image encoder according to claim 5, characterized in that, The mask image difference modeling training task includes: Perform random region masking on the current chest X-ray image to obtain the masked current chest X-ray image; The masked current chest X-ray image and historical chest X-ray images are input into a single image encoder to obtain the corresponding difference-perceived visual features. The difference-perceived visual features and the corresponding text features of the radiology report are input into the image-text cross-modal encoder to obtain the prediction results; Using the unmasked input to obtain the prediction result as the target, the reconstruction error between the predicted features and the target features is calculated only at the masked location, and the parameters of the medical image report generation model are updated accordingly.

8. The method for generating medical image reports based on a dual-image encoder according to claim 5, characterized in that, The image-report matching prediction training task includes: Use real paired image pairs and their corresponding reports as positive samples; Within a mini-batch, image pairs are randomly recombined with non-corresponding reports to obtain negative samples; A matching marker is added before the text sequence, and the text features with the matching marker and the difference-aware visual features are input into the cross-modal coding module for joint modeling; The multimodal features corresponding to the matching markers, which incorporate visual difference information, are used as the representation of the image-text matching state. The binary classification prediction head is then input to output the matching probability. The loss is calculated based on the matching probability and the true matching label, and the parameters of the medical image report generation model are updated.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the medical image report generation method based on a dual-image encoder as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the medical image report generation method based on a dual-image encoder as described in any one of claims 1-8.

Images