A radiology image report generation method for medical image sequences
By using grouped causal feature modeling and cross-modal alignment learning methods, the problem of characterizing the long-term disease evolution of longitudinal medical image sequences is solved. The generated reports have higher accuracy and consistency, and reduce computational complexity and the risk of information leakage.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TONGJI UNIV
- Filing Date
- 2026-02-03
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies are difficult to effectively process longitudinal medical imaging sequences from multiple follow-up examinations, cannot accurately depict the long-term evolution trend of diseases, and have high computational complexity, insufficient utilization of temporal information, and risk of information leakage.
By employing a grouped causal feature modeling mechanism and an assisted cross-modal alignment learning method, combined with a text data augmentation strategy based on temporal reversal, a structurally coherent and temporally consistent radiological report is generated.
It enhances the ability to depict long-term disease evolution trends, and the generated reports are more consistent with real reports in terms of disease entities, anatomical structures, and descriptions of disease changes, while reducing computational complexity and the risk of information leakage.
Smart Images
Figure CN122135873A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent medical image analysis technology, and in particular to a method for generating radiological image reports for medical image sequences. Background Technology
[0002] Longitudinal medical imaging sequences refer to multi-temporal medical imaging data acquired from the same subject at different times. These sequences contain crucial information about the occurrence, development, and outcome of diseases, serving as important evidence for clinical diagnosis, efficacy evaluation, and follow-up management. In clinical practice, radiologists typically need to comprehensively analyze multiple imaging examinations and corresponding reports obtained by patients at different time points, and compare them with previous examination results to determine the dynamic changes in the patient's condition. This type of longitudinal comparative analysis is universal and necessary in clinical practice.
[0003] However, most existing medical image analysis and automated report generation technologies are still primarily designed for modeling single-phase images or image pairs with limited time points. For example, traditional medical image analysis methods typically extract features and identify lesions from images from a single examination, making it difficult to depict the trend of disease evolution over time. While some methods that incorporate temporal information can handle image pairs from the current to previous time points, their modeling paradigm is still limited to comparing differences between two time points, making it difficult to effectively extend to long-term medical image sequences that include multiple follow-up examinations.
[0004] In recent years, with the development of multimodal large language models, some studies have attempted to combine pre-trained visual encoders with large-scale language models for medical image understanding and radiology report generation tasks. Related methods have made some progress in medical image description generation through image-text alignment. However, due to significant differences between medical images and general natural images in terms of imaging mechanisms, semantic levels, and terminology, directly applying general multimodal large language models often only yields relatively coarse-grained semantic understanding results, which is insufficient to meet the requirements of medical image analysis for professionalism, accuracy, and fine-grained description of changes. Therefore, some existing techniques improve the models through continuous pre-training or medical domain adaptation, but they still have significant shortcomings in modeling complex disease changes and longitudinal reasoning capabilities.
[0005] To address the issue of insufficient utilization of temporal information, existing methods have introduced time-aware modules or adapter structures to explicitly model the semantic relationships between current and previous images, achieving some success in tasks involving two-time-point medical image comparison and report generation. For example, some methods enhance change perception capabilities by aligning the feature representations of the current image with those of a single previous image. However, these methods essentially still rely on the assumption of two-time-point comparison, making it difficult to handle longitudinal medical image sequences consisting of multiple follow-up examinations, and also difficult to characterize the long-term evolution of a patient's disease at a holistic level.
[0006] Furthermore, in long-sequence medical image modeling scenarios, directly inputting multiple longitudinal images into a multimodal large language model is not only computationally unsustainable, but also prone to insufficient utilization of temporal information or even temporal order confusion due to the lack of a clear temporal modeling mechanism. Meanwhile, some existing variation modeling methods employ unimodal cross-coding or fully connected attention structures to model image sequences, whose computational complexity typically increases quadratically with time length. This can easily lead to efficiency and stability issues as sequence length increases, and there is a potential risk of temporal information leakage, limiting their practicality and scalability in real-world clinical longitudinal scenarios. Summary of the Invention
[0007] The technical problem to be solved by the present invention is to provide a method for generating radiological image reports for medical image sequences, which can realize the causal difference modeling of medical image sequences across time points.
[0008] The technical solution adopted by this invention to solve its technical problem is: to provide a method for generating radiological image reports for medical image sequences, including the following steps:
[0009] Acquire multiple medical images of the same subject at different times;
[0010] The multiple medical images are input into a radiology report generation model to obtain a radiology report that is structurally coherent and temporally consistent; wherein, the radiology report generation model includes:
[0011] The feature extraction section for a single medical image is used to extract features from each medical image to obtain multiple single-temporal visual features;
[0012] The longitudinal medical image sequence causal feature modeling part is used to perform longitudinal dependency modeling on the multiple single-phase visual features under time causal constraints to obtain longitudinal visual representation information that integrates historical information.
[0013] The report generation section is used to generate a structurally coherent and temporally consistent radiological report based on the longitudinal visual representation information of the fused historical information.
[0014] The longitudinal medical image sequence causal feature modeling part includes:
[0015] The time information fusion module is used to integrate time information into the multiple single-phase visual features to obtain a feature representation containing image semantic information and time information, and to splice them in chronological order to obtain a vertical feature sequence.
[0016] The historical information fusion module is used to apply a multi-layer grouped causal attention layer to the longitudinal feature sequence, and aggregate historical medical image information layer by layer to obtain longitudinal visual representation information fused with historical information.
[0017] The time information fusion module uses the examination time of the first medical image of the subject as a benchmark, calculates the time offset of each medical image relative to the first medical image, and maps the time offset into a time embedding vector; the time embedding vector is fused with the corresponding single-phase visual features to obtain a feature representation containing image semantic information and time information.
[0018] The grouped causal attention layer is used to group the feature units in the longitudinal feature sequence according to their source, so that all feature units corresponding to the same medical image belong to the same group; each group is arranged according to the examination time sequence of the medical images, and causal constraints are applied at the group level so that the feature unit corresponding to any time point can only interact with the feature units of its own medical image and other medical images whose examination time is earlier than that medical image.
[0019] The report generation section includes:
[0020] A visual adaptation module is used to map the longitudinal visual representation information to a feature space consistent with the internal representation of the multimodal language generation model;
[0021] The prompt generation module is used to build vertical conditional text prompts;
[0022] The report generation module is used to input the mapped longitudinal visual representation information and the longitudinal conditional text prompts into the multimodal language generation model to obtain a structurally coherent and temporally consistent radiological report.
[0023] The vertical conditional text prompts constructed by the prompt generation module include:
[0024] The system prompts information to define the role of the multimodal language generation model in the report generation process and to clarify that the input visual information has been integrated with historical medical image examination clues.
[0025] Historical report prompts are used to import radiology report content from historical follow-up examinations of the same subject, providing longitudinal semantic reference for the language generation process;
[0026] Visual feature occupancy information is used to indicate the position of the mapped longitudinal visual representation information in the input of the multimodal language generation model, so that the multimodal language generation model can receive longitudinal visual feature input at the corresponding position;
[0027] Clinically generated prompts guide the multimodal language generation model to generate radiological descriptions of the current examination images, and, when necessary, incorporate longitudinal follow-up information to provide variable descriptions.
[0028] The multimodal language generation model employs an assisted cross-modal alignment learning method during training. This method maps the longitudinal visual representation information and the radiographic reports at corresponding time points to a unified semantic representation space, and uses cross-modal alignment constraints to ensure that the visual representation and textual representation at the same time point maintain consistency in the semantic space.
[0029] The multimodal language generation model employs a time-reversal-based text data augmentation strategy during training, which includes:
[0030] Semantic analysis was performed on the original radiology report, dividing the report content into a static description part and a dynamic change description part. The static description part is used to depict the anatomical structure and lesion state, while the dynamic change description part is used to depict the evolution trend of the disease relative to previous examinations.
[0031] The descriptions used to characterize the direction of disease evolution in the dynamic change description section are reversed in time sequence, while the descriptions used to characterize the stability of the disease in the dynamic change description section retain their original semantics.
[0032] Consistency checks are performed on the descriptions used to characterize the direction of disease evolution after reversal processing, and only reversal results that are reasonable in medical semantics, logically consistent, and do not introduce new lesions or omit original lesions are retained.
[0033] The static description of previous examinations is combined with the description used to characterize the direction of disease evolution after consistency verification to construct a new radiology report text. The time sequence of the corresponding medical images is reversed simultaneously to form a longitudinal training sample that is consistent with the reversed report semantics.
[0034] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned method for generating radiological image reports for medical image sequences.
[0035] The technical solution adopted by the present invention to solve its technical problem is: to provide a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the above-mentioned method for generating radiological image reports for medical image sequences are implemented.
[0036] Beneficial effects
[0037] By adopting the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art:
[0038] This invention introduces a grouped causal feature modeling mechanism into the longitudinal medical image sequence modeling process. Under the premise of ensuring temporal causal consistency, it achieves effective aggregation of multiple follow-up image information, avoids future information leakage, and thus improves the ability to depict long-term disease evolution trends. By using the visual features obtained from longitudinal causal modeling as conditions input into a multimodal language generation model, the generated radiological report has higher consistency with the real report in terms of disease entities, anatomical structures, and descriptions of disease changes, and can more accurately reflect the changing relationships between images at different time points.
[0039] This invention introduces an assisted cross-modal alignment learning method and a text data augmentation strategy based on time reversal, which effectively alleviates the semantic drift problem that may occur in the process of long sequence modeling, enhances the model's ability to distinguish different disease evolution directions, and thus improves the stability and generalization performance on complex clinical data. Attached Figure Description
[0040] Figure 1 This is a flowchart of the first embodiment of the radiological image report generation method for medical image sequences of the present invention;
[0041] Figure 2 This is a framework diagram of the radiological image report generation model in the first embodiment of the present invention;
[0042] Figure 3 This is a schematic diagram of a data augmentation strategy based on time reversal in the first embodiment of the present invention. Detailed Implementation
[0043] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.
[0044] The first embodiment of the present invention relates to a method for generating radiological image reports for medical image sequences, such as... Figure 1 As shown, it includes the following steps:
[0045] Step 1: Acquire multiple medical images of the same subject at different times;
[0046] Step 2: Input the multiple medical images into the radiology image report generation model to obtain a radiology report with a coherent structure and consistent timing.
[0047] The radiological image report generation model in this embodiment takes a cross-time point medical image sequence as input and achieves modeling of long-term disease evolution characteristics and structured report generation through the collaborative design of multi-level feature modeling and multi-modal generation techniques. For example... Figure 2 As shown, the radiological image report generation model includes:
[0048] The feature extraction section for a single medical image is used to extract features from each medical image to obtain multiple single-temporal visual features.
[0049] In this embodiment, the feature extraction part of a single medical image uses a unified image feature extraction method for multiple medical images of the same subject acquired at different times, that is, extracting the features of multiple medical images separately. The input is fed into a single-image encoder for feature extraction to obtain the corresponding single-temporal visual features, which represent the spatial semantic information used to describe the internal structure of a single image inspection.
[0050] The longitudinal medical image sequence causal feature modeling part is used to perform longitudinal dependency modeling on the multiple single-phase visual features under time causal constraints to obtain longitudinal visual representation information that integrates historical information.
[0051] The longitudinal medical image sequence causal feature modeling part in this embodiment includes:
[0052] The temporal information fusion module integrates temporal information into the multiple single-phase visual features to obtain feature representations containing both image semantic information and temporal information. These representations are then concatenated in chronological order to obtain a vertical feature sequence. Specifically, the temporal information fusion module uses the examination time of the first medical image of the subject as a benchmark, calculates the time offset of each medical image relative to the first medical image, and maps this time offset into a temporal embedding vector. Subsequently, the temporal embedding vector is fused with the corresponding single-phase visual features to obtain feature representations that simultaneously contain both image semantic information and temporal information. Finally, these feature representations are concatenated in chronological order to obtain a vertical feature sequence.
[0053] The historical information fusion module applies multi-layered grouped causal attention layers to the longitudinal feature sequence, aggregating historical medical image information layer by layer to obtain longitudinal visual representation information fused with historical information. To balance spatial modeling capabilities within a single image with causal dependencies between images across time periods during longitudinal modeling, this implementation introduces grouped causal attention layers to process the longitudinal feature sequence. Specifically, feature units in the longitudinal feature sequence are grouped according to their source images, ensuring that all feature units corresponding to the same medical image belong to the same group. Each group is then arranged according to the examination time sequence of the medical images, and causal constraints are applied at the group level so that a feature unit at any given time point can only interact with feature units of its own medical image and other medical images with examination times earlier than that image, but cannot access feature units of medical images with examination times later than that image.
[0054] By employing the aforementioned grouping causal constraint method, while ensuring sufficient interaction of features within a single image, orderly modeling of dependencies between medical images across time is achieved. This avoids future information leakage issues and reduces the computational complexity of longitudinal sequence modeling. Finally, the longitudinal visual representation information obtained after processing by the historical information fusion module implicitly encodes the examinee's historical examination information and clues to changes in their condition, serving as the visual input basis for subsequent multimodal report generation.
[0055] The report generation section is used to generate a structurally coherent and temporally consistent radiological report based on the longitudinal visual representation information of the fused historical information.
[0056] In this embodiment, to ensure that the generated radiology report accurately reflects the changes in the patient's condition within the longitudinal medical imaging sequence, the visual representation obtained from longitudinal causal modeling is combined with structured text prompts to achieve automated report generation for medical imaging sequences spanning multiple time points. The report generation component includes:
[0057] A visual adaptation module is used to map the longitudinal visual representation information to a feature space consistent with the internal representation of the multimodal language generation model.
[0058] To enable the vertical visual representation information to be effectively utilized by the multimodal language generation model, this embodiment sets up a visual adaptation module to map the vertical visual features. This module maps the vertical visual representation information to a feature space consistent with the internal representation of the multimodal language generation model, allowing the vertical visual representation information to be directly received and processed by the multimodal language generation model as a visual input sequence. Through this adaptation method, effective integration of vertical visual information and the language generation process is achieved without altering the main structure of the multimodal language generation model.
[0059] A prompt generation module is used to construct longitudinal conditional text prompts. To guide the multimodal language generation model to focus on the time dimension and disease progression when generating reports, the longitudinal conditional text prompts constructed in this implementation provide clear semantic constraints and contextual information for the report generation process. These longitudinal conditional text prompts include at least the following components:
[0060] The system prompts information to define the role of the multimodal language generation model in the report generation process and to clarify that the input visual information has been integrated with historical medical image examination clues.
[0061] Historical report prompts are used to import radiology report content from historical follow-up examinations of the same subject, providing longitudinal semantic reference for the language generation process;
[0062] Visual feature occupancy information is used to indicate the position of the mapped longitudinal visual representation information in the input of the multimodal language generation model, so that the multimodal language generation model can receive longitudinal visual feature input at the corresponding position;
[0063] Clinically generated prompts guide the multimodal language generation model to generate radiological descriptions of the current examination images, and, when necessary, incorporate longitudinal follow-up information to provide variable descriptions.
[0064] By combining the above-mentioned structured prompts, the multimodal language generation model can simultaneously utilize longitudinal visual features and historical linguistic context when generating reports, thereby enhancing the performance of the generated results in terms of temporal consistency and medical semantic rationality.
[0065] The report generation module is used to input the mapped longitudinal visual representation information and the longitudinal conditional text prompts into the multimodal language generation model to obtain a structurally coherent and temporally consistent radiological report.
[0066] In this embodiment, the longitudinal visual representation information processed by the visual adaptation module and the longitudinal conditional text prompts are input together into the multimodal language generation model, which then generates the radiology report text step by step in an autoregressive manner.
[0067] During the generation process, by limiting the range of model parameter updates, the language generation process relies more on the conditional information provided by longitudinal visual features, thus avoiding the report generation results from being based solely on language priors and ignoring longitudinal change clues in medical images. The resulting radiology reports maintain structural coherence and reflect the relationship between current examination images and historical follow-up images in terms of content.
[0068] In the process of generating reports from longitudinal medical image sequences, as the sequence length increases, the multimodal language generation model may experience instability in the correspondence between visual representations and linguistic semantics. To address this, this implementation introduces an auxiliary cross-modal alignment learning method outside the report generation task to strengthen the consistency between longitudinal visual features and radiological text semantics. Specifically, during the training phase of the multimodal language generation model, the longitudinal visual representation information and the radiological reports at corresponding time points are mapped to a unified semantic representation space. Cross-modal alignment constraints ensure that visual representations and textual representations at the same time point maintain consistency in the semantic space, while representations at different time points or for different subjects remain distinguishable in the semantic space.
[0069] By introducing the aforementioned auxiliary cross-modal alignment learning method, the risk of semantic drift that may occur during longitudinal modeling is reduced without affecting the main report generation process, thereby improving the stability and reliability of longitudinal visual representations when inputted into multimodal language generation models across time.
[0070] To address the issues of uneven distribution of disease change types and scarcity of samples showing the direction of change in longitudinal medical imaging data, this implementation employs a text data augmentation strategy based on time-series reversal to expand the coverage of disease change directions in the longitudinal training samples. For example... Figure 3 As shown, the data augmentation strategy includes the following steps:
[0071] The semantic decomposition steps of the report are as follows: Semantic analysis is performed on the original radiology report, dividing the report content into a static description section and a dynamic change description section. The static description section depicts anatomical structures and lesion states, while the dynamic change description section depicts the evolution trend of the condition relative to previous examinations. This decomposition process, as a weakly supervised semantic partitioning method, is used to explicitly extract longitudinal change cues.
[0072] The dynamic change description reversal step involves reversing the temporal direction of descriptions used to characterize the direction of disease evolution in the dynamic change description section. For example, descriptions indicating disease worsening are converted to descriptions indicating disease improvement, or vice versa. Descriptions characterizing disease stability in the dynamic change description section retain their original semantics. This reversal process changes only the direction of change while maintaining consistency with the disease entity, anatomical location, and medical facts.
[0073] Consistency verification steps for inversion results: To avoid introducing unreasonable medical semantic biases during the inversion process, a consistency verification is performed on the descriptions used to represent the direction of disease evolution after inversion. Only inversion results that are medically semantically reasonable, logically consistent, and do not introduce new lesions or omit existing lesions are retained.
[0074] The steps for constructing the reversed sample are as follows: Combine the static description of the previous examination with the description used to characterize the direction of disease evolution after consistency verification to construct a new radiology report text, and simultaneously reverse the time sequence of the corresponding medical images to form a longitudinal training sample that is consistent with the semantics of the reversed report.
[0075] By employing the aforementioned text data augmentation strategy based on time-series reversal, it is possible to construct longitudinal samples with opposite disease evolution directions without disrupting the basic correspondence between images and text, thereby balancing the distribution of different change directions in the training data.
[0076] The aforementioned assisted cross-modal alignment learning method and the time-reversal-based text data augmentation strategy can be used synergistically during the model training phase to enhance the model's ability to discriminate the longitudinal disease evolution direction and the stability of its representation of stage-specific changes. Through this synergistic design, this implementation improves the robustness and generalization performance of longitudinal medical image sequence modeling and report generation tasks without altering the main model structure.
[0077] The overall training approach for the radiological image report generation model in this embodiment is as follows:
[0078] To reduce the training difficulty of longitudinal medical image modeling and language generation tasks, and to improve the stability and convergence of model training, a phased training strategy based on curriculum learning is adopted. This strategy gradually transitions the model from longitudinal visual representation learning to the generation of radiological reports under longitudinal conditions through a process of "representation modeling first, conditional generation second, and overall fine-tuning third".
[0079] Phase 1: Vertical Causal Representation Learning Phase.
[0080] In the first training phase, the focus is on training the causal feature modeling capability of longitudinal medical image sequences. Specifically, only the causal modeling module and the temporal information encoding module for longitudinal medical image sequences are enabled, and the model is trained by combining cross-modal alignment learning tasks and data augmentation strategies. In this phase, radiology report generation tasks are not introduced to avoid interference from language generation objectives with longitudinal representation learning.
[0081] This training phase enables the model to learn long-term dependencies and temporal evolution structures between multiple follow-up images, relying solely on medical imaging information, thereby obtaining a stable longitudinal visual representation. This phase can employ a relatively large number of training steps and a learning rate to accelerate the convergence of the longitudinal representation.
[0082] Phase Two: The generation transition phase under vertical conditions.
[0083] In the second training phase, while keeping the main parameters of the multimodal language generation model frozen, a radiographic report generation task is introduced, and longitudinal visual representations are gradually integrated into the language generation process. Specifically, in this phase, an adaptation module for mapping longitudinal visual features to the language model input space is enabled, and the adaptation module and low-rank parameters are fine-tuned to achieve effective alignment between longitudinal visual information and the language generation process. Simultaneously, the cross-modal alignment learning task is retained to maintain consistency between longitudinal visual representations and text semantics.
[0084] Through this training phase, the model can smoothly transition from pure representation learning to conditional generation tasks, enabling the language generation process to gradually adapt to the input of longitudinal medical image features.
[0085] Phase 3: Vertical generation and fine-tuning phase.
[0086] In the third training phase, the model is fine-tuned through radiology report generation and cross-modal alignment learning tasks. During this phase, language generation-related parameters are continuously updated with small increments, while a small learning rate constraint is applied to the longitudinal medical image causal modeling module to avoid disrupting the longitudinal dependency structure learned in the previous phases. This approach allows the model to maintain longitudinal modeling stability while further improving the accuracy and consistency of radiology report generation.
[0087] The training process described above can be completed on computing devices with graphics processing capabilities. Gradient-based optimization algorithms are used to update model parameters, and learning rate decay and warm-up strategies are combined to improve training stability. The longitudinal medical image sequence causal modeling module can be implemented using a multi-layer attention structure, and the language generation stage can employ decoding strategies such as beam search to generate radiological report text.
[0088] It is easy to see that this invention introduces a grouped causal feature modeling mechanism in the longitudinal medical image sequence modeling process, which, while ensuring temporal causal consistency, effectively aggregates information from multiple follow-up images, avoids future information leakage, and thus improves the ability to depict long-term disease evolution trends. By using the visual features obtained from longitudinal causal modeling as conditions input into a multimodal language generation model, the generated radiological report has higher consistency with the real report in terms of disease entities, anatomical structures, and descriptions of disease changes, and can more accurately reflect the changing relationships between images at different time points.
[0089] A second embodiment of the present invention relates to an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the radiological image report generation method for medical image sequences of the first embodiment.
[0090] The third embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the radiological image report generation method for medical image sequences of the first embodiment.
[0091] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0092] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction methods implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0095] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for generating radiological image reports for medical image sequences, characterized in that, Includes the following steps: Acquire multiple medical images of the same subject at different times; The multiple medical images are input into a radiology report generation model to obtain a radiology report that is structurally coherent and temporally consistent; wherein, the radiology report generation model includes: The feature extraction section for a single medical image is used to extract features from each medical image to obtain multiple single-temporal visual features; The longitudinal medical image sequence causal feature modeling part is used to perform longitudinal dependency modeling on the multiple single-phase visual features under time causal constraints to obtain longitudinal visual representation information that integrates historical information. The report generation section is used to generate a structurally coherent and temporally consistent radiological report based on the longitudinal visual representation information of the fused historical information.
2. The method for generating radiological image reports for medical image sequences according to claim 1, characterized in that, The longitudinal medical image sequence causal feature modeling part includes: The time information fusion module is used to integrate time information into the multiple single-phase visual features to obtain a feature representation containing image semantic information and time information, and to splice them in chronological order to obtain a vertical feature sequence. The historical information fusion module is used to apply a multi-layer grouped causal attention layer to the longitudinal feature sequence, and aggregate historical medical image information layer by layer to obtain longitudinal visual representation information fused with historical information.
3. The method for generating radiological image reports for medical image sequences according to claim 3, characterized in that, The time information fusion module uses the examination time of the first medical image of the subject as a benchmark to calculate the time offset of each medical image relative to the first medical image, and maps the time offset into a time embedding vector. The temporal embedding vector is fused with the corresponding single-phase visual features to obtain a feature representation that includes both image semantic information and temporal information.
4. The method for generating radiological image reports for medical image sequences according to claim 3, characterized in that, The grouped causal attention layer is used to group the feature units in the longitudinal feature sequence according to their source, so that all feature units corresponding to the same medical image belong to the same group; each group is arranged according to the examination time sequence of the medical images, and causal constraints are applied at the group level so that the feature unit corresponding to any time point can only interact with the feature units of its own medical image and other medical images whose examination time is earlier than that medical image.
5. The method for generating radiological image reports for medical image sequences according to claim 1, characterized in that, The report generation section includes: A visual adaptation module is used to map the longitudinal visual representation information to a feature space consistent with the internal representation of the multimodal language generation model; The prompt generation module is used to build vertical conditional text prompts; The report generation module is used to input the mapped longitudinal visual representation information and the longitudinal conditional text prompts into the multimodal language generation model to obtain a structurally coherent and temporally consistent radiological report.
6. The method for generating radiological image reports for medical image sequences according to claim 5, characterized in that, The vertical conditional text prompts constructed by the prompt generation module include: The system prompts information to define the role of the multimodal language generation model in the report generation process and to clarify that the input visual information has been integrated with historical medical image examination clues. Historical report prompts are used to import radiology report content from historical follow-up examinations of the same subject, providing longitudinal semantic reference for the language generation process; Visual feature occupancy information is used to indicate the position of the mapped longitudinal visual representation information in the input of the multimodal language generation model, so that the multimodal language generation model can receive longitudinal visual feature input at the corresponding position; Clinically generated prompts guide the multimodal language generation model to generate radiological descriptions of the current examination images, and, when necessary, incorporate longitudinal follow-up information to provide variable descriptions.
7. The method for generating radiological image reports for medical image sequences according to claim 5, characterized in that, The multimodal language generation model employs an assisted cross-modal alignment learning method during training. This method maps the longitudinal visual representation information and the radiographic reports at corresponding time points to a unified semantic representation space, and uses cross-modal alignment constraints to ensure that the visual representation and textual representation at the same time point maintain consistency in the semantic space.
8. The method for generating radiological image reports for medical image sequences according to claim 5, characterized in that, The multimodal language generation model employs a time-reversal-based text data augmentation strategy during training, which includes: Semantic analysis was performed on the original radiology report, dividing the report content into a static description part and a dynamic change description part. The static description part is used to depict the anatomical structure and lesion state, while the dynamic change description part is used to depict the evolution trend of the disease relative to previous examinations. The descriptions used to characterize the direction of disease evolution in the dynamic change description section are reversed in time sequence, while the descriptions used to characterize the stability of the disease in the dynamic change description section retain their original semantics. Consistency checks are performed on the descriptions used to characterize the direction of disease evolution after reversal processing, and only reversal results that are reasonable in medical semantics, logically consistent, and do not introduce new lesions or omit original lesions are retained. The static description of previous examinations is combined with the description used to characterize the direction of disease evolution after consistency verification to construct a new radiology report text. The time sequence of the corresponding medical images is reversed simultaneously to form a longitudinal training sample that is consistent with the reversed report semantics.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the radiological image report generation method for medical image sequences as described in any one of claims 1-8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the radiological image report generation method for medical image sequences as described in any one of claims 1-8.