Medical image report generation method and device

Abstract

The invention discloses a medical image report generation method and device. The method comprises the following steps: acquiring to-be-analyzed medical image data and examination item information and non-image clinical data thereof; standardizing image data, and structuring non-image clinical data; constructing or updating a dynamic medical knowledge graph; inputting the image data into an anatomical region segmentation model to obtain a segmentation mask containing an anatomical part identifier, and obtaining an abnormal region spatial position identifier; querying the knowledge graph based on examination item information and anatomical site identification to obtain medical knowledge entries; generating a cue word text according to a cue word template based on the segmentation mask, the abnormal label, the structured clinical data and the medical knowledge entry; and inputting the image data, the segmentation mask and the cue word text into a multi-modal report generation model, and outputting a report text and linkage information of the abnormal description and the spatial position identifier. According to the method, region guidance, knowledge enhancement and multi-modal fusion are realized, and local anomaly capture and text-region correspondence are improved.

Description

Technical Field

[0001] This application relates to the field of medical image report generation technology, and in particular to a method and apparatus for generating medical image reports. Background Technology

[0002] Medical imaging reports are professional documents created by doctors based on the imaging data generated by medical imaging equipment, after observation and analysis. Their content typically provides diagnostic evidence, describes the patient's condition, and assists in developing treatment plans. They also serve as an important medium for communication between doctors and patients. With the continuous growth of medical informatization and the scale of imaging examinations, how to leverage artificial intelligence technology to improve report writing efficiency, reduce repetitive work, and enhance diagnostic consistency has become an important development direction for the intelligent application of medical imaging.

[0003] Current methods for automatically generating medical image reports mostly employ an encoder-decoder architecture, encoding the complete image into a global feature vector and then directly generating a text report via the decoder. This end-to-end generation approach still has significant shortcomings in practical applications: First, when based solely on global features, the model struggles to capture subtle abnormalities in specific areas such as the heart, pelvis, and scapula, easily overlooking crucial local lesions for diagnosis. Second, the lack of a clear correspondence between the generated text and specific lesion areas in the image leads to insufficient interpretability, making it difficult for clinicians to verify and trace the diagnostic basis generated by the model. Third, the integration of cross-modal information (such as images, medical history, and laboratory indicators) is low, often relying on a single image data source while ignoring multi-dimensional diagnostic evidence. Fourth, the lack of a dynamic optimization channel after model training makes it difficult to promptly absorb clinical feedback and updates to the latest treatment guidelines, potentially leading to lag in generation logic and performance degradation over long-term operation.

[0004] Therefore, in the automatic generation of medical image reports, the difficulty in effectively capturing subtle abnormalities in key local areas, the unclear correspondence between report text and lesion areas, the insufficient integration of cross-modal diagnostic and treatment information, and the lack of dynamic optimization of the generation logic based on clinical feedback and the latest guidelines have become urgent problems to be solved. Summary of the Invention

[0005] This application provides a method and apparatus for generating medical image reports, aiming to solve the problems in the automatic generation of medical image reports in the prior art, such as the difficulty in effectively capturing subtle abnormalities in local key areas, the unclear correspondence between report text and lesion area, insufficient integration of cross-modal diagnosis and treatment information, and the lack of ability to dynamically optimize the generation logic with clinical feedback and the latest guidelines.

[0006] In a first aspect, a method for generating a medical image report, the method comprising: Acquire the medical imaging data to be analyzed, as well as the examination item information and non-imaging clinical data corresponding to the medical imaging data, wherein the non-imaging clinical data includes medical history information and / or laboratory indicator information; The medical imaging data is subjected to image standardization processing to obtain processed medical imaging data, and the non-imaging clinical data is subjected to structured processing to obtain processed non-imaging clinical data. Acquire medical knowledge data and construct or update a dynamic medical knowledge graph, which includes the relationship between anatomical locations, abnormality types, examination items and diagnostic conclusions; The processed medical image data is input into the anatomical region segmentation model, and the anatomical region segmentation mask is output, which includes anatomical location identifiers. Obtain the abnormal region annotation information corresponding to the processed medical image data, wherein the abnormal region annotation information includes the spatial location identifier of the abnormal region; Based on the examination item information and the anatomical location identifier, the dynamic medical knowledge graph is queried to obtain medical knowledge entries that match the examination item information and the anatomical location identifier; wherein, the medical knowledge entries include abnormality type information and / or diagnostic conclusion information; Based on the anatomical region segmentation mask, the abnormal region annotation information, the processed non-image clinical data, and the medical knowledge entries, prompt text is generated according to a preset prompt text template. The processed medical image data, the anatomical region segmentation mask, and the prompt text are input into the multimodal report generation model, which outputs a medical image report text and linkage information for associating the abnormal descriptions in the medical image report text with the spatial location identifiers of the abnormal regions.

[0007] Optionally, in the above scheme, the image standardization processing of the medical image data includes: Convert the medical image data into a preset medical image standard format; Consistency correction is performed on the scanning parameters and slice thickness differences of the medical image data.

[0008] Optionally, in the above scheme, the structuring processing of the non-imaging clinical data includes at least one of the following: The key information of the medical history information is extracted using a structured form to obtain a set of structured medical history fields; The laboratory indicator information is standardized according to the format of indicator name, indicator value, and reference range to obtain a set of structured indicator fields. Establish an association mapping between the structured medical history field set and the structured indicator field set and the patient identifier.

[0009] Optionally, in the above scheme, the step of acquiring medical knowledge data and constructing or updating a dynamic medical knowledge graph includes: A basic disease atlas is constructed based on medical terminology standards. The basic disease atlas is used to indicate the four-dimensional mapping relationship between anatomical location, abnormality type, examination items and diagnostic conclusion. A rare disease sub-atlas is established based on clinical guidelines and / or rare disease diagnosis and treatment standards. The rare disease sub-atlas is used to indicate the relationship between rare disease types, imaging features and non-imaging indicators. The basic disease atlas and the rare disease sub-atlas are integrated into the dynamic medical knowledge graph; Obtain updated content of the clinical guidelines, rare disease diagnosis and treatment standards, and / or medical terminology standards, and update the mapping and / or association relationships in the dynamic medical knowledge graph based on the updated content.

[0010] In the above scheme, optionally, the anatomical region segmentation model is an encoder-decoder segmentation network, which includes an encoder, a decoder, residual connections, and a dynamic attention optimization module; The encoder is a lightweight backbone network, which includes MobileNetV3; The residual connection is used to connect the output features of the encoder with the input features of the decoder; The dynamic attention optimization module is configured to determine the importance weight of anatomical regions in the dynamic medical knowledge graph based on the examination item information, and adjust the attention allocation of segmentation features according to the importance weight of the anatomical regions.

[0011] In the above scheme, optionally, the abnormal area labeling information includes an abnormality type identifier, morphological parameters of the abnormal area, and / or risk level parameters of the abnormal area; The abnormal region annotation information is obtained by at least two annotators independently annotating the abnormal region and undergoing consistency review, or the abnormal region annotation information is obtained by an anomaly detection model detecting the processed medical image data.

[0012] Optionally, in the above scheme, the step of generating prompt text based on the anatomical region segmentation mask, the abnormal region annotation information, the processed non-image clinical data, and the medical knowledge entries, according to a preset prompt text template, includes: The examination item information, the anatomical location identification, the abnormal area annotation information, and the processed non-imaging clinical data are filled into the preset prompt word template to generate basic prompt words; Write the abnormality type information and / or diagnostic conclusion information from the medical knowledge entries into the basic prompt words to generate the prompt word text; The preset prompt word template includes fields for examination items, anatomical locations, abnormal information, and non-imaging clinical data.

[0013] Optionally, in the above scheme, the multimodal report generation model includes an image encoder, a text encoder, a cross-attention layer, and a text decoder; The image encoder is used to encode the processed medical image data to obtain image features; The text encoder is used to encode the prompt word text to obtain prompt word text features; The cross-attention layer is used to fuse the image features and the cue word text features, as well as the guiding features formed by the anatomical region segmentation mask, to obtain the fused features; The text decoder is used to generate the medical image report text based on the fusion features.

[0014] In the above scheme, optionally, the fusion of the image features, the prompt text features, and the guiding features formed by the anatomical region segmentation mask to obtain the fused features includes: The segmentation mask of the anatomical region is mapped to an attention guidance map to obtain the segmentation mask features; The segmentation mask features, image features, and prompt word text features are input into the cross-attention layer to calculate the fusion of the image features, segmentation mask features, and prompt word text features based on attention weights, and the fused features are output.

[0015] Secondly, a medical image report generation device, the device comprising: The acquisition module is used to acquire the medical imaging data to be analyzed, as well as the examination item information and non-imaging clinical data corresponding to the medical imaging data. The non-imaging clinical data includes medical history information and / or laboratory indicator information. The processing module is used to perform image standardization processing on the medical image data to obtain processed medical image data, and to perform structured processing on the non-image clinical data to obtain processed non-image clinical data. The knowledge graph module is used to acquire medical knowledge data and construct or update a dynamic medical knowledge graph, which includes the relationship between anatomical sites, abnormality types, examination items and diagnostic conclusions. The segmentation module is used to input the processed medical image data into the anatomical region segmentation model and output an anatomical region segmentation mask, wherein the anatomical region segmentation mask includes anatomical location identifiers. An anomaly information acquisition module is used to acquire anomaly region annotation information corresponding to the processed medical image data, wherein the anomaly region annotation information includes the spatial location identifier of the anomaly region. The query module is used to query the dynamic medical knowledge graph based on the examination item information and the anatomical location identifier to obtain medical knowledge entries that match the examination item information and the anatomical location identifier; wherein, the medical knowledge entries include abnormality type information and / or diagnostic conclusion information; The prompt word generation module is used to generate prompt word text according to a preset prompt word template based on the anatomical region segmentation mask, the abnormal region annotation information, the processed non-image clinical data, and the medical knowledge entries. The report generation module is used to input the processed medical image data, the anatomical region segmentation mask, and the prompt text into the multimodal report generation model, output medical image report text, and linkage information for associating the abnormal descriptions in the medical image report text with the spatial location identifiers of the abnormal regions.

[0016] Compared with the prior art, this application has at least the following beneficial effects: This application, based on further analysis and research of existing technical problems, recognizes that existing technologies in the automatic generation of medical image reports suffer from several shortcomings. First, they struggle to effectively capture subtle abnormalities in key local areas. Second, the correspondence between report text and lesion areas is unclear. Third, cross-modal diagnostic and treatment information fusion is insufficient, and the generation logic lacks the ability to dynamically optimize with clinical feedback and the latest guidelines. This application addresses these issues by acquiring medical image data and its corresponding examination information, as well as non-image clinical data containing medical history and / or laboratory indicators, before report generation. Image data is standardized, and non-image clinical data is structured, ensuring a unified data representation foundation for subsequent model inputs. Simultaneously, this solution constructs or updates a dynamic medical knowledge graph containing the relationships between anatomical locations, abnormality types, examination items, and diagnostic conclusions. Based on examination item information and anatomical location identifiers contained in the anatomical region segmentation mask, this knowledge graph is queried to obtain medical knowledge entries containing abnormality type information and / or diagnostic conclusion information. This allows for the formation of a knowledge constraint source matching the current examination items and anatomical locations before the generation process. Finally, the processed medical image data is input into the anatomical region segmentation model to obtain anatomical regions containing anatomical location identifiers. The algorithm uses a domain segmentation mask to obtain anomaly region annotation information, including spatial location identifiers of abnormal regions. This allows for the simultaneous acquisition of two types of spatial guidance information at the data level: "anatomical region range" and "abnormal region location." Based on this, the algorithm generates prompt text by combining the anatomical region segmentation mask, anomaly region annotation information, structured non-image clinical data, and medical knowledge entries according to a preset prompt text template. This organizes examination items, anatomical locations, abnormal locations, clinical information, and knowledge entries into a unified input semantic that the model can utilize. Finally, the algorithm inputs the processed medical image data, anatomical region segmentation mask, and prompt text into a multimodal report generation model to output a medical image report text. Simultaneously, it outputs linkage information to associate the abnormal descriptions in the report text with the spatial location identifiers of abnormal regions. Therefore, report generation no longer relies solely on the global representation of the entire image. Instead, it generates abnormal descriptions related to local regions under the spatial guidance of the anatomical region segmentation mask and abnormal region location identifiers. The prompt text integrates non-image clinical data and medical knowledge entries obtained from a dynamic medical knowledge graph to participate in the semantic organization of the report. Simultaneously, the linkage information establishes a correspondence between the textual abnormal descriptions and the abnormal image regions.

[0017] Based on the input constraints and information organization methods of the above process, it can be deduced that this solution can address the problems in the background technology, such as the difficulty in capturing subtle local anomalies caused by end-to-end global feature generation, the unclear correspondence between text and lesion areas, and the insufficient integration of cross-modal diagnostic and treatment information and the difficulty in absorbing new knowledge with guideline updates. It forms a report generation and location-related output mechanism driven by regional guidance, knowledge item constraints, and multimodal information, thereby solving the aforementioned background technology problems. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method for generating medical image reports according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] In one embodiment, such as Figure 1 As shown, a method for generating medical image reports is provided, including the following steps: Acquire the medical imaging data to be analyzed, as well as the examination item information and non-imaging clinical data corresponding to the medical imaging data, wherein the non-imaging clinical data includes medical history information and / or laboratory indicator information; The medical imaging data is subjected to image standardization processing to obtain processed medical imaging data, and the non-imaging clinical data is subjected to structured processing to obtain processed non-imaging clinical data. Acquire medical knowledge data and construct or update a dynamic medical knowledge graph, which includes the relationship between anatomical locations, abnormality types, examination items and diagnostic conclusions; The processed medical image data is input into the anatomical region segmentation model, and the anatomical region segmentation mask is output, which includes anatomical location identifiers. Obtain the abnormal region annotation information corresponding to the processed medical image data, wherein the abnormal region annotation information includes the spatial location identifier of the abnormal region; Based on the examination item information and the anatomical location identifier, the dynamic medical knowledge graph is queried to obtain medical knowledge entries that match the examination item information and the anatomical location identifier; wherein, the medical knowledge entries include abnormality type information and / or diagnostic conclusion information; Based on the anatomical region segmentation mask, the abnormal region annotation information, the processed non-image clinical data, and the medical knowledge entries, prompt text is generated according to a preset prompt text template. The processed medical image data, the anatomical region segmentation mask, and the prompt text are input into the multimodal report generation model, which outputs a medical image report text and linkage information for associating the abnormal descriptions in the medical image report text with the spatial location identifiers of the abnormal regions.

[0021] In one specific implementation, the medical imaging data to be analyzed and its corresponding examination information are first acquired. The medical imaging data can be two-dimensional or three-dimensional image data generated by examinations such as CT, MRI, and X-ray; the examination information is used to characterize the examination type, examination site, or examination protocol. Simultaneously, non-imaging clinical data corresponding to the medical imaging data is acquired. This non-imaging clinical data includes at least medical history information and / or laboratory indicator information, and a correspondence is established with patient identification and examination time to ensure data consistency.

[0022] In one specific implementation, image standardization is performed on medical image data to obtain processed medical image data; structured processing is performed on non-image clinical data to obtain processed non-image clinical data. Image standardization is used to unify the image data format and differences in imaging parameters; structured processing is used to convert medical history, test indicators, etc. in text or tabular form into field-based data, which facilitates subsequent prompt word construction and model input.

[0023] In one specific implementation, medical knowledge data is acquired and a dynamic medical knowledge graph is constructed or updated. The dynamic medical knowledge graph is used to organize and maintain the relationships between anatomical locations, abnormality types, examination items, and diagnostic conclusions, and supports subsequent retrieval by examination item and anatomical location identifier, outputting medical knowledge entries relevant to the current sample. The construction, integration, and updating methods of the dynamic medical knowledge graph are described in subsequent implementation method descriptions.

[0024] In one specific implementation, the processed medical image data is input into an anatomical region segmentation model, which outputs an anatomical region segmentation mask. The anatomical region segmentation mask identifies one or more anatomical regions in the image and includes anatomical location identifiers (e.g., anatomical division identifiers for lung lobes, mediastinum, liver segments, etc., under the current examination item). The anatomical region segmentation model can employ a structure including an encoder, decoder, residual connections, and dynamic attention optimization modules to output mask results corresponding to the anatomical location identifiers.

[0025] In one specific implementation, abnormal region annotation information corresponding to the processed medical image data is obtained. This annotation information includes at least the spatial location identifier of the abnormal region. The spatial location identifier can be specifically represented as at least one of the following forms: two-dimensional bounding box coordinates, three-dimensional bounding box coordinates, pixel / voxel-level mask index, slice number, and contour point set, in order to establish a location relationship between the abnormal description and the image region. The abnormal region annotation information can be obtained through manual annotation and consistency review, or through detection using an anomaly detection model.

[0026] In one specific implementation, a dynamic medical knowledge graph is queried based on examination item information and anatomical location identifiers to obtain medical knowledge entries that match the examination item information and anatomical location identifiers. Each medical knowledge entry includes at least abnormality type information and / or diagnostic conclusion information, which can serve as knowledge constraints or supplementary content for subsequent prompts. The query method can be: using examination items as a primary index and anatomical location identifiers as a secondary index to retrieve relevant abnormality type sets and diagnostic conclusion sets, or using a combination of "examination items and anatomical location identifiers" as the search key to return the corresponding set of knowledge entries.

[0027] In one specific implementation, prompt text is generated based on anatomical region segmentation masks, abnormal region annotation information, processed non-imaging clinical data, and medical knowledge entries, according to a preset prompt text template. The prompt text template may include fields for examination items, anatomical locations, abnormal information, non-imaging clinical data, and knowledge entries. Basic prompts are generated by filling the template with examination item information, anatomical location identifiers, abnormal region spatial location identifiers, and medical history / laboratory indicator fields. Then, abnormality type information and / or diagnostic conclusion information from the medical knowledge entries are written into the basic prompts to form the prompt text.

[0028] In one specific implementation, processed medical image data, anatomical region segmentation masks, and prompt text are input into a multimodal report generation model, which outputs medical image report text and associated information. The multimodal report generation model may include an image encoder, a text encoder, a cross-attention layer, and a text decoder, used to fuse image features, prompt text features, and guiding features formed by the anatomical region segmentation mask to generate the report text. The associated information is used to associate the anomaly descriptions in the report text with the spatial location identifiers of the anomaly regions. Its implementation can be a mapping table of "text fragment index / segment number—spatial location identifier," a correspondence table between structured report entries and mask / coordinate IDs, or a data structure file containing the aforementioned correspondence, to support the linked presentation of the report text and image regions.

[0029] This embodiment achieves regional guidance and location association by introducing anatomical region segmentation masks and abnormal region spatial location identifiers; it enhances the prompt text with knowledge by querying medical knowledge entries obtained from a dynamic medical knowledge graph; and it combines non-image clinical data with a multimodal report generation model to achieve collaborative generation of image and clinical information, thereby enabling the report generation process to have the ability to interpret and constrain based on region, knowledge and multimodal information, and forming a linkage and association output between the report abnormal description and the image abnormal region.

[0030] In this embodiment, the image standardization processing of the medical image data includes: Convert the medical image data into a preset medical image standard format; Consistency correction is performed on the scanning parameters and slice thickness differences of the medical image data.

[0031] In one specific implementation, image standardization processing includes converting medical image data into a preset medical image standard format. The preset medical image standard format is defined as DICOM 3.0 format; for image data from different devices or systems, a format conversion module can encapsulate it into a unified standard format file while retaining necessary examination metadata (such as examination time, sequence information, etc.).

[0032] In one specific implementation, image standardization processing also includes consistency correction for scanning parameters and slice thickness differences in medical image data. Scanning parameters may include sets of acquisition parameters from different sequences; slice thickness differences can be unified to a preset slice thickness through resampling or interpolation, ensuring that the input scale of subsequent segmentation models and multimodal models is consistent, thus avoiding spatial resolution differences caused by inconsistent slice thickness.

[0033] This embodiment provides a consistent input data foundation for subsequent anatomical region segmentation and multimodal report generation by unifying the image data format and performing consistency correction on scanning parameters and slice thickness differences, thereby reducing the impact of input inconsistencies caused by differences in image data across devices and centers.

[0034] In this embodiment, the structuring processing of the non-imaging clinical data includes at least one of the following: The key information of the medical history information is extracted using a structured form to obtain a set of structured medical history fields; The laboratory indicator information is standardized according to the format of indicator name, indicator value, and reference range to obtain a set of structured indicator fields. Establish an association mapping between the structured medical history field set and the structured indicator field set and the patient identifier.

[0035] In one specific implementation, structuring non-imaging clinical data includes extracting key information from medical history using structured forms to obtain a set of structured medical history fields. This set of structured medical history fields may include fields such as past medical history, chief complaint, key points of present illness, and relevant medication or surgical history. When the medical history source is free text, field template extraction or rule / model extraction can be performed first, and then the data can be written into the structured form fields.

[0036] In one specific implementation, the laboratory indicator information is standardized, and a structured indicator field set is output according to the format of indicator name, indicator value, and reference range. The structured indicator field set can be saved in the form of "Indicator Item ID / Name—Value—Unit—Reference Range—Sampling Time".

[0037] In one specific implementation, an association mapping is established between the structured medical history field set and the structured indicator field set and the patient identifier. The patient identifier is defined as a patient ID or equivalent identifier; the association mapping can adopt an index structure of "patient identifier, examination time, and data field set", which enables accurate retrieval and binding of corresponding non-imaging clinical data according to the patient identifier and examination time when generating prompt text.

[0038] This embodiment, by structuring medical history and laboratory indicators into fields and establishing a mapping relationship with patient identifiers, enables non-imaging clinical data to be stably and traceably incorporated into the prompt word construction and multimodal generation process, thereby enhancing the consistency between multi-source clinical information and the direction of image analysis.

[0039] In this embodiment, acquiring medical knowledge data and constructing or updating a dynamic medical knowledge graph includes: A basic disease atlas is constructed based on medical terminology standards. The basic disease atlas is used to indicate the four-dimensional mapping relationship between anatomical location, abnormality type, examination items and diagnostic conclusion. A rare disease sub-atlas is established based on clinical guidelines and / or rare disease diagnosis and treatment standards. The rare disease sub-atlas is used to indicate the relationship between rare disease types, imaging features and non-imaging indicators. The basic disease atlas and the rare disease sub-atlas are integrated into the dynamic medical knowledge graph; Obtain updated content of the clinical guidelines, rare disease diagnosis and treatment standards, and / or medical terminology standards, and update the mapping and / or association relationships in the dynamic medical knowledge graph based on the updated content.

[0040] In one specific implementation, a basic disease atlas is constructed based on medical terminology standards to establish a four-dimensional mapping relationship between anatomical locations, abnormality types, examination items, and diagnostic conclusions. The basic disease atlas can establish a mapping index of "anatomical location—abnormality type—examination item—diagnostic conclusion" based on the conceptual system in the medical terminology standards. For example, starting with an examination item, the corresponding set of anatomical locations can be retrieved, and then the set of abnormality types that can occur at that anatomical location and their corresponding candidate sets of diagnostic conclusions can be associated.

[0041] In one specific implementation, a rare disease sub-atlas is established based on clinical guidelines and / or rare disease diagnosis and treatment standards to organize the correlation between rare disease types, imaging features, and non-imaging indicators. The rare disease sub-atlas can adopt a correlation structure of "rare disease type - typical imaging features - relevant laboratory indicators / clinical manifestations," so that when examination items and anatomical sites suggest that a rare disease may be involved, the corresponding imaging features and non-imaging indicators can be returned for reference in the prompt text.

[0042] In one specific implementation, the basic disease atlas and the rare disease sub-atlas are integrated into a dynamic medical knowledge graph. Integration methods may include: maintaining a relationship index between the basic disease atlas and the rare disease sub-atlas in a unified knowledge storage and providing a unified query interface; or merging the two into a single graph structure at the logical layer, distinguishing between "basic mapping" and "rare disease association" through a type field.

[0043] In one specific implementation, updated content from clinical guidelines, rare disease diagnosis and treatment standards, and / or medical terminology standards is obtained, and the mapping and / or association relationships in the dynamic medical knowledge graph are updated accordingly. Updated content can be obtained through API access, file synchronization, or version comparison; update operations may include adding mapping entries, modifying the correspondence between abnormality types and diagnostic conclusions, and updating association entries between rare disease types and imaging features / non-imaging indicators, ensuring that the content of the dynamic medical knowledge graph iterates along with updates to standards and guidelines.

[0044] This embodiment constructs and integrates basic disease atlases and rare disease sub-atlases in a hierarchical manner, and updates the atlas relationships according to the updated content of guidelines / norms / terminology standards. This allows knowledge constraints and knowledge supplements to be maintained as the external medical knowledge system changes, thereby providing updatable knowledge support for prompt word generation and report generation.

[0045] In this embodiment, the anatomical region segmentation model is an encoder-decoder segmentation network, which includes an encoder, a decoder, residual connections, and a dynamic attention optimization module. The encoder is a lightweight backbone network, which includes MobileNetV3; The residual connection is used to connect the output features of the encoder with the input features of the decoder; The dynamic attention optimization module is configured to determine the importance weight of anatomical regions in the dynamic medical knowledge graph based on the examination item information, and adjust the attention allocation of segmentation features according to the importance weight of the anatomical regions.

[0046] In one specific implementation, the anatomical region segmentation model is an encoder-decoder segmentation network, comprising an encoder, a decoder, residual connections, and a dynamic attention optimization module. The encoder extracts multi-scale features from the image, the decoder upsamples the encoded features stepwise and outputs pixel / voxel-level segmentation results, and the residual connections connect the encoder output features with the decoder input features to preserve spatial details and enhance feature transfer stability.

[0047] In one specific implementation, the encoder is a lightweight backbone network, which includes MobileNetV3. MobileNetV3 can be used as the encoder backbone to extract features from the processed medical image data and output multi-level feature maps for the decoder to fuse; the lightweight backbone can be used to complete the extraction of anatomical region features while maintaining the deployability of the segmentation network.

[0048] In one specific implementation, the dynamic attention optimization module is configured to determine the importance weights of anatomical regions in a dynamic medical knowledge graph based on examination item information, and adjust the attention allocation of segmentation features according to these importance weights. Specifically, examination item information can be used as a query condition to obtain the weight configurations of each anatomical region under that examination item from the dynamic medical knowledge graph; the dynamic attention optimization module then adjusts the channel weights and / or spatial attention weights related to the corresponding anatomical regions in the segmentation network accordingly, so that the segmentation network reflects a region attention strategy matching the examination item during the feature extraction and fusion stages.

[0049] This embodiment uses an encoder-decoder segmentation structure that includes a MobileNetV3 encoder, residual connections, and a dynamic attention optimization module. This enables the segmentation model to structurally support attention adjustment based on inspection items and knowledge graph weights, and outputs an anatomical region segmentation mask containing anatomical site identifiers, providing region-guided input for subsequent prompt word generation and multimodal report generation.

[0050] In this embodiment, the abnormal area labeling information includes an abnormality type identifier, morphological parameters of the abnormal area, and / or risk level parameters of the abnormal area; The abnormal region annotation information is obtained by at least two annotators independently annotating the abnormal region and undergoing consistency review, or the abnormal region annotation information is obtained by an anomaly detection model detecting the processed medical image data.

[0051] In one specific implementation, the abnormal region labeling information includes an anomaly type identifier, morphological parameters of the abnormal region, and / or risk level parameters of the abnormal region. Morphological parameters may include size, major and minor axes, boundary shape description parameters, etc.; risk level parameters can be output according to preset classification rules (e.g., classification according to size thresholds or morphological feature rules), and together with the anomaly type identifier, describe the attributes of the abnormal region.

[0052] In one specific implementation, the annotation information for abnormal regions can be obtained by at least two annotators independently annotating the abnormal regions and undergoing consistency review. Independent annotation can adopt a double-blind annotation method, with annotators outputting the spatial location identifier and attribute annotation of the abnormal region respectively; consistency review can be conducted by the reviewer to make a final decision in case of disagreement, so as to ensure the reliability of the annotation information for abnormal regions.

[0053] In another specific implementation, the abnormal region annotation information is obtained by an anomaly detection model detecting the processed medical image data. The anomaly detection model can output the spatial location identifier of the abnormal region, and can further output at least one of the following: anomaly type identifier, morphological parameters, and risk level parameters. When the output of the anomaly detection model and manual annotation coexist, one can be selected or fused according to preset rules to form abnormal region annotation information for prompt word generation and linkage information output.

[0054] This embodiment defines abnormal region annotation information that includes spatial location and abnormal attributes, and provides an acquisition path for manual consistency review or model detection. This enables abnormal region information to enter the prompt word generation and linkage information construction process in a structured form, providing a data foundation for reporting abnormal descriptions and image location association.

[0055] In this embodiment, the step of generating prompt text based on the anatomical region segmentation mask, the abnormal region annotation information, the processed non-image clinical data, and the medical knowledge entries, according to a preset prompt text template, includes: The examination item information, the anatomical location identification, the abnormal area annotation information, and the processed non-imaging clinical data are filled into the preset prompt word template to generate basic prompt words; Write the abnormality type information and / or diagnostic conclusion information from the medical knowledge entries into the basic prompt words to generate the prompt word text; The preset prompt word template includes fields for examination items, anatomical locations, abnormal information, and non-imaging clinical data.

[0056] In one specific implementation, the preset prompt template includes at least the following fields: examination item field, anatomical location field, abnormal information field, and non-imaging clinical data field. The examination item field describes the current examination type and examination site; the anatomical location field is used to enter the anatomical location identifier; the abnormal information field is used to enter the spatial location identifier of the abnormal area and its abnormal attributes (such as at least one of the abnormality type, morphological parameters, and risk level parameters); and the non-imaging clinical data field is used to enter key fields from the structured medical history field set and the structured indicator field set.

[0057] In one specific implementation, the examination project information, anatomical location identification, abnormal area annotation information, and processed non-imaging clinical data are filled into a preset prompt word template to generate basic prompt words. The filling rules may be as follows: the structured fields are transcribed into natural language fragments or structured label fragments according to the order of the template fields, and the spatial location identifier of the abnormal area is written in a parsable form (such as mask ID, coordinate range, slice number, etc.).

[0058] In one specific implementation, the abnormality type information and / or diagnostic conclusion information from the medical knowledge entries are written into the basic prompt words to generate prompt word text. The writing method can be as follows: the medical knowledge entries are appended to the end of the basic prompt words in the form of "candidate abnormality type / candidate diagnostic conclusion / related knowledge prompt" or inserted after the abnormality information field, so that the prompt word text simultaneously carries the joint constraint information of "image—region—clinical data—knowledge entry".

[0059] This embodiment uses a template-based approach to organize examination items, anatomical locations, abnormal annotations, non-imaging clinical data, and medical knowledge items into prompt text, enabling the multimodal report generation model to obtain a structured and controllable input semantic framework, providing a consistent input representation for subsequent cross-attention fusion and report generation.

[0060] In this embodiment, the multimodal report generation model includes an image encoder, a text encoder, a cross-attention layer, and a text decoder; The image encoder is used to encode the processed medical image data to obtain image features; The text encoder is used to encode the prompt word text to obtain prompt word text features; The cross-attention layer is used to fuse the image features and the cue word text features, as well as the guiding features formed by the anatomical region segmentation mask, to obtain the fused features; The text decoder is used to generate the medical image report text based on the fusion features.

[0061] In one specific implementation, the multimodal report generation model includes an image encoder, a text encoder, a cross-attention layer, and a text decoder. The image encoder encodes the processed medical image data and outputs image features; the text encoder encodes the prompt text and outputs prompt text features; the cross-attention layer establishes attention interactions between image features, prompt text features, and guidance features, and outputs fused features; the text decoder generates medical image report text based on the fused features.

[0062] In one specific implementation, the image encoder can extract features from two-dimensional images or three-dimensional volume data, outputting multi-scale or single-scale image feature representations; the text encoder encodes cue text, outputting text feature representations corresponding to the positions in the text sequence. The above features are received across the attention layer and combined with the guiding features for attention calculation, forming fused features; the text decoder can generate report text word by word in an autoregressive manner, or generate report text content by paragraph / structured field.

[0063] This embodiment sets up a modular structure of image encoding, text encoding, cross-attention fusion, and text decoding within the same model, enabling the prompt text and image features to interact across the cross-attention layer, and the text decoder outputs the report text, thereby realizing a report generation process based on multimodal input.

[0064] In this embodiment, the fusion of the image features, the prompt text features, and the guiding features formed by the anatomical region segmentation mask to obtain the fused features includes: The segmentation mask of the anatomical region is mapped to an attention guidance map to obtain the segmentation mask features; The segmentation mask features, image features, and prompt word text features are input into the cross-attention layer to calculate the fusion of the image features, segmentation mask features, and prompt word text features based on attention weights, and the fused features are output.

[0065] In one specific implementation, the anatomical region segmentation mask is mapped to an attention guidance map to obtain segmentation mask features. The mapping method may include: scaling the mask to match the spatial resolution of the image features, or characterizing the mask to obtain a guidance representation that is aligned with the image features; the attention guidance map is used to represent the degree of attention or region selection weight at different spatial locations.

[0066] In one specific implementation, segmentation mask features, image features, and prompt text features are input together into a cross-attention layer. Attention weights are then used to calculate the fusion of these three features, outputting a fused feature. Cross-attention calculation allows the prompt text features to reference image features and segmentation mask features during query / key-value interactions, thus reflecting the constraint of mask guidance on feature fusion within the attention mechanism. The fused feature serves as input for the subsequent text decoder to generate the report text.

[0067] This embodiment maps the anatomical region segmentation mask to an attention guidance map and uses it as a segmentation mask feature to participate in cross-attention fusion. This allows the cross-attention layer to introduce region guidance information when fusing image features and prompt word text features, thereby forming a "mask-guided multimodal fusion" implementation path at the model structure level.

[0068] In one embodiment, a method for generating large-scale medical image reports based on region guidance, dynamic knowledge enhancement, and multimodal collaborative optimization is provided, including the following steps: Step 1: Obtain multicenter clinical data, including: Imaging data: 2D / 3D CT, MRI, X-ray, etc., including complete examination reports and examination categories; Non-imaging data: Patient medical history (such as past illnesses and surgical history), laboratory indicators (such as complete blood count and tumor markers), and medical records (such as previous diagnoses and treatment plans). Knowledge data sources: the latest clinical guidelines, SNOMED-CT terminology (2024 edition), and guidelines for the diagnosis and treatment of rare diseases.

[0069] Step 2, data standardization and structured cleaning, including: Step 2.1, Image Data Standardization: Convert to DICOM 3.0 format and correct differences in scanning parameters, layer thickness, etc. Step 2.2, Text Report Cleaning: Redundant information is removed based on NER technology, and medical terminology is standardized. Step 2.3, Non-imaging data structuring: Use structured forms to extract key information from the patient's medical history (e.g., "5-year history of diabetes"), standardize laboratory indicators according to the format of "indicator name - value - reference range", and establish a mapping between non-imaging data and patient ID.

[0070] Step 3: Pixel-level segmentation of the anatomical region, including: An improved U-Net network (with a MobileNetV3 encoder and a newly added residual connection module) is used to generate anatomical site masks, and a dynamic attention optimization module is introduced. The importance weights of anatomical regions are defined based on medical knowledge graphs (e.g., in a lung CT scan, the weight of "lung lobe" is 0.8, and the weight of "chest wall" is 0.2). During segmentation, attention weights are dynamically adjusted based on the importance of regions to enhance the feature extraction accuracy of key regions, as shown in the following formula:

[0071] in As regional importance weight, For local pixel features, Global features

[0072] Step 4: In addition to the standard anatomical labels, additionally mark the visually abnormal areas (such as masses, calcifications) on the images from Step 2. The abnormal areas are marked by two senior radiologists in a double-blind manner, and the markings are reviewed and confirmed by a third-party physician. The coordinates, shape, and risk level of the abnormal areas are recorded (e.g., "high risk: lung nodules with a diameter > 8 mm").

[0073] Step 5, dynamic medical knowledge graph construction and updating, including: Step 5.1: Construct a basic disease atlas based on a medical knowledge graph (such as the SNOMED-CT terminology set), which includes a four-dimensional mapping of "anatomical location - abnormality type - examination item - diagnostic conclusion"; Step 5.2: Integrate the "Guidelines for the Diagnosis and Treatment of Rare Diseases (2024 Edition)" and establish the association between "rare disease type - imaging characteristics - non-imaging indicators" (e.g., "pulmonary lymphangioleiomyomatosis: diffuse cystic lesions in both lungs + chylothorax"). Step 5.3: Dynamic update. Synchronize the latest guidelines and terminology standards through the interface to automatically update the map mapping relationship.

[0074] Step 6: Associate the report data from Step 1 with the segmentation results from Step 2 using a unique identifier (e.g., patient ID + examination time) to generate a data record containing: Image data: original image, segmentation mask, anomaly region annotation; Text data: structured reports, medical history information; Knowledge data: Knowledge graph association information (including rare disease sub-graphs).

[0075] Step 7: When dividing all the processed data obtained in Step 6 into training / test / validation sets in a 7:2:1 ratio, stratified sampling by disease type is used. Disease stratification: Ensure a balanced proportion of common diseases (such as pneumonia and myocardial infarction); Modal stratification: Ensures uniform distribution of different image modalities; Case type stratification: Ensure that rare diseases and complex complications account for no less than 5% of each set to avoid underfitting the model to special cases.

[0076] Step 8: Use U-Net as the base framework, but replace its encoder with the lightweight MobileNetV3 to reduce the number of parameters; use Dice Loss as the loss function, train the segmentation network on the data from Step 7, set the batch size to 16, use AdamW as the optimizer, set the initial learning rate to 0.001 and use cosine annealing scheduling, and train for 100 rounds.

[0077] in, and Representing pixels The label value and the predicted value, where N is the total number of pixels.

[0078] Step 9: Perform morphological operations (opening operation to remove noise and hole filling) on ​​the segmentation result seg_res output by the model in Step 8, and remove false detection regions with too small area through connected component analysis.

[0079] Step 10: Convert the segmentation results from Step 9 and the anomaly annotations from Step 7 into natural language prompts, in the format: "[Examination Item] Image, [Anatomical Location] region found [Abnormality Type] (Risk Level: [High / Medium / Low]); Patient's [Medical History] and [Laboratory Indicators] are abnormal; According to the latest guidelines, this abnormality may correspond to [Diagnostic Direction], which needs to be further confirmed in conjunction with [Non-Imaging Indicators]; Please focus on analyzing this region and generate a report."

[0080] Step 11, Training data augmentation and expansion, including: Step 11.1: Pair the prompt words with the corresponding report text and use them as input to the large model. Data augmentation methods include synonym replacement (e.g., replacing "nodule" with "placeholder") and sentence generalization (converting active and passive sentences). Step 11.2: Based on the rare disease knowledge sub-graph, use the "feature transfer" technique to generate simulated data (transfer the image features of known rare diseases to normal images to generate rare disease simulated samples) to expand the amount of training data for special cases.

[0081] Step 12: Use the BLIP model, with ViT-L / 16 for image encoder and BERT-base (fusion of medical pre-trained weights) for text encoder. Fusion is performed using a cross-attention layer. Inputs include: the prompt text from Step 11, the original image from Step 7, and the segmentation mask (as an attention guide map). These inputs are fed into the cross-attention layer, and dynamic fusion of multimodal features is achieved through attention weight calculation, as shown in the following formula:

[0082] in For text features, Features of the original image For segmentation mask features Attention weights for each modality (learned automatically through model training).

[0083] Step 13: Calculate the average training loss value BCELoss using the cross-entropy loss function;

[0084]

[0085] in For the Softmax function, For the sample The probability prediction value, For the sample The labels are N, where N is the total number of samples.

[0086] Step 14: Repeat steps 12 and 13 until the loss function converges to less than the set threshold, then end the model training and obtain the report generation model. Step 15, Model Application: Step 15.1 First, input the image to be used to generate the report into the segmentation model to obtain the segmentation result; Step 15.2 Convert the style results obtained in Step 15.1 into large model prompt words, input them into the main report generation model, and obtain the generated report results.

[0087] Step 16, Model Application, including: Step 16.1 Report Classification: Based on clinical scenario needs, generate a "Quick Report" (within 3 minutes, core abnormality description) and a "Detailed Report" (within 10 minutes, including imaging features, non-imaging indicators, diagnostic basis, and recommended treatment). Step 16.2 Visual Association: Embed an "image-text" linkage link in the report. Doctors can click on the abnormal description to jump to the corresponding area of ​​the image, and at the same time display the segmentation mask and abnormal feature annotation of the area; This embodiment differs from existing technologies that directly input complete medical image data into a large model for end-to-end report generation. Reports generated by this method suffer from insufficient quality and accuracy. The root cause lies in the fact that the model only processes raw pixel data, lacking an understanding of the inherent anatomical structure of medical images and failing to effectively distinguish and focus on key local areas of diagnostic significance. This invention constructs a region-guided mechanism by introducing spatial prior information about anatomical sites. This mechanism enables the large model to focus on specific anatomical regions for feature extraction and report generation, thereby achieving precise capture and description of subtle local lesions. This technical approach not only significantly improves the clinical relevance and accuracy of the generated reports but also possesses excellent interpretability due to the clear correlation between the generation process and anatomical structures, solving the trust problem of end-to-end black-box models in clinical applications.

[0088] In one embodiment, a medical image report generation device is provided, comprising: The acquisition module is used to acquire the medical imaging data to be analyzed, as well as the examination item information and non-imaging clinical data corresponding to the medical imaging data. The non-imaging clinical data includes medical history information and / or laboratory indicator information. The processing module is used to perform image standardization processing on the medical image data to obtain processed medical image data, and to perform structured processing on the non-image clinical data to obtain processed non-image clinical data. The knowledge graph module is used to acquire medical knowledge data and construct or update a dynamic medical knowledge graph, which includes the relationship between anatomical sites, abnormality types, examination items and diagnostic conclusions. The segmentation module is used to input the processed medical image data into the anatomical region segmentation model and output an anatomical region segmentation mask, wherein the anatomical region segmentation mask includes anatomical location identifiers. An anomaly information acquisition module is used to acquire anomaly region annotation information corresponding to the processed medical image data, wherein the anomaly region annotation information includes the spatial location identifier of the anomaly region. The query module is used to query the dynamic medical knowledge graph based on the examination item information and the anatomical location identifier to obtain medical knowledge entries that match the examination item information and the anatomical location identifier; wherein, the medical knowledge entries include abnormality type information and / or diagnostic conclusion information; The prompt word generation module is used to generate prompt word text according to a preset prompt word template based on the anatomical region segmentation mask, the abnormal region annotation information, the processed non-image clinical data, and the medical knowledge entries. The report generation module is used to input the processed medical image data, the anatomical region segmentation mask, and the prompt text into the multimodal report generation model, output medical image report text, and linkage information for associating the abnormal descriptions in the medical image report text with the spatial location identifiers of the abnormal regions.

[0089] The specific implementation details of each module can be found in the above description of the limitations on the medical image report generation method, and will not be repeated here.

[0090] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for generating medical image reports, characterized in that, The method includes: Acquire the medical imaging data to be analyzed, as well as the examination item information and non-imaging clinical data corresponding to the medical imaging data, wherein the non-imaging clinical data includes medical history information and / or laboratory indicator information; The medical imaging data is subjected to image standardization processing to obtain processed medical imaging data, and the non-imaging clinical data is subjected to structured processing to obtain processed non-imaging clinical data. Acquire medical knowledge data and construct or update a dynamic medical knowledge graph, which includes the relationship between anatomical locations, abnormality types, examination items and diagnostic conclusions; The processed medical image data is input into the anatomical region segmentation model, and the anatomical region segmentation mask is output, which includes anatomical location identifiers. Obtain the abnormal region annotation information corresponding to the processed medical image data, wherein the abnormal region annotation information includes the spatial location identifier of the abnormal region; Based on the examination item information and the anatomical location identifier, the dynamic medical knowledge graph is queried to obtain medical knowledge entries that match the examination item information and the anatomical location identifier; wherein, the medical knowledge entries include abnormality type information and / or diagnostic conclusion information; Based on the anatomical region segmentation mask, the abnormal region annotation information, the processed non-image clinical data, and the medical knowledge entries, prompt text is generated according to a preset prompt text template. The processed medical image data, the anatomical region segmentation mask, and the prompt text are input into the multimodal report generation model, which outputs a medical image report text and linkage information for associating the abnormal descriptions in the medical image report text with the spatial location identifiers of the abnormal regions.

2. The method according to claim 1, characterized in that, The image standardization processing of the medical image data includes: Convert the medical image data into a preset medical image standard format; Consistency correction is performed on the scanning parameters and slice thickness differences of the medical image data.

3. The method according to claim 1, characterized in that, The structuring of the non-imaging clinical data includes at least one of the following: The key information of the medical history information is extracted using a structured form to obtain a set of structured medical history fields; The laboratory indicator information is standardized according to the format of indicator name, indicator value, and reference range to obtain a set of structured indicator fields. Establish an association mapping between the structured medical history field set and the structured indicator field set and the patient identifier.

4. The method according to claim 1, characterized in that, The acquisition of medical knowledge data and the construction or updating of a dynamic medical knowledge graph include: A basic disease atlas is constructed based on medical terminology standards. The basic disease atlas is used to indicate the four-dimensional mapping relationship between anatomical location, abnormality type, examination items and diagnostic conclusion. A rare disease sub-atlas is established based on clinical guidelines and / or rare disease diagnosis and treatment standards. The rare disease sub-atlas is used to indicate the relationship between rare disease types, imaging features and non-imaging indicators. The basic disease atlas and the rare disease sub-atlas are integrated into the dynamic medical knowledge graph; Obtain updated content of the clinical guidelines, rare disease diagnosis and treatment standards, and / or medical terminology standards, and update the mapping and / or association relationships in the dynamic medical knowledge graph based on the updated content.

5. The method according to claim 1, characterized in that, The anatomical region segmentation model is an encoder-decoder segmentation network, which includes an encoder, a decoder, residual connections, and a dynamic attention optimization module. The encoder is a lightweight backbone network, which includes MobileNetV3; The residual connection is used to connect the output features of the encoder with the input features of the decoder; The dynamic attention optimization module is configured to determine the importance weight of anatomical regions in the dynamic medical knowledge graph based on the examination item information, and adjust the attention allocation of segmentation features according to the importance weight of the anatomical regions.

6. The method according to claim 1, characterized in that, The abnormal area labeling information includes an abnormality type identifier, morphological parameters of the abnormal area, and / or risk level parameters of the abnormal area. The abnormal region annotation information is obtained by at least two annotators independently annotating the abnormal region and undergoing consistency review, or the abnormal region annotation information is obtained by an anomaly detection model detecting the processed medical image data.

7. The method according to claim 1, characterized in that, The step of generating prompt text based on the anatomical region segmentation mask, the abnormal region annotation information, the processed non-imaging clinical data, and the medical knowledge entries, according to a preset prompt text template, includes: The examination item information, the anatomical location identification, the abnormal area annotation information, and the processed non-imaging clinical data are filled into the preset prompt word template to generate basic prompt words; Write the abnormality type information and / or diagnostic conclusion information from the medical knowledge entries into the basic prompt words to generate the prompt word text; The preset prompt word template includes fields for examination items, anatomical locations, abnormal information, and non-imaging clinical data.

8. The method according to claim 1, characterized in that, The multimodal report generation model includes an image encoder, a text encoder, a cross-attention layer, and a text decoder; The image encoder is used to encode the processed medical image data to obtain image features; The text encoder is used to encode the prompt word text to obtain prompt word text features; The cross-attention layer is used to fuse the image features and the cue word text features, as well as the guiding features formed by the anatomical region segmentation mask, to obtain the fused features; The text decoder is used to generate the medical image report text based on the fusion features.

9. The method according to claim 8, characterized in that, The fused features, obtained by fusing the image features, the prompt text features, and the guidance features formed by the anatomical region segmentation mask, include: The segmentation mask of the anatomical region is mapped to an attention guidance map to obtain the segmentation mask features; The segmentation mask features, image features, and prompt word text features are input into the cross-attention layer to calculate the fusion of the image features, segmentation mask features, and prompt word text features based on attention weights, and the fused features are output.

10. A medical image report generation device, characterized in that, The device includes: The acquisition module is used to acquire the medical imaging data to be analyzed, as well as the examination item information and non-imaging clinical data corresponding to the medical imaging data. The non-imaging clinical data includes medical history information and / or laboratory indicator information. The processing module is used to perform image standardization processing on the medical image data to obtain processed medical image data, and to perform structured processing on the non-image clinical data to obtain processed non-image clinical data. The knowledge graph module is used to acquire medical knowledge data and construct or update a dynamic medical knowledge graph, which includes the relationship between anatomical sites, abnormality types, examination items and diagnostic conclusions. The segmentation module is used to input the processed medical image data into the anatomical region segmentation model and output an anatomical region segmentation mask, wherein the anatomical region segmentation mask includes anatomical location identifiers. An anomaly information acquisition module is used to acquire anomaly region annotation information corresponding to the processed medical image data, wherein the anomaly region annotation information includes the spatial location identifier of the anomaly region. The query module is used to query the dynamic medical knowledge graph based on the examination item information and the anatomical location identifier to obtain medical knowledge entries that match the examination item information and the anatomical location identifier; wherein, the medical knowledge entries include abnormality type information and / or diagnostic conclusion information; The prompt word generation module is used to generate prompt word text according to a preset prompt word template based on the anatomical region segmentation mask, the abnormal region annotation information, the processed non-image clinical data, and the medical knowledge entries. The report generation module is used to input the processed medical image data, the anatomical region segmentation mask, and the prompt text into the multimodal report generation model, output medical image report text, and linkage information for associating the abnormal descriptions in the medical image report text with the spatial location identifiers of the abnormal regions.

Images