Image report generation method, system and equipment based on multi-granularity knowledge fusion

Abstract

The invention discloses an image report generation method, system and device based on multi-granularity knowledge fusion, and the method comprises an image report pre-generation unit which is used for receiving a medical image as input, and processing the medical image through an image report generation model to obtain an initial image report text; the knowledge retrieval unit is used for retrieving related medical entities and definition knowledge thereof from a public medical knowledge base, a local self-built medical knowledge base and a real-time data source based on the initial image report text to obtain multi-source medical knowledge; the knowledge fusion unit is used for fusing the multi-source medical knowledge to obtain a fused knowledge text; and the final report generation unit is used for combining the medical image, the initial image report text and the fused knowledge text, and regenerating an image report through an image report generation model to obtain a final image report. According to the method, diagnosis errors can be remarkably reduced, the working efficiency of doctors is improved, and meanwhile reports easier to understand are provided for patients.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an image report generation method, system and device based on multi-granularity knowledge fusion. Background Art

[0002] In the field of medical imaging diagnosis and report generation, the development of automation and intelligence has become an important means to improve medical efficiency and accuracy. At present, medical imaging report generation systems mainly rely on visual feature extraction and natural language processing technologies, but these systems still have many shortcomings in multi-granularity knowledge fusion, clinical relevance, and semantic accuracy. Existing systems usually extract features of medical images through a single visual encoder and generate reports using a pre-trained language model. However, this method often fails to fully utilize the rich knowledge in the medical knowledge base when dealing with complex images and multimodal information, resulting in a large gap in the professionalism, accuracy, and interpretability of the generated reports.

[0003] Specifically, the existing medical imaging report generation systems mainly face the following problems: First, a single visual feature extraction method is difficult to capture the complex details in medical images, especially in multi-scale and multi-modal imaging data. Second, most of the existing knowledge retrieval and fusion methods rely on static medical knowledge bases and cannot dynamically integrate the latest medical research results and knowledge in real-time data sources. This makes the generated reports lack clinical relevance and easily miss important information. In addition, the existing report generation model has bottlenecks in multimodal information fusion and cannot effectively integrate image and text information, resulting in defects in the generated reports in terms of semantic accuracy and coherence. Summary of the invention

[0004] In view of this, the present invention aims at the problems of insufficient semantic accuracy and low clinical relevance in the existing medical image analysis and report generation technology. To this end, the present invention adopts the following technical solution: an image report generation system based on multi-granularity knowledge fusion, which includes: An image report pre-generation unit, for receiving a medical image as an input, and processing the medical image through an image report generation model to obtain an initial image report text; A knowledge retrieval unit is used to retrieve relevant medical entities and their definition knowledge from public medical knowledge bases, local self-built medical knowledge bases and real-time data sources based on the initial imaging report text to obtain multi-source medical knowledge; A knowledge fusion unit, used for fusing the multi-source medical knowledge to obtain a fused knowledge text; The final report generation unit is used to combine the medical image, the initial image report text and the fusion knowledge text, and regenerate the image report through the image report generation model to obtain the final image report.

[0005] Preferably, the image report generation model includes a visual encoder, an adapter module, a bidirectional Transformer encoder and a large language model; wherein: The visual encoder extracts multi-scale visual features of medical images through an improved residual network. The improved residual network introduces an adaptive feature fusion module based on the traditional ResNet to dynamically fuse visual features of different levels, thereby enhancing the ability to capture image details; The adapter module adapts the features extracted by the visual encoder to effectively interact with the subsequent large language model; The bidirectional Transformer encoder performs word segmentation and embedding processing on the initial imaging report text, and encodes the text through a bidirectional Transformer encoder. The bidirectional Transformer encoder introduces context-aware position encoding to capture long-distance dependencies in the text and contextual semantics of medical entities; The large language model receives inputs from the visual encoder and the text feature encoding layer in the pre-generation stage, and adopts a multimodal hierarchical attention mechanism to achieve two-way interaction between visual features and text features, including visual-to-text attention and text-to-visual attention, so as to dynamically adjust the weight distribution of visual features and text features.

[0006] Preferably, the knowledge retrieval unit extracts medical entities from the initial imaging report text by using entity recognition technology or deep learning technology, and performs knowledge retrieval based on these entities to obtain medical entity definitions and knowledge related to the imaging.

[0007] Preferably, the knowledge fusion unit performs knowledge fusion in one or more of the following ways: direct concatenation, textual representation of knowledge based on structured data to text, and integration of large language models.

[0008] Preferably, the textual representation of knowledge based on structured data to text is achieved by the following steps: For structured data based on medical knowledge graph, the entities and their relationships in the knowledge graph are encoded through graph neural network to generate graph embedding representation, which is converted into natural language text through graph-to-sequence model. The graph-to-sequence model adopts multi-hop attention mechanism to capture the semantic information of multi-hop relationships in the knowledge graph and dynamically generate coherent medical description text.

[0009] Preferably, the final report generating unit supports generating reports in different formats, including concise reports, detailed reports, professional reports, vernacular reports and visually annotated reports.

[0010] Preferably, it also includes: The report presentation unit is used to combine the medical image and the final image report for segmentation, and mark the areas, problems and symptoms involved in the final image report to achieve an interpretable effect of combining pictures and texts.

[0011] Preferably, the report presentation unit includes a multimodal alignment and segmentation module, a region-aware segmentation model, and a graphic-text interaction generation module; wherein: The multimodal alignment and segmentation module realizes accurate alignment of medical images and report text through a multimodal alignment network; The region-aware segmentation model accurately segments medical images through a region-aware segmentation network; The image-text interaction generation module realizes mutual interpretation between images and texts through an image-text interaction generator.

[0012] A second embodiment of the present invention provides a method for generating an image report based on multi-granularity knowledge fusion, which includes: receiving a medical image as an input, and processing the medical image through an image report generation model to obtain an initial image report text; Based on the initial imaging report text, relevant medical entities and their definition knowledge are retrieved from public medical knowledge bases, local self-built medical knowledge bases and real-time data sources to obtain multi-source medical knowledge; Fusing the multi-source medical knowledge to obtain a fused knowledge text; Combining medical images, initial image report text and fused knowledge text, the image report is regenerated through an image report generation model to obtain the final image report.

[0013] The third embodiment of the present invention also provides an image report generation device based on multi-granularity knowledge fusion, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the image report generation method based on multi-granularity knowledge fusion as described above.

[0014] In summary, the present invention organically combines visual features, medical knowledge base knowledge, real-time auxiliary knowledge and pre-generated content through a multi-granularity knowledge fusion mechanism, so that the automated analysis and report generation of medical images have higher semantic accuracy and clinical relevance. The system can significantly reduce diagnostic errors and improve the work efficiency of doctors, while providing patients with more understandable reports, and promoting the automation and intelligent development of the medical imaging field. By dynamically updating the knowledge base, the system can always generate reports based on the latest medical knowledge to ensure the timeliness and accuracy of the reports. In addition, the system supports the generation of reports in different formats to meet the needs of different clinical scenarios and patients, thereby improving the flexibility and practicality of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is an overall schematic diagram of an image report generation system based on multi-granularity knowledge fusion according to a first embodiment of the present invention; Figure 2 It is a working principle diagram of the image report generation system based on multi-granularity knowledge fusion according to the first embodiment of the present invention; Figure 3 Schematic diagram of the flow of the method for generating an image report based on multi-granularity knowledge fusion according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] The present invention provides an image report generation system based on multi-granularity knowledge fusion, aiming to improve the accuracy and intelligence level of medical image data analysis and report generation. Figure 1 To Attachment Figure 2 , the specific implementation methods of the present invention are described in detail.

[0019] like Figure 1 As shown, in this embodiment, the image report generating system includes: The image report pre-generating unit 10 is used to receive a medical image as an input, and process the medical image through an image report generating model to obtain an initial image report text.

[0020] Specifically, in this embodiment, the image report pre-generation unit 10 is used to process the medical image through the image report generation model to obtain the initial image report text.

[0021] In this embodiment, if Figure 2As shown in the figure, the imaging report generation model adopts a multimodal hierarchical attention mechanism (MHAM), and its specific structure includes a visual feature encoding layer, a text feature encoding layer and a multimodal interaction layer. The visual feature encoding layer extracts multi-scale visual features of medical images through an improved residual network (ResNet). The improved residual network introduces an adaptive feature fusion module (AFFM) on the basis of traditional ResNet, which is used to dynamically fuse visual features of different levels to enhance the ability to capture image details. Specifically, the adaptive feature fusion module dynamically adjusts the weights of features at each level by learning the correlation between features at different levels, so that important details in the image can be better retained when extracting high-dimensional features. The text feature encoding layer encodes the initial imaging report text through a bidirectional Transformer encoder. The bidirectional Transformer encoder introduces context-aware position encoding (CAPE) on the basis of traditional Transformer to capture long-distance dependencies in the text and the contextual semantics of medical entities. Context-aware position encoding introduces context information into position encoding, enabling the model to better understand medical terms and long-distance dependencies in the text, thereby improving the accuracy of text encoding. The multimodal interaction layer realizes the bidirectional interaction between visual features and text features through a multimodal hierarchical attention mechanism, which includes vision-to-text attention (VTA) and text-to-vision attention (TVA), which is used to dynamically adjust the weight distribution of visual features and text features to enhance the model's ability to integrate image and text information. Specifically, the vision-to-text attention mechanism dynamically adjusts the weight of visual features in text generation by calculating the similarity between visual features and text features, so that the generated text can more accurately reflect the details in the image. The text-to-vision attention mechanism dynamically adjusts the weight of text features in image segmentation by calculating the similarity between text features and visual features, so that image segmentation can more accurately reflect the content described in the text.

[0022] The knowledge retrieval unit 20 is used to retrieve relevant medical entities and their definition knowledge from a public medical knowledge base, a local self-built medical knowledge base and a real-time data source based on the initial imaging report text to obtain multi-source medical knowledge.

[0023] Specifically, the knowledge retrieval unit 20 retrieves relevant medical knowledge from multi-source data based on the initial imaging report text. The multi-source data includes a public medical knowledge base, a local medical knowledge base, and a real-time data source. Among them, the public medical knowledge base includes UMLS, PubMed, ICD10, SNOMEDCT, etc., which are used to retrieve entities, definitions, and knowledge related to imaging. Specifically, UMLS is used to obtain standardized representations of medical entities, PubMed is used to obtain the latest medical research results, ICD10 is used to obtain international classification standards for diseases, and SNOMEDCT is used to obtain standardized representations of clinical terms and concepts. The local medical knowledge base includes medical guidelines, rare disease diagnosis and treatment guidelines, patient medical records, medical degree theses, etc., and obtains imaging-related entities, definitions, and knowledge through text retrieval or vector retrieval. Specifically, text retrieval obtains medical knowledge related to the initial imaging report text from the local medical knowledge base through keyword matching and semantic analysis; vector retrieval converts text and image features into vector representations and calculates the similarity between vectors, thereby obtaining medical knowledge related to image features and the initial imaging report text from the local medical knowledge base. Real-time data sources include general search engines or medical engines, such as Baidu, Sogou, Bing, Google, Wanfang, CNKI, etc., which are used to obtain auxiliary knowledge related to image features and initial descriptions. Specifically, real-time data sources use web crawler technology to obtain the latest medical research results, clinical guidelines, patient medical records and other auxiliary knowledge from general search engines or medical engines to supplement the lack of medical knowledge in the initial imaging report text.

[0024] The knowledge fusion unit 30 is used to fuse the multi-source medical knowledge to obtain a fused knowledge text.

[0025] Specifically, the knowledge fusion unit 30 fuses multi-source medical knowledge based on artificial intelligence technology to obtain a fused knowledge text. The knowledge fusion method includes direct splicing or segmented splicing, textual representation of structured data to text, and integration of large language models. The direct splicing or segmented splicing method generates a fused knowledge text by splicing or segmented splicing the acquired knowledge units in a certain order. Specifically, the direct splicing method splices all acquired knowledge units into a complete text in sequence; the segmented splicing method splices the knowledge units in segments according to the type and content of the knowledge units to improve the readability and logic of the text. The textual representation of structured data to text is achieved in the following way: the knowledge graph-driven text generation module encodes the entities and their relationships in the knowledge graph through the graph neural network (GNN) based on the structured data of the medical knowledge graph to generate a graph embedding representation. The graph embedding representation is converted into natural language text through a graph-to-sequence (G2S) model, which uses a multi-hop attention mechanism (MHAM) to capture the semantic information of multi-hop relationships in the knowledge graph and dynamically generate coherent medical description text. Specifically, the multi-hop attention mechanism gradually captures the multi-hop relationships in the knowledge graph through multiple attention layers, thereby generating more accurate and coherent medical description text. The structured data parsing and semantic mapping module parses non-text data such as tables, key-value pairs, and tree structures in the medical knowledge base into intermediate representations through a structured data parser (SDP), and the intermediate representations are converted into natural language text through a semantic mapping network (SMN), which uses a hierarchical attention mechanism (HAM) to perform semantic parsing and dynamic weight assignment on different levels of structured data to generate accurate and readable medical text. Specifically, the hierarchical attention mechanism gradually parses different levels of structured data through multiple attention layers to generate more accurate and readable medical texts. Large model enhancement technologies include Domain-Adaptive Pretraining (DAP), Knowledge Injection Mechanism (KIM), and Multi-Task Joint Training (MTJT), which are used to improve the medical accuracy and professionalism of generated texts.Domain adaptive pre-training uses massive text data in the medical field (such as medical literature, clinical guidelines, and medical records) for secondary pre-training on the basis of a general large language model to enhance the model's ability to understand the terminology, grammar, and semantics in the medical field. Specifically, domain adaptive pre-training introduces a large amount of text data in the medical field into the general large language model, so that the model can better understand the terminology, grammar, and semantics in the medical field, thereby improving the medical accuracy of the generated text. The knowledge injection mechanism dynamically injects entity relationships, clinical rules, and the latest medical research results in the medical knowledge graph into the generation process of the large language model through the External Knowledge Injection Module (EKIM) to improve the medical accuracy and professionalism of the generated text. Specifically, the external knowledge injection module injects entity relationships, clinical rules, and the latest medical research results in the medical knowledge graph into the large language model in the form of vectors or text, so that the model can refer to these external knowledge when generating text, thereby generating more accurate and professional medical text. Multi-task joint training completes multiple medical tasks (such as entity recognition, relationship extraction, and text generation) by jointly training a large language model to enhance the model's comprehensive understanding and application capabilities of medical knowledge. Specifically, multi-task joint training introduces multiple medical tasks into the large language model, allowing the model to comprehensively consider multiple medical knowledge when generating text, thereby improving the medical accuracy and professionalism of the generated text. Finally, the fused knowledge text generated by the above knowledge fusion method is passed to the final report generation unit for generating the final report.

[0026] The final report generating unit 40 is used to combine the medical image, the initial image report text and the fused knowledge text, and regenerate the image report through the image report generating model to obtain the final image report.

[0027] Specifically, the final report generation unit 40 regenerates the image report through the image report generation model according to the medical image and the initial image report text, the retrieved medical knowledge or the fused knowledge text. The image report generation model adopts a multimodal hierarchical attention mechanism, and its specific structure includes a visual feature encoding layer, a text feature encoding layer and a multimodal interaction layer. The visual feature encoding layer extracts the multi-scale visual features of the medical image through an improved residual network, wherein the improved residual network introduces an adaptive feature fusion module on the basis of the traditional residual network, which is used to dynamically fuse visual features of different levels to enhance the ability to capture image details. The text feature encoding layer encodes the initial image report text and the fused knowledge text through a bidirectional Transformer encoder, wherein the bidirectional Transformer encoder introduces context-aware position encoding on the basis of the traditional Transformer, which is used to capture long-distance dependencies in the text and the contextual semantics of medical entities. The multimodal interaction layer realizes the bidirectional interaction between visual features and text features through the multimodal hierarchical attention mechanism, wherein the multimodal hierarchical attention mechanism includes visual-to-text attention and text-to-visual attention, which is used to dynamically adjust the weight distribution of visual features and text features to enhance the model's ability to integrate image and text information. Specifically, the visual-to-text attention mechanism dynamically adjusts the weight of visual features in text generation by calculating the similarity between visual features and text features, so that the generated text can more accurately reflect the details in the image; the text-to-visual attention mechanism dynamically adjusts the weight of text features in image segmentation by calculating the similarity between text features and visual features, so that image segmentation can more accurately reflect the content described in the text. The final report generation unit supports the generation of reports in different formats, including concise reports, detailed reports, professional reports, vernacular reports, and visual annotation reports to meet the needs of different clinical scenarios and patients. Specifically, concise reports are suitable for rapid diagnosis and preliminary screening, detailed reports are suitable for comprehensive diagnosis and detailed analysis, professional reports are suitable for reference and research by medical experts, vernacular reports are suitable for patients' understanding and communication, and visual annotation reports are suitable for accurate correspondence and interpretation of images and texts.

[0028] The report presentation unit 50 combines the medical image and the final report to perform precise segmentation, mark the involved areas, problems and symptoms, and achieve an interpretable effect of combining pictures and texts.

[0029] The report presentation unit 50 includes a multimodal alignment and segmentation module, a region-aware segmentation model, and a graphic-text interaction generation module. The multimodal alignment and segmentation module realizes the precise alignment of medical images and report texts through a multimodal alignment network (MAN). The multimodal alignment network adopts a cross-modality attention mechanism (CMAM) to dynamically capture the semantic association between the image region and the text description, and generate an aligned multimodal feature representation. Specifically, the cross-modal attention mechanism dynamically adjusts the weight distribution between the image region and the text description by calculating the similarity between the medical image and the report text, thereby realizing the precise alignment of the image region and the text description. The region-aware segmentation model accurately segments the medical image through a region-aware segmentation network (RASN). The region-aware segmentation network introduces a region attention mechanism (RAM) based on the traditional segmentation model, which is used to dynamically adjust the weight distribution of image segmentation according to the keywords and semantic information in the report text to enhance the segmentation accuracy of the target area. Specifically, the regional attention mechanism dynamically adjusts the weight distribution of image segmentation by calculating the similarity between the keywords in the report text and the image region, so that the image segmentation can more accurately reflect the description content in the report text. The image-text interactive generation module realizes the mutual interpretation of image and text through the image-text interactive generator (ITIG), which adopts a bidirectional generation strategy, including image-to-text generation and text-to-image generation. Image-to-text generation generates corresponding text descriptions according to the segmented image region, and the generation process dynamically adjusts the semantic consistency of the generated text through the image-to-text attention mechanism (ITA). Specifically, the image-to-text attention mechanism dynamically adjusts the weight distribution of the generated text by calculating the similarity between the segmented image region and the generated text, so that the generated text can more accurately describe the details in the image. Text-to-image generation generates corresponding image tags according to the description in the report text, and the generation process dynamically adjusts the position and shape of the image tags through the text-to-image attention mechanism (TIA). Specifically, the text-to-image attention mechanism dynamically adjusts the position and shape of the image markers by calculating the similarity between the description in the report text and the image markers, so that the image markers can more accurately reflect the description content in the report text.

[0030] It should be noted that in order to further improve the accuracy and real-time performance of the system, the system supports dynamic updating of the knowledge base, uses vector retrieval technology to ensure the consistency between the knowledge base and real-time auxiliary knowledge, and gives priority to the use of the latest knowledge sources to improve the accuracy of the report. Specifically, the knowledge base is dynamically updated by regularly obtaining the latest medical knowledge from the public medical knowledge base, the local medical knowledge base and the real-time data source to update the content in the knowledge base. Vector retrieval technology converts medical knowledge and real-time auxiliary knowledge into vector representations and calculates the similarity between vectors, thereby ensuring the consistency between the knowledge base and the real-time auxiliary knowledge. Prioritizing the use of the latest knowledge sources by setting weights or priorities makes it possible to give priority to the latest medical knowledge and real-time auxiliary knowledge when generating reports, thereby improving the accuracy of the reports.

[0031] The system and method of the present invention are described in detail below in conjunction with a specific application scenario.

[0032] Suppose a hospital needs to analyze a patient's lung CT image and generate a report. First, the image report pre-generation unit receives the patient's lung CT image and extracts the multi-scale visual features of the image through the improved residual network (ResNet). The improved residual network introduces the adaptive feature fusion module (AFFM) on the basis of the traditional ResNet, dynamically fuses visual features of different levels, and enhances the ability to capture image details. Subsequently, the initial image report text is encoded by the bidirectional Transformer encoder. The bidirectional Transformer encoder introduces context-aware position encoding (CAPE) on the basis of the traditional Transformer to capture the long-range dependencies in the text and the contextual semantics of medical entities. The adapter module adapts the features extracted by the visual feature encoding layer to effectively interact with the large language model. In the pre-generation stage, the large language model receives inputs from the visual feature encoding layer and the text feature encoding layer, and uses the multimodal hierarchical attention mechanism (MHAM) to achieve bidirectional interaction between visual features and text features to generate the initial image report text.

[0033] Next, the knowledge retrieval unit retrieves relevant medical entities and their definition knowledge from public medical knowledge bases, local self-built medical knowledge bases, and real-time data sources based on the initial imaging report text. For example, medical entities and their definition knowledge related to lung CT images are retrieved from public medical knowledge bases such as UMLS and PubMed; medical entities and their definition knowledge related to images are obtained through text retrieval or vector retrieval from local self-built medical knowledge bases such as medical guidelines, rare disease diagnosis and treatment guidelines, and patient medical records; auxiliary knowledge related to image features and initial descriptions is obtained from general search engines such as Baidu and Sogou or medical engines such as Wanfang and HowNet. The knowledge retrieval unit extracts medical entities from the initial imaging report text through entity recognition technology or deep learning technology, and performs knowledge retrieval based on these entities to obtain medical entities related to images and their definition knowledge.

[0034] Then, the knowledge fusion unit fuses multi-source medical knowledge through artificial intelligence technology to generate fused knowledge text. Specifically, the knowledge fusion unit directly splices or segmented splices the acquired knowledge units by direct splicing to form a preliminary fused knowledge text. At the same time, the knowledge fusion unit uses technology based on deep neural networks or large language models to complete the knowledge textual representation of structured data to text (Data2Text). Based on the structured data of the medical knowledge graph, the entities and their relationships in the knowledge graph are encoded through the graph neural network (GNN) to generate a graph embedding representation, which is converted into natural language text through the graph to sequence (G2S) model. The graph to sequence model uses a multi-hop attention mechanism (MHAM) to capture the semantic information of multi-hop relationships in the knowledge graph and dynamically generate coherent medical description text. In addition, the non-text data such as tables, key-value pairs, and tree structures in the medical knowledge base are parsed into intermediate representations through the structured data parser (SDP) and converted into natural language text through the semantic mapping network (SMN). The semantic mapping network uses a hierarchical attention mechanism (HAM) to perform semantic parsing and dynamic weight assignment on different levels of structured data to generate accurate and readable medical text. Finally, the knowledge fusion unit uses a large language model to integrate all acquired knowledge and generate fused knowledge text.

[0035] The final report generation unit combines medical images, initial image report text and fused knowledge text, and regenerates the image report through the image report generation model. The image report generation model includes an image report generation model based on a Transformer model, a residual network model, a convolutional neural network, a recurrent neural network, a visual language large model (VLLM) or a multimodal large model. The model adopts a multimodal hierarchical attention mechanism (MHAM), including visual-to-text attention (VTA) and text-to-visual attention (TVA), to dynamically adjust the weight distribution of visual features and text features to enhance the model's ability to integrate image and text information. The final report generation unit supports the generation of reports in different formats, including concise reports, detailed reports, professional reports, vernacular reports and visual annotation reports to meet the needs of different clinical scenarios and patients. For example, for doctors, detailed professional reports can be generated; for patients, vernacular reports can be generated to facilitate patient understanding.

[0036] Finally, the report presentation unit combines the medical image and the final report to perform precise segmentation and mark the areas, problems and symptoms involved in the image report to achieve an interpretable effect of combining images and text. The report presentation unit includes a multimodal alignment and segmentation module, a region-aware segmentation model and a graphic-text interaction generation module. The multimodal alignment and segmentation module achieves precise alignment of medical images and report texts through a multimodal alignment network (MAN). The multimodal alignment network uses a cross-modal attention mechanism (CMAM) to dynamically capture the semantic association between image regions and text descriptions and generate aligned multimodal feature representations. The region-aware segmentation model accurately segments medical images through a region-aware segmentation network (RASN). The region-aware segmentation network introduces a regional attention mechanism (RAM) based on the traditional segmentation model to dynamically adjust the weight distribution of image segmentation according to keywords and semantic information in the report text to enhance the segmentation accuracy of the target area. The image-text interactive generation module realizes the mutual interpretation of images and texts through the image-text interactive generator (ITIG). The image-text interactive generator adopts a bidirectional generation strategy, including image-to-text generation and text-to-image generation, to dynamically adjust the semantic consistency of the generated text and the position and shape of the image markers. Image-to-text generation generates corresponding text descriptions based on the segmented image area. The generation process dynamically adjusts the semantic consistency of the generated text through the image-to-text attention mechanism (ITA). Text-to-image generation generates corresponding image markers based on the description in the report text. The generation process dynamically adjusts the position and shape of the image markers through the text-to-image attention mechanism (TIA). Through the above steps, the report presentation unit generates an interpretable medical image report that combines images and texts and provides it to doctors and patients.

[0037] The present invention organically combines visual features, medical knowledge base knowledge, real-time auxiliary knowledge and pre-generated content through a multi-granularity knowledge fusion mechanism, so that the automated analysis and report generation of medical images have higher semantic accuracy and clinical relevance. The system can significantly reduce diagnostic errors and improve the work efficiency of doctors, while providing patients with more understandable reports, and promoting the automation and intelligent development of the medical imaging field. By dynamically updating the knowledge base, the system can always generate reports based on the latest medical knowledge to ensure the timeliness and accuracy of the reports. In addition, the system supports the generation of reports in different formats to meet the needs of different clinical scenarios and patients, thereby improving the flexibility and practicality of the system.

[0038] See also Figure 3 The second embodiment of the present invention provides an image report generation method based on multi-granularity knowledge fusion, which includes: S201, receiving a medical image as input, and processing the medical image through an image report generation model to obtain an initial image report text; S202, based on the initial imaging report text, relevant medical entities and their definition knowledge are retrieved from a public medical knowledge base, a local self-built medical knowledge base, and a real-time data source to obtain multi-source medical knowledge; S203, fusing the multi-source medical knowledge to obtain a fused knowledge text; S204, combining the medical image, the initial image report text and the fused knowledge text, regenerating the image report through an image report generation model to obtain a final image report.

[0039] The third embodiment of the present invention also provides an image report generation device based on multi-granularity knowledge fusion, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the image report generation method based on multi-granularity knowledge fusion as described above.

[0040] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0041] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0042] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, electronic device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0043] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image report generation system based on multi-granularity knowledge fusion, characterized in that: include: An image report pre-generation unit, for receiving a medical image as an input, and processing the medical image through an image report generation model to obtain an initial image report text; A knowledge retrieval unit is used to retrieve relevant medical entities and their definition knowledge from public medical knowledge bases, local self-built medical knowledge bases and real-time data sources based on the initial imaging report text to obtain multi-source medical knowledge; A knowledge fusion unit, used for fusing the multi-source medical knowledge to obtain a fused knowledge text; The final report generation unit is used to combine the medical image, the initial image report text and the fusion knowledge text, and regenerate the image report through the image report generation model to obtain the final image report.

2. The imaging report generation system based on multi-granularity knowledge fusion according to claim 1, characterized in that: The image report generation model includes a visual encoder, an adapter module, a bidirectional Transformer encoder and a large language model; wherein: The visual encoder extracts multi-scale visual features of medical images through an improved residual network. The improved residual network introduces an adaptive feature fusion module based on the traditional ResNet to dynamically fuse visual features of different levels, thereby enhancing the ability to capture image details; The adapter module adapts the features extracted by the visual encoder to effectively interact with the subsequent large language model; The bidirectional Transformer encoder performs word segmentation and embedding processing on the initial imaging report text, and encodes the text through a bidirectional Transformer encoder. The bidirectional Transformer encoder introduces context-aware position encoding to capture long-distance dependencies in the text and contextual semantics of medical entities; The large language model receives inputs from the visual encoder and the text feature encoding layer in the pre-generation stage, and adopts a multimodal hierarchical attention mechanism to achieve two-way interaction between visual features and text features, including visual-to-text attention and text-to-visual attention, so as to dynamically adjust the weight distribution of visual features and text features.

3. The imaging report generation system based on multi-granularity knowledge fusion according to claim 1, characterized in that: The knowledge retrieval unit extracts medical entities from the initial imaging report text through entity recognition technology or deep learning technology, and performs knowledge retrieval based on these medical entities to obtain medical entity definitions and knowledge related to the imaging.

4. The imaging report generation system based on multi-granularity knowledge fusion according to claim 1, characterized in that: The knowledge fusion unit performs knowledge fusion in one or more of the following ways: direct concatenation, textual representation of knowledge based on structured data to text, and integration of large language models.

5. The imaging report generation system based on multi-granularity knowledge fusion as claimed in claim 4, characterized in that: The textual representation of knowledge based on structured data to text is achieved by the following steps: For structured data based on medical knowledge graph, the entities and their relationships in the knowledge graph are encoded through graph neural network to generate graph embedding representation, which is converted into natural language text through graph-to-sequence model. The graph-to-sequence model adopts multi-hop attention mechanism to capture the semantic information of multi-hop relationships in the knowledge graph and dynamically generate coherent medical description text.

6. The imaging report generation system based on multi-granularity knowledge fusion according to claim 1, characterized in that: The final report generating unit supports generating reports in different formats, including concise reports, detailed reports, professional reports, vernacular reports and visually annotated reports.

7. The imaging report generation system based on multi-granularity knowledge fusion according to claim 1, characterized in that: Also includes: The report presentation unit is used to combine the medical image and the final image report for segmentation, and mark the areas, problems and symptoms involved in the final image report to achieve an interpretable effect of combining pictures and texts.

8. The imaging report generation system based on multi-granularity knowledge fusion according to claim 7, characterized in that: The report presentation unit includes a multimodal alignment and segmentation module, a region-aware segmentation model, and a graphic-text interaction generation module; wherein: The multimodal alignment and segmentation module realizes accurate alignment of medical images and report text through a multimodal alignment network; The region-aware segmentation model accurately segments medical images through a region-aware segmentation network; The image-text interaction generation module realizes mutual interpretation between images and texts through an image-text interaction generator.

9. A method for generating an image report based on multi-granularity knowledge fusion, characterized in that: include: receiving a medical image as an input, and processing the medical image through an image report generation model to obtain an initial image report text; Based on the initial imaging report text, relevant medical entities and their definition knowledge are retrieved from public medical knowledge bases, local self-built medical knowledge bases and real-time data sources to obtain multi-source medical knowledge; Fusing the multi-source medical knowledge to obtain a fused knowledge text; Combining medical images, initial image report text and fused knowledge text, the image report is regenerated through an image report generation model to obtain the final image report.

10. An imaging report generation device based on multi-granularity knowledge fusion, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the image report generation method based on multi-granularity knowledge fusion as described in claim 9.

Images