Image report generation method, system and equipment based on multi-modal learning optimization

Abstract

The invention discloses an image report generation method, system and device based on multi-modal learning optimization, and the method comprises an image retrieval module which is used for carrying out the image feature extraction of an input medical image, and obtaining similar medical images and reports; the simulation generation module is used for guiding an image report generation model to generate an initial draft report by utilizing the medical image and the report; the personalized diagnosis module is used for carrying out personalized adjustment on the report generation model according to the historical report preference of the doctor or the historical medical data of the patient; the reinforcement learning optimization module is used for dynamically adjusting a text generation mode based on a reinforcement learning report optimization strategy and optimizing an image report generation model in combination with doctor scoring, automatic evaluation and clinical feedback; and the iterative refining module is used for performing text retrieval by using the initial draft report as a query, acquiring text fragments which are semantically similar to the initial draft report, and performing neural symbol reasoning optimization to generate a final report.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, system and device for generating an image report based on multimodal learning optimization. Background Art

[0002] In the field of medical imaging diagnosis, the accuracy and efficiency of imaging reports are crucial to doctors' clinical decision-making. Currently, the generation of imaging reports mainly relies on manual input by doctors, which is not only time-consuming and labor-intensive, but also easily affected by doctors' personal experience and fatigue, resulting in large differences in the accuracy and consistency of reports. In recent years, with the development of deep learning and natural language processing technologies, automated imaging report generation systems have gradually become a research hotspot. These systems usually use deep learning models to extract features from imaging data and generate corresponding text reports.

[0003] However, existing automated imaging report generation systems have some shortcomings. First, these systems often lack an in-depth understanding and dynamic updating of medical knowledge when generating reports, resulting in reports that are not accurate enough in terms of medical semantic expression. Second, existing systems usually use fixed generation models that cannot be adjusted according to the individual needs of doctors, making it difficult to meet the diagnostic habits of different doctors and the individual needs of patients. In addition, the generated reports also have deficiencies in medical logic and reasoning consistency, especially when dealing with complex cases, the generated reports may lack accurate medical reasoning and diagnostic recommendations. Finally, the existing systems lack an effective feedback mechanism and cannot continuously optimize the generation model through clinical feedback from doctors, resulting in poor clinical adaptability and self-adaptation of the system. Summary of the invention

[0004] In view of this, the present invention provides a method, system and device for generating an image report based on multimodal learning optimization to improve the above problems. An embodiment of the present invention provides an image report generating system based on multimodal learning optimization, which includes: Image retrieval module, used to extract image features from input medical images, perform semantic enhancement in combination with dynamic medical knowledge base, perform similar image retrieval based on image feature vectors, and obtain medical images that are visually semantically similar to the input medical images and their corresponding reports; An imitation generation module is used to use the retrieved medical images and their reports to guide the image report generation model to generate an initial draft report; wherein the image report generation model adopts semantic deep learning guided by multimodal attention and optimizes the semantic fusion of image features and text through multi-level attention layers; Personalized diagnosis module, which is used to personalize the report generation model based on the doctor's historical report preferences or the patient's historical medical data, so that the generated report is more in line with individual diagnosis needs; The reinforcement learning optimization module is used to optimize the report strategy based on reinforcement learning, dynamically adjust the text generation method, and optimize the imaging report generation model by combining doctor scoring, automated evaluation, and clinical feedback; The iterative refinement module is used to use the initial draft report as a query to perform text retrieval, obtain text fragments that are semantically similar to the initial draft report, and use them as context information for neural symbolic reasoning optimization to generate a final report.

[0005] Preferably, the image retrieval module includes a visual encoder and a dynamic medical knowledge base update mechanism, wherein the visual encoder converts the input image into a feature vector, and the dynamic medical knowledge base update mechanism automatically extracts relevant medical knowledge from the latest published medical literature and clinical data, and updates the knowledge base content in real time.

[0006] Preferably, the image report generation model is constructed based on Transformer, residual network, recurrent neural network, visual language large model or multimodal large model, and adopts multimodal attention mechanism to enhance the semantic alignment capability of image and text.

[0007] Preferably, the imitation generation module also has an adversarial training optimization mechanism, which evaluates the authenticity of the generated initial draft report through a discriminator network. The discriminator network generates the distribution difference between the initial draft report and the real report based on the comparison, and provides a feedback signal based on the distribution difference to optimize the generation quality of the image report generation model.

[0008] Preferably, the reinforcement learning optimization module includes a reinforcement learning reward function and an adaptive training mechanism, wherein the reinforcement learning reward function optimizes the accuracy, clinical relevance and semantic consistency of the generated report in combination with doctor scores, user feedback and automated evaluation indicators.

[0009] Preferably, it also includes: The visual explanation and doctor feedback module includes a thermal map annotation function and a doctor interactive modification function. The thermal map annotation function marks key diagnostic areas, and the doctor interactive modification function supports doctors to modify the generated report. The modified content can be used as training data to optimize the imaging report generation model.

[0010] The embodiment of the present invention also provides a method for generating an image report based on multimodal learning optimization, which includes: Extract image features from the input medical image, perform semantic enhancement in combination with the dynamic medical knowledge base, perform similar image retrieval based on the image feature vector, and obtain medical images that are visually semantically similar to the input medical image and their corresponding reports; Using the retrieved medical images and their reports, guiding the image report generation model to generate an initial draft report; wherein the image report generation model adopts semantic deep learning guided by multimodal attention, and optimizes the semantic fusion of image features and text through multi-level attention layers; Personalize the report generation model based on the doctor's historical reporting preferences or the patient's historical medical data, so that the generated report is more in line with individual diagnostic needs; A report optimization strategy based on reinforcement learning dynamically adjusts the text generation method and optimizes the imaging report generation model by combining doctor scoring, automated evaluation, and clinical feedback; Using the initial draft report as a query, text retrieval is performed to obtain text fragments that are semantically similar to the initial draft report, and these fragments are used as context information for neural symbolic reasoning optimization to generate the final report.

[0011] An embodiment of the present invention also provides an image report generation device based on multimodal learning optimization, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the image report generation method based on multimodal learning optimization as described above.

[0012] The present invention maintains the timeliness and accuracy of medical knowledge and enhances the medical semantic expression of images through a dynamic medical knowledge base update mechanism; optimizes the semantic alignment between images and texts through a deep multimodal attention mechanism to generate high-quality preliminary reports; meets individualized diagnosis needs through a personalized adjustment mechanism; dynamically adjusts the text generation strategy through reinforcement learning, combines doctor scores and automated evaluation indicators to improve the accuracy and clinical relevance of reports; ensures the consistency of medical logic and reasoning of generated reports through a neural symbolic reasoning mechanism; and further optimizes the clinical adaptability of the model by marking key diagnostic areas with heat maps and supporting doctor interactive modifications.

[0013] The present invention is highly innovative and has clinical application value, and can significantly improve the quality and efficiency of image report generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 It is an overall schematic diagram of an imaging report generation system based on deep multimodal learning optimization according to a first embodiment of the present invention; Figure 2This is a flowchart of an image report generation system based on deep multimodal learning optimization according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The present invention provides an imaging report generation system, method and device based on deep multimodal learning optimization, aiming to significantly improve the innovation and accuracy of imaging report generation.

[0018] The specific implementation modes of the present invention are described in detail below with reference to the accompanying drawings.

[0019] like Figure 1 As shown, the imaging report generation system based on deep multimodal learning optimization of the embodiment of the present invention includes an imaging retrieval module 1, an imitation generation module 2, a personalized diagnosis module 3, a reinforcement learning optimization module 4, an iterative refinement module 5, and a visualization explanation and doctor feedback module 6. The specific functions and implementation methods of each module are as follows: Image retrieval module 1 In this embodiment, the image retrieval module 1 is used to extract features from the input medical image and perform semantic enhancement in combination with the dynamic medical knowledge base. Specifically, the image retrieval module 1 includes a visual encoder and a dynamic medical knowledge base update mechanism. The visual encoder converts the input medical image into a feature vector, which contains information such as the color, texture, shape and image semantic vector of the image. The visual encoder can use an existing deep learning model, such as a convolutional neural network (CNN) or other visual encoding models, to convert image data into a high-dimensional feature vector.

[0020] The dynamic medical knowledge base update mechanism is used to automatically extract relevant medical knowledge from the latest published medical literature and clinical data, and update the knowledge base content in real time to maintain the timeliness and accuracy of the system's medical knowledge. The dynamic medical knowledge base update mechanism can regularly obtain the latest medical knowledge from medical databases, journals, and clinical reports through crawler technology, parse and structure this knowledge using natural language processing (NLP) technology, and then update it to the knowledge base. The content of the knowledge base can include various medical terms, diagnostic criteria, treatment plans, etc., which will be used for subsequent image feature enhancement and report generation optimization.

[0021] In actual applications, when a doctor inputs a patient's imaging data into the system, the visual encoder first extracts features from the imaging data and generates a feature vector. Then, the dynamic medical knowledge base update mechanism retrieves images and their corresponding reports that are visually and semantically similar to the input image based on the feature vector. For example, if a lung CT image is input, the system will retrieve other lung CT images and their reports from the knowledge base, which can be used as a reference for generating an initial draft report.

[0022] Imitation generation module 2: In this embodiment, the imitation generation module 2 is used to use the retrieved medical images and their reports to guide the image report generation model to generate an initial draft report. This module includes an image report generation model and an adversarial training optimization mechanism. The image report generation model can be built based on Transformer, residual network, recurrent neural network, visual language large model (VLLM) or multimodal large model, and adopts a multimodal attention mechanism to further enhance the semantic alignment ability of images and texts through fine-grained and coarse-grained attention layers.

[0023] Specifically, the imaging report generation model aligns the feature vector of the input image with the retrieved similar images and their reports to generate a preliminary draft report. The multimodal attention mechanism achieves fine-grained and coarse-grained semantic fusion through multi-level attention layers, thereby optimizing the semantic alignment between image features and text. The fine-grained attention layer focuses on local details in the image, such as specific lesion areas, while the coarse-grained attention layer focuses on the overall features of the image, such as the overall layout of the anatomical structure. Through this multi-level attention mechanism, the generated report can more accurately describe the key information in the image.

[0024] The adversarial training optimization mechanism evaluates the authenticity of the generated preliminary draft reports through the discriminator network. The discriminator network provides feedback signals based on the distribution difference between the generated reports and the real reports to optimize the generation quality of the imaging report generation model. The discriminator network can adopt a deep learning model, such as the discriminator in GAN (Generative Adversarial Network), which can be trained to distinguish between the generated reports and the real reports, thereby providing effective feedback signals. These feedback signals are used to adjust the parameters of the generation model to improve the accuracy and naturalness of the generated reports.

[0025] Personalized diagnostic module In this embodiment, the personalized diagnosis module 3 is used to personalize the report generation model according to the doctor's historical report preferences or the patient's historical medical data. The module includes a personalized adjustment mechanism. The personalized adjustment mechanism fine-tunes the imaging report generation model by analyzing the doctor's historical report preferences or the patient's medical history data to ensure that the report content better meets the individual's clinical needs.

[0026] Specifically, the personalized adjustment mechanism first collects the doctor's historical report data and analyzes the terms, sentences, and diagnostic logic commonly used by doctors in their reports. Then, the generation model is fine-tuned based on these preferences so that the reports it generates are closer to the doctor's style. In addition, the personalized adjustment mechanism can also collect the patient's medical history data, such as previous test results, treatment plans, and medical history, and use it as input to the generation model to generate more personalized reports. For example, if a patient has a specific medical history, the system can generate a report containing this medical history information to help doctors better understand the patient's condition.

[0027] Reinforcement Learning Optimization Module 4: In this embodiment, the reinforcement learning optimization module 4 is used to dynamically adjust the text generation method based on the report optimization strategy of reinforcement learning. This module includes a reinforcement learning reward function and an adaptive training mechanism. The reinforcement learning reward function combines doctor scores, user feedback, and automated evaluation indicators (such as BLEU, ROUGE, CIDEr) to optimize the accuracy, clinical relevance, and semantic consistency of the generated report. Doctor scores and user feedback can be scored by the doctor's satisfaction with the generated report, and the automated evaluation indicators evaluate the report quality by calculating the similarity between the generated report and the real report.

[0028] Specifically, the reinforcement learning reward function calculates the reward value for generating a report based on the combined results of doctor scores, user feedback, and automated evaluation indicators. The generation model adjusts its generation strategy based on these reward values, gradually improving the quality of the generated report. The adaptive training mechanism is based on the dynamic optimization strategy of reinforcement learning, combined with personalized feedback from doctors and patients, to automatically adjust the text generation strategy, thereby improving the quality and clinical applicability of the generated report. For example, if a doctor is not satisfied with a part of the generated report, the system will record this feedback and optimize it in the subsequent generation process.

[0029] Iterative Refining Module 5: In this embodiment, the iterative refinement module 5 is used to use the draft report as a query to perform text retrieval, obtain text fragments that are semantically similar to the draft report, and use them as context information for neural symbolic reasoning optimization. This module includes a neural symbolic reasoning mechanism and text retrieval. The neural symbolic reasoning mechanism combines the symbolic reasoning rules and deep learning models in the medical field to optimize the medical reasoning process in the report and ensure that the generated report meets the medical logic and reasoning requirements.

[0030] Specifically, after the generative model generates a preliminary report, the iterative refinement module 5 uses the report as a query and retrieves text fragments that are semantically similar to the draft report from the dynamic medical knowledge base through text retrieval. These text fragments are input into the neural symbolic reasoning mechanism as context information, and the report is further optimized through symbolic reasoning rules and deep learning models. For example, if a lesion is described in the preliminary report but its possible cause is not detailed, the neural symbolic reasoning mechanism can supplement the cause analysis through the retrieved relevant text fragments to ensure the completeness and accuracy of the report content.

[0031] Visual explanation and doctor feedback module In this embodiment, the visual interpretation and doctor feedback module 6 is used to provide a visual mapping of images and texts, use heat maps to mark key diagnostic areas, and support doctors to interactively modify report content. This module includes a heat map marking function and a doctor interactive modification function. The heat map marking function helps doctors understand the key parts of the report by marking key diagnostic areas. The doctor interactive modification function supports doctors to modify the generated report, and the modified content can be used as training data to optimize the image report generation model.

[0032] Specifically, after the generation model generates a preliminary report, the visual interpretation and doctor feedback module 6 aligns the report with the image data and generates a heat map. The heat map displays key diagnostic areas in the image, such as lesions, abnormal structures, etc., through annotations of different colors. Doctors can quickly locate the key content in the report through the heat map to improve the readability and comprehensibility of the report. In addition, doctors can interactively modify the generated report, such as adding or deleting certain content. These modifications will be recorded and used to train the model, thereby continuously optimizing the performance of the generation model. For example, if the doctor thinks that the description of a lesion in the generated report is not detailed enough, it can be supplemented through the interactive modification function. The system will record these modifications and optimize them in the subsequent generation process.

[0033] The following will take a practical application scenario of generating a lung CT image report as an example to describe the implementation process of the present invention in detail: Suppose a radiologist in a hospital needs to generate a lung CT image report for a patient. The doctor first inputs the patient's lung CT image data into the system, and the visual encoder extracts features from the image data to generate a feature vector containing the image color, texture, shape, and image semantic vector. The dynamic medical knowledge base update mechanism retrieves lung CT images and their corresponding reports that are visually and semantically similar to the input image based on the feature vector. These similar images and their reports will serve as reference data for generating the initial draft report.

[0034] The imitation generation module 2 uses the retrieved similar images and their reports to guide the image report generation model to generate a preliminary report. The generation model uses a multimodal attention mechanism to optimize the semantic alignment between image features and text through fine-grained and coarse-grained attention layers. The preliminary report will contain a preliminary description of the patient's lung CT images and diagnostic recommendations.

[0035] The reinforcement learning optimization module 4 calculates the reward value for generating the report based on the comprehensive results of the doctor's score, user feedback, and automated evaluation indicators. The generation model adjusts its generation strategy based on these reward values ​​and gradually improves the quality of the generated report. The personalized diagnosis module 3 fine-tunes the generation model based on the doctor's historical report preferences or the patient's historical medical data to ensure that the report content is more in line with the individual's clinical needs. For example, if the patient has a history of smoking, the system can generate a report that includes the effects of smoking on the lungs.

[0036] Iterative Refinement Module 5 uses the preliminary report as a query and retrieves text fragments that are semantically similar to the draft report from the dynamic medical knowledge base through text retrieval. These text fragments are input into the neural symbolic reasoning mechanism as context information, and the report is further optimized through symbolic reasoning rules and deep learning models. For example, if a lung nodule is described in the preliminary report but its possible cause is not detailed, the neural symbolic reasoning mechanism can supplement the cause analysis through the retrieved relevant text fragments to ensure the completeness and accuracy of the report content.

[0037] After the generation model generates a preliminary report, the visual interpretation and doctor feedback module 6 aligns the report with the image data and generates a heat map. The heat map displays key diagnostic areas in the image, such as lung nodules, abnormal structures, etc., through annotations of different colors. Doctors can use heat maps to quickly locate key content in the report and improve the readability and comprehensibility of the report. In addition, doctors can interactively modify the generated report, such as adding or deleting certain content. These modifications will be recorded and used to train the model, thereby continuously optimizing the performance of the generation model. For example, if the doctor believes that the description of a lung nodule in the generated report is not detailed enough, it can be supplemented through the interactive modification function. The system will record these modifications and optimize them in the subsequent generation process.

[0038] See also Figure 2 The second embodiment of the present invention further provides a method for generating an image report based on multimodal learning optimization, which includes: S201, extracting image features from the input medical image, performing semantic enhancement in combination with a dynamic medical knowledge base, performing similar image retrieval based on image feature vectors, and obtaining medical images that are visually semantically similar to the input medical image and their corresponding reports; S202, using the retrieved medical images and their reports to guide an image report generation model to generate an initial draft report; wherein the image report generation model adopts semantic deep learning guided by multimodal attention, and optimizes the semantic fusion of image features and text through a multi-level attention layer; S203, personalizing the report generation model according to the doctor's historical report preference or the patient's historical medical data, so that the generated report better meets individual diagnosis needs; S204, a report optimization strategy based on reinforcement learning, dynamically adjusts the text generation method, and optimizes the imaging report generation model by combining doctor scoring, automated evaluation, and clinical feedback; S205, using the initial draft report as a query, performing text retrieval, obtaining text fragments that are semantically similar to the initial draft report, and using them as context information to perform neural symbolic reasoning optimization to generate a final report.

[0039] The third embodiment of the present invention also provides an image report generation device based on multimodal learning optimization, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the image report generation method based on multimodal learning optimization as described above.

[0040] The present invention maintains the timeliness and accuracy of medical knowledge and enhances the medical semantic expression of images through a dynamic medical knowledge base update mechanism; optimizes the semantic alignment between images and texts through a deep multimodal attention mechanism to generate high-quality preliminary reports; meets individualized diagnosis needs through a personalized adjustment mechanism; dynamically adjusts the text generation strategy through reinforcement learning, combines doctor scores and automated evaluation indicators to improve the accuracy and clinical relevance of reports; ensures the consistency of medical logic and reasoning of generated reports through a neural symbolic reasoning mechanism; and further optimizes the clinical adaptability of the model by marking key diagnostic areas with heat maps and supporting doctor interactive modifications.

[0041] The present invention is highly innovative and has clinical application value, and can significantly improve the quality and efficiency of image report generation.

[0042] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0043] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0044] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, electronic device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0045] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An imaging report generation system based on multimodal learning optimization, characterized in that: include: The image retrieval module is used to extract image features from the input medical image, perform semantic enhancement in combination with the dynamic medical knowledge base, perform similar image retrieval based on the image feature vector, and obtain medical images that are visually semantically similar to the input medical image and their corresponding reports; An imitation generation module is used to use the retrieved medical images and their reports to guide the image report generation model to generate an initial draft report; wherein the image report generation model adopts semantic deep learning guided by multimodal attention to optimize the semantic fusion of image features and text through multi-level attention layers; Personalized diagnosis module, which is used to personalize the report generation model based on the doctor's historical report preferences or the patient's historical medical data, so that the generated report is more in line with individual diagnosis needs; The reinforcement learning optimization module is used to optimize the report strategy based on reinforcement learning, dynamically adjust the text generation method, and optimize the imaging report generation model by combining doctor scoring, automated evaluation, and clinical feedback; The iterative refinement module is used to use the initial draft report as a query to perform text retrieval, obtain text fragments that are semantically similar to the initial draft report, and use them as context information for neural symbolic reasoning optimization to generate a final report.

2. The system according to claim 1, characterized in that The image retrieval module includes a visual encoder and a dynamic medical knowledge base update mechanism. The visual encoder converts the input image into a feature vector. The dynamic medical knowledge base update mechanism automatically extracts relevant medical knowledge from the latest published medical literature and clinical data, and updates the knowledge base content in real time.

3. The system according to claim 1, characterized in that The image report generation model is built based on Transformer, residual network, recurrent neural network, visual language large model or multimodal large model, and adopts multimodal attention mechanism to enhance the semantic alignment ability of image and text.

4. The system according to claim 3, characterized in that The imitation generation module also has an adversarial training optimization mechanism, which evaluates the authenticity of the generated initial draft report through a discriminator network. The discriminator network generates distribution differences between the initial draft report and the real report based on comparison, and provides feedback signals based on the distribution differences to optimize the generation quality of the image report generation model.

5. The system according to claim 1, wherein: The reinforcement learning optimization module includes a reinforcement learning reward function and an adaptive training mechanism. The reinforcement learning reward function combines doctor scores, user feedback, and automated evaluation indicators to optimize the accuracy, clinical relevance, and semantic consistency of generated reports.

6. The system according to claim 1, wherein: Also includes: The visual explanation and doctor feedback module includes a thermal map annotation function and a doctor interactive modification function. The thermal map annotation function marks key diagnostic areas, and the doctor interactive modification function supports doctors to modify the generated report. The modified content is used as training data to optimize the imaging report generation model.

7. A method for generating an image report based on multimodal learning optimization, characterized in that: include: Extract image features from the input medical image, perform semantic enhancement in combination with the dynamic medical knowledge base, perform similar image retrieval based on the image feature vector, and obtain medical images that are visually semantically similar to the input medical image and their corresponding reports; Using the retrieved medical images and their reports, guiding the image report generation model to generate an initial draft report; wherein the image report generation model adopts semantic deep learning guided by multimodal attention, and optimizes the semantic fusion of image features and text through multi-level attention layers; Personalize the report generation model based on the doctor's historical reporting preferences or the patient's historical medical data, so that the generated report is more in line with individual diagnostic needs; A report optimization strategy based on reinforcement learning dynamically adjusts the text generation method and optimizes the imaging report generation model by combining doctor scoring, automated evaluation, and clinical feedback; Using the initial draft report as a query, text retrieval is performed to obtain text fragments that are semantically similar to the initial draft report, and these fragments are used as context information for neural symbolic reasoning optimization to generate the final report.

8. An imaging report generation device based on multimodal learning optimization, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the image report generation method based on multimodal learning optimization as described in claim 7.

Images