Interactive methods, devices, computer equipment, storage media, and software products for medical imaging and image reporting.

Abstract

The present application relates to a method, device, computer equipment, storage medium and program product for the interaction of medical images and image reports, the method comprising: obtaining a medical image and an image report of the medical image; segmenting the medical image according to the text of interest selected from the image report to obtain an area of ​​interest; searching a database according to the area of ​​interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report. The method can provide different image presentations of similar lesion descriptions, and provide interactive image report reading according to similar typical cases.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium and computer program product for interaction between medical images and image reports. Background Art

[0002] In the medical field, after collecting medical images, doctors write imaging reports and read the images, which are two separate tasks. Information cannot be transmitted between the two links, resulting in each link taking more time.

[0003] Therefore, the traditional method has the problem of poor interactivity between medical images and imaging reports, which affects the speed of writing imaging reports and reading images. Summary of the invention

[0004] Based on this, it is necessary to provide an interactive method, device, computer equipment, computer-readable storage medium and computer program product for medical images and image reports to address the technical problems that the interactivity between medical images and image reports in the above-mentioned method is poor, which affects the speed of writing image reports and reading images.

[0005] In a first aspect, the present application provides a method for interacting medical images and image reports. The method comprises:

[0006] Obtaining medical images and imaging reports of said medical images;

[0007] According to the text of interest selected from the image report, the medical image is segmented to obtain a region of interest;

[0008] A database is searched according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report.

[0009] In one embodiment, the segmenting of the medical image to obtain the region of interest according to the text of interest selected from the image report includes:

[0010] Inputting the text of interest and the medical image into a first segmentation model for segmentation processing to obtain the region of interest;

[0011] The first segmentation model is obtained by training a pre-trained segmentation model using a sample medical image as an input variable, a sample text of interest of the sample medical image as prompt information, and a segmentation result of the sample medical image as an output variable.

[0012] In one of the embodiments, the database stores mapping vectors of a plurality of image report pairs, each image report pair comprising a historical medical image and an image report corresponding to the historical medical image;

[0013] The step of searching a database according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report includes:

[0014] Determining the degree of match between the mapping vector of each image report pair in the database and the coded feature of the region of interest respectively;

[0015] From the database, a corresponding image report pair with a matching degree higher than a threshold is determined, and the historical image report and the historical medical image corresponding to the image report pair are used as the target historical image report and the historical medical image corresponding to the target historical image report.

[0016] In one embodiment, after searching the database according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report, the method further includes:

[0017] converting the historical medical image into text description information, and determining a target report text associated with the region of interest in the target historical image report;

[0018] Adjusting the target report text in the target historical image report according to the historical medical image and the text description information converted from the historical medical image by a multimodal large language model to obtain an adjusted target historical image report;

[0019] The region of interest is marked and displayed in the medical image, and the adjusted target historical image report and the historical medical image corresponding to the target historical image report are displayed.

[0020] In a second aspect, the present application also provides a method for interacting medical images and image reports. The method comprises:

[0021] Obtaining medical images and imaging reports of said medical images;

[0022] Determining corresponding text of interest from the image report according to the region of interest selected from the medical image;

[0023] A database is searched according to the text of interest to obtain a target historical medical image similar to the region of interest and a historical image report corresponding to the target historical medical image.

[0024] In one embodiment, before determining the corresponding text of interest from the image report based on the region of interest selected from the medical image, the method further includes:

[0025] Obtaining location prompt information of the medical image;

[0026] Inputting the location prompt information and the medical image into a second segmentation model for segmentation processing to obtain the region of interest; the second segmentation model is obtained by training a pre-trained segmentation model using sample medical images;

[0027] The step of determining corresponding text of interest from the image report according to the region of interest selected from the medical image includes:

[0028] The region of interest and the image report are input into a comparative language image pre-training model, and the similarity between the region of interest and each text paragraph in the image report is detected by the comparative language image pre-training model. Based on the similarity, the text of interest corresponding to the region of interest is determined from the image report.

[0029] In one embodiment, the database stores mapping vectors of multiple image report pairs, each image report pair includes a historical medical image and an image report corresponding to the historical medical image; searching the database according to the text of interest to obtain a target historical medical image similar to the region of interest and a historical image report corresponding to the target historical medical image includes:

[0030] Determining the degree of match between the mapping vector of each image report pair in the database and the encoding feature of the text of interest respectively;

[0031] From the database, a corresponding image report pair with a matching degree higher than a threshold is determined, and the historical medical image and the historical image report corresponding to the image report pair are used as the target historical medical image and the historical image report corresponding to the target historical medical image.

[0032] In one embodiment, after searching a database according to the text of interest to obtain a target historical medical image similar to the region of interest and a historical image report corresponding to the target historical medical image, the method further includes:

[0033] Converting the target historical medical image into text description information, and determining a target report text associated with the region of interest in a historical image report corresponding to the target historical medical image;

[0034] Adjusting the target report text in the historical image report corresponding to the target historical medical image according to the target historical medical image and the text description information converted from the target historical medical image by a multimodal large language model to obtain an adjusted historical image report;

[0035] The text of interest is marked and displayed in the image report, and the target historical medical image and the adjusted historical image report corresponding to the target historical medical image are displayed.

[0036] In a third aspect, the present application also provides an interactive device for medical images and image reports. The device comprises:

[0037] An acquisition unit, used for acquiring medical images and image reports of the medical images;

[0038] A segmentation unit, configured to segment the medical image according to the text of interest selected from the image report to obtain a region of interest;

[0039] A search unit is used to search a database according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report.

[0040] In a fourth aspect, the present application also provides an interactive device for medical images and image reports. The device comprises:

[0041] An acquisition unit, used for acquiring medical images and image reports of the medical images;

[0042] A determination unit, configured to determine a corresponding text of interest from the image report according to the region of interest selected from the medical image;

[0043] A search unit is used to search a database according to the text of interest to obtain a target historical medical image similar to the region of interest and a historical image report corresponding to the target historical medical image.

[0044] According to a fifth aspect of an embodiment of the present disclosure, a computer device is provided, comprising: a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a method as described in any one of the above items.

[0045] According to a sixth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0046] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0047] The above-mentioned interactive method, device, computer equipment, storage medium and computer program product for medical images and image reports, after obtaining medical images and image reports of medical images, segment the medical images according to the text of interest selected from the image report to obtain the region of interest; search the database according to the region of interest to obtain the target historical image report similar to the text of interest and the historical medical image corresponding to the target historical image report. In this way, the region of interest in the medical image is accurately located based on the text of interest in the image report, and the target historical image report and historical medical image similar to the text of interest selected by the doctor are retrieved by searching the database, thereby providing the doctor with different image presentations of similar lesion descriptions, realizing the provision of interactive image report reading according to similar typical cases, which can improve the efficiency of diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a schematic diagram of the overall architecture of an interactive system for medical images and image reports in one embodiment;

[0049] Figure 2 A schematic diagram of a flow chart of a method for interacting with medical images and image reports in one embodiment;

[0050] Figure 3 A flowchart of an interactive method of text-driven medical images and image reports using brain MRI images as an example in an application example;

[0051] Figure 4 is a schematic diagram of a model architecture of a first segmentation model in an embodiment;

[0052] Figure 5 A detailed flowchart of a text-driven medical image and image report interaction method in one embodiment;

[0053] Figure 6 A schematic flow chart of a method for interacting with medical images and image reports in another embodiment;

[0054] Figure 7 A flowchart of an interactive method of medical imaging and imaging reports based on image-driven technology using brain MRI images as an example in an application example;

[0055] Figure 8 A detailed flowchart of an interactive method for medical images and image reports based on image-driven in one embodiment;

[0056] Fig. 9 is a structural block diagram of an interactive device for medical images and image reports in one embodiment;

[0057] Fig.10 A structural block diagram of an interactive device for medical images and image reports in another embodiment;

[0058] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0061] It should be noted that the technical problem addressed by this application is that in the traditional method, doctors writing imaging reports and reading images are two separate things. Currently, there are only some simple intelligent systems that generate pseudo reports based on image reading. Therefore, this application proposes a system for intelligent interaction between imaging reports and medical images based on multimodal modeling. The core of this interactive system is the dynamic interaction between the text of interest (TOI) and the region of interest (ROI), which specifically includes two functions: one is to locate the corresponding region of interest for the highlighted text, and to retrieve similar reports and related images, which can be achieved through a text-driven artificial intelligence model. The text-driven artificial intelligence model can automatically accurately locate the corresponding region of interest for the selected TOI. The other is to determine the corresponding report text for the selected region of interest and retrieve similar images and related reports, which can be achieved through an image-driven artificial intelligence model. The image-driven artificial intelligence model can automatically segment and process ROI and provide relevant report text. This interactive reading and reporting system can not only help radiologists write imaging reports efficiently, but also provide interactive image text reading for reference based on the learned radiological knowledge and similar typical cases.

[0062] refer to Figure 1 , is a schematic diagram of the overall architecture of a system for intelligent interaction between image reports and medical images, as shown in an embodiment. Figure 1As shown, in the text-driven image processing model, the input is the medical image and the text of interest in the image report of the medical image, the corresponding region of interest is found from the medical image through the text of interest, and then the database is searched according to the region of interest to obtain similar reports and related images, thereby realizing automatic accurate positioning of the image region for the selected TOI and retrieving similar reports and images corresponding to the similar reports. In the image-driven report generation model, the input is the image report of the medical image and the region of interest in the medical image, the corresponding text of interest is found from the image report through the region of interest, and then the database is searched according to the text of interest to obtain similar images and related reports, thereby realizing positioning of the corresponding report text for the selected ROI and retrieving similar images and reports corresponding to the similar images. It should also be noted that in the text-driven image processing model, the type of the input medical image and the type of the related image retrieved from the database (i.e., the historical medical image corresponding to the target historical image report) can be the same or different. For example, the input medical image is an MRI (Magnetic Resonance Imaging) image, and the related image retrieved from the database is a CT (Computed Tomography) image or an X-ray image, etc. By showing different image types to doctors, a more comprehensive representation is provided to facilitate subsequent diagnosis. Similarly, in the image-driven report generation model, the type of the input medical image can be the same as or different from the target historical medical image retrieved from the database, and its purpose is also to provide a more comprehensive representation.

[0063] In the system of intelligent interaction between image reports and medical images, a database is also provided, which stores mapping vectors of multiple image report pairs, each of which includes a historical medical image and an image report corresponding to the historical medical image. In order to maintain patient privacy while realizing data analysis, a local database can be constructed to pair anonymous medical images with de-identified image reports, and these image-report pairing data can be converted into a vector database using an embedding model, thereby maintaining the confidentiality of patient information while promoting efficient data retrieval.

[0064] Specifically, when constructing the database, the initial data set of medical images and imaging reports is first obtained, and then the medical images and imaging reports are standardized to create a unified data set to facilitate model training. This data is further encoded into a high-dimensional vector space through an embedding model, and natural language processing is used for the text data in the imaging report, and computer vision technology is used for the medical images. This dual embedding method can map the two modes into a shared vector space, making it possible to semantically link medical images with corresponding imaging reports. Moreover, the embedding model can combine the latest advances in artificial intelligence and adopt a continuous learning mechanism to adapt to new medical data and clinical use feedback, and the performance of the embedding model can be verified by precision and recall indicators to ensure high retrieval accuracy. In addition, in order to ensure the data security of the database, the database can be encrypted and the access rights to the database can be controlled. This database will serve as the backbone of the workflow of this application and is essential for supporting subsequent text-to-image and image-to-text workflows. It provides a safe and efficient method for associating and retrieving medical images and imaging reports, thereby improving the diagnosis process.

[0065] The following is a detailed description of the implementation methods corresponding to the two functions. It can be understood that the implementation methods corresponding to the two functions can be applied to the terminal, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers.

[0066] refer to Figure 2 , is a flow chart of a method for interacting medical images and image reports based on a text-driven image processing model according to an embodiment. In this embodiment, the method includes the following steps:

[0067] Step S210, obtaining medical images and image reports of the medical images.

[0068] Step S220, segmenting the medical image according to the text of interest selected from the image report to obtain a region of interest.

[0069] Step S230 , searching a database according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report.

[0070] The types of medical images may include MRI images (magnetic resonance images), CT images (ultrasound images), X-ray images, etc.

[0071] There may be one or more target historical image reports.

[0072] In this embodiment, the text of interest is the text selected by the doctor, and the region of interest is the region determined according to the text of interest.

[0073] Specifically, the interaction between medical images and imaging reports based on the text-driven image processing model mainly includes two steps: (1) image segmentation based on text of interest (TOI), that is, segmenting the corresponding region of interest in the medical image according to the text of interest selected by the doctor from the imaging report; (2) ROI-based database retrieval, that is, using the segmented ROI as a query vector to retrieve the vectorized imaging report database, obtain the imaging reports of historical cases similar to the text of interest and the medical images corresponding to the report, so as to provide comparative insights for clinical medicine and provide a basis for subsequent diagnosis.

[0074] Specifically, the acquired medical images and image reports are paired, and the terminal displays the medical images and the image reports of the medical images. Figure 3 This embodiment is described. When the terminal obtains the text of interest selected by the doctor from the brain MRI report (for example Figure 3 The text of interest in the image is “pathy high signal intensity in bilateral thalamus”, and the text of interest can be encoded by a text encoder to create a TOI representation (or TOI feature) of the text of interest as the initial feature of TOI for subsequent processing.

[0075] Then, the brain MRI image is segmented according to the TOI representation. Specifically, the TOI representation and the brain MRI image can be input into the first segmentation model for segmentation. During segmentation, the TOI representation serves as segmentation prompt information of the first segmentation model to guide the segmentation process of the brain MRI image to delineate the region of interest indicated by the text of interest selected by the doctor (i.e., Figure 3 Output 1 in ), and mark and display the region of interest in the brain MRI image, such as Figure 3 The marked region in the figure is 30. This mechanism can identify and emphasize the ROI specified by the doctor in the medical image, effectively converting the text input into a visual segmentation and realizing interactive image analysis.

[0076] After the segmentation is completed, the region of interest can be used as a query vector to query the local pre-built database. After the query is successful, the interactive system will retrieve relevant knowledge and obtain text descriptions similar to the text of interest selected by the doctor (i.e., the target historical imaging report), as well as historical medical images that are similar but different from the segmented ROI. Different historical medical images can present different lesion signs. The purpose of this ROI-based retrieval is to help doctors by providing confirmed information and differential diagnosis pointers. The retrieved target historical imaging report and the corresponding historical medical images will then be displayed adjacent to the ROI on the interactive visualization platform, thereby achieving seamless integration of text and visual data and enriching the diagnostic experience.

[0077] Among them, when the terminal obtains the text of interest selected by the doctor from the image report, before the medical image is segmented, the text of interest can be firstly subjected to grammatical correction processing to obtain the corrected text of interest, and then the corrected text of interest is encoded, and the medical image is segmented according to the obtained text of interest representation to obtain the region of interest. The correction processing can eliminate possible grammatical errors and ambiguities, making the text of interest clearer and more accurate, which is conducive to more accurate segmentation of medical images. The encoding processing can convert text information into a computer-recognizable data format, which is convenient for analyzing and processing the text of interest, thereby realizing accurate segmentation of medical images, effectively extracting regions of interest, and providing more accurate and reliable data support for medical image diagnosis and analysis.

[0078] In the above-mentioned interactive method of medical images and image reports, after obtaining the medical images and image reports of medical images, the medical images are segmented according to the text of interest selected from the image report to obtain the region of interest; the database is searched according to the region of interest to obtain the target historical image report similar to the text of interest and the historical medical image corresponding to the target historical image report. In this way, the region of interest in the medical image is accurately located based on the text of interest in the image report, and the target historical image report and historical medical image similar to the text of interest selected by the doctor are retrieved by searching the database, thereby providing the doctor with different image presentations of similar lesion descriptions, realizing the provision of interactive image report reading according to similar typical cases, which can improve the efficiency of diagnosis.

[0079] In an exemplary embodiment, the above step S220, according to the text of interest selected from the image report, performs segmentation processing on the medical image to obtain the region of interest. The specific implementation method is: input the text of interest and the medical image into the first segmentation model for segmentation processing to obtain the region of interest.

[0080] Among them, the first segmentation model takes the sample medical image as the input variable, the sample text of interest of the sample medical image as the prompt information, and the segmentation result of the sample medical image as the output variable, and is obtained by training the pre-trained open vocabulary segmentation model (Open-Vocabulary MedSAM).

[0081] The first segmentation model, that is, the trained open vocabulary segmentation model, includes the MedSAM module and the CLIP module. Among them, the MedSAM model and the CLIP model are both excellent visual base models. SAM (Segment Anything Model, a large image segmentation model) is well-known for its excellent segmentation capabilities in different fields, while CLIP (Contrastive Language-Image Pre-Training) is highly praised for its zero-shot capabilities (Zero-Shot refers to a model that can complete a task without receiving training data for a specific task). MedSAM is a specialized iterative version of SAM designed specifically for the field of medical imaging. It has been carefully trained on a comprehensive dataset consisting of more than 1.5 million image-mask pairs covering 10 imaging modes and more than 30 cancer types. This extensive training enables MedSAM to perform accurate segmentation tasks that are critical to medical diagnosis. In the process of advancing MedSAM, this application constructs an open vocabulary segmentation model (Open-VocabularyMedSAM, such as Figure 4 As shown in Figure 2, it combines the multimodal advantages of CLIP with the segmentation capabilities of MedSAM. Using knowledge distillation technology, an Adapter is constructed to combine the segmentation knowledge of MedSAM with the multimodal recognition capabilities in CLIP.

[0082] refer to Figure 4 , is a schematic diagram of a model architecture of a first segmentation model shown in an embodiment. Figure 4 As shown in the figure, when the medical image is segmented according to the text of interest selected from the image report, the text of interest is encoded by the CLIP text encoder; the medical image is first encoded by the encoder of the MedSAM model, and then the parameters are fine-tuned by the adapter of the MedSAM model, and then the adjustment result and the medical image are input into the CLIP visual encoder for encoding, and then the parameters are fine-tuned again by the adapter. The fine-tuning result and the text representation obtained by encoding the text of interest are input into the decoder of the MedSAM model for decoding, and the region of interest corresponding to the text of interest in the medical image is output.

[0083] In this embodiment, the text representation of the text of interest (TOI) is used as a prompt to train the first segmentation model. This training method enables the first segmentation model to accurately identify and segment the corresponding region of interest in the medical imaging data.

[0084] In an exemplary embodiment, a database stores mapping vectors of multiple image report pairs, each image report pair includes a historical medical image and an image report corresponding to the historical medical image; in the above step S230, the database is searched according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report, specifically including: determining the matching degree between the mapping vector of each image report pair in the database and the encoding feature of the region of interest respectively; from the database, determining the image report pair whose corresponding matching degree is higher than a threshold, and using the historical image report and the historical medical image corresponding to the image report pair as the target historical image report and the historical medical image corresponding to the target historical image report.

[0085] In a specific implementation, after completing the segmentation of the region of interest, the interactive system can be combined with the embedding model constructed in the database to convert the image report into a vector representation, and use a similarity search algorithm to find the most relevant historical image report that matches the features of the region of interest. For example, the cosine similarity algorithm can be used to determine the matching degree between the mapping vector of each image report pair in the database and the encoded features of the region of interest; the matching degree is sorted from high to low, and the corresponding image report pairs with a matching degree higher than a threshold are selected, and the historical image reports and historical medical images corresponding to such image report pairs are used as the target historical image reports and the historical medical images corresponding to the target historical image reports, thereby providing doctors with report texts similar to the text of interest, and providing doctors with different image presentations of similar lesion descriptions.

[0086] In this embodiment, the target historical image reports are screened by matching the mapping vector of each image report pair in the database with the coding features of the region of interest, so that the historical image reports corresponding to the text of interest can be more accurately matched, thereby ensuring the accuracy and reliability of the screening results.

[0087] In an exemplary embodiment, after the above step S230 searches the database according to the region of interest to obtain the target historical image report similar to the text of interest and the historical medical image corresponding to the target historical image report, the following is further included:

[0088] Step S231, converting the historical medical image into text description information, and determining the target report text associated with the region of interest in the target historical image report;

[0089] Step S232, adjusting the target report text in the target historical image report according to the historical medical images and the text description information converted from the historical medical images by using the multimodal large language model to obtain an adjusted target historical image report;

[0090] Step S233: marking and displaying the region of interest in the medical image, and displaying the adjusted target historical image report and the historical medical image corresponding to the target historical image report.

[0091] In a specific implementation, after obtaining the target historical image report and the corresponding historical medical image from the database according to the region of interest, the historical medical image is converted into text description information, and the target report text associated with the region of interest in the target historical image report is determined. The text description information corresponding to the historical medical image will be made into prompt information of the multimodal large language model (LLM), and then the multimodal large language model uses its huge medical knowledge corpus to adjust the target report text in the target historical image report to obtain a more accurate and detailed text description, and finally outputs a text description similar to the TOI selected by the doctor, and a historical medical image that is similar but not identical to the segmented ROI. The text description similar to the TOI in the adjusted target historical image report, the historical medical image corresponding to the text description, and the ROI highlighted on the medical image input by the doctor will be presented together to provide the doctor with a comprehensive diagnostic view.

[0092] In this embodiment, after completing the database search, the historical medical images and the text description information converted from the historical medical images are used as prompt information, and the target report text in the target historical image report is adjusted through a multimodal large language model, so that the text description of the adjusted target historical image report is more accurate, thereby ensuring the reliability of the target historical image report as a basis for diagnosis.

[0093] To facilitate those skilled in the art to understand the above text-driven interactive process, the following Figure 5 This process is further described.

[0094] refer to Figure 5, showing a detailed flow diagram of a text-driven interactive method for medical imaging and imaging reports. The text-driven artificial intelligence model can be called CLIP2MedSAM (open vocabulary-based text-driven segmentation model), which uses natural language representation to segment regions of interest (ROIs) in medical images using the powerful description processing capabilities of the CLIP model. Unlike traditional segmentation tools that require a lot of manual input and have limited understanding of object attributes, CLIP2MedSAM can automatically locate and segment ROIs, and understand and highlight ROIs based entirely on the text description provided by the doctor.

[0095] The CLIP2MedSAM model operates in two stages. First, when doctors interact with imaging reports, relevant text, i.e., text of interest (TOI), is captured. The CLIP text encoder processes the TOI, performs semantic analysis, and generates a context-rich representation, including extracting key medical entities, their relationships, and attributes from the text as text representations. In addition, the TOI recognition algorithm is used to standardize the text input in the imaging report to improve the consistency of the input data. Subsequently, the segmentation model is used to obtain the ROI based on the TOI and the medical image. Second, the ROI is seamlessly integrated with the retrieval system of the imaging report database as a query vector to quickly query similar imaging report pairs from the database, and the relevant case library stored in the database is used to provide doctors with relevant examples, laying the foundation for evidence-based diagnosis and treatment planning.

[0096] refer to Figure 6 , is a flow chart of a method for interacting medical images and image reports based on an image-driven report generation model according to an embodiment. In this embodiment, the method includes the following steps:

[0097] Step S610, obtaining medical images and image reports of medical images;

[0098] Step S620, determining corresponding text of interest from the image report according to the region of interest selected from the medical image;

[0099] Step S630 , searching a database according to the text of interest to obtain a target historical medical image similar to the region of interest and a historical image report corresponding to the target historical medical image.

[0100] The types of medical images may include MRI images (magnetic resonance images), CT images (ultrasound images), X-ray images, etc.

[0101] There may be one or more target historical medical images.

[0102] In this embodiment, the region of interest is a region selected by a doctor, and the text of interest is a text determined according to the region of interest.

[0103] Specifically, the image-driven report generation model is designed to generate text of interest (TOI) based on a specified region of interest (ROI) and retrieve relevant imaging report pairs that can aid in the diagnostic process. The interactive process consists of two main steps: (1) ROI-based text extraction. When a physician selects a point or draws a box on a medical image (such as MRI or CT), the model is initiated to segment the physician-defined region and generate an ROI. At the same time, it also processes the accompanying imaging report to extract the TOI corresponding to the physician-determined ROI. (2) TOI-based database retrieval. Using the extracted TOI as a query vector, the vectorized imaging report database is retrieved to obtain historical cases that reflect current medical imaging and report data, and relevant precedents are used to enrich clinical evaluation.

[0104] Specifically, the terminal displays paired medical images and image reports of the medical images. For ease of understanding, the following uses brain MRI images and brain MRI reports as examples. Figure 7 This embodiment is described. Figure 7 As shown in the operation in the brain MRI image, when the doctor interacts with the brain MRI image by clicking a point or selecting a box, the image-driven report generation model will respond and use this input as a hint for ROI segmentation through the segmentation network (such as MedSAM) to obtain the region of interest (ROI) in the brain MRI image, such as Figure 7 The area 70 in FIG. 1 is shown as the region of interest.

[0105] Then, according to the region of interest, the corresponding text information is extracted from the brain MRI report as the text of interest (TOI). Figure 7 As shown in Output 1 in the figure, the key text in the brain MRI report that is directly related to the identified ROI is highlighted, thereby focusing on the text of interest.

[0106] After extracting the text of interest, the text of interest is used as a query vector to query the pre-built data to find historical medical images similar to the region of interest as the target historical medical image, thereby obtaining lesion marks similar to the ROI. At the same time, the historical image reports corresponding to the target historical medical image can also be obtained. These historical image reports contain text descriptions that can distinguish the current ROI from the historical ROI. These search results can be displayed near the extracted TOI, providing a comprehensive reference for related and different cases.

[0107] In the above-mentioned interactive method of medical images and image reports, medical images and image reports of medical images are obtained, and corresponding texts of interest are determined from the image reports according to the region of interest selected from the medical images; the database is searched according to the texts of interest to obtain target historical medical images similar to the region of interest and historical image reports corresponding to the target historical medical images. Thus, the texts of interest in the image reports corresponding to the regions of interest in the medical images are located, and the target historical medical images similar to the regions of interest selected by the doctor and historical image reports corresponding to the target historical medical images are retrieved by searching the database, thereby providing different text descriptions of the same lesion signs to the doctor, realizing interactive image report reading according to similar typical cases, which can improve diagnostic efficiency.

[0108] In an exemplary embodiment, the above-mentioned step S620 further includes: obtaining location prompt information of the medical image before determining the corresponding text of interest from the image report based on the region of interest selected from the medical image; inputting the location prompt information and the medical image into a second segmentation model for segmentation processing to obtain the region of interest; and the second segmentation model is obtained by training a pre-trained segmentation model using sample medical images.

[0109] Specifically, the generation of the region of interest in the medical image can be achieved through the second segmentation model. When the doctor selects a specific area on the medical image through a point or box indication, the interactive segmentation begins, and the second segmentation model interprets this input and performs segmentation to accurately define the region of interest (ROI). The second segmentation model can be a MedSAM model, a specialized iterative version of SAM designed specifically for the field of medical imaging, which is obtained by training the pre-trained SAM model through sample medical images.

[0110] Step S620 determines the corresponding text of interest from the imaging report based on the region of interest selected from the medical image, including: inputting the region of interest and the imaging report into a comparative language image pre-training model, detecting the similarity between the region of interest and each text paragraph in the imaging report through the comparative language image pre-training model, and determining the text of interest corresponding to the region of interest from the imaging report based on the similarity.

[0111] Specifically, according to the region of interest selected from the medical image, the corresponding text of interest from the imaging report can be determined by the contrastive language image pre-training model (CLIP). The encoder in the contrastive language image pre-training model can understand and match images and related texts, and therefore, can be used to identify the paragraphs in the imaging report related to the region of interest (ROI). This process can be specifically achieved by converting the imaging report and the region of interest into a common feature space, in which the similarity between the text paragraph in the imaging report and the ROI can be measured. Then the text paragraph with the highest similarity score is extracted as the text of interest corresponding to the ROI.

[0112] This method refines the physician-marked ROI into a clear text description by extracting text of interest, which not only summarizes the key clinical findings but also directly aligns with the diagnostic details highlighted in the ROI, thereby improving the model's accuracy in matching specific image features with text descriptions, thereby ensuring that the extracted text accurately identifies the region of interest marked in the medical image.

[0113] In an exemplary embodiment, the above step S630 searches a database based on the text of interest to obtain a target historical medical image similar to the region of interest and a historical image report corresponding to the target historical medical image, including: determining the degree of match between the mapping vector of each image report pair in the database and the encoding feature of the text of interest respectively; determining from the database an image report pair whose corresponding matching degree is higher than a threshold, and using the historical medical image and the historical image report corresponding to the image report pair as the target historical medical image and the historical image report corresponding to the target historical medical image.

[0114] In a specific implementation, the purpose of searching the database based on the text of interest is to identify historical medical images similar to the selected ROI, and report texts that are similar but not identical to the text description of the ROI. This type of retrieval can not only identify similarities, but also emphasize different features, which are then displayed to the doctor together with the TOI. Similar to text-driven database retrieval, the historical medical images most relevant to the region of interest can be found through a similarity search algorithm (such as cosine similarity), the matching degree between the mapping vector of each image report pair and the encoded features of the text of interest is calculated, the matching degrees are sorted from high to low, and the corresponding image report pairs with matching degrees higher than the threshold are screened out, and the historical medical images and historical image reports corresponding to such image report pairs are used as the target historical medical images and the historical image reports corresponding to the target historical medical images, thereby providing doctors with the presentation of lesion features similar to the region of interest, and providing doctors with different report texts for similar lesions.

[0115] In this embodiment, the target historical medical images are screened by matching the mapping vector of each image report pair in the database with the encoding features of the text of interest, so that the historical medical images corresponding to the area of ​​interest can be matched more accurately, thereby ensuring the accuracy and reliability of the screening results.

[0116] In an exemplary embodiment, after the above step S630 searches the database according to the text of interest to obtain the target historical medical image similar to the region of interest and the historical image report corresponding to the target historical medical image, the following is further included:

[0117] Step S631, converting the target historical medical image into text description information, and determining the target report text associated with the region of interest in the historical image report corresponding to the target historical medical image;

[0118] Step S632, adjusting the target report text in the historical image report corresponding to the target historical medical image according to the target historical medical image and the text description information converted from the target historical medical image by using the multimodal large language model to obtain an adjusted historical image report;

[0119] Step S633 , marking and displaying the text of interest in the image report, and displaying the target historical medical image and the adjusted historical image report corresponding to the target historical medical image.

[0120] In a specific implementation, after the database is searched for the text of interest, the multimodal large language model (LLM) begins to intervene. The retrieved target historical medical image is input as prompt information of the multimodal large language model, and the multimodal large language model adjusts the target report text in the historical image report corresponding to the target historical medical image according to the prompt information to obtain the adjusted historical image report. Finally, the text of interest is marked and displayed in the image report, and the target historical medical image and the adjusted historical image report corresponding to the target historical medical image are displayed. Among them, before the retrieval result is input as prompt information into the multimodal large language model, the target historical medical image can be first converted into text description information that can be recognized by the multimodal large language model, and the target report text associated with the text of interest in the historical image report corresponding to the target historical medical image is determined.

[0121] In this embodiment, the multimodal large language model integrates the retrieved results and refines them into a coherent, medically relevant narrative. This synthesis step can transform complex medical imaging results into understandable and actionable reports. It adds descriptive insights directly to the imaging report, thereby enhancing the original data. Therefore, the addition of the multimodal large language model promotes the generation of TOI details and provides doctors with a comprehensive report that combines visual imaging markers (regions of interest) with rich diagnostic narratives. This comprehensive approach can greatly improve the interpretability of medical images and strengthen the clinical decision-making process.

[0122] In one embodiment, reference Figure 8 , shows a detailed flow chart of an interactive method for medical imaging and imaging reports based on image-driven. The image-driven artificial intelligence model can be called MedSAM2CLIP, which is an image-driven model that can generate text descriptions for ROIs detected in images. This model works in conjunction with the text-driven CLIP2MedSAM in the aforementioned embodiment, that is, text provides information for imaging, and imaging enriches text data.

[0123] The interactive image-text artificial intelligence system provided in the above-mentioned embodiments of the present application is used for radiological image reading and reporting. The key innovative technological breakthroughs include joint artificial intelligence modeling of natural language and image segmentation to provide text-driven and image-driven knowledge understanding and reasoning. By extracting and associating detailed medical insights in two traditionally different forms of clinical data, on the one hand, the corresponding image area can be located for the highlighted text, and similar reports and related images can be retrieved, and on the other hand, the corresponding report description can be located for the selected area of ​​interest, and similar images and related reports can be retrieved, thereby providing a novel interactive imaging-text reporting system. This can improve the level of clinical diagnosis, change the extraction and association of detailed medical insights, and bridge the gap between traditionally independent forms of clinical data.

[0124] Moreover, the system can automatically expand imaging reports through detailed descriptions of imaging results generated by artificial intelligence, which is a major innovation in medical documentation. This not only saves doctors' time, but also provides them with comprehensive and easily accessible insights to support more informed decisions and personalized patient care. The interactive interface designed by the system allows doctors to easily switch between text reports and visual imaging data, which promotes an intuitive diagnostic process, enhances the user experience, and makes clinical analysis and interpretation more targeted. This innovative tool that integrates text and image data can enable doctors to make complex medical diagnoses more easily, accurately, and efficiently.

[0125] The interactive system uses two key datasets: one is a segmentation dataset that integrates various imaging modalities (such as CT, MRI, PET, DR) and anatomical systems (such as breast, cardiovascular, chest, gastrointestinal tract, head and neck) and their corresponding regions of interest, which is used to evaluate the performance of MedSAM in accurately segmenting and identifying relevant anatomical features of different diseases and imaging technologies. The other is an image report pairing dataset that includes labeled cases of similar patients to evaluate the system's retrieval performance. This pairing dataset is used to evaluate the system's accuracy in matching medical images and image reports, thereby improving the consistency between visual results and clinical descriptions. In addition, a comprehensive retrieval performance evaluation of the interactive system can be performed using open source datasets. In other words, the evaluation of the interactive system mainly includes three aspects: (1) Testing the system's text-driven and image-driven modeling capabilities based on standard datasets of medical images and corresponding image reports to verify the system's accuracy in identifying, segmenting ROIs, and generating accurate text descriptions. (2) Conducting a user study on a group of clinicians to evaluate the system's usability, interface intuitiveness, and overall user experience, and improving the system based on the evaluation results to better meet user needs. (3) Through pilot studies in clinical settings, the effectiveness of the system in improving the diagnostic process, decision-making, and patient care will be evaluated, such as analysis of indicators such as saved diagnostic time, improved diagnostic accuracy, and clinician satisfaction. Through these methods, the system will be comprehensively evaluated, focusing on its clinical efficacy, user experience, and impact on medical practice, to ensure that the advantages of the system are fully utilized in the actual medical environment.

[0126] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0127] Based on the same inventive concept, the embodiment of the present application also provides a medical image and image report interaction device for implementing the medical image and image report interaction method involved in the above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more medical image and image report interaction device embodiments provided below can refer to the limitations of the medical image and image report interaction method above, and will not be repeated here.

[0128] In one embodiment, Fig. 9 As shown, a device for interacting medical images and image reports is provided, comprising:

[0129] An acquisition unit 910 is used to acquire medical images and image reports of medical images;

[0130] The segmentation unit 920 is used to segment the medical image according to the text of interest selected from the image report to obtain a region of interest;

[0131] The search unit 930 is used to search the database according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report.

[0132] In one embodiment, the above-mentioned segmentation unit 920 is specifically used to input the text of interest and the medical image into the first segmentation model for segmentation processing to obtain the region of interest; the first segmentation model uses the sample medical image as the input variable, the sample text of interest of the sample medical image as the prompt information, and the segmentation result of the sample medical image as the output variable, and is obtained by training the pre-trained segmentation model.

[0133] In one embodiment, the above-mentioned search unit 930 is specifically used to determine the matching degree between the mapping vector of each image report pair in the database and the coding feature of the region of interest; from the database, determine the corresponding image report pair with a matching degree higher than a threshold, and use the historical image report and the historical medical image corresponding to the image report pair as the target historical image report and the historical medical image corresponding to the target historical image report.

[0134] In one embodiment, the above-mentioned search unit 930 is also used to convert the historical medical image into text description information, and to determine the target report text associated with the region of interest in the target historical image report; adjust the target report text in the target historical image report according to the historical medical image and the text description information converted from the historical medical image through the multimodal large language model to obtain an adjusted target historical image report; mark and display the region of interest in the medical image, and display the adjusted target historical image report and the historical medical image corresponding to the target historical image report.

[0135] In another embodiment, Fig.10 As shown, a device for interacting medical images and image reports is provided, comprising:

[0136] An acquisition unit 1010 is used to acquire medical images and image reports of medical images;

[0137] A determination unit 1020, configured to determine a corresponding text of interest from an image report according to a region of interest selected from the medical image;

[0138] The search unit 1030 is used to search the database according to the text of interest to obtain the target historical medical image similar to the region of interest and the historical image report corresponding to the target historical medical image.

[0139] In one embodiment, the above-mentioned determination unit 1020 is also used to obtain location prompt information of the medical image; the location prompt information and the medical image are input into the second segmentation model for segmentation processing to obtain the region of interest; the second segmentation model is obtained by training the pre-trained segmentation model through sample medical images; the region of interest and the image report are input into the comparative language image pre-trained model, and the similarity between the region of interest and each text paragraph in the image report is detected by the comparative language image pre-trained model, and the text of interest corresponding to the region of interest is determined from the image report according to the similarity.

[0140] In one embodiment, the search unit 1030 is specifically used to determine the matching degree between the mapping vector of each image report pair in the database and the encoding feature of the text of interest; determine the corresponding image report pair with a matching degree higher than a threshold from the database, and use the historical medical image and the historical image report corresponding to the image report pair as the target historical medical image and the historical image report corresponding to the target historical medical image.

[0141] In one embodiment, the search unit 1030 is also used to convert the target historical medical image into text description information, and to determine the target report text associated with the region of interest in the historical image report corresponding to the target historical medical image; adjust the target report text in the historical image report corresponding to the target historical medical image according to the target historical medical image and the text description information converted from the target historical medical image through a multimodal large language model to obtain an adjusted historical image report; mark and display the text of interest in the image report, and display the target historical medical image and the adjusted historical image report corresponding to the target historical medical image.

[0142] Each module in the above-mentioned medical image and image report interaction device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0143] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for interacting with medical images and image reports is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0144] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0146] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0147] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0149] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0150] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for interacting medical images and image reports, characterized in that: The method comprises: Obtaining medical images and imaging reports of said medical images; The medical image and the text of interest selected from the image report are input into a first segmentation model, and the medical image is segmented by the first segmentation model to obtain a region of interest; the first segmentation model uses a sample medical image as an input variable, a sample text of interest of the sample medical image as prompt information, and a segmentation result of the sample medical image as an output variable, and is obtained by training a pre-trained segmentation model; Search the database according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report; the database stores mapping vectors of multiple image report pairs, each image report pair including a historical medical image and an image report corresponding to the historical medical image; the mapping vector of each image report pair is obtained by mapping the historical medical image and the corresponding image report in the image report pair into a shared high-dimensional vector space through an embedding model, wherein the historical medical image is mapped using computer vision technology, and the image report is mapped using natural language processing technology; The step of searching a database according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report includes: Determining the degree of match between the mapping vector of each image report pair in the database and the coded feature of the region of interest respectively; From the database, a corresponding image report pair with a matching degree higher than a threshold is determined, and the historical image report and the historical medical image corresponding to the image report pair are used as the target historical image report and the historical medical image corresponding to the target historical image report.

2. The method according to claim 1, characterized in that After searching the database according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report, the method further includes: converting the historical medical image into text description information, and determining a target report text associated with the region of interest in the target historical image report; Adjusting the target report text in the target historical image report according to the historical medical image and the text description information converted from the historical medical image by a multimodal large language model to obtain an adjusted target historical image report; The region of interest is marked and displayed in the medical image, and the adjusted target historical image report and the historical medical image corresponding to the target historical image report are displayed.

3. A method for interaction between medical images and image reports, characterized in that: The method comprises: Obtaining medical images and imaging reports of said medical images; Inputting the region of interest selected from the medical image and the image report into a comparative language image pre-training model, detecting the similarity between the region of interest and each text paragraph in the image report through the comparative language image pre-training model, and determining the text of interest corresponding to the region of interest from the image report according to the similarity; Search a database according to the text of interest to obtain a target historical medical image similar to the region of interest and a historical image report corresponding to the target historical medical image; the database stores mapping vectors of multiple image report pairs, each image report pair including a historical medical image and an image report corresponding to the historical medical image; the mapping vector of each image report pair is obtained by mapping the historical medical image and the corresponding image report in the image report pair into a shared high-dimensional vector space through an embedding model, wherein the historical medical image is mapped using computer vision technology, and the image report is mapped using natural language processing technology; The step of searching a database according to the text of interest to obtain a target historical medical image similar to the region of interest and a historical image report corresponding to the target historical medical image includes: Determining the degree of match between the mapping vector of each image report pair in the database and the encoding feature of the text of interest respectively; From the database, a corresponding image report pair with a matching degree higher than a threshold is determined, and the historical medical image and the historical image report corresponding to the image report pair are used as the target historical medical image and the historical image report corresponding to the target historical medical image.

4. The method according to claim 3, characterized in that Before determining the corresponding text of interest from the image report according to the region of interest selected from the medical image, the method further includes: Obtaining location prompt information of the medical image; The location prompt information and the medical image are input into a second segmentation model for segmentation processing to obtain the region of interest; the second segmentation model is obtained by training a pre-trained segmentation model using sample medical images.

5. The method according to claim 3, characterized in that: After searching the database according to the text of interest to obtain a target historical medical image similar to the region of interest and a historical image report corresponding to the target historical medical image, the method further includes: Converting the target historical medical image into text description information, and determining a target report text associated with the region of interest in a historical image report corresponding to the target historical medical image; Adjusting the target report text in the historical image report corresponding to the target historical medical image according to the target historical medical image and the text description information converted from the target historical medical image by a multimodal large language model to obtain an adjusted historical image report; The text of interest is marked and displayed in the image report, and the target historical medical image and the adjusted historical image report corresponding to the target historical medical image are displayed.

6. An interactive device for medical images and image reports, characterized in that: The device comprises: An acquisition unit, used for acquiring medical images and image reports of the medical images; A segmentation unit is used to input the text of interest selected from the image report into a first segmentation model, and perform segmentation processing on the medical image through the first segmentation model to obtain a region of interest; the first segmentation model uses a sample medical image as an input variable, a sample text of interest of the sample medical image as prompt information, and a segmentation result of the sample medical image as an output variable, and is obtained by training a pre-trained segmentation model; A search unit is used to search a database according to the region of interest to obtain a target historical image report similar to the text of interest and a historical medical image corresponding to the target historical image report; the database stores mapping vectors of multiple image report pairs, each image report pair includes a historical medical image and an image report corresponding to the historical medical image; the mapping vector of each image report pair is obtained by mapping the historical medical image and the corresponding image report in the image report pair into a shared high-dimensional vector space through an embedding model, wherein the historical medical image is mapped using computer vision technology, and the image report is mapped using natural language processing technology; The search unit is also used to respectively determine the degree of match between the mapping vector of each image report pair in the database and the coding feature of the region of interest; determine from the database the corresponding image report pair whose matching degree is higher than a threshold, and use the historical image report and historical medical image corresponding to the image report pair as the target historical image report and the historical medical image corresponding to the target historical image report.

7. An interactive device for medical images and image reports, characterized in that: The device comprises: An acquisition unit, used for acquiring medical images and image reports of the medical images; A determination unit, configured to input a region of interest selected from the medical image and the image report into a comparative language image pre-training model, detect similarities between the region of interest and each text paragraph in the image report through the comparative language image pre-training model, and determine a text of interest corresponding to the region of interest from the image report according to the similarities; A search unit is used to search a database according to the text of interest to obtain a target historical medical image similar to the region of interest and a historical image report corresponding to the target historical medical image; the database stores mapping vectors of multiple image report pairs, each image report pair includes a historical medical image and an image report corresponding to the historical medical image; the mapping vector of each image report pair is obtained by mapping the historical medical image and the corresponding image report in the image report pair into a shared high-dimensional vector space through an embedding model, wherein the historical medical image is mapped using computer vision technology, and the image report is mapped using natural language processing technology; The search unit is also used to respectively determine the degree of match between the mapping vector of each image report pair in the database and the encoding features of the text of interest; determine from the database the corresponding image report pairs with a degree of match higher than a threshold, and use the historical medical image and historical image report corresponding to the image report pairs as the target historical medical image and the historical image report corresponding to the target historical medical image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for interacting with medical images and image reports according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for interacting with medical images and image reports according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for interacting with medical images and image reports described in any one of claims 1 to 5 are implemented.

Images