A method for lesion localization based on a large model

Abstract

This invention discloses a lesion localization method based on a large model, belonging to the field of medical image analysis technology. After the image is passed through an image encoder, it is input into a lesion detection module and a lesion segmentation module, which output lesion features. Each lesion feature is processed by a region encoder and organized into a feature map of uniform size, which is then input into a large language model. The large language model integrates the feature map, global features, similar cases, and user prompts to output a medical image report. Compared with the prior art, this invention avoids the problem of manually comparing the lesion location in the image with the text description in the report, thus improving diagnostic efficiency.

Description

Technical Field

[0001] This invention belongs to the field of medical image analysis technology, and in particular relates to a lesion localization method based on a large model. Background Technology

[0002] In the field of medical image analysis, computer-aided diagnostics (CAD) technology is gradually becoming an important tool for clinicians, assisting doctors in lesion detection and diagnosis through automated processing of image data. Current image analysis technologies primarily rely on deep learning algorithms for object detection and segmentation. The development of these technologies has laid the foundation for the automated analysis of medical images.

[0003] Current image analysis technologies are limited to providing lesion detection or segmentation results, which lack a direct correlation with the final diagnostic report written by the physician. In practice, physicians need to manually compare the location of lesions in the images with the textual descriptions in the report. This fragmented workflow prevents seamless integration of image analysis results and clinical reports, increasing the physician's workload and potentially leading to misjudgments or omissions in the diagnostic process. Especially in complex cases or cases with multiple lesions, physicians need to frequently switch between images and reports, which is not only time-consuming but also increases workload and reduces diagnostic efficiency.

[0004] To address this issue, we propose a lesion localization method based on a large model. Summary of the Invention

[0005] The purpose of this invention is to solve the problem of low diagnostic efficiency caused by the need to manually compare the location of lesions in images with the text descriptions in reports in the existing technology, and to propose a lesion localization method based on a large model.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A lesion localization method based on a large model includes the following steps:

[0008] S1: Input the original image into the image encoder in the large model processing module. The image encoder extracts global features.

[0009] S2: Input the global features processed by the image encoder into the lesion detection module and the lesion segmentation module of the input processing module;

[0010] S3: The lesion detection module detects the approximate location of the lesion and marks the region box;

[0011] S4: The lesion segmentation module uses the region bounding box of the lesion detection module as a prompt to perform fine segmentation of the lesion;

[0012] S5: The region encoder in the input processing module processes each lesion feature and organizes them into feature maps of uniform size;

[0013] S6: The lesion feature codes in the region encoder are input into the knowledge base module for comparison;

[0014] S7: The global features in the image encoder, the lesion feature encoding in the region encoder, the similar lesion features in the knowledge base module, and the text prompt input by the user are all input into the large language model (LLM) in the large model processing module;

[0015] S8: The Large Language Model (LLM) integrates content through multimodal task adaptation and outputs lesion localization images and text descriptions.

[0016] Preferably, step S3 includes:

[0017] S31: The lesion detection module marks the approximate location of the detected lesion with a bounding box;

[0018] S32: Using nonmaximum suppression and knowledge filtering methods, retain the 20 regions with the highest confidence.

[0019] Preferably, step S4 includes:

[0020] S41: The lesion segmentation module segments the lesion and submits it to the user for viewing;

[0021] S42: The user inputs a prompt message, and the lesion segmentation module performs further fine-grained segmentation.

[0022] Preferably, step S5 includes:

[0023] S51: The region encoder extracts four feature layers from the image encoder encoding process;

[0024] S52: The region encoder extracts the region bounding box detected by the lesion detection module;

[0025] S53: The region encoder enhances attention to extract potential regions based on the weight information of the mask of the target lesion, while retaining some background information;

[0026] S54: The region encoder uses the ROI Align method to organize each target lesion into a feature map of uniform size, 14x14. <pn>.

[0027] Preferably, in step S7, similar lesion features in the knowledge base are simultaneously provided to the user, and the user provides similarity feedback.

[0028] Preferably, the lesion detection module is based on a retrained dinov2 model from medical image data and shares an image encoder with the lesion segmentation module.

[0029] Preferably, the lesion detection module is composed of a SAM2 model-adjusted image encoder.

[0030] In summary, the technical effects and advantages of this invention are as follows: Compared with existing devices, this lesion localization method based on a large model processes each lesion feature through a regional encoder and organizes it into a feature map of uniform size, which is then input into a large language model. The large language model integrates the feature map, global features, similar cases, and the user's prompt, and outputs a medical image report. Compared with existing technologies, this avoids the problem of manually comparing the lesion location in the image with the text description in the report, thus improving diagnostic efficiency. Attached Figure Description

[0031] Figure 1 This is a structural block diagram of the present invention. Detailed Implementation

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0033] Reference Figure 1 A lesion localization system based on a large model includes an input processing module, a large model processing module, and a knowledge base module. The input processing module includes a lesion detection module, a lesion segmentation module, and a region encoder. The large model processing module includes a large model module (LLM) and an image encoder.

[0034] The knowledge base module stores global features, regional features, and medical report annotations for easy retrieval later.

[0035] Image encoders are used to extract global features from medical images, providing high-quality input for subsequent analysis.

[0036] The lesion detection module is used for preliminary detection, identifying the approximate location of lesions in medical images.

[0037] The lesion segmentation module is used to further refine the segmentation of the output of the lesion detection module.

[0038] The region encoder is used to encode the features of the lesions so that they can be fed into a larger model later.

[0039] Large Model Module (LLM) is used to integrate content and output the results of inference.

[0040] A lesion localization method based on a large model includes the following steps:

[0041] S1: Input the original image into the image encoder in the large model processing module. The image encoder extracts global features.

[0042] S2: Input the global features processed by the image encoder into the lesion detection module and lesion segmentation module of the input processing module.

[0043] S3: The lesion detection module detects the approximate location of the lesion and marks the region bounding box.

[0044] S4: The lesion segmentation module uses the region bounding box from the lesion detection module as a prompt to perform fine segmentation of the lesion.

[0045] S5: The region encoder in the input processing module processes each lesion feature and organizes them into feature maps of uniform size for subsequent use in a large model.

[0046] S6: The lesion feature codes in the region encoder are input into the knowledge base module for comparison.

[0047] S7: Global features from the image encoder, lesion feature encoding from the region encoder, similar lesion features from the knowledge base module, and the user-inputted text prompt are all input into the large language model in the large model processing module.

[0048] S8: Large Language Model (LLM) integrates content through multimodal task adaptation and outputs lesion localization images and text descriptions. Large Language Model (LLM) can perform tasks such as medical report generation, lesion detection, and image diagnosis.

[0049] The large model processing module includes a large model module (LLM) and a separate image encoder, as well as a regular projection layer module and a parsing module.

[0050] The projection layer module performs feature mapping on the output of the image encoder to adapt it to the feature requirements of subsequent tasks. This module maps global and regional features to an encoding format suitable for large-scale model processing through dimensionality reduction and feature transformation. This mainly includes: feature transformation and adaptation of global and regional features; and mapping...<global image> The labels are replaced with encoded global image features; <p1> 、 <p2>The region markers are replaced with the corresponding region coding features.

[0051] The parsing module is responsible for decoding and presenting the task results. This module decodes the output of the large model and performs structured processing. This includes parsing the encoded information into structured report information, and... <p1>The system matches region markers to specific region image markers and generates visualized images based on global information.

[0052] Since the projection layer module and the parsing module are quite common in the field of large models, they will not be discussed in detail and are not shown in the diagram.

[0053] This lesion localization method based on large models combines large model technology with various interactive tools to provide doctors with the ability to automatically generate reports and accurately locate lesions.

[0054] Step S3 includes...

[0055] S31: The lesion detection module marks the approximate location of the detected lesion with a bounding box.

[0056] S32: Using nonmaximum suppression and knowledge filtering methods, retain the 20 regions with the highest confidence.

[0057] Step S4 includes...

[0058] S41: The lesion segmentation module segments the lesions and submits them to the user for viewing;

[0059] S42: The user inputs a prompt message, and the lesion segmentation module performs further fine-grained segmentation.

[0060] As a prompting-based segmentation module, the lesion segmentation module offers users more modal options for input, going beyond just the region bounding boxes from the aforementioned lesion detection module. It can also directly accept user-inputted points, drawing areas, bounding boxes, and other prompts as input, allowing for more refined segmentation based on user needs. This interaction with the user helps determine more precise segmentation targets.

[0061] Step S5 includes...

[0062] S51: The region encoder extracts four feature layers from the image encoder's encoding process. In the large model processing module, four feature layers are extracted from the image encoder during the encoding process to form a hierarchical feature pyramid.

[0063] S52: The region encoder extracts the region bounding box detected by the lesion detection module.

[0064] S53: The region encoder enhances attention to extract potential regions based on the weight information of the mask of the target lesion, while retaining some background information.

[0065] S54: The region encoder uses the ROI Align method to organize each target lesion into a uniform 14x14 feature map. <pn>This is so that it can be fed into a larger model later.

[0066] In step S7, similar lesion features from the knowledge base are simultaneously provided to the user, who then provides feedback on the similarity. After the doctor provides feedback, the lesion is added to a modification list for future updates to the knowledge base, thus improving the accuracy and adaptability of lesion localization.

[0067] The lesion detection module is based on a retrained dinov2 model from medical image data, sharing the image encoder with the lesion segmentation module. The training data for the lesion detection module primarily consists of ground truth (GT) bounding boxes expanded by a factor of 1.2 from medical images. During training, it is first pre-trained using large-scale medical image data, and then fine-tuned using detailed data specific to the corresponding scene.

[0068] The lesion detection module consists of an image encoder adjusted from the SAM2 model. After the lesion detection module has been trained, the image encoding module of the dinov2 model is extracted as the basic parameters of the image encoding module of the SAM2 model.

[0069] Example of medical imaging report generation (including report generation and question matching boxes, segmentation):

[0070] Prompt:

[0071] [Detail Report Generation]<global image> For the above image, the following locations <p1> , <p2> , <p3> ,…… <pn>There may be lesions present. Please analyze them and write a detailed medical imaging report.

[0072] Res:

[0073] Findings:

[0074] The chest is symmetrical, the trachea is in the middle, and the bony structure of the ribs above the diaphragm is intact. <d> Increased lung markings< / d> <p1> 。 <d> High-density nodular shadows were seen in the upper right lung field, lower left lung field, and outer zone of the lower left lung field.< / d> <p4> <p5>The boundaries are relatively clear. <d> Increased density of hilar shadows in both lungs< / d> <p6> 。 <d> The heart's silhouette is boot-shaped, with a concave waist and an enlarged apex pointing to the left.< / d> <p7> ， <d> Aortic knob protrusion with calcification, and local calcification shifting inward.< / d> <p8> 。 <d> blunting of the right costophrenic angle< / d> <p9>。

[0075] Impression:

[0076] 1. <d> Old lesions in the upper right lung and lower left lung< / d> <p4> <p5>

[0077] 2. <d> Aortic type cardiac silhouette< / d> <p7> ， <d> Aortic node protrusion and calcification< / d> <p8>

[0078] 3. <d> Pleural effusion< / d> <p9>

[0079] This allows us to identify the descriptions of the problems, retrieve the previously generated bounding boxes and segmentation information based on these markers, and then provide a visual analysis based on these.

[0080] Example of a regional Q&A:

[0081] Prompt:

[0082] [VQA]<global image> For the above image, the following locations <p4> , <p9>Please analyze whether lesions are present.

[0083] Res:

[0084] The following issues were found

[0085] 1. <d> Old lesions in the upper right lung< / d> <p4>

[0086] 2. <d> blunting of the right costophrenic angle< / d> <p9>

[0087] Regarding the Prompt mentioned above,<global image> It will be replaced by full-image encoded information, and <p4> , <p9>These two tags will then be followed by local information that has been encoded by the region encoder.

[0088] After large-scale model inference, the aforementioned Res information can be output.

[0089] Regarding the output Res information, we can use the information in the data to... <p4>or <p9>By using tags to retrieve the coordinates of the corresponding area, and then visualizing the information through color matching, the problem and the relationship between the area can be better displayed.

[0090] Other general questions and answers:

[0091] Thanks to the support of multimodal large models, other common problems can also be solved, such as:

[0092] Report Correction Prompt:

[0093] [Correction] Please correct the following description based on the images and report content.

[0094] The large model compares the image features with the original report segment by segment, corrects inaccurate or missing diagnostic information, and generates a new revised report.

[0095] Multi-period comparative analysis Prompt:

[0096] [Multi-phase comparison] Please compare the changes in image A and image B.

[0097] Multi-stage image codes are extracted, and then feature matching and time series reasoning are used to identify the progression or changes of lesions and generate comparative analysis results.

[0098] Treatment recommendations generate a Prompt:

[0099] [Recommendations] Provide treatment options based on the patient's images and reports.

[0100] By leveraging a large-scale medical knowledge base and combining disease characteristics from images and reports, personalized treatment recommendations are automatically generated.

[0101] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention. < / p4> < / p4> < / p4> < / p8> < / p7> < / p5> < / p4> < / p8> < / p7> < / p6> < / p4> < / p1> < / pn> < / p3> < / p2> < / p1> < / pn> < / p1> < / pn>

Claims

1. A lesion localization method based on a large model, characterized in that, Including the following steps: S1: Input the original image into the image encoder in the large model processing module. The image encoder extracts global features. S2: Input the global features processed by the image encoder into the lesion detection module and the lesion segmentation module of the input processing module; S3: The lesion detection module detects the approximate location of the lesion and marks the region box; S4: The lesion segmentation module uses the region bounding box of the lesion detection module as a prompt to perform fine segmentation of the lesion; S5: The region encoder in the input processing module processes each lesion feature and organizes them into feature maps of uniform size; S6: The lesion feature codes in the region encoder are input into the knowledge base module for comparison; S7: The global features in the image encoder, the lesion feature encoding in the region encoder, the similar lesion features in the knowledge base module, and the text prompt input by the user are all input into the large language model in the large model processing module; S8: The large language model integrates content through multimodal task adaptation and outputs lesion localization images and text descriptions.

2. The lesion localization method based on a large model according to claim 1, characterized in that, Step S3 includes: S31: The lesion detection module marks the approximate location of the detected lesion with a bounding box; S32: Using nonmaximum suppression and knowledge filtering methods, retain the 20 regions with the highest confidence.

3. The lesion localization method based on a large model according to claim 1, characterized in that, Step S4 includes: S41: The lesion segmentation module segments the lesions and submits them to the user for viewing; S42: The user inputs a prompt message, and the lesion segmentation module performs further fine-grained segmentation.

4. The lesion localization method based on a large model according to claim 1, characterized in that, Step S5 includes: S51: The region encoder extracts four feature layers from the image encoder encoding process; S52: The region encoder extracts the region bounding box detected by the lesion detection module; S53: The region encoder enhances attention to extract potential regions based on the weight information of the mask of the target lesion, while retaining some background information; S54: The region encoder uses the ROIAlign method to organize each target lesion into a feature map of uniform size, 14x14.

5. The lesion localization method based on a large model according to claim 1, characterized in that, In step S7, similar lesion features in the knowledge base are simultaneously provided to the user, and the user provides similarity feedback.

6. The lesion localization method based on a large model according to claim 1, characterized in that, The lesion detection module is based on a retrained dinov2 model from medical image data and shares an image encoder with the lesion segmentation module.

7. The lesion localization method based on a large model according to claim 6, characterized in that, The lesion detection module consists of an image encoder adjusted by the SAM2 model.

Images