Fusion retrieval method and system based on medical image and report text

Through the CLIP model, the graphic and text characteristics and confusion matrix are established, which solves the problem of data quality and diversity limitations, realizes high-precision and scalability of picture and text matching, and improves auxiliary diagnosis accuracy.

CN119066218APending Publication Date: 2024-12-03SHANGHAI SHEN KANG HOSPITAL DEV CENT
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411062250.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art has limitations on data quality and diversity in medical image and report text retrieval, and deep learning feature extraction has limitations and lack of interpretation, which affects system performance and generalization capabilities.

Method used

The CLIP model is used to fused search of medical images and report texts, and through feature extraction, training and storing graphic feature vectors, a confusion matrix and a search database are established, and a cosine similarity is used for network reasoning and searching to achieve matching images and texts.

Benefits of technology

It improves matching accuracy, has good scalability, can adapt to graphic and text databases of different sizes, supports dynamic expansion and content updates, and enhances the accuracy of graphic and text matching and auxiliary diagnosis capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066218A_ABST
    Figure CN119066218A_ABST
Patent Text Reader

Abstract

The invention discloses a fusion retrieval method and system based on a medical image and a report text, and the method comprises the following steps: S1, carrying out the training according to an image and a text pair based on a CLIP model, compressing and storing the features of the image and the text, and building the image-text features and an image database matched with the image-text features; and S2, performing network reasoning and searching according to the text or the image. By adopting the method, the images and the reports are subjected to large-scale supervised training, and the image and text features are compressed and stored, so that similar cases can be quickly retrieved, and doctors can be assisted in reading films and formulating treatment schemes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a fusion retrieval method and system based on medical images and report texts. Background Art

[0002] Medical image and report retrieval supports the retrieval of image and report texts by quickly extracting relevant information from large amounts of image data and text reports, assisting clinical diagnosis, treatment decisions, research and education. Similar cases can provide references to reduce the risk of misdiagnosis, and can also help use retrieval data for disease research and efficacy evaluation, helping doctors track changes in patients' conditions for effective follow-up and long-term monitoring, and overall improving the quality and efficiency of medical services.

[0003] Traditional image retrieval methods achieve retrieval by extracting features such as grayscale, texture, and shape, while deep learning achieves retrieval by automatically extracting deep features. Commonly used networks include ResNet and VGG. In terms of text retrieval, traditional methods perform retrieval through keyword matching and semantic analysis, while deep learning methods achieve retrieval through neural networks or large language models.

[0004] However, both traditional methods and deep learning methods have some shortcomings. The first is the limitation of data quality and diversity. Due to the quantity and quality of medical imaging and report data, there are restrictions on the training and use of models, which affects the performance and generalization ability of the system. In addition, deep learning feature extraction has limitations and lacks explanatory guidance. Although deep learning methods perform well in feature extraction, they require big data and ensure that the deep network can accurately extract information from images or texts, so as to truly learn the characteristics of disease-related image areas and report content. Summary of the invention

[0005] The purpose of the present invention is to provide a fusion retrieval method and system based on medical images and report texts to solve the problems raised in the above background technology.

[0006] To achieve the above-mentioned object of the invention, one aspect of the present invention provides a fusion retrieval method based on medical images and report texts, comprising the following steps:

[0007] Step S1, based on the CLIP model, training is performed according to the image and text pairs, image and text features are compressed and stored, and image and text features and matching image databases are established;

[0008] Step S2: perform network reasoning and search based on text or images.

[0009] Further, step S1 includes the following steps:

[0010] Step S101, performing feature extraction for an input image and text pair;

[0011] Step S102, training the image and text pairs;

[0012] Step S103, input a number of image and text pairs, establish a confusion matrix of image and text feature vectors, and set the target value of the confusion matrix based on whether the image and text pairs are paired, with the paired value set to 1 and the unpaired value set to 0; the confusion matrix is ​​the cosine similarity of the image and text feature vectors;

[0013] Step S104: After the training is completed, all the image and text pairs of the training samples are projected, and their projected feature vectors are stored as a retrieval database.

[0014] Furthermore, the network reasoning and search are performed based on the text. During reasoning, the text is input, and the feature vector of the text is obtained through the system's text encoder. The cosine similarity is used to obtain the corresponding best matching image and output it.

[0015] Furthermore, the network reasoning and search are performed based on the image. During reasoning, the image is input, and the feature vector of the image is obtained through the system's image encoder. The cosine similarity is used to obtain the corresponding best matching text report and output it.

[0016] Another aspect of the present invention provides a fusion retrieval system based on medical images and report texts, comprising a training module and a search module, wherein:

[0017] The training module is based on the CLIP model, which is trained according to the image and text pairs, compresses and stores the image and text features, and establishes the image database with image and text features and their matching.

[0018] The search module performs network reasoning and search based on text or images.

[0019] Furthermore, the training module includes the following components:

[0020] Feature extraction component, used to complete feature extraction for input image and text pairs;

[0021] A training component for training image and text pairs;

[0022] The confusion component is used to establish a confusion matrix of image and text feature vectors for several input image and text pairs, and set the target value of the confusion matrix based on whether the image and text pairs are paired, which is set to 1 for paired and 0 for unpaired. The confusion matrix is ​​the cosine similarity of the image and text feature vectors.

[0023] The storage component is used to project the image and text pairs of all training samples and store their projected feature vectors as a retrieval database.

[0024] Furthermore, the network reasoning and search are performed based on the text. During reasoning, the text is input, and the feature vector of the text is obtained through the system's text encoder. The cosine similarity is used to obtain the corresponding best matching image and output it.

[0025] Furthermore, the network reasoning and search are performed based on the image. During reasoning, the image is input, and the feature vector of the image is obtained through the system's image encoder. The cosine similarity is used to obtain the corresponding best matching text report and output it.

[0026] Compared with the prior art, the present system and method have the following advantages:

[0027] 1. The present invention improves the matching accuracy of the CLIP model by extracting the deep features of the text report. In addition, the method has strong scalability and can be easily adapted and expanded to image-text databases of different sizes. When processing a new text report, it is only necessary to infer the text report through deep learning to obtain its features. When adding a new image-text pair for learning, the existing training queue can be learned as needed. If the network is not updated, there is no need to recalculate the feature vectors of the existing image-text data pairs.

[0028] 2. The method of the present invention can be widely used in various imaging images and reports, such as XR (X-ray), CT (computed tomography), MR (magnetic resonance imaging), etc. It not only improves the accuracy of image-text matching, but also has good scalability, can easily adapt to image-text databases of different sizes, and supports dynamic expansion and content update.

[0029] 3. The method of the present invention is not only applicable to images and reports, such as XR (X-ray), CT (computed tomography), MR (magnetic resonance imaging), etc., but can also be extended to other medical fields, such as hospitalization reports, treatment reports, etc. By learning these reports, the CLIP model can more accurately match the key content of the report. This not only improves the accuracy of image-text matching, but also has good scalability, can easily adapt to image-text databases of different sizes, and supports dynamic expansion and content updates. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the fusion retrieval method based on medical images and report texts.

[0031] Figure 2 Schematic diagram of the principle of step S1 in the fusion retrieval method based on medical images and report texts.

[0032] Figure 3 Schematic diagram of the principle of network reasoning and search based on text.

[0033] Figure 4 Schematic diagram of the principle of network reasoning and search based on images. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] like Figure 1 Shown is a flow chart of the method of the present invention, and the specific steps are as follows:

[0036] Step S1, based on the existing open source CLIP model, training is performed based on image and text pairs to establish a database corresponding to image and text features and the images that match them.

[0037] like Figure 2 The schematic diagram of step S1 is shown, which includes the following steps:

[0038] Step S101, for an input image and report pair, i.e., Image_i and Text_i, i=1,…,N, complete feature extraction training.

[0039] Step S102: during training, input image and text pairs Image_i and Text_i.

[0040] Step S103, during training, M pairs of images and texts are input simultaneously, and a confusion matrix of image and text feature vectors is established. The target value of the confusion matrix is ​​set based on whether the image and text pairs are paired, and the paired values ​​are set to 1, and the unpaired values ​​are set to 0; the confusion matrix is ​​the cosine similarity of the image and text feature vectors. Cosine similarity is used to measure the similarity of the direction of the vector without measuring the size of the vector.

[0041] Step S104: After the training is completed, all the image and text pairs of the training samples are projected, and their projected feature vectors are stored as a retrieval database.

[0042] like Figure 3 The figure shows the principle diagram of network reasoning and search based on text. The process is as follows:

[0043] During inference, input text, and obtain the feature vector of the text through the system's text encoder. Using cosine similarity, obtain 1 to K best matching vectors in the text feature vector library, and obtain the corresponding best matching image after table lookup; 1 to K matching results that best match the input text are given at the system output. The matching result is the image-text pairing that is most similar to the input text, which is helpful for doctors to provide reference during the film reading process.

[0044] like Figure 4 The figure shows the principle diagram of network reasoning and search based on images. The process is as follows:

[0045] During inference, the image is input, and the image feature vector is obtained through the system's image encoder. Using cosine similarity, 1 to K best matching vectors are obtained in the image feature vector library. After looking up the table, the corresponding best matching text report is obtained; 1 to K matching results that best match the input image are given at the system output. The matching result is the image-text pairing that is most similar to the input image, which is helpful for doctors to provide reference during the film reading process.

[0046] In the embodiment of the present invention, the database of the big data training facility is used for training. The database contains images and reports of various diseases, covering a wide range of medical conditions. Specifically, the database contains more than ten kinds of medical image image sequences and corresponding detailed report texts. These image data include X-rays, CT scans, MRI images, ultrasound images, etc., covering diagnostic images of various diseases such as lung cancer, liver cancer, and stroke. At the same time, each image data is accompanied by a report text written by a professional doctor, which records the diagnosis results, lesion areas, description of the condition, and treatment recommendations in detail. These rich image data and report texts can be used as training data sets for the network of the present invention. By using these data, the network can be effectively trained, learn and master the ability to identify and diagnose various diseases from images, thereby improving the accuracy and reliability of auxiliary diagnosis. The database not only provides a large amount of sample data for network training, but also ensures the diversity and comprehensiveness of data in the training process, laying a solid foundation for the ultimate realization of precision medicine.

[0047] The data used in the present invention include:

[0048] a) Pathological images: Pathological images are required for auxiliary diagnosis of lung cancer pathology, and the report includes cancer cell detection and grading, lesion area annotation, and cancer staging. Pathological images are required for digital pathology screening of cervical cancer, and the report covers lesion cell detection, the possibility of precancerous lesions or cancer, and examination recommendations. Pathological images are required for auxiliary diagnosis of gastrointestinal pathology, and the report describes the nature, scope, severity, and pathological grading of gastrointestinal lesions. Pathological images are required for auxiliary diagnosis of prostate pathology, and the report describes the type, grading, and lesion range of cancerous cells.

[0049] b) X-ray image (DR): DR chest film multi-disease auxiliary detection requires chest X-ray, and the report covers the test results and diagnostic conclusions of various diseases (such as pneumonia, tuberculosis, tumors, pneumothorax, etc.). DR screening of limb fractures requires DR images of the limbs, and the report describes the location, type, displacement and treatment plan of the fracture. X-ray image-assisted detection of children's bone age requires DR images of children's hands, and the report describes the bone development, the comparison of bone age with actual age, and the detection of developmental abnormalities.

[0050] c) CT images: Liver CT images are required for preoperative planning of liver surgery, and the report includes preoperative planning, tumor location, size, surrounding important structures, surgical path planning and risk assessment. Liver CT images are required for CT-assisted diagnosis of liver cancer, and the report includes the size, location, number, stage and recommended treatment plan of the tumor. Brain CT images are required for auxiliary diagnosis of stroke, and the report describes the type, location, area and emergency treatment recommendations of hemorrhagic or ischemic stroke. Coronary CTA auxiliary detection requires coronary CTA images, and the report describes the stenosis or blockage of the coronary artery and cardiovascular disease risk assessment. Pneumonia CT-assisted diagnosis requires chest CT images, and the report describes the location, range, severity and treatment recommendations of inflammation. Vertebral disease examination requires CT images, and the report describes abnormalities of structures such as intervertebral discs, vertebral bodies, and spinal cords (such as intervertebral disc herniation, spinal stenosis, etc.) and treatment recommendations. Thyroid cancer screening requires CT images, and the report describes the size, location, nature (benign or malignant) and treatment recommendations of thyroid nodules. Rib fracture auxiliary examination requires chest CT images, and the report describes the location, severity and treatment plan of the fracture. CT imaging-assisted detection of early lung cancer requires chest CT images, and the report describes the size, location, nature (benign or malignant) of the suspicious nodules and further examination recommendations. Preoperative evaluation of kidney transplantation requires CT images, and the report describes kidney function, vascular status, potential surgical risks, and suitability analysis.

[0051] d) Magnetic resonance imaging (MR): MR-assisted diagnosis of liver cancer requires liver MR images, and the report includes detailed information about the tumor (size, location, type), staging, and treatment recommendations. MR imaging of the knee joint requires knee MR images, and the report describes the condition of the joint soft tissue, bone structure, joint lesions (such as tears, wear, etc.) and treatment recommendations. Diagnosis of intracranial tumors requires MR images, and the report describes the size, location, type, staging, and treatment recommendations of the tumor. MR images are required for auxiliary diagnosis of elderly neurodegenerative diseases, and the report describes the progression of neurodegenerative diseases (such as Alzheimer's disease, Parkinson's disease, etc.) and treatment recommendations.

[0052] e) Endoscopic images: Endoscopic analysis of the upper gastrointestinal tract requires endoscopic images, and the report includes image analysis of the lesion area, the nature of the lesion (such as inflammation, ulcer, tumor, etc.) and treatment recommendations. Endoscopic diagnosis of the colon and rectum requires endoscopic images, and the report includes the detection and classification of lesions such as polyps and tumors and treatment recommendations.

[0053] In this embodiment, a large number of clinical images and reports covering various diseases are collected and equipped with large model training facilities. These diseases include important information such as head, chest, bone, tumor, pathology, colonoscopy, etc. By conducting large-scale supervised training on these images and reports, compressing and storing image and text features, similar cases can be quickly retrieved to assist doctors in reading films and formulating treatment plans.

[0054] The present invention uses the CLIP model to train image and text pairs. The method processes image and text data simultaneously and encodes them into the same embedding space. By learning the semantic connection between images and text, CLIP can achieve excellent performance in retrieval tasks. Its multimodal embedding and text-image comparison methods make the model have strong generalization ability and can be applied to various retrieval tasks, including image search, text search and multimodal search.

[0055] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A fusion retrieval method based on medical images and report texts, characterized in that: The following steps are involved: Step S1, based on the CLIP model, training is performed according to the image and text pairs, image and text features are compressed and stored, and image and text features and matching image databases are established; Step S2: perform network reasoning and search based on text or images.

2. The fusion retrieval method based on medical images and report text according to claim 1 is characterized in that: Step S1 includes the following steps: Step S101, performing feature extraction for an input image and text pair; Step S102, training the image and text pairs; Step S103, input a number of image and text pairs, establish a confusion matrix of image and text feature vectors, and set the target value of the confusion matrix based on whether the image and text pairs are paired, with the paired value set to 1 and the unpaired value set to 0; the confusion matrix is ​​the cosine similarity of the image and text feature vectors; Step S104: project the image and text pairs of all training samples and store their projected feature vectors as a retrieval database.

3. The fusion retrieval method based on medical images and report text according to claim 1 is characterized in that: The network reasoning and search are performed based on the text. During reasoning, the text is input, and the feature vector of the text is obtained through the system's text encoder. The cosine similarity is used to obtain the corresponding best matching image and output it.

4. The fusion retrieval method based on medical images and report text according to claim 1 is characterized in that: The network reasoning and search are performed based on the image. During reasoning, the image is input, and the feature vector of the image is obtained through the system's image encoder. The cosine similarity is used to obtain the corresponding best matching text report and output it.

5. A fusion retrieval system based on medical images and report texts, characterized by: It includes training module and search module, where: The training module is based on the CLIP model, which is trained according to the image and text pairs, compresses and stores the image and text features, and establishes the image database with image and text features and their matching. The search module performs network reasoning and search based on text or images.

6. The fusion retrieval system based on medical images and report texts according to claim 5 is characterized in that: The training module includes the following components: Feature extraction component, used to complete feature extraction for input image and text pairs; A training component for training image and text pairs; The confusion component is used to establish a confusion matrix of image and text feature vectors for several input image and text pairs, and set the target value of the confusion matrix based on whether the image and text pairs are paired, which is set to 1 for paired and 0 for unpaired. The confusion matrix is ​​the cosine similarity of the image and text feature vectors. The storage component is used to project the image and text pairs of all training samples and store their projected feature vectors as a retrieval database.

7. The fusion retrieval system based on medical images and report texts according to claim 5 is characterized in that: The network reasoning and search are performed based on the text. During reasoning, the text is input, and the feature vector of the text is obtained through the system's text encoder. The cosine similarity is used to obtain the corresponding best matching image and output it.

8. The fusion retrieval system based on medical images and report texts according to claim 5 is characterized in that: The network reasoning and search are performed based on the image. During reasoning, the image is input, and the feature vector of the image is obtained through the system's image encoder. The cosine similarity is used to obtain the corresponding best matching text report and output it.

Citation Information

Cited By

  • Method and system for synthesizing CT images into MR images based on similar reference cases

    CN119904547A

  • Multi-modal health medical data management method and system based on cloud platform

    CN121075530A