Lung MRI Image Consultation System Based on Large Language Model

By extracting location description information of lung lobes and nodule regions from a lung magnetic resonance imaging system and generating consultation suggestions using a large language model, the problem of relying on physician experience and compliance risks in existing technologies is solved, and more accurate lung imaging diagnosis is achieved.

CN116825389BActive Publication Date: 2026-11-17SHANGHAI SOUNDWISE TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310687501.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-06-09
Publication Date
2026-11-17
Estimated Expiration
2043-06-09

AI Technical Summary

Technical Problem

Existing lung imaging diagnostic systems based on artificial intelligence models rely on doctors' experience, and GPT-like models have issues with interpretability and compliance risks, failing to effectively focus on key areas of lesions.

Method used

The image extraction module extracts lung lobe images and nodule regions from lung MRI images, generates nodule location description information, and inputs it into a large language model to generate supplementary consultation suggestions, allowing doctors to conduct further consultations to obtain comprehensive information.

Benefits of technology

It improves the accuracy of lung imaging diagnosis, and the prompting effect of large language models makes it focus on key areas, reducing reliance on doctors' experience and lowering compliance risks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116825389B_ABST
    Figure CN116825389B_ABST
Patent Text Reader

Abstract

This invention relates to the field of imaging diagnostic technology, specifically to a lung magnetic resonance imaging (MRI) image consultation system based on a large language model. The system includes: an image extraction module for acquiring lung MRI images and extracting lobar images from the images; a nodule description module for generating nodule location description information based on the lobar images; a language model interaction module for inputting the lobar images and nodule location description information into an external large language model to obtain supplementary consultation suggestions from the large language model; and an inquiry module for questioning the patient based on the supplementary consultation suggestions to obtain additional consultation information. The beneficial effect is that by pre-extracting the lobar images and nodule regions and generating corresponding location description information for the nodule regions, and then inputting this information into the large language model, a better prompting effect is achieved, allowing the large language model to better focus on key areas.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of imaging diagnostic technology, specifically to a lung magnetic resonance imaging consultation system based on a large language model. Background Technology

[0002] MRI is a commonly used medical computed tomography method that uses the magnetic resonance phenomenon to obtain electromagnetic signals from the human body and reconstruct human information. This technology utilizes the principle of nuclear magnetic resonance, based on the different attenuations of released energy in different structural environments within a substance. By detecting the emitted electromagnetic waves through an external gradient magnetic field, the location and type of atomic nuclei that make up the object can be determined, and an image of the object's internal structure can be created.

[0003] In the existing technology, there are technical solutions for assisted diagnosis based on artificial intelligence models. Such technical solutions usually involve pre-collecting a large amount of medical image data to establish a training set, and then using artificial intelligence models to detect and label lesions to assist doctors in making diagnoses. With the development of multimodal models and GPT-like models, such artificial intelligence models can also be used to directly output corresponding descriptive information.

[0004] However, in actual implementation, the inventors found that traditional artificial intelligence models, after detecting and classifying lesions, still rely on doctors' experience to interpret images and make diagnoses, and are limited by doctors' experience; while GPT-like models, due to the uninterpretability of artificial intelligence models themselves, may lead to corresponding compliance risks, and it is impossible to determine whether they can correctly focus on key areas. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, a lung magnetic resonance imaging consultation system based on a large language model is provided.

[0006] The specific technical solution is as follows:

[0007] A lung magnetic resonance imaging (MRI) diagnostic system based on a large language model includes:

[0008] An image extraction module acquires lung magnetic resonance images and extracts lung lobe images from the lung magnetic resonance images;

[0009] A nodule description module, which generates nodule location description information based on the lung lobe image;

[0010] The language model interaction module inputs the lung lobe image and the nodule location description information into an external large language model to obtain supplementary consultation suggestions from the large language model.

[0011] The inquiry module asks questions of the patient based on the supplementary consultation suggestions to obtain additional consultation information.

[0012] On the other hand, the image extraction module includes:

[0013] A thoracic cavity extraction module, which extracts the thoracic cavity region from multiple frames of the lung magnetic resonance imaging sequence;

[0014] The lung lobe extraction module processes the thoracic cavity region into a time series model to obtain lung lobe images corresponding to the lung lobe region.

[0015] On the other hand, in the lung lobe extraction module, the time series model includes:

[0016] A convolutional network that detects the thoracic cavity region in an input single frame to segment and obtain a single-frame lung lobe region;

[0017] An input gate, which is connected to the convolutional network, receives the single-frame lung lobe region;

[0018] An output gate that outputs the predicted lung lobe image;

[0019] The forget gate receives the lung lobe image of the previous frame output by the output gate, and processes the lung lobe image to form a memory image output.

[0020] The output gate receives the single-frame lung lobe region and the memory image respectively, and predicts the lung lobe image.

[0021] On the other hand, the nodule description module includes:

[0022] A nodule extraction module, which extracts multiple nodule regions from the lung lobe image;

[0023] The location extraction module inputs the nodule region and the lung lobe image into the location description model to obtain the relative positional relationship of multiple nodule regions and the distribution relationship of the nodule regions in the lung lobe image.

[0024] A location description module generates the nodule location description information according to the relative positional relationship and the distribution relationship.

[0025] On the other hand, the nodule description module also includes a vector graphics acquisition module, which includes:

[0026] The first edge extraction module acquires nodule example images and extracts the edges of example nodules from each nodule example image.

[0027] A graphics processing module processes the edges of the example nodules to obtain vector graphics of the nodules;

[0028] The nodule extraction module includes:

[0029] The second edge extraction module extracts the edges of the lung lobe image to obtain an edge line image.

[0030] The image matching module matches the edge line image based on the nodule vector image to obtain multiple matching positions;

[0031] The nodule output module outputs the enclosed area corresponding to the matching position as the nodule region.

[0032] On the other hand, the location description model includes:

[0033] A feature extraction network is used to extract features from the lung lobe image with the nodule region labeled to obtain location features;

[0034] A clustering module, which is connected to the feature extraction network, performs clustering on the location features to obtain nodule clustering results as the relative positional relationship;

[0035] A radiomics model, wherein the radiomics model is connected to the feature extraction network, and the radiomics model generates the distribution relationship according to the location features.

[0036] On the other hand, the location description module includes:

[0037] A grading module, which determines the spread of nodules in the lung lobe region according to the distribution relationship and a predefined distribution level;

[0038] A template invocation module, which invokes the corresponding overall template according to the distribution level;

[0039] The description information generation module fills the overall template with the relative positional relationship and the distribution level to obtain the nodule location description information.

[0040] On the other hand, the language model interaction module includes:

[0041] The output module inputs the lung lobe image into the large language model;

[0042] The prompting module generates prompt words based on the nodule location description information and inputs the prompt words into the large language model;

[0043] A consultation suggestion generation module obtains the supplementary consultation suggestions fed back by the large language model.

[0044] The above technical solution has the following advantages or beneficial effects:

[0045] To address the shortcomings of existing workflows based on artificial intelligence models, this solution improves upon the aforementioned process. It pre-extracts lung lobe images and nodule regions from lung MRI images and generates corresponding location descriptions for these nodule regions. This information is then input into a large-scale language model, resulting in better prompting for the model and enabling it to focus on key areas. Furthermore, by modifying the prompts, the large-scale language model outputs only supplementary consultation suggestions requiring further medical examination. Doctors then conduct consultations to obtain more comprehensive information before making a diagnosis, thus improving diagnostic accuracy. Attached Figure Description

[0046] Embodiments of the invention will be described more fully with reference to the accompanying drawings. However, the drawings are for illustration and explanation only and do not constitute a limitation on the scope of the invention.

[0047] Figure 1 This is an overall schematic diagram of an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the image extraction module in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of a time series model in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the nodule description module in an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of the vector graphics acquisition model in an embodiment of the present invention;

[0052] Figure 6 This is a schematic diagram of the nodule extraction module in an embodiment of the present invention;

[0053] Figure 7 This is a schematic diagram of the location description model in an embodiment of the present invention;

[0054] Figure 8 This is a schematic diagram of the location description module in an embodiment of the present invention;

[0055] Figure 9This is a schematic diagram of the language model interaction module in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0058] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0059] This invention includes:

[0060] A lung magnetic resonance imaging diagnostic system based on a large language model, such as Figure 1 As shown, it includes:

[0061] Image extraction module 1 acquires lung magnetic resonance images and extracts lung lobe images from the lung magnetic resonance images;

[0062] Nodule description module 2 generates nodule location description information based on lung lobe images;

[0063] Language model interaction module 3 inputs lung lobe images and nodule location description information into an external large language model to obtain supplementary consultation suggestions from the large language model.

[0064] Inquiry module 4 asks questions of the patient based on supplementary consultation suggestions to obtain additional consultation information.

[0065] Specifically, addressing the shortcomings of existing workflows based on artificial intelligence models, this solution improves the aforementioned process by pre-extracting lung lobe images and nodule regions from lung MRI images and generating corresponding location descriptions for the nodule regions. This information is then input into a large language model, resulting in better prompting for the large language model and enabling it to focus on key areas. Furthermore, by modifying the prompting words, the large language model outputs only supplementary consultation suggestions requiring further medical examination. Doctors then conduct consultations to obtain more comprehensive information before making a diagnosis, thus improving diagnostic accuracy.

[0066] In the implementation process, lung MRI images are obtained by pre-scanning the patient, typically presented as multi-frame images in a time series. Lobar images refer to single or multiple frames of lung lobe images obtained through processing and filtering based on the lung MRI images; these images remove background areas such as the pleural cavity compared to the original MRI images. Nodule location description information is textual information obtained by descriptively processing the extracted nodule regions' locations within the lobar images. This text can be directly input as prompts into the Large Language Model (LLM) to inform the LLM about the information it needs to focus on. The LLM can be an existing model implemented on a public cloud, such as the GPT series models provided by OpenAI, or a locally deployed LLM built using specific medical datasets, depending on actual needs. Supplementary consultation suggestions are based on feedback from specific prompts, such as "What diseases might be involved based on the current information, what symptoms correspond to, and what information needs to be confirmed further?", used to assist doctors in obtaining more comprehensive consultation information for better diagnosis.

[0067] In one embodiment, such as Figure 2 As shown, the image extraction module 1 includes:

[0068] The thoracic cavity extraction module 11 extracts the thoracic cavity region from multiple frames of images in the image sequence of lung magnetic resonance imaging.

[0069] The lung lobe extraction module 12 processes the input of the thoracic cavity region into a time series model to obtain lung lobe images corresponding to the lung lobe region.

[0070] Specifically, in response to the problem that respiratory activity can easily create numerous afterimages in the images during the acquisition of MRI images, this embodiment employs a two-step extraction process. This includes pre-removing the background portion of the lung MRI images, retaining only the thoracic cavity region; subsequently, a time-series model is used to extract the periodically moving lung lobe regions in the thoracic cavity region across multiple frames, thereby segmenting a clearer lung lobe region and achieving a better extraction effect.

[0071] In one embodiment, such as Figure 3 As shown, the time series model includes:

[0072] Convolutional network 101 detects the thoracic cavity region of the input single frame to segment and obtain the lung lobe region of the single frame;

[0073] Input gate 102 is connected to convolutional network 101 and receives a single frame of lung lobe region;

[0074] Output gate 103 outputs the predicted lung lobe image;

[0075] Forget gate 104 receives the lung lobe image of the previous frame output by output gate 103, and processes the lung lobe image to form a memory image output.

[0076] Output gate 103 receives a single frame of lung lobe region and memory image respectively and predicts the lung lobe image.

[0077] Specifically, to achieve better extraction of the periodically moving lung lobe region, in this embodiment, the lung lobe region of each frame is detected by a convolutional network 101 to segment and obtain a single-frame lung lobe region. Subsequently, the single-frame lung lobe image of each frame is processed by a memory network model composed of an input gate 102, an output gate 103, and a forget gate 104 to obtain the predicted lung lobe image. At the same time, the lung lobe region in the previous frame image is selected and retained by controlling the forget gate 104. Since the breathing process is a periodic process, the lung lobe image can be obtained more accurately after repeated extraction, thereby eliminating the afterimage part at the edge.

[0078] In one embodiment, such as Figure 4 As shown, the nodule description module 2 includes:

[0079] Nodule extraction module 21 extracts multiple nodule regions from lung lobe images;

[0080] Location extraction module 22 inputs nodule regions and lung lobe images into the location description model to obtain the relative positional relationships of multiple nodule regions and the distribution relationship of nodule regions in lung lobe images.

[0081] Location description module 23 generates nodule location description information according to relative positional relationships and distribution relationships.

[0082] Specifically, in order to achieve better prompting effect for large language models, in this embodiment, the nodule regions are extracted in advance, and then the relative positional relationship of multiple nodule regions and the distribution relationship of nodule regions in lung lobe images are obtained through the location description model. This makes the generated nodule location description information more in line with actual diagnostic habits, which is convenient for determining the location of lesions and the overall spread of lesions in the lungs, thereby generating more accurate consultation suggestions.

[0083] In one embodiment, the nodule description module 2 further includes a vector graphics acquisition module 24, such as... Figure 5 As shown, the vector graphics acquisition module 24 includes:

[0084] The first edge extraction module 241 acquires nodule example images and extracts the edges of example nodules from each nodule example image.

[0085] The graphics processing module 242 processes the edges of the example nodules to obtain vector graphics of the nodules;

[0086] like Figure 6 As shown, the nodule extraction module 21 includes:

[0087] The second edge extraction module 211 performs edge extraction on the lung lobe image to obtain an edge line image.

[0088] The image matching module 212 matches the edge line image based on the nodule vector image to obtain multiple matching positions;

[0089] The nodule output module 213 outputs the enclosed area corresponding to the matching position as the nodule area.

[0090] Specifically, to achieve better acquisition of nodule locations in the lung lobes, this embodiment uses a method of pre-maintaining vector graphics and then performing graphic matching to detect nodules. The nodule example images are standard medical images obtained from multiple sources, including images of several typical nodules. Through algorithms such as binarization and edge extraction, the edges of the example nodules are easily extracted, and then vectorized to obtain nodule vector images. In the actual matching process, appropriate image processing methods are used to binarize and extract edges from the images, resulting in edge line images. Some suspected nodule locations have enclosed areas similar to the nodule vector images, bounded by edge lines. The matched areas are output as nodule regions, achieving better matching results.

[0091] In one embodiment, such as Figure 7 As shown, the location description model includes:

[0092] Feature extraction network 201 extracts features from lung lobe images with marked nodular regions to obtain location features;

[0093] Clustering module 202 is connected to feature extraction network 201. Clustering module 202 clusters positional features to obtain nodule clustering results as relative positional relationships.

[0094] Radiomics model 203 is connected to a feature extraction network and generates distribution relationships based on location features.

[0095] Specifically, to achieve a better description of the nodule location, this embodiment constructs the aforementioned model to process the extracted location features, thereby describing the distribution location of the nodules. Before extraction, the nodule region and lung lobe images are synthesized, and the nodule region in the lung lobe images is indicated by annotations. Subsequently, the feature extraction network 201 extracts the location features in the image based on the annotations through a series of convolutions, upsampling, and downsampling operations. These features are then input into the clustering module 202 and the radiomics model 203, respectively. The clustering module 202 clusters nodules according to their relative location features based on the annotations, stopping when a certain threshold is reached. This is used to determine the number of lesions in the lung lobe and their approximate location, thus forming a relative location relationship. The radiomics module 203 predicts the distribution of nodules in the lung lobe using the acquired location features and pre-trained lung lobe images, thereby forming a distribution relationship.

[0096] In one embodiment, such as Figure 8 As shown, the location description module 23 includes:

[0097] Grading module 231 determines the spread of nodules in the lobar region according to the distribution relationship and the predefined distribution level.

[0098] Template calling module 232 calls the corresponding overall template according to the distribution level;

[0099] The description information generation module 233 fills the overall template with relative positional relationships and distribution levels to obtain nodule location description information.

[0100] Specifically, to achieve better descriptive results, this embodiment further classifies the distribution of nodules that may be encountered in clinical practice into levels, generating distribution grades. Subsequently, the actual spread is determined based on the distribution relationships and distribution grades, and the corresponding template is invoked to facilitate the use of appropriate descriptive methods according to the disease progression. Finally, the overall template is filled based on relative positional relationships and distribution grades to form nodule location description information, achieving a better prompting process.

[0101] In one embodiment, such as Figure 9 As shown, the language model interaction module 3 includes:

[0102] Output module 31 inputs lung lobe images into a large language model;

[0103] The prompting module 32 generates prompt words based on the nodule location description information and inputs the prompt words into a large language model;

[0104] The diagnostic suggestion generation module 33 obtains supplementary diagnostic suggestions from the feedback of a large language model.

[0105] Specifically, to achieve better prompting results, in this embodiment, lung lobe images are pre-input into a large language model, enabling the large language model to perform preliminary recognition of the images; subsequently, segmented prompting words are generated according to the nodule location description information and input into the large language model one by one, so that the large language model can gradually focus on the location information of the nodules, thereby generating more accurate supplementary consultation suggestions.

[0106] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A lung magnetic resonance imaging diagnostic system based on a large language model, characterized in that, include: An image extraction module acquires lung magnetic resonance images and extracts lung lobe images from the lung magnetic resonance images; A nodule description module, which generates nodule location description information based on the lung lobe image; The language model interaction module inputs the lung lobe image and the nodule location description information into an external large language model to obtain supplementary consultation suggestions from the large language model. The inquiry module asks questions of the patient based on the supplementary consultation suggestions to obtain additional consultation information; The nodule description module includes: A nodule extraction module, which extracts multiple nodule regions from the lung lobe image; The location extraction module inputs the nodule region and the lung lobe image into the location description model to obtain the relative positional relationship of multiple nodule regions and the distribution relationship of the nodule regions in the lung lobe image. A location description module generates the nodule location description information according to the relative positional relationship and the distribution relationship.

2. The lung magnetic resonance imaging consultation system according to claim 1, characterized in that, The image extraction module includes: A thoracic cavity extraction module, which extracts the thoracic cavity region from multiple frames of the lung magnetic resonance imaging sequence; The lung lobe extraction module processes the thoracic cavity region into a time series model to obtain lung lobe images corresponding to the lung lobe region.

3. The lung magnetic resonance imaging consultation system according to claim 2, characterized in that, In the lung lobe extraction module, the time series model includes: A convolutional network that detects the thoracic cavity region in an input single frame to segment and obtain a single-frame lung lobe region; An input gate, which is connected to the convolutional network, receives the single-frame lung lobe region; An output gate that outputs the predicted lung lobe image; The forget gate receives the lung lobe image of the previous frame output by the output gate, and processes the lung lobe image to form a memory image output. The output gate receives the single-frame lung lobe region and the memory image respectively, and predicts the lung lobe image.

4. The lung magnetic resonance imaging consultation system according to claim 1, characterized in that, The nodule description module further includes a vector graphics acquisition module, which includes: The first edge extraction module acquires nodule example images and extracts the edges of example nodules from each nodule example image. A graphics processing module processes the edges of the example nodules to obtain vector graphics of the nodules; The nodule extraction module includes: The second edge extraction module extracts the edges of the lung lobe image to obtain an edge line image. The image matching module matches the edge line image based on the nodule vector image to obtain multiple matching positions; The nodule output module outputs the enclosed area corresponding to the matching position as the nodule region.

5. The lung magnetic resonance imaging consultation system according to claim 1, characterized in that, The location description model includes: A feature extraction network is used to extract features from the lung lobe image with the nodule region labeled to obtain location features; A clustering module, which is connected to the feature extraction network, performs clustering on the location features to obtain nodule clustering results as the relative positional relationship; A radiomics model, wherein the radiomics model is connected to the feature extraction network, and the radiomics model generates the distribution relationship according to the location features.

6. The lung magnetic resonance imaging consultation system according to claim 2, characterized in that, The location description module includes: A grading module, which determines the spread of nodules in the lung lobe region according to the distribution relationship and a predefined distribution level; A template invocation module, which invokes the corresponding overall template according to the distribution level; The description information generation module fills the overall template with the relative positional relationship and the distribution level to obtain the nodule location description information.

7. The lung magnetic resonance imaging consultation system according to claim 6, characterized in that, The language model interaction module includes: The output module inputs the lung lobe image into the large language model; The prompting module generates prompt words based on the nodule location description information and inputs the prompt words into the large language model; A consultation suggestion generation module obtains the supplementary consultation suggestions fed back by the large language model.

Citation Information

Patent Citations

  • A method for identifying early lung adenocarcinoma manifested as frosted glass nodules

    CN111539946A

  • New coronal pneumonia intelligent diagnosis system based on deep learning

    CN112786189A