Medical video region-of-interest feature detection and classification method based on large language model
Through the online deployment of large language model, the characteristics of medical videos of interest are detected and classified, which solves the problem of relying on doctors' subjective experience and time-consuming in the existing technology, realizes automatic analysis and efficient identification of areas of interest, and improves clinical Doctor's work efficiency.
Patent Information
- Application Number
- CN202411084356.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-12
AI Technical Summary
Classification of regions of interest in existing medical videos requires a long time dependent on doctors' subjective experience, and the detection and classification methods of regions of interest in medical videos based on large language models have not been effectively developed.
By deploying a large language model based on medical video corpus training online, providing a video data upload interface and a prompt word input interface, using a large language model to identify and feature detection of areas of interest in medical videos, and output classification results and feature expressions.
Automatic resolution of areas of interest in medical videos is realized, which reduces the need for additional network models and improves the efficiency of clinicians in identifying and classifying areas of interest.
Smart Images

Figure CN118941872A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical video feature detection and classification, and in particular to a method for detecting and classifying features of a region of interest in a medical video based on a large language model. Background Art
[0002] The existing classification problem of regions of interest in medical videos usually requires the doctor to locate the region of interest in the video with his naked eyes, and then identify the representation of the region of interest for feature detection and analysis to give the classification result of the region of interest. This process requires the subjective experience of doctors and manual participation is time-consuming. In the field of artificial intelligence-assisted medical video data classification, it is currently necessary to design a more complex video parsing neural network, but it still relies on high-performance computing hardware and massive training data, and easily accessible medical video classification methods are still scarce.
[0003] Large Language Models (LLMs) are a core technology in the field of Natural Language Processing (NLP). These models can understand and generate human-readable natural language text and corresponding output by training a large amount of text and speech data. LLMs play an important role in the field of NLP and are widely used in various tasks such as text generation, machine translation, sentiment analysis, and question-answering systems. However, the current method of feature detection and classification of regions of interest in medical videos based on large language models is basically a technical gap.
[0004] Therefore, we urgently need a medical video classification method based on a large language model. Summary of the invention
[0005] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.
[0006] To this end, the purpose of the present invention is to propose a method for detecting and classifying features of regions of interest in medical videos based on a large language model.
[0007] In order to achieve the above-mentioned purpose, the technical solution of the present invention provides a method for detecting and classifying features of regions of interest in medical videos based on a large language model, the method comprising: deploying a large language model trained based on medical video corpus online through a model deployment module, and providing a video data upload interface and a prompt word input interface; uploading the acquired medical video to the deployed large language model through the video data upload interface through the video data upload module, so that the regions of interest in the medical video can be subsequently identified and features detected through the deployed large language model; sending prompt words containing information about the regions of interest in the medical video to the deployed large language model through the prompt word input module, so that the deployed large language model can locate the regions of interest; identifying the regions of interest in the medical video based on the uploaded medical video and the prompt words using the deployed large language model through the video parsing and output module, so as to detect the features of the regions of interest, and outputting the classification results and feature descriptions of the regions of interest according to the detected features.
[0008] Preferably, the large language model is a model capable of performing video analysis.
[0009] Preferably, the large language model includes: a large language model based on the Transformers architecture.
[0010] Preferably, the medical video includes: videos composed of ultrasound, CT, MRI, PET radiographic images, endoscopic examination videos, and videos collected by wearable devices.
[0011] Preferably, the format of the medical video is a universal video format.
[0012] Preferably, the video data uploading module includes a function of selecting a local or online video file to upload to the deployed large language model.
[0013] Preferably, the prompt word input module includes a function of inputting text information into the deployed large language model; and the text information includes: prompt words containing information about the region of interest in the medical video.
[0014] Preferably, the video analysis and output module includes a function of identifying a target according to the region of interest information contained in the prompt word, a function of detecting features based on the identified region of interest target, and a function of classifying the region of interest based on the detected features.
[0015] Preferably, the video analysis and output module further includes: a function of performing morphological analysis on the region of interest in the medical video and outputting the result of feature description of the region of interest in the medical video.
[0016] Preferably, the medical video region of interest includes: organ tissue, tumor lesions, abnormal structures, and inflammation areas in the medical video.
[0017] Beneficial effects of the present invention:
[0018] The method for detecting and classifying features of regions of interest in medical videos based on a large language model provided by the present invention will realize the automatic analysis of key features of regions of interest in medical videos, and reduce the work of designing additional network models for identifying and detecting regions of interest. Furthermore, while automatically completing the target identification and feature detection of regions of interest, the method outputs the classification results of the video and outputs the above target identification and feature detection of regions of interest as classification evidence, thereby improving the efficiency of clinicians in visually identifying and classifying regions of interest in medical videos.
[0019] Additional aspects and advantages of the invention will become apparent from the following description, or may be learned by practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 A schematic flow chart of a method for detecting and classifying features of regions of interest in medical videos based on a large language model according to an embodiment of the present invention is shown;
[0021] Figure 2 A schematic diagram showing an embodiment of the present invention using a deployed large language model to locate a lung lesion region of interest in a video composed of lung CT scan images and mark the location of the lung lesion. DETAILED DESCRIPTION
[0022] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0024] Figure 1 FIG. 1 is a schematic flow chart showing a method for detecting and classifying features of a region of interest in a medical video based on a large language model according to an embodiment of the present invention. Figure 1 As shown, the method for detecting and classifying features of regions of interest in medical videos based on a large language model includes:
[0025] Step S1: deploying a large language model trained based on medical video corpus online through the model deployment module, and providing a video data upload interface and a prompt word input interface;
[0026] Step S2: The acquired medical video is uploaded to the deployed large language model through the video data upload interface by the video data upload module, so that the deployed large language model can be used to identify and detect features of the region of interest in the medical video;
[0027] Step S3: sending the prompt words containing the information of the region of interest in the medical video to the deployed large language model through the prompt word input module, so that the deployed large language model can locate the region of interest;
[0028] Step S4: Through the video analysis and output module, the region of interest in the medical video is identified based on the uploaded medical video and the prompt words to detect the features of the region of interest, and the classification result and feature description of the region of interest are output according to the detected features.
[0029] In this embodiment, the large language model is a new artificial intelligence technology that can be deployed online and easily used by users. Through the video upload and prompt word upload interface provided by the deployed large language model, the large language model can perform frame-by-frame analysis of medical video data. Combined with the description of the target region of interest in the prompt word, the large language model is expected to accurately identify the region of interest, and then automatically detect the key imaging representation of the region of interest, and finally achieve accurate classification based on the extracted image features.
[0030] In this embodiment, the large language model is a model trained by using medical video corpus.
[0031] In this embodiment, the specific process of step S1 is as follows: by loading the large language model on the online server, the large language model is debugged online to be able to receive and analyze the uploaded video and prompt words. At the same time, a video data upload interface and a prompt word input interface are provided. Specifically, the current common Transformers-based large language model can be connected or the locally trained large language model can be directly loaded on the online server.
[0032] In this embodiment, the overall concept of the method for detecting and classifying features of regions of interest in medical videos based on a large language model provided by the present invention is to deploy a large language model trained based on medical video corpus online, use a video data upload module to upload medical videos to the deployed large language model, inform the deployed large language model of the regions of interest that need to be identified in the uploaded medical video through a prompt word input module, use a video parsing and output module to identify the regions of interest in the uploaded medical video through the large language model, detect features of the regions of interest, and output classification results and feature descriptions of the regions of interest based on the detected features.
[0033] In this embodiment, the method for detecting and classifying features of regions of interest in medical videos based on a large language model provided by the present invention will realize the automatic analysis of key features of regions of interest in medical videos, reducing the work of designing additional network models for identifying and detecting regions of interest. Furthermore, while automatically completing the target identification and feature detection of regions of interest, the method outputs the classification results of the video and outputs the above target identification and feature detection of regions of interest as classification evidence, thereby improving the efficiency of clinicians in visually identifying and classifying regions of interest in medical videos.
[0034] In one embodiment of the present invention, the large language model is a model capable of performing video analysis.
[0035] In one embodiment of the present invention, the large language model includes: a large language model based on a Transformers architecture.
[0036] In one embodiment of the present invention, the medical video includes: a video composed of ultrasound, CT, MRI, PET radiological images, an endoscopic examination video, and a video collected by a wearable device.
[0037] In one embodiment of the present invention, the format of the medical video is a universal video format.
[0038] In one embodiment of the present invention, the video data uploading module includes a function of selecting a local or online video file to upload to the deployed large language model.
[0039] In one embodiment of the present invention, the prompt word input module includes a function of inputting text information into the deployed large language model; and the text information includes: prompt words containing information about the region of interest in the medical video.
[0040] In one embodiment of the present invention, the video analysis and output module includes a function of identifying a target based on the region of interest information contained in the prompt word, a function of performing feature detection based on the identified region of interest target, and a function of classifying the region of interest based on the detected features.
[0041] In one embodiment of the present invention, the video analysis and output module further includes: a function of performing morphological analysis on the region of interest in the medical video and outputting the result of feature description of the region of interest in the medical video.
[0042] In one embodiment of the present invention, the medical video region of interest includes: organ tissue, tumor lesions, abnormal structures, and inflammation areas in the medical video.
[0043] The technical solution of the present invention will be demonstrated below with a specific embodiment. The method for detecting and classifying features of regions of interest in medical videos based on a large language model of the present invention uses a deployed large language model to classify benign and malignant lung nodules in a video composed of a certain medical data, namely, lung CT scan images, in the following specific implementation steps:
[0044] 1. Upload a local patient's lung CT scan image video through the video data upload module. The video is composed of layers from the abdomen to the neck and named "107_300.mp4"; and send the prompt words of the region of interest information in the uploaded video to the deployed large language model through the prompt word input module, so that the deployed large language model can locate the region of interest;
[0045] Specifically, the prompt words of the region of interest information specifically include: basic information such as gender and age of the CT scanner.
[0046] The large language model includes: a large language model based on the Transformers architecture.
[0047] The large language model based on the Transformers architecture specifically includes: a neural network model composed of multiple layers of multi-head self-attention modules, including GPT (Generative Pre-trained Transformer).
[0048] 2. If Figure 2 As shown, the deployed large language model is used to locate the region of interest of lung lesions in the "107_300.mp4" video, and the location of the lung lesions identified by the large language model is output and marked with a bounding box.
[0049] 3. Output the feature description and classification results of the imaging features of the region of interest detected by the deployed large language model. Specifically, output a radiology report on lung nodule analysis:
[0050] CT Scan Information:
[0051] Video length: 300 frames
[0052] Pixel size: 0.74×0.74×1mm
[0053] Frame rate: 20 frames per second
[0054] Frame with maximum nodule diameter: about 11 seconds (around frame 220)
[0055] Characteristics of pulmonary nodules: size of pulmonary nodules, morphology of pulmonary nodules, edge of pulmonary nodules, internal structure of pulmonary nodules, characteristics of enhanced areas and surrounding structures, reporting of whether pulmonary nodules are solid, mixed or ground-glass opacities, etc.
[0056] Quantitative probability of malignancy of pulmonary nodules: Output the predicted value of the malignancy probability of the pulmonary nodule accurate to two decimal places.
[0057] In summary, Figure 2 It can be seen that the method proposed in the present invention can identify the location of the region of interest such as the lesion area in the medical video based on the large language model, detect the imaging features of the area, classify the region of interest and output the classification results and feature descriptions, and at the same time can assist in improving the feature analysis and classification efficiency of clinicians in medical videos.
[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for detecting and classifying features of regions of interest in medical videos based on a large language model. It is characterized in that include: Through the model deployment module, a large language model trained based on medical video corpus is deployed online, and a video data upload interface and a prompt word input interface are provided; The acquired medical video is uploaded to the deployed large language model through the video data upload interface by the video data upload module, so that the region of interest in the medical video can be subsequently identified and features detected by the deployed large language model; Sending the prompt words containing the information of the region of interest in the medical video to the deployed large language model through the prompt word input module, so that the deployed large language model can locate the region of interest; Through the video analysis and output module, the region of interest in the medical video is identified based on the uploaded medical video and the prompt words to detect the characteristics of the region of interest, and the classification result and feature description of the region of interest are output according to the detected characteristics.
2. The method for detecting and classifying features of regions of interest in medical videos based on a large language model according to claim 1, It is characterized in that The large language model is a model capable of performing video analysis.
3. The method for detecting and classifying features of regions of interest in medical videos based on a large language model according to claim 2, It is characterized in that The large language model includes: a large language model based on the Transformers architecture.
4. The method for detecting and classifying features of regions of interest in medical videos based on a large language model according to claim 1, wherein the medical video include: Videos composed of ultrasound, CT, MRI, PET radiology images, endoscopic examination videos, and videos collected by wearable devices.
5. The method for detecting and classifying features of regions of interest in medical videos based on a large language model according to claim 1, It is characterized in that The format of the medical video is a universal video format.
6. The method for detecting and classifying features of regions of interest in medical videos based on a large language model according to claim 1, It is characterized in that The video data uploading module includes a function of selecting a local or online video file to upload to the deployed large language model.
7. The method for detecting and classifying features of regions of interest in medical videos based on a large language model according to claim 1, It is characterized in that The prompt word input module includes a function of inputting text information into the deployed large language model; and the text information includes: prompt words containing information about the region of interest in the medical video.
8. The method for detecting and classifying features of regions of interest in medical videos based on a large language model according to claim 1, It is characterized in that The video analysis and output module includes a function of identifying a target according to the region of interest information contained in the prompt word, a function of detecting features based on the identified region of interest target, and a function of classifying the region of interest based on the detected features.
9. The method for detecting and classifying features of regions of interest in medical videos based on a large language model according to claim 1, Features: The video analysis and output module also includes: a function of performing morphological analysis on the region of interest in the medical video and outputting the result of feature description of the region of interest in the medical video.
10. The method for detecting and classifying features of regions of interest in medical videos based on a large language model according to any one of claims 1 to 9, It is characterized in that The medical video regions of interest include: organ tissues, tumor lesions, abnormal structures, and inflammation areas in the medical video.
Citation Information
Patent Citations
Medical image auxiliary diagnosis method and system combining image recognition and report editing
CN109583440A
Medical image report generation method and device and storage medium
CN116864064A
Information extraction method, training method of information extraction system, and information extraction system
CN117079298A
Conversational target positioning method and device based on arbitrary granularity text input
CN118350464A
Systems and methods for interacting with a large language model
US12051205B1
