Needle passage detection method and device based on multi-modal large language model

Abstract

The invention discloses a needle passage detection method and device based on a multi-modal large language model (MLLM). The method comprises the following steps: constructing a rotation sensing target detection data set for training and an instruction fine tuning data set of the multi-modal large language model; analyzing the two-dimensional slice of the CT image by using a rotary sensing needle point and needle handle detection algorithm, and extracting space information of a needle point and a needle handle; an initial detection result is compared and verified by means of MLLM, global image context analysis is carried out on a target with low confidence coefficient by utilizing a text prompt function guide model, and a false positive target is eliminated; and performing browsing analysis on the slice sequence by using MLLM, and realizing accurate matching of the needle tip and the needle handle in combination with spatial consistency reasoning of the cross-slice sequence. Compared with the prior art, the scheme of the invention effectively relieves the problem that the existing needle passage automatic pickup method fails under the complicated conditions of fuzzy needle passage gray scale distribution, adhesion and the like, and can improve the reliability and accuracy of needle passage detection.

Description

Technical Field

[0001] The present invention relates to the field of simulation technology, and particularly to a needle track detection method and device based on a multimodal large language model. Background Art

[0002] During particle implantation surgery, it is necessary to real-time locate the implanted needle tracks and particles based on intraoperative CT, so as to dynamically optimize the plan based on the dose of the implanted particles. However, automatic needle track detection faces many problems: First, the gray distribution of the needle tracks is blurred, which not only creates obstacles for automatic detection, but also makes it quite difficult for manual annotation. Moreover, there is an adhesion phenomenon between the needle tracks, further increasing the difficulty of separating and identifying the needle tracks. In recurrent cases, the previously implanted particles and the surrounding bone structures will interfere with intraoperative CT imaging and have a significant impact on needle track positioning.

[0003] Currently, in the field of automatic needle picking, most existing studies focus on the positioning of needle tracks implanted in the perineal head-to-foot direction. Typical application scenarios include brachytherapy and biopsy puncture for prostate and gynecological tumors. In such scenarios, the trajectory of the needle often appears as a clear small circle in the axial slice, with a low probability of adhesion and easy to locate. There are also some studies that only detect a single implanted needle. Compared with the complex positioning of multiple needle tracks, the task difficulty is greatly reduced. Generally speaking, for the automatic detection technology of needle tracks for body particle implantation based on intraoperative CT, there are still many unsolved problems at present, and key breakthroughs are urgently needed to meet the actual clinical needs. Summary of the Invention

[0004] In view of this, the present invention proposes a needle track detection method and device based on a multimodal large language model, and the simulation results provide reference for clinicians to alleviate the deficiencies of the prior art.

[0005] In the first aspect, first, a dataset for training a rotation-aware object detection network is constructed, including the rotation box annotation information of the needle tips and needle handles in all CT two-dimensional slices, and a dataset for instruction fine-tuning of the multimodal large language model, covering two task modes of low-confidence screening and multi-slice pairing; subsequently, the intraoperative CT two-dimensional slices are input into the rotation-aware object detection network to extract the rotation box coordinates and confidence scores of the needle tips and needle handles, and a first detection result is obtained; the first detection result is input into the multimodal large language model, and semantic contrast reasoning is performed on the low-confidence objects by combining the image context information and high-confidence object prompts to screen out suspected false positives and obtain a second detection result; finally, the remaining objects and the corresponding image sequences are input into the multimodal large language model together, and reasoning pairing is performed by combining the prior prompt of the number of needle tracks to output the corresponding relationship between the needle tips and the needle handles, and a structured needle track detection result is generated, which completely expresses the three-dimensional starting and ending positions and direction information of each needle track.

[0006] In a second aspect, the present invention provides a needle track detection device based on rotation perception and multi-modal reasoning, comprising: a data construction and management module for generating and maintaining a rotation perception object detection training data set and a multi-modal instruction fine-tuning data set, providing standardized data support for the joint training of a detection model and a language model; a rotation perception needle tip-needle handle detection module for performing object detection on intraoperative CT two-dimensional slices, extracting the rotation box coordinates and confidence levels of the needle tip and needle handle of each needle; a detection target comparison and verification module for inputting an initial detection result into a multi-modal large language model, and eliminating false positive targets by performing image semantic comparison and analysis on high and low confidence targets; a needle tip-needle handle pairing module for performing cross-slice needle tip and needle handle pairing according to the position and direction information of the detection target, and combining multi-slice image input and needle track number prompt information, and outputting structured needle track pairing information.

[0007] The beneficial effects of the present invention are as follows:

[0008] The technical solution provided by the present invention may include the following beneficial effects: a needle track detection method and device based on a multi-modal large language model are proposed, which combines rotation perception object detection and multi-modal semantic reasoning for the first time, realizing high-precision recognition and automatic pairing of needle tips and needle handles in intraoperative CT images. By introducing a multi-modal large language model to perform semantic verification on low-confidence targets, the false positive rate in complex scenarios is significantly reduced; at the same time, based on the cross-slice pairing and reasoning ability of the model, the accurate reconstruction of the multi-needle track spatial structure is completed. This method combines local accuracy and global consistency, breaking through the robustness bottleneck of traditional methods based on gray or geometric features in scenarios such as metal artifact interference and dense adhesion of needle tracks, and can provide highly reliable input support for intraoperative particle dose assessment and treatment plan optimization, with significant clinical application value and promotion prospects.

[0009] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are one embodiment of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0011] Figure 1 It is a schematic flowchart of a needle track detection method based on a multi-modal large language model according to the first embodiment of the present invention;

[0012] Figure 2Schematic diagram of the structure of a needle track detection simulation device based on a multimodal large language model according to the second embodiment of the present invention. Detailed implementation manners

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are some, but not all, of the embodiments of the present invention.

[0014] First embodiment:

[0015] Figure 1 Schematic flowchart of a needle track detection method based on a multimodal large language model according to the first embodiment of the present invention. As Figure 1 shown, the method includes steps S1, S2, S3, and S4.

[0016] Step S1: Construct a rotation-aware object detection subset and a multimodal large language model (MLLM) instruction fine-tuning subset. Among them, the rotation-aware object detection subset includes the rotation box annotations of the needle tips and needle handles in all two-dimensional CT slices; the MLLM fine-tuning subset contains two sub-task modules, which are respectively used for low-confidence object screening and multi-slice object pairing, and the training samples are organized in the ShareGPT format combining text and images.

[0017] Step S2: Rotation-aware object detection. In this step, the input is the two-dimensional slice image of intraoperative CT. The rotation-aware object detection network is used to detect the needle tips and needle handles of each needle in the image respectively, extract the rotation box coordinates, confidence, and other information, and construct the initial object detection result.

[0018] Step S3: Verification of low-confidence objects. The detection result is input into the multimodal large language model (MLLM), a prompt information template is constructed, the high-confidence object list and the image positions, directions, and other information of low-confidence objects are input, and semantic understanding and image context comparison are performed through the MLLM to determine whether it is a false positive object. If it is a false positive, the object is removed, thereby improving the overall detection accuracy and reliability.

[0019] Step S4: Cross-slice needle tip-needle handle pairing. The detection results of multiple CT slices are used as inputs, and together with the total number of needle tracks NUM that has been prompted, they are input into the MLLM. The model combines the image spatial distribution, target direction consistency, and position continuity, and completes the pairing between the needle tips and needle handles through cross-slice inference, and outputs a structured needle track pairing result.

[0020] Step S5: Output the needle track reconstruction result. Convert the pairing result into a structured output form, including the starting coordinates (needle handle) and ending coordinates (needle tip) of each needle track and their spatial direction information, to achieve three-dimensional reconstruction of the intraoperative needle track. In an optional embodiment, the MLLM is based on the LoRA parameter efficient fine-tuning technology, and supervised instruction fine-tuning training is performed on the low-confidence screening task and the needle tip-needle handle pairing task respectively.

[0021] Second Embodiment:

[0022] Figure 2 It is a schematic structural diagram of a needle track detection device based on a multimodal large language model described in an embodiment of the present invention. As Figure 2 shown, the device includes a data construction and management module 201, a rotation-aware needle tip-needle handle detection module 202, a detection target comparison and verification module 203, and a needle tip-needle handle pairing module 204.

[0023] The data construction and management module 201 is used to construct a rotation-aware object detection training data set and an MLLM instruction fine-tuning data set, and perform unified format management and storage on data samples to support the model training and inference processes.

[0024] The rotation-aware needle tip-needle handle detection module 202 is used to perform rotation object detection on the two-dimensional images of intraoperative CT, extract the rotation box coordinates and confidence scores of the needle tip and needle handle of each needle, and generate an initial detection result.

[0025] The initial result comparison and verification module 203 is used to input the detection result into a multimodal large language model, combine the spatial prompt information of high-confidence targets, perform semantic reasoning and comparison on low-confidence candidate targets, judge their effectiveness, screen out false positive targets, and output an accurate target list.

[0026] The needle tip-needle handle pairing module 204 is used to input the screened target result and the image slice sequence into the MLLM together, and achieve accurate matching of the needle tip and needle handle by combining spatial continuity, direction consistency, and prior prompt information of the needle track number, and output a structured needle track reconstruction result.

[0027] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments, or can easily think of changes, or make equivalent replacements for some of the technical features; and these modifications, changes or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A needle track detection method and device based on a multimodal large language model, characterized in that It includes the following steps: Construct a rotation-aware object detection dataset and an instruction fine-tuning dataset for the multi-modal large language model; Input the two-dimensional CT slices into the rotation-aware object detection network to obtain the first detection result; wherein, the first detection result includes the rotation box coordinates and confidence scores of the needle tip and the needle handle; Input the first detection result into the multi-modal large language model, perform context semantic analysis on the low-confidence targets and compare them with the high-confidence targets, screen out the false positive targets, and obtain the second detection result; wherein, the second detection result includes the rotation box coordinates of the needle tip and the needle handle; Input the second detection result and the corresponding image sequence into the multi-modal large language model, and through paired reasoning with the prior hint of the given number of needle tracks, output the pairing result of the needle tip and the needle handle to obtain the needle track reconstruction result.

2. The method according to claim 1, wherein the rotation-aware object detection dataset annotates the rotation boxes of the needle tips and the needle handles in all two-dimensional CT slices for training a detection model with rotation awareness ability.

3. The method according to claim 1, wherein the multi-modal large language model instruction fine-tuning dataset includes two task modules: one is the screening task for low-confidence targets, and the other is the needle tip-needle handle pairing task across multiple slices.

4. The method according to claim 1, wherein The format of the multi-modal large language model instruction fine-tuning dataset is ShareGPT, including instruction text, model responses, and image paths.

5. The method according to claim 1, wherein The rotation-aware object detection network is a rotation box detection model based on the HRNet structure, and the rotation-aware object detection network is constrained based on the intersection over union and classification loss.

6. The method according to claim 1, wherein, Targets with confidence scores between 0.1 and 0.25 are regarded as low-confidence targets, those greater than 0.25 are high-confidence targets, and targets below 0.1 are directly ignored.

7. The method according to claim 1, wherein the multi-modal large language model receives image feature encoding and target prompt text, and completes the low-confidence screening or needle tip-needle handle pairing task by using low-rank adaptation (LoRA) instruction fine-tuning. The MLLM is constrained based on binary cross-entropy.

8. The method according to claim 1, the instruction template used for the low-confidence screening is: "Given the list of {high-confidence objects} and their {positions and orientations}, identify whether the target located at {position} and with an azimuth angle of {angle} is a valid target".

9. The method according to claim 1, the instruction template for the multi-slice pairing is: "Given the list of targets in multiple slices and their directions and positions, combined with the hint of the total number of needles {NUM}, perform needle tip-needle handle pairing".

10. The method according to claim 1, the number of input corresponding image sequences is 5.

11. A needle track detection method and device based on a multimodal large language model, characterized in that, It includes: A data construction and management module for constructing and managing a rotation-aware object detection training dataset and an instruction fine-tuning dataset for the multi-modal large language model; A rotation-aware needle tip-needle handle detection module for performing rotation object detection on the two-dimensional slices of the CT image, and extracting the rotation box information and confidence scores of the needle tips and the needle handles of each implanted needle; A detection target comparison and verification module, which is used to perform multi-modal language inference and comparative analysis on the targets with low confidence in the initial detection results and the targets with high confidence, and combine the global image semantic information to screen out suspected false positive targets, improving the detection accuracy and robustness; A needle tip-needle handle pairing module, which is used to receive the target detection results from the multi-slice sequence, combine the given prior of the number of needle tracks, analyze the spatial position and direction consistency between the targets through a multi-modal large language model, complete the automatic pairing of the needle tip and the needle handle, and output the needle track reconstruction result.

Images