Double-model fusion polyp segmentation quality control method and system

Abstract

The invention discloses a double-model fusion polyp segmentation quality control method and system, and the method comprises the steps: fusing a general video segmentation model and a special model for polyp segmentation, and constructing a polyp video segmentation quality control frame SQC-SAM; the first function of the method is to evaluate the polyp segmentation quality of a polyp segmentation model on a colonoscope video without a real label; and the second function is that the high-quality segmentation result is used as a prompt, the low-quality segmentation result is segmented again, and the polyp segmentation quality is improved under the condition that training or fine adjustment is not needed. A novel double-model fusion framework is provided, the segmentation result of the polyp segmentation model is effectively combined with the video segmentation capability and promptability of the basic video segmentation model, and effective utilization of time and space clues in the colonoscope video is realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a dual-model fusion polyp segmentation quality control method and system. Background Art

[0002] As the third most common malignant tumor and the second most lethal cancer, colorectal cancer poses a huge threat to human health. Colorectal adenomatous polyps are the key precursors of the disease. Identifying, locating, and removing diseased polyps can significantly reduce the morbidity and mortality of colorectal cancer, and play an important role in the diagnosis and treatment of colorectal cancer. Thanks to the development of artificial intelligence, automated polyp segmentation technology is expected to become an effective tool to assist in the diagnosis and treatment of colorectal cancer.

[0003] However, there are still some challenges for reliable clinical application of automated polyp segmentation: 1) Current polyp segmentation methods are mostly based on colonoscopic images. Although some progress has been made, these methods ignore the temporal and spatial cues between consecutive frames in colonoscopic videos; 2) Polyps vary in color, size, and shape, and current polyp segmentation methods often perform poorly when faced with diverse polyps in complex scenarios; 3) Current polyp segmentation methods usually directly output prediction results without judging the quality of the prediction, resulting in a lack of confidence in artificial intelligence technology among doctors and patients in clinical applications. Summary of the invention

[0004] The object of the present invention is to provide a dual-model fusion polyp segmentation quality control method and system.

[0005] The technical solution to achieve the purpose of the present invention is: a dual-model fusion polyp segmentation quality control method, comprising:

[0006] (1) Input a colonoscopy video; split the colonoscopy video into consecutive frames and generate a binary mask for each frame using the polyp segmentation model;

[0007] (2) After the hint extractor, the bounding box and center point of each mask are extracted to generate a hybrid enhanced hint;

[0008] (3) Each frame of image and its visual cue are input into the SAM2 model, and the forward and backward propagation of each Frame, get the mapping of the segmentation result of the current image on the previous and next frames;

[0009] (4) Calculate the Dice coefficient between the segmentation result of each frame and the mapping of the segmentation results of the previous and next frames in the current frame, and take the average of all Dice coefficients as the segmentation quality evaluation score of each frame. The higher the score, the better the quality of the segmentation result.

[0010] (5) Calculate the average of the segmentation quality assessment scores of all frames in the colonoscopy video, and frames with scores less than the average are considered low-quality frames;

[0011] (6) Take the visual cues of non-low-quality frames and use SAM2 to segment them. If the Dice coefficient between the self-cued segmentation result and the original segmentation result is greater than the threshold, and the quality assessment score is greater than or equal to the average score, the frame is considered a high-quality frame. For each low-quality frame, select the c high-quality frames closest to it in the video frame sequence to form a cue sequence for re-segmenting the low-quality frame.

[0012] (7) Map the segmentation results of each frame in the prompt sequence to the low-quality frame using the SAM2 model, and generate a set of binary masks on the low-quality frame;

[0013] (8) Take the average mask of all maps as the segmentation result of each low-quality frame, and achieve re-segmentation of all low-quality frames in the entire colonoscopy video.

[0014] Based on the same inventive concept, the present invention also provides a dual-model fusion polyp segmentation quality control system for implementing the above method, the system comprising:

[0015] The quality assessment module evaluates the polyp segmentation quality of the polyp segmentation model on colonoscopy videos without real annotations by propagating and comparing the polyp segmentation results of each frame in the video;

[0016] The re-segmentation module uses the high-quality segmentation results to re-segment the low-quality segmentation results.

[0017] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the computer program is loaded into the processor.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] (1) This paper proposes a novel dual-model fusion framework, which effectively combines the segmentation results of the polyp segmentation model with the video segmentation capability and suggestibility of the basic video segmentation model, thereby realizing the effective utilization of temporal and spatial cues in colonoscopy videos.

[0020] (2) The present invention proposes an accurate and efficient polyp segmentation quality evaluation system. By propagating and comparing the polyp segmentation results of each frame in the video, it is possible to accurately evaluate the polyp segmentation quality of colonoscopy videos without real annotation.

[0021] (3) The present invention proposes an effective method for improving the quality of polyp segmentation, which uses high-quality segmentation results to re-segment low-quality segmentation results, thereby improving the quality of polyp segmentation without the need for training or fine-tuning.

[0022] (4) The present invention can replace different models according to different usage scenarios and tasks, can cope with various downstream tasks, and has extremely high flexibility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 1 is a structural diagram of the dual-model fusion polyp segmentation quality control framework SQC-SAM of the present invention, wherein (a) is a structural diagram of the segmentation quality assessment module, and (b) is a structural diagram of the re-segmentation.

[0024] Figure 2 This is a flow chart of SQC-SAM for segmentation quality assessment and re-segmentation.

[0025] Figure 3 It is a schematic diagram of the results of performing a polyp segmentation test on a set of colonoscopy videos and comparing the present invention with two existing polyp segmentation models, PolypPVT and HSNet. DETAILED DESCRIPTION

[0026] The present invention integrates the general video segmentation model Segment Anything Model 2 (SAM2) and a polyp segmentation model to construct a polyp video segmentation quality control framework SQC-SAM, which realizes two functions. The first function is to evaluate the polyp segmentation quality of the polyp segmentation model on colonoscopy videos without real annotations; the second function is to use high-quality segmentation results as prompts to re-segment the low-quality segmentation results, so as to improve the polyp segmentation quality without training or fine-tuning.

[0027] The dual-model fusion polyp segmentation quality control method and the corresponding system of the present invention are respectively described below.

[0028] The present invention proposes a dual-model fusion polyp segmentation quality control method, comprising the following steps:

[0029] First, the polyp segmentation quality of the polyp segmentation model was evaluated on colonoscopy videos without real annotations;

[0030] Split a colonoscopy video into Consecutive frames , polyp segmentation model Generate a binary mask for each frame .

[0031]

[0032] Extract each frame Mask Bounding box and center point of the generated hybrid enhanced visual cues .

[0033] For each frame , select a fixed-size window, the window consists of and The previous and next frames composition, ,in Represents the size of the tracking window. Using SAM2 based on visual cues exist Tracking The segmentation results are Generate a binary mask on .

[0034]

[0035] Through this step, we project the segmentation results on two different images onto the same image. and The similarity of the segmentation results can reflect the stability of the polyp segmentation model in the previous and next frames.

[0036] The Dice coefficient is used to quantify the similarity between the two segmentation results. The higher the Dice coefficient, the more similar the segmentation results of the polyp segmentation model on the two frames are.

[0037]

[0038] exist All previous and next frames in the entire window Use the above formula to obtain the mapping of each frame and After calculating the Dice coefficients between the segmentation results, the average of all Dice coefficients is calculated.

[0039]

[0040] This average value represents The degree of stability in the window reflects the segmentation quality of the polyp segmentation model to a large extent. Therefore, this score can be used as an evaluation indicator to measure the segmentation quality without real annotation. Based on the quality evaluation score, we can evaluate the performance of the polyp segmentation model on each frame of the colonoscopy video, which improves the credibility of artificial intelligence in clinical practice.

[0041] Secondly, the high-quality segmentation results are used as hints to re-segment the low-quality segmentation results, thereby improving the polyp segmentation quality without the need for training or fine-tuning.

[0042] After generating a quality assessment score for each frame in a colonoscopy video through the above function, we filter based on the score. The specific screening method is: calculate the average score of all frames in a colonoscopy video:

[0043]

[0044] Frames with scores less than the average score are considered low-quality frames. All low-quality frames Construct a low-quality frame set L:

[0045]

[0046] Frames with scores greater than or equal to the average score have higher segmentation quality and can be used as a reference for re-segmenting low-quality frames. In order to ensure the reliability of high-quality frames in re-segmentation, we additionally introduce SAM2 stability evaluation. The segmentation mask The bounding box and center point of , that is, the image itself is segmented:

[0047]

[0048] The self-prompt segmentation result Compared with the original segmentation result The Dice coefficient between An indicator of stability.

[0049]

[0050] Therefore, a high-quality frame has a quality assessment score greater than or equal to the average score, and a SAM2 stability score greater than the threshold. Frame.

[0051]

[0052] For each low-quality frame , we choose the distance in the video frame sequence Recent high-quality frames, forming pairs Resegmentation hint sequence .

[0053]

[0054] We map the mask of each high-quality frame to Mapping is done.

[0055]

[0056] The average mask of all mapping results is obtained as The new segmentation result.

[0057]

[0058] Based on the same inventive concept, the present invention also provides a dual-model fusion polyp segmentation quality control system, characterized in that, for implementing the above method, the system comprises:

[0059] The quality assessment module evaluates the polyp segmentation quality of the polyp segmentation model on the colonoscopy video without real annotation by propagating and comparing the polyp segmentation results of each frame in the video. Specifically, a colonoscopy video is input; the colonoscopy video is split into consecutive frames, and a binary mask is generated for each frame by the polyp segmentation model; the bounding box and center point of each mask are extracted through the hint extractor to generate a mixed reinforcement hint; each frame image and its visual hint are input into the SAM2 model, and each frame image and its visual hint are propagated forward and backward. Frames, get the mapping of the segmentation result of the current image on the previous and next frames; calculate the Dice coefficient between the segmentation result of each frame and the mapping of the segmentation results of the previous and next frames on the current frame, take the average of all Dice coefficients as the segmentation quality evaluation score of the current frame; perform the above operation on all frames to get the segmentation quality evaluation score of each frame; the higher the score, the better the quality of the segmentation result.

[0060] The re-segmentation module uses the high-quality segmentation results to re-segment the low-quality segmentation results, thereby improving the polyp segmentation quality without training or fine-tuning. Specifically, the average segmentation quality assessment scores of all frames on the colonoscopy video are calculated, and frames with scores less than the average are considered low-quality frames; visual cues of non-low-quality frames are taken and self-segmented using SAM2; if the Dice coefficient between the self-prompted segmentation result and the original segmentation result is greater than the threshold, and the frame with a quality assessment score greater than or equal to the average score is considered a high-quality frame; for each low-quality frame, the c high-quality frames closest to it in the video frame sequence are selected to form a prompt sequence for re-segmenting the low-quality frame; the segmentation results of each frame in the prompt sequence are mapped to the low-quality frame using the SAM2 model, and a set of binary masks are generated on the low-quality frame; the average mask of all mappings is taken as the segmentation result of the current low-quality frame; the above process is performed on all low-quality frames to achieve re-segmentation of all low-quality frames on the entire colonoscopy video.

[0061] The specific implementation method of the above module is the same as the specific method of the aforementioned dual-model fusion polyp segmentation quality control method, and the present invention will not repeat it again.

[0062] The present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0063] Example

[0064] like Figure 1 , Figure 2 As shown in the figure, the specific implementation process of SQC-SAM is as follows:

[0065] 1) Input a colonoscopy video. SQC-SAM splits the colonoscopy video into consecutive frames and generates a binary mask for each frame through a dedicated model for polyp segmentation.

[0066] 2) After the hint extractor, the bounding box and center point of each mask are extracted to generate a hybrid enhanced hint.

[0067] 3) Input each frame of the image and its visual cues into Segment Anything Model 2 (SAM2), propagate forward and backward for 10 frames each, and obtain the mapping of the segmentation result of the current image on the previous and next frames.

[0068] 4) Calculate the Dice similarity coefficient between the segmentation result of each frame and the mapping of the segmentation results of the previous and next frames in the current frame, and take the average value of all Dice coefficients as the segmentation quality evaluation score of the current frame. Perform the above operation on all frames to obtain the quality evaluation score of each frame. The higher the score, the better the quality of the segmentation result, and vice versa.

[0069] 5) Calculate the average quality assessment score of all frames in the colonoscopy video, and frames with scores less than the average score are considered low-quality frames.

[0070] 6) Take the visual cues of non-low-quality frames and use SAM2 to segment them. If the Dice similarity coefficient between the self-cued segmentation result and the original segmentation result is greater than the threshold 0.75, and the frame with a quality assessment score greater than the average score is considered a high-quality frame. For each low-quality frame, SQC-SAM selects the 6 high-quality frames closest to it in the video frame sequence to form a cue sequence for re-segmenting the low-quality frame.

[0071] 7) Use SAM2 to map the segmentation results of each frame in the prompt sequence to the low-quality frame and generate a set of binary masks on the low-quality frame.

[0072] 8) Take the average mask of all maps as the segmentation result of the current low-quality frame. By performing the above process on all low-quality frames, SQC-SAM achieves the re-segmentation of all frames with poor segmentation effects on the entire colonoscopy video, improving the overall segmentation quality of the colonoscopy video.

[0073] Figure 3 is a set of simulation schematic diagrams of this embodiment. It can be seen that the method of the present invention can improve the overall segmentation quality of colonoscopy video.

[0074] The above embodiments only show several specific implementation methods of the present application. Although described in more detail, they should not be regarded as limiting the scope of the present invention. It should be emphasized that, for those of ordinary skill in the art, various modifications and improvements can be made without departing from the basic concept of the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of this patent should be based on the attached claims.

Claims

1. A dual-model fusion polyp segmentation quality control method, characterized in that: include: (1) Input a colonoscopy video; The colonoscopy video is split into consecutive frames, and a binary mask is generated for each frame using a polyp segmentation model; (2) After the hint extractor, the bounding box and center point of each mask are extracted to generate a hybrid enhanced hint; (3) Each frame of image and its visual cue are input into the SAM2 model, and the forward and backward propagation of each Frame, get the mapping of the segmentation result of the current image on the previous and next frames; (4) Calculate the Dice coefficient between the segmentation result of each frame and the mapping of the segmentation results of the previous and next frames in the current frame, and take the average of all Dice coefficients as the segmentation quality evaluation score of each frame. The higher the score, the better the quality of the segmentation result. (5) Calculate the average of the segmentation quality assessment scores of all frames in the colonoscopy video, and frames with scores less than the average are considered low-quality frames; (6) Take the visual cues of non-low-quality frames and use SAM2 to segment them. If the Dice coefficient between the self-cued segmentation result and the original segmentation result is greater than the threshold, and the quality assessment score is greater than or equal to the average score, the frame is considered a high-quality frame. For each low-quality frame, select the c high-quality frames closest to it in the video frame sequence to form a cue sequence for re-segmenting the low-quality frame. (7) Map the segmentation results of each frame in the prompt sequence to the low-quality frame using the SAM2 model, and generate a set of binary masks on the low-quality frame; (8) Take the average mask of all maps as the segmentation result of each low-quality frame, and achieve re-segmentation of all low-quality frames in the entire colonoscopy video.

2. The dual-model fusion polyp segmentation quality control method according to claim 1, characterized in that: Split a colonoscopy video into Consecutive frames , polyp segmentation model Generate a binary mask for each frame ; 。 3. The dual-model fusion polyp segmentation quality control method according to claim 2, characterized in that: Extract each frame Mask Bounding box and center point of the generated hybrid enhanced visual cues .

4. The dual-model fusion polyp segmentation quality control method according to claim 2, characterized in that: For each frame , select a fixed-size window, the window consists of and The previous and next frames Composition; based on visual cues using the SAM2 model exist Tracking The segmentation results are Generate a binary mask on : ； in Represents the radius of the tracking window. Through this step, the segmentation results on two different images are mapped to the same image. and The similarity of the segmentation results can reflect the stability of the polyp segmentation model in the previous and next frames.

5. The dual-model fusion polyp segmentation quality control method according to claim 4, characterized in that: The Dice coefficient is used to quantify the similarity between the two segmentation results. The higher the Dice coefficient, the more similar the segmentation results of the polyp segmentation model on the two frames are. ； exist All previous and next frames in the entire window Mapping; Use the above formula to obtain the mapping and After finding the Dice coefficients between the segmentation results, the average value of all Dice coefficients is calculated; ； The average value represents The degree of stability in the window is used as the segmentation quality evaluation score of the current frame.

6. The dual-model fusion polyp segmentation quality control method according to claim 1, characterized in that: The method further includes: using the high-quality segmentation result as a hint to re-segment the low-quality segmentation result; After generating a quality assessment score for each frame in the colonoscopy video, we filter based on the score. The specific filtering method is to calculate the average score of all frames in a colonoscopy video: ； Frames with scores less than the average score are considered low-quality frames; all low-quality frames Construct a low-quality frame set L: ； Frames with scores greater than or equal to the average score are used as references for re-segmenting low-quality frames; SAM2 stability evaluation is introduced; non-low-quality frames are extracted The segmentation mask The bounding box and center point of , that is, the image itself is segmented: ； The self-prompt segmentation result Compared with the original segmentation result The Dice coefficient is used as a measure of SAM2 The stability index : ； Therefore, a high-quality frame has a quality assessment score greater than or equal to the average score, and a SAM2 stability score greater than the threshold. Frames, ； For each low-quality frame , select the distance in the video frame sequence Recent high-quality frames, forming pairs Resegmentation hint sequence ; ； For each high-quality frame, the mask To map: ； The average mask of all mapping results is obtained as The new segmentation result: 。 7. A dual-model fusion polyp segmentation quality control system, characterized in that: For implementing the method described in any one of claims 1 to 6, the system comprises: The quality assessment module evaluates the polyp segmentation quality of the polyp segmentation model on colonoscopy videos without real annotations by propagating and comparing the polyp segmentation results of each frame in the video; The re-segmentation module uses the high-quality segmentation results to re-segment the low-quality segmentation results.

8. The dual-model fusion polyp segmentation quality control system according to claim 7, characterized in that: The quality assessment module evaluates the polyp segmentation quality of the polyp segmentation model on the colonoscopy video without real annotations by propagating and comparing the polyp segmentation results of each frame in the video, specifically: Input a colonoscopy video; split the colonoscopy video into consecutive frames, and generate a binary mask for each frame through the polyp segmentation model; After the hint extractor, the bounding box and center point of each mask are extracted to generate a hybrid enhanced hint; Each frame of image and its visual cues are fed into the SAM2 model, and the forward and backward propagation of each Frame, get the mapping of the segmentation result of the current image on the previous and next frames; The Dice coefficient between the segmentation result of each frame and the mapping of the segmentation results of the previous and next frames in the current frame is calculated respectively, and the average value of all Dice coefficients is taken as the segmentation quality evaluation score of each frame; the higher the score, the better the quality of the segmentation result.

9. The dual-model fusion polyp segmentation quality control system according to claim 8, characterized in that: The re-segmentation module re-segments the low-quality segmentation results using the high-quality segmentation results, specifically: The average of the segmentation quality assessment scores of all frames on the colonoscopy video is calculated, and the frames with scores less than the average are considered to be low-quality frames; Take the visual cues of non-low-quality frames and use SAM2 to segment them. If the Dice coefficient between the self-cued segmentation result and the original segmentation result is greater than the threshold, and the frame with a quality assessment score greater than or equal to the average score is considered a high-quality frame. For each low-quality frame, select the c high-quality frames closest to it in the video frame sequence to form a cue sequence for re-segmenting the low-quality frame. The segmentation results of each frame in the prompt sequence are mapped to the low-quality frames using the SAM2 model, and a set of binary masks are generated on the low-quality frames; The average mask of all maps is taken as the segmentation result of each low-quality frame to achieve re-segmentation of all low-quality frames on the entire colonoscopy video.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images