Dual-Model Fusion Polyp Segmentation Quality Control Method and System

Abstract

The present invention discloses a dual-model fusion polyp segmentation quality control method and system. The present invention fuses a general video segmentation model and a special polyp segmentation model to construct a polyp video segmentation quality control framework SQC-SAM. The first function of the present invention is to evaluate the polyp segmentation quality of the polyp segmentation model on colonoscopy videos without real annotations. The second function is to use the high-quality segmentation results as prompts to re-segment the low-quality segmentation results, so as to improve the polyp segmentation quality without training or fine-tuning. The present invention proposes a novel dual-model fusion framework, which effectively combines the segmentation results of the polyp segmentation model with the video segmentation ability and promptability of the basic video segmentation model, realizing the effective utilization of temporal and spatial cues in colonoscopy videos.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and particularly to a method and system for quality control of polyp segmentation by fusing two models. Background Art

[0002] Colorectal cancer, as the third most common malignant tumor and the second leading cause of cancer death, poses a great threat to human health. Colorectal adenomatous polyps, as the key precursor of this disease, identifying, locating, and removing the diseased polyps can significantly reduce the incidence and mortality of colorectal cancer and play an important role in the diagnosis and treatment of colorectal cancer. Thanks to the development of artificial intelligence, automated polyp segmentation technology is expected to become an effective tool for assisting in the diagnosis and treatment of colorectal cancer.

[0003] However, there are still some challenges for automated polyp segmentation to be reliable for clinical applications: 1) Most current polyp segmentation methods are based on colonoscopy images. Although certain progress has been made, these methods ignore the temporal and spatial cues between consecutive frames in colonoscopy videos; 2) The color, size, and shape of polyps vary greatly, and current polyp segmentation methods often perform poorly in the face of diverse polyps in complex scenarios; 3) Current polyp segmentation methods usually directly output prediction results without evaluating the quality of the prediction, resulting in a lack of confidence in artificial intelligence technology by doctors and patients in clinical applications. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for quality control of polyp segmentation by fusing two models.

[0005] The technical solution for achieving the purpose of the present invention is: A method for quality control of polyp segmentation by fusing two models, comprising:

[0006] (1) Input a colonoscopy video; split the colonoscopy video into consecutive frames, and generate a binary mask for each frame through a polyp segmentation model;

[0007] (2) Through a prompt extractor, extract the bounding box and center point of each mask to generate a hybrid enhanced prompt;

[0008] (3) Input each frame image and its visual prompt into the SAM2 model, and propagate forward and backward for each frame to obtain the mapping of the segmentation result of the current image on the front and back frames;

[0009] (4) Calculate the Dice coefficient between the segmentation result of each frame and the mapping of the segmentation results of the front and back frames on the current frame respectively, and take the average value of all Dice coefficients as the segmentation quality evaluation score for each frame. The higher the score, the better the quality of the segmentation result of the frame;

[0010] (5) Calculate the average value of the segmentation quality assessment scores for all frames in the colonoscopy video. Frames with scores less than this average value are regarded as low-quality frames;

[0011] (6) Take the visual cues of non-low-quality frames and use SAM2 to segment itself; if the Dice coefficient between the segmentation result of the self-cue and the original segmentation result is greater than the threshold, and the quality assessment score is greater than or equal to the average score, the frame is regarded as a high-quality frame; for each low-quality frame, select the c high-quality frames closest to it in the video frame sequence to form a cue sequence for re-segmenting the low-quality frame;

[0012] (7) Map the segmentation results of each frame in the cue sequence to the low-quality frame using the SAM2 model to generate a set of binary masks on the low-quality frame;

[0013] (8) Take the average mask of all the mappings as the segmentation result of each low-quality frame to achieve re-segmentation of all low-quality frames in the entire colonoscopy video.

[0014] Based on the same inventive concept, the present invention also provides a dual-model fusion polyp segmentation quality control system for implementing the above method. The system includes:

[0015] A quality assessment module that evaluates the polyp segmentation quality of the polyp segmentation model on a colonoscopy video without true annotations by propagating and comparing the polyp segmentation results of each frame in the video;

[0016] A re-segmentation module that re-segments the low-quality segmentation results using the high-quality segmentation results.

[0017] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the above method are implemented.

[0018] Compared with the prior art, the beneficial effects of the present invention are:

[0019] (1) The present invention proposes a novel dual-model fusion framework that effectively combines the segmentation results of the polyp segmentation model with the video segmentation ability and promptability of the basic video segmentation model, realizing the effective utilization of temporal and spatial cues in the colonoscopy video.

[0020] (2) The present invention proposes an accurate and efficient polyp segmentation quality evaluation system that realizes accurate evaluation of the polyp segmentation quality of the colonoscopy video without true annotations by propagating and comparing the polyp segmentation results of each frame in the video.

[0021] (3)The present invention proposes an effective method for improving the quality of polyp segmentation. By using high-quality segmentation results to re-segment low-quality segmentation results, it realizes the improvement of polyp segmentation quality without training or fine-tuning.

[0022] (4)The present invention can replace different models according to different usage scenarios and tasks, can handle various downstream tasks, and has extremely high flexibility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural diagram of the dual-model fusion polyp segmentation quality control framework SQC-SAM of the present invention. Among them, (a) is the structural diagram of the segmentation quality evaluation module, and (b) is the re-segmentation structural diagram.

[0024] Figure 2 It is a flowchart of SQC-SAM for segmentation quality evaluation and re-segmentation.

[0025] Figure 3 It is a schematic diagram of the results of the present invention for polyp segmentation testing on a group of colonoscopy videos and comparison with two existing polyp segmentation models, PolypPVT and HSNet. DETAILED DESCRIPTION OF THE INVENTION

[0026] The present invention integrates the general video segmentation model Segment Anything Model 2 (SAM2) and a polyp segmentation model to construct a polyp video segmentation quality control framework SQC-SAM. This framework realizes two functions. The first function is to evaluate the polyp segmentation quality of the polyp segmentation model on colonoscopy videos without ground truth annotations. The second function is to use high-quality segmentation results as prompts to re-segment low-quality segmentation results, so as to improve the polyp segmentation quality without training or fine-tuning.

[0027] The following will separately describe the dual-model fusion polyp segmentation quality control method and the corresponding system of the present invention.

[0028] The present invention proposes a dual-model fusion polyp segmentation quality control method, including the following steps:

[0029] First, evaluate the polyp segmentation quality of the polyp segmentation model on colonoscopy videos without ground truth annotations;

[0030] Divide a colonoscopy video into successive frames , and the polyp segmentation model generates a binary mask for each frame .

[0031]

[0032] Extract each frame of the mask and the bounding box and center point to generate a hybrid enhanced visual cue .

[0033] For each frame , select a window of a fixed size, which consists of and the front and back frames . Among them , represents the size of the tracking window. Use SAM2 to track on the segmentation result of and generate a binary mask on .

[0034]

[0035] Through this step, we map the segmentation results on two different images to the same image. and The similarity degree of the segmentation results can reflect the stability of the polyp segmentation model in the front and back frames.

[0036] Use the Dice coefficient to quantify the similarity degree of the two segmentation results. The higher the Dice coefficient, the more similar the segmentation results of the polyp segmentation model on these two frames are.

[0037]

[0038] In all the front and back frames within the full window range are mapped to . After obtaining the Dice coefficient between the mapping of each frame and the segmentation result of

[0039]

[0040] This average value represents the stability degree in the window, and whether it is stable largely reflects the segmentation quality of the polyp segmentation model. Therefore, this score can be used as an evaluation index to measure the segmentation quality in the case of no ground truth annotation. According to the quality evaluation score, we can evaluate the performance of the polyp segmentation model on each frame of the colonoscopy video, improving the credibility of artificial intelligence in clinical practice.

[0041] Secondly, using the high-quality segmentation results as a hint, the low-quality segmentation results are re-segmented to improve the polyp segmentation quality without training or fine-tuning.

[0042] After generating the quality assessment scores for each frame in the colonoscopy video through the above-mentioned function pair, we perform screening based on the scores. The specific screening method is as follows: Calculate the average score of all frames in a colonoscopy video:

[0043]

[0044] Frames with scores less than the average score are regarded as low-quality frames, and all low-quality frames constitute the low-quality frame set L:

[0045]

[0046] While frames with scores greater than or equal to the average score have higher segmentation quality and can be used as a reference for re-segmenting low-quality frames. To ensure the reliability of high-quality frames in re-segmentation, we additionally introduce the SAM2 stability assessment. Extract the bounding boxes and center points of the segmentation masks of non-low-quality frames as visual hints, and use SAM2 to , that is, segment the image itself:

[0047]

[0048] Take the Dice coefficient between the self-hint segmentation result and the original segmentation result as an index to measure the stability of SAM2 on .

[0049]

[0050] Therefore, high-quality frames are those with quality assessment scores greater than or equal to the average score and SAM2 stability scores greater than the threshold .

[0051]

[0052] For each low-quality frame , we select the nearest high-quality frames in the frame sequence of the video to form the hint sequence for re-segmentation.

[0053]

[0054] We perform the mask of each high-quality frame towards Perform mapping.

[0055]

[0056] Calculate the average mask for all mapping results as the new segmentation result.

[0057]

[0058] Based on the same inventive concept, the present invention also provides a dual-model fusion polyp segmentation quality control system, which is characterized in that it is used to implement the above method. The system includes:

[0059] A quality evaluation module, which evaluates the polyp segmentation quality of the polyp segmentation model on a colonoscopy video without real annotations by propagating and comparing the polyp segmentation results of each frame in the video. Specifically: input a colonoscopy video; split the colonoscopy video into consecutive frames, and generate a binary mask for each frame through the polyp segmentation model; after passing through the prompt extractor, extract the bounding box and center point of each mask to generate a hybrid enhanced prompt; input each frame image and its visual prompt into the SAM2 model, and propagate forward and backward for each frame to obtain the mapping of the segmentation result of the current image on the front and back frames; calculate the Dice coefficient between the segmentation result of each frame and the mapping of the segmentation results of the front and back frames on the current frame respectively, and take the average value of all Dice coefficients as the segmentation quality evaluation score of the current frame; perform the above operations on all frames to obtain the segmentation quality evaluation score of each frame; the higher the score, the better the quality of the segmentation result of the frame.

[0060] A re-segmentation module, which re-segments the low-quality segmentation results using the high-quality segmentation results, and realizes the improvement of polyp segmentation quality without training or fine-tuning. Specifically: calculate the average value of the segmentation quality evaluation scores of all frames on the colonoscopy video, and the frames with scores less than this average value are regarded as low-quality frames; take the visual prompts of non-low-quality frames and use SAM2 to segment itself; if the Dice coefficient between the segmentation result of the self-prompt and the original segmentation result is greater than the threshold, and the frames with quality evaluation scores greater than or equal to the average score are regarded as high-quality frames; for each low-quality frame, select the c nearest high-quality frames in the video frame sequence to form a prompt sequence for re-segmenting the low-quality frame; map the segmentation results of each frame in the prompt sequence to the low-quality frame using the SAM2 model to generate a set of binary masks on the low-quality frame; take the average mask of all mappings as the segmentation result of the current low-quality frame; perform the above process on all low-quality frames to realize the re-segmentation of all low-quality frames on the entire colonoscopy video.

[0061] The specific implementation method of the above module is the same as that of the aforementioned dual-model fusion polyp segmentation quality control method, and the present invention will not elaborate further.

[0062] The following will combine the accompanying drawings and embodiments to elaborate in detail on the content of the present invention.

[0063] Embodiment

[0064] As Figure 1 、 Figure 2 shown, the specific implementation process of SQC-SAM is as follows:

[0065] 1) Input a colonoscopy video. SQC-SAM splits the colonoscopy video into consecutive frames and generates a binary mask for each frame through a dedicated polyp segmentation model.

[0066] 2) Through the prompt extractor, extract the bounding box and center point of each mask to generate a mixed enhanced prompt.

[0067] 3) Input each frame image and its visual prompt into Segment Anything Model 2 (SAM2), and perform forward and backward propagation for 10 frames each to obtain the mapping of the segmentation result of the current image on the front and back frames.

[0068] 4) Calculate the Dice similarity coefficient between the segmentation result of each frame and the mapping of the segmentation results of the front and back frames on the current frame respectively, and take the average value of all Dice coefficients as the segmentation quality evaluation score of the current frame. Perform the above operations on all frames to obtain the quality evaluation score of each frame. The frame with a higher score has a better segmentation result quality, and vice versa.

[0069] 5) Calculate the average quality evaluation score of all frames in the colonoscopy video, and frames with a score less than the average score are regarded as low-quality frames.

[0070] 6) Take the visual prompts of non-low-quality frames and use SAM2 to segment itself. If the Dice similarity coefficient between the segmentation result of the self-prompt and the original segmentation result is greater than the threshold of 0.75, and at the same time the frame with a quality evaluation score greater than the average score is regarded as a high-quality frame. For each low-quality frame, SQC-SAM selects the 6 high-quality frames closest to it in the video frame sequence to form a prompt sequence for re-segmenting the low-quality frame.

[0071] 7) Map the segmentation result of each frame in the prompt sequence to the low-quality frame using SAM2 to generate a set of binary masks on the low-quality frame.

[0072] 8) Take the average mask of all mappings as the segmentation result of the current low-quality frame. Perform the above process on all low-quality frames. SQC-SAM realizes the re-segmentation of all frames with poor segmentation effects in the entire colonoscopy video, improving the overall segmentation quality of the colonoscopy video.

[0073] Figure 3 This is a set of simulation schematic diagrams of this embodiment. It can be seen that the method of the present invention can improve the overall segmentation quality of the colonoscopy video.

[0074] The above embodiments only show several specific implementation manners of the present application. Although described in detail, they should not be regarded as a limitation to the scope of the invention patent of the present application. It should be emphasized that for those of ordinary skill in the art, without departing from the basic concept of the present application, various modifications and improvements can still be made, and these should be included within the protection scope of the present application. Therefore, the protection scope of this patent shall be subject to the appended claims.

Claims

1. A dual-model fusion polyp segmentation quality control method, characterized in that, Including: (1) Input a colonoscopy video; Split the colonoscopy video into consecutive frames, and generate a binary mask for each frame through a polyp segmentation model; (2) Through a prompt extractor, extract the bounding box and center point of each mask to generate a mixed enhanced prompt; (3) Input each frame of the image and its visual cues into the SAM2 model, and propagate forward and backward for frames to obtain the mapping of the segmentation result of the current image on the previous and next frames; (4) Calculate the Dice coefficient between the segmentation result of each frame and the mapping of the segmentation results of the previous and next frames in the current frame respectively, and take the average value of all Dice coefficients as the segmentation quality evaluation score of each frame. The higher the score, the better the segmentation result quality of the frame; (5) Calculate the average value of the segmentation quality evaluation scores of all frames on the colonoscopy video. Frames with scores less than this average value are regarded as low-quality frames; (6) Take the visual prompts of non-low-quality frames and use SAM2 to segment itself; if the Dice coefficient between the segmentation result of the self-prompt and the original segmentation result is greater than the threshold, and the quality evaluation score is greater than or equal to the average score, the frame is regarded as a high-quality frame; for each low-quality frame, select the c high-quality frames closest to it in the video frame sequence to form a prompt sequence for re-segmenting the low-quality frame; (7) Map the segmentation results of each frame in the prompt sequence to the low-quality frame using the SAM2 model to generate a set of binary masks on the low-quality frame; (8) Take the average mask of all mappings as the segmentation result of each low-quality frame to achieve re-segmentation of all low-quality frames on the entire colonoscopy video.

2. The polyp segmentation quality control method for dual-model fusion according to claim 1, wherein, Divide a colonoscopy video into successive frames , and the polyp segmentation model generates a binary mask for each frame ; 。 3. The polyp segmentation quality control method for dual-model fusion according to claim 2, wherein Extract each frame of the mask to generate a mixed enhanced visual cue .

4. The double-model fusion polyp segmentation quality control method according to claim 2, wherein For each frame , a window of a fixed size is selected, and the window is composed of and the previous and next frames of ; Using the SAM2 model, based on visual cues to track the segmentation result of and generate a binary mask on : ​ ； where represents the radius of the tracking window; through this step, the segmentation results on two different images are mapped onto the same image; and The similarity degree of the segmentation results can reflect the stability of the polyp segmentation model in the segmentation of the front and back frames.

5. The polyp segmentation quality control method for dual-model fusion according to claim 4, wherein Use the Dice coefficient to quantify the similarity degree of the two segmentation results. The higher the Dice coefficient, the more similar the segmentation results of the polyp segmentation model on these two frames; ； Within All the front and back frames within the full window are mapped to After obtaining the Dice coefficient between the mapping of each frame and the segmentation result using the above formula, the average value of all Dice coefficients is calculated; ； The average value represents the degree of stability within the window and serves as the segmentation quality assessment score for the current frame.

6. The quality control method for polyp segmentation by dual model fusion according to claim 1, wherein This method further includes: using the high-quality segmentation result as a prompt to re-segment the low-quality segmentation result; After generating the quality evaluation score for each frame in the colonoscopy video, perform screening according to the scores. The specific screening method is: calculate the average score of all frames on a colonoscopy video: ； Frames with a score lower than the average score are considered low-quality frames; all low-quality frames constitute the set L of low-quality frames: ； Frames greater than or equal to the average score are used as references for re-segmenting low-quality frames; SAM2 stability assessment is introduced; non-low-quality frames are extracted of the segmentation mask of the bounding box and center point as visual cues, and use SAM2 to , that is, segment the image itself: ； The self-prompt segmentation results and the original segmentation results have their Dice coefficients used as an indicator to measure the stability of SAM2 on : ； Therefore, a high-quality frame is a frame whose quality assessment score is greater than or equal to the average score and whose SAM2 stability score is greater than the threshold frame ； For each low-quality frame , select the nearest high-quality frames in the frame sequence of the video to form a prompt sequence for double segmentation ; ； Map the mask for each high-quality frame to perform mapping: ； Calculate the average mask for all mapping results as the new segmentation result: 。 7. A dual-model fusion polyp segmentation quality control system, characterized in that For implementing the method according to any one of claims 1-6, the system includes: A quality evaluation module that evaluates the polyp segmentation quality of the polyp segmentation model on a colonoscopy video without real annotations by propagating and comparing the polyp segmentation results of each frame in the video; A re-segmentation module that uses the high-quality segmentation result to re-segment the low-quality segmentation result.

8. The dual-model fusion polyp segmentation quality control system according to claim 7, characterized in that The quality evaluation module described above evaluates the polyp segmentation quality of the polyp segmentation model on a colonoscopy video without real annotations by propagating and comparing the polyp segmentation results of each frame in the video, specifically: Input a colonoscopy video; split the colonoscopy video into consecutive frames, and generate a binary mask for each frame through a polyp segmentation model; Through a prompt extractor, extract the bounding box and center point of each mask to generate a mixed enhanced prompt; Input each frame of the image and its visual cues into the SAM2 model, and propagate forward and backward for each frame to obtain the mapping of the segmentation result of the current image on the previous and next frames; Calculate the Dice coefficient between the segmentation result of each frame and the mapping of the segmentation results of the previous and next frames in the current frame respectively, and take the average value of all Dice coefficients as the segmentation quality evaluation score; the higher the score, the better the segmentation result quality of the frame.

9. The dual-model fusion polyp segmentation quality control system according to claim 8, wherein The re-segmentation module described above uses the high-quality segmentation result to re-segment the low-quality segmentation result, specifically: Calculate the average of the segmentation quality assessment scores for all frames in the colonoscopy video, and frames with scores less than the average are considered low-quality frames; Obtain the visual cues of non-low-quality frames and use SAM2 to segment itself; if the Dice coefficient between the segmentation result of the self-cue and the original segmentation result is greater than the threshold, and the frames with a quality assessment score greater than or equal to the average score are considered high-quality frames; for each low-quality frame, select the c high-quality frames closest to it in the video frame sequence to form a cue sequence for re-segmenting the low-quality frame; Map the segmentation results of each frame in the cue sequence to the low-quality frame using the SAM2 model to generate a set of binary masks on the low-quality frame; Take the average mask of all the mappings as the segmentation result of each low-quality frame, and achieve the re-segmentation of all low-quality frames in the entire colonoscopy video.

10. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the steps of the method described in any one of claims 1-6.

Images