Video polyp segmentation system and method based on self-supervised task diffusion model

Abstract

The invention discloses a video polyp segmentation system and method based on a self-supervised task diffusion model, and the system comprises a processing module which is used for obtaining a polyp video, carrying out the data set labeling of the polyp video, and obtaining N polyp image frames and a segmentation mask corresponding to each polyp image frame; the conditional diffusion model is used for carrying out coding and noise processing on the segmentation mask corresponding to the last polyp image frame to obtain a noise segmentation mask; the multi-task diffusion model is used for generating multi-scale spatial features and multi-scale temporal features, and integrating and inputting the multi-scale spatial features and the multi-scale temporal features to the feature pyramid network to obtain multi-level spatial-temporal features; the MLP layer is used for splicing and fusing the multi-level spatial-temporal features and the noise segmentation masks to obtain fused features; and the de-noising decoder is used for carrying out reverse de-noising processing on the fusion features, outputting a prediction segmentation mask and determining a prediction segmentation result according to the prediction segmentation mask. According to the application, the accuracy of polyp diagnosis and polyp segmentation is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image segmentation processing, and in particular to a video polyp segmentation system and method based on a self-supervised task diffusion model. Background Art

[0002] Colorectal cancer has become the second leading cause of cancer-related deaths worldwide, accounting for approximately 10% of all cancer cases. Colonoscopy plays a vital role in detecting polyps and reducing the incidence and mortality of colorectal cancer. Currently, the most effective way to prevent colorectal cancer is to perform regular endoscopic examinations and timely polypectomy. By taking video images through the colon endoscope, doctors can evaluate the location and appearance of polyp tissue and remove it before the polyp becomes cancerous. However, due to the variable shapes and colors of polyps, the complexity of blurred boundaries, and external shooting conditions such as low boundary contrast, mirror reflections, and artifacts, polyps are highly concealed and have a high rate of missed diagnosis during diagnosis.

[0003] In the related art, most of the polyp segmentation work only trains and evaluates the model on static images, ignoring the additional information about polyp positioning brought by the temporal nature of the video, and not making full use of the temporal information between endoscopic video frames. In the video polyp segmentation task, it is obviously not enough to focus only on independent static images. In the video polyp image tracking and segmentation task, it is first necessary to pre-train on a large number of static images, and then fine-tune on the video images. This training strategy requires a large number of high-quality annotations, but the current scale of video polyp data is still small. At the same time, because the polyp boundaries are fuzzy and similar to the background tissue, even skilled clinicians may not be able to reach a consensus on the annotations of consecutive frames. Finally, the current open source polyp dataset is a sparse sequence, and some of its adjacent frames vary greatly. Although the endoscopic video focuses on the same polyp tissue, due to different camera angles or lighting, the contextual environment of the polyp, such as cavity, highlight, mucosal tissue, etc., will change, which may affect the prediction results of adjacent frames, thereby affecting the prediction results of polyp segmentation and reducing the accuracy of polyp segmentation diagnosis. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the embodiment of the present application provides a video polyp segmentation system and method based on a self-supervised task diffusion model, which is conducive to accurately identifying the location of polyps, improving the accuracy of polyp diagnosis, and improving the accuracy of polyp segmentation.

[0005] In a first aspect, an embodiment of the present application provides a video polyp segmentation system based on a self-supervised multi-task diffusion model, comprising:

[0006] A processing module, used for acquiring a polyp video, and performing data set annotation processing on the polyp video to obtain N polyp image frames and a segmentation mask corresponding to each polyp image frame;

[0007] The conditional diffusion model is used to encode the segmentation mask corresponding to the last polyp image frame to obtain the segmentation mask of the hidden state sequence, and perform forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask;

[0008] A multi-task diffusion model, comprising an image encoder and a temporal encoder, wherein the multi-task diffusion model is used to use the last polyp image frame as an input of the image encoder to generate multi-scale spatial features, use the N polyp image frames as an input of the temporal encoder to generate multi-scale temporal features, and integrate the multi-scale spatial features and the multi-scale temporal features and input them into a feature pyramid network to obtain multi-level spatiotemporal features;

[0009] The MLP layer is used to concatenate and fuse the multi-level spatiotemporal features and the noise segmentation mask to obtain a fused feature;

[0010] A denoising decoder is used to perform inverse denoising on the fused features, output a predicted segmentation mask of the same dimension as the last polyp image frame, determine a predicted segmentation result based on the predicted segmentation mask, and determine the polyp position based on the predicted segmentation result.

[0011] According to some embodiments of the present application, the video polyp segmentation system based on the self-supervised multi-task diffusion model also includes: a temporal reasoning module, which is used to average the N frames of polyp image frames along the time axis and summarize them into the multi-scale temporal features, and input the multi-scale temporal features into the reconstruction decoder to predict the next frame of polyp image frame to obtain the target polyp image frame.

[0012] According to some embodiments of the present application, a video polyp segmentation system based on a self-supervised multi-task diffusion model also includes: an adversarial self-supervision module, including a generator and a discriminator, the generator is the temporal reasoning module, the discriminator is used to distinguish between the target polyp image frame and the real frame polyp image frame, and the adversarial loss is determined based on the target polyp image frame and the real frame polyp image frame; the discriminator is optimized by minimizing the adversarial loss.

[0013] In a second aspect, an embodiment of the present application provides a video polyp segmentation method based on a self-supervised task diffusion model, which is applied to the video polyp segmentation system described in the technical solution of the first aspect. The method includes:

[0014] Acquire a polyp video, and perform data set annotation processing on the polyp video to obtain N polyp image frames and a segmentation mask corresponding to each polyp image frame;

[0015] Performing encoding processing on the segmentation mask corresponding to the last polyp image frame to obtain a segmentation mask of a hidden state sequence, and performing forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask;

[0016] Using the last polyp image frame as an input of an image encoder to generate multi-scale spatial features, using the N polyp image frames as an input of a temporal encoder to generate multi-scale temporal features, integrating the multi-scale spatial features and the multi-scale temporal features and inputting them into a feature pyramid network to obtain multi-level spatiotemporal features;

[0017] Concatenating and fusing the multi-level spatiotemporal features and the noise segmentation mask to obtain a fused feature;

[0018] The fusion feature is subjected to inverse denoising processing, and a predicted segmentation mask having the same dimension as the last polyp image frame is outputted; a predicted segmentation result is determined according to the predicted segmentation mask; and a polyp position is determined according to the predicted segmentation result.

[0019] According to some embodiments of the present application, the N polyp image frames are averaged along the time axis and summarized into the multi-scale temporal features;

[0020] The multi-scale temporal features are input into a reconstruction decoder to predict a next polyp image frame, thereby obtaining a target polyp image frame.

[0021] According to some embodiments of the present application, the video polyp segmentation based on the self-supervised task diffusion model also includes an adversarial self-supervisory module, the adversarial self-supervisory module includes a generator and a discriminator, the generator is the temporal reasoning module, and the method also includes:

[0022] distinguishing the target polyp image frame from the real polyp image frame,

[0023] Determine an adversarial loss according to the target polyp image frame and the real polyp image frame;

[0024] The discriminator is optimized by minimizing the adversarial loss.

[0025] According to some embodiments of the present application, after the multi-level spatiotemporal features and the noise segmentation mask are concatenated and fused to obtain the fused features, the method further includes:

[0026] Performing inverse denoising on the fused features, and outputting a classification score and a bounding box;

[0027] Determining the category of the target polyp image frame according to the output classification score and the bounding box;

[0028] determining a loss for a multi-task diffusion model based on the classification scores and the bounding boxes;

[0029] The total loss of the video polyp segmentation system based on the self-supervised task diffusion model is determined according to the loss of the task diffusion model and the loss of the adversarial self-supervised module.

[0030] According to some embodiments of the present application, the forward noise processing is performed on the predicted segmentation mask of the hidden state sequence to obtain the noise segmentation mask, including:

[0031] Introduce Gaussian noise;

[0032] Performing forward noise processing on the predicted segmentation mask of the hidden state sequence by using the Gaussian noise to obtain a probability distribution of the noise segmentation mask;

[0033] The probability distribution of the noise segmentation mask is expressed by the following formula:

[0034]

[0035] Where I is the unit matrix, z0 is the segmentation mask corresponding to the input initial polyp image frame, and z t is the noise segmentation mask obtained after t steps of noise addition, is a Gaussian distribution, is the mean vector, is the covariance matrix, q(z t |z0) represents the state noise segmentation mask z at time t given the segmentation mask z0 corresponding to the initial polyp image frame t The conditional probability distribution of .

[0036] According to some embodiments of the present application, the performing inverse denoising processing on the fused features to output a predicted segmentation mask of the same dimension as the last polyp image frame includes:

[0037] A CNN-based denoising head is introduced, which is a mapping function f θ (z t ,x,t);

[0038] Under the guidance of condition x, the segmentation mask corresponding to the input initial polyp image frame is predicted from the noise segmentation mask obtained after t steps of noise addition by minimizing the objective function, thereby obtaining the predicted segmentation mask;

[0039] The predicted segmentation mask is reconstructed from the Gaussian noise by using the denoising head conditioned on x to obtain a predicted segmentation mask of the same dimension as the last polyp image frame, and the predicted segmentation mask is output in a Markov manner. The predicted segmentation mask is represented by the following formula:

[0040]

[0041] Among them, p θ is the parameterized probability distribution, θ is the denoising head f θ Parameters, is the estimated segmentation mask of the hidden state sequence from time 0 to T-1, is the estimated segmentation mask of the hidden state at time t, T is the length of the hidden state sequence, and p(z T ) is the initial probability of the hidden state at time T, is the noise segmentation mask z after given noise addition T The probability of the hidden state at time T-1 under condition x is, is the hidden state transition probability at each time point t from time 1 to T-1, given the hidden state at the previous time point t+1 and the condition x.

[0042] In a third aspect, an embodiment of the present application provides a controller comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method described in the technical solution of the second aspect above when running the computer program.

[0043] The video polyp segmentation system and method of the self-supervised multi-task diffusion model provided in the embodiments of the present application have at least one of the following advantages or beneficial effects: a conditional diffusion model is applied in the video polyp segmentation system, and a Markov random process is used to encode and denoise the segmentation mask corresponding to the last polyp image frame acquired by the acquisition module in a hidden state sequence or latent space to obtain a noise segmentation mask, and at the same time, multi-scale features of the polyp image frame are extracted to obtain a segmentation mask of the same resolution for the polyp image of the current frame of the polyp video; a multi-task diffusion model is introduced into the video polyp segmentation system, and polyp classification and polyp detection tasks are performed on the polyp image frame by the multi-task diffusion model, which can effectively utilize the high-level semantic information of the objects in the polyp image frame, thereby providing context and discrimination information for the segmentation task, improving the discrimination and generalization capabilities of the multi-task diffusion model, and improving the accuracy of polyp recognition. The last polyp image frame is used as the input of the image encoder to generate multi-scale spatial features. At the same time, N polyp image frames are used as the input of the temporal encoder to generate multi-scale temporal features. Then, the multi-scale spatial features and multi-scale temporal features are integrated and input into the feature pyramid network to obtain multi-level spatiotemporal features. The multi-level spatiotemporal features and the noise segmentation mask are spliced ​​and fused using the MLP layer, and the fused features are denoised by the denoising decoder to obtain a predicted segmentation mask with the same dimension as the last polyp image frame. The predicted segmentation result is obtained through the predicted segmentation mask, and the polyp position is determined based on the predicted segmentation result. The temporal information related to the previous and next frames in the polyp video is used to assist in locating the polyp position in a complex environment, thereby achieving accurate positioning of the polyp position, improving the accuracy of polyp diagnosis, and improving the accuracy of polyp segmentation.

[0044] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a structural schematic diagram of a video polyp segmentation system based on a self-supervised multi-task diffusion model provided in an embodiment of the present application;

[0046] Figure 2 is a structural schematic diagram of another video polyp segmentation system based on a self-supervised multi-task diffusion model provided in an embodiment of the present application;

[0047] Figure 3 is a flow chart of a video polyp segmentation method based on a self-supervised multi-task diffusion model provided in an embodiment of the present application;

[0048] Figure 4It is a flow chart of a method for reconstructing a target polyp image frame provided by an embodiment of the present application;

[0049] Figure 5 is a flow chart of a method for optimizing a discriminator provided in an embodiment of the present application;

[0050] Figure 6 is a flow chart of a method for determining the total loss of a video polyp segmentation system based on a self-supervised task diffusion model provided by an embodiment of the present application;

[0051] Figure 7 It is a structural diagram of a controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, the characteristics, operations or features described in the specification can be combined in any appropriate manner to form various implementation methods. At the same time, the steps or actions in the method description can also be replaced or adjusted in order in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the accompanying drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a necessary sequence, unless otherwise specified that a certain sequence must be followed.

[0053] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0054] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0055] Colorectal cancer has become the second leading cause of cancer-related deaths worldwide, accounting for about 10% of all cancer cases. Colonoscopy plays a vital role in detecting polyps and reducing the incidence and mortality of colorectal cancer. In related technologies, most polyp segmentation work only trains and evaluates the model on static images, ignoring the additional information for polyp localization brought by the temporal nature of the video, and not making full use of the temporal information between endoscopic video frames. In the task of video polyp segmentation, it is obviously not enough to focus only on independent static images. In the task of video polyp image tracking and segmentation, it is first necessary to pre-train on a large number of static images and then fine-tune on video images. This training strategy requires a large amount of high-quality annotations, but the current scale of video polyp data is still small. At the same time, because the boundaries of polyps are blurred and similar to background tissues, even skilled clinicians may not be able to reach a consensus on the annotations of consecutive frames. Finally, the current open source polyp dataset is a sparse sequence, and some of its adjacent frames vary greatly. Although the endoscopic video focuses on the same polyp tissue, due to different camera angles or lighting, the contextual environment of the polyp, such as cavity, highlight, mucosal tissue, etc., will change, which may affect the prediction results of adjacent frames, thereby affecting the prediction results of polyp segmentation and reducing the accuracy of polyp segmentation diagnosis.

[0056] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0057] Reference Figure 1 As shown, Figure 11 is a structural schematic diagram of a video polyp segmentation system based on a self-supervised multi-task diffusion model provided in an embodiment of the present application. The video polyp segmentation system based on a self-supervised multi-task diffusion model includes: a processing module, a conditional diffusion model, a multi-task diffusion model, an MLP layer and a denoising decoder; the processing module is used to obtain a polyp video, and perform data set annotation processing on the polyp video to obtain N frames of polyp image frames and a segmentation mask corresponding to each polyp image frame; the conditional diffusion model is used to encode the segmentation mask corresponding to the last frame of the polyp image frame to obtain a segmentation mask of a hidden state sequence, and perform forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask; the multi-task diffusion model includes FIG. Like the encoder and the temporal encoder, the multi-task diffusion model is used to take the last frame of the polyp image frame as the input of the image encoder to generate multi-scale spatial features, and take N frames of polyp image frames as the input of the temporal encoder to generate multi-scale temporal features. It is also used to integrate the multi-scale spatial features and the multi-scale temporal features and input them into the feature pyramid network to obtain multi-level spatiotemporal features; the MLP layer is used to concatenate and fuse the multi-level spatiotemporal features and the noise segmentation mask to obtain the fused features; the denoising decoder is used to perform inverse denoising on the fused features, output a predicted segmentation mask of the same dimension as the last frame of the polyp image frame, determine the predicted segmentation result based on the predicted segmentation mask, and determine the polyp position based on the predicted segmentation result.

[0058] The acquisition module acquires N frames of polyp image frames and the segmentation masks corresponding to each frame of the polyp image frame. The conditional diffusion model is applied in the video polyp segmentation system in the embodiment of the present application. The segmentation mask corresponding to the last frame of the polyp image frame acquired by the acquisition module is encoded in the hidden state sequence or latent space by using the Markov random process, and is subjected to noise processing to obtain a noise segmentation mask. At the same time, the multi-scale features of the polyp image frame are extracted to obtain a segmentation mask of the same resolution for the polyp image of the current frame of the polyp video. A multi-task diffusion model is introduced into the video polyp segmentation system. The multi-task diffusion model is used to perform polyp classification and polyp detection tasks on the polyp image frames. The high-level semantic information of the objects in the polyp image frames can be effectively utilized, thereby providing context and discrimination information for the segmentation task, improving the discrimination and generalization capabilities of the multi-task diffusion model, and improving the accuracy of polyp recognition. The last polyp image frame is used as the input of the image encoder to generate multi-scale spatial features. At the same time, N polyp image frames are used as the input of the temporal encoder to generate multi-scale temporal features. Then, the multi-scale spatial features and multi-scale temporal features are integrated and input into the feature pyramid network to obtain multi-level spatiotemporal features. The multi-level spatiotemporal features and the noise segmentation mask are spliced ​​and fused using the MLP layer, and the fused features are denoised by the denoising decoder to obtain a predicted segmentation mask with the same dimension as the last polyp image frame. The predicted segmentation result is obtained through the predicted segmentation mask, and the polyp position is determined based on the predicted segmentation result. The temporal information related to the previous and next frames in the polyp video is used to assist in locating the polyp position in a complex environment, thereby achieving accurate positioning of the polyp position, improving the accuracy of polyp diagnosis, and improving the accuracy of polyp segmentation.

[0059] Reference Figure 1 and Figure 2 As shown, in some embodiments of the present application, the video polyp segmentation system based on the self-supervised multi-task diffusion model also includes a temporal reasoning module, which is used to average N frames of polyp image frames along the time axis and summarize them into multi-scale temporal features, and input the multi-scale temporal features into a reconstruction decoder to predict the next frame of polyp image frame to obtain a target polyp image frame.

[0060] The temporal reasoning module is introduced into the video polyp segmentation system, which can capture the dynamic appearance and maintain temporal coherence when segmenting polyps. The generative adversarial self-supervision strategy is adopted to reconstruct the current target polyp image frame from the temporal information of the previous polyp image frame, and the polyp image frames are averaged along the time axis and summarized into multi-scale temporal features. At the same time, the temporal multi-scale features extracted during the reconstruction process are fused with the spatial multi-scale features of the conditional diffusion model and input into the reconstruction decoder to predict the next polyp image frame and obtain the target polyp image frame. The temporal reasoning module explores the temporal dependency from the polyp video sequence by inferring the unseen target polyp image frame, and learns the spatiotemporal discriminant features to provide guidance for discriminating and segmenting highly camouflaged polyps.

[0061] In some embodiments of the present application, a video polyp segmentation system based on a self-supervised multi-task diffusion model also includes an adversarial self-supervision module, which includes a generator and a discriminator. The generator is a temporal reasoning module, and the discriminator is used to distinguish between target polyp image frames and real frame polyp image frames. The adversarial loss is determined based on the target polyp image frames and the real frame polyp image frames; and the discriminator is optimized by minimizing the adversarial loss.

[0062] The video polyp segmentation system introduces an adversarial self-supervision module, which can generate more realistic frame polyp image frames. In the process of reconstructing the current target polyp image frame from the timing information of the previous polyp image frame, the generative adversarial self-supervision function is introduced. The temporal reasoning module is used as the generator, and the Inception3 network pre-trained on ImageNet is used as the discriminator. The discriminator is used to distinguish between the target polyp image frame and the real frame polyp image frame. The adversarial loss is determined by the target polyp image frame and the real frame polyp image frame, and then the discriminator is optimized by minimizing the adversarial loss, thereby improving the discrimination and generalization ability of the video polyp segmentation system, while improving the accuracy of polyp recognition.

[0063] Reference Figure 3 , Figure 3 is a flowchart of a video polyp segmentation method based on a self-supervised task diffusion model provided in an embodiment of the present application. The video polyp segmentation method based on the self-supervised task diffusion model includes but is not limited to steps S100 to S500. Specifically,

[0064] Step S100: obtaining a polyp video, and performing data set annotation processing on the polyp video to obtain N polyp image frames and a segmentation mask corresponding to each polyp image frame;

[0065] Step S200: performing encoding processing on the segmentation mask corresponding to the last polyp image frame to obtain a segmentation mask of a hidden state sequence, and performing forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask;

[0066] Step S300: taking the last polyp image frame as the input of the image encoder to generate multi-scale spatial features, taking N polyp image frames as the input of the temporal encoder to generate multi-scale temporal features, integrating the multi-scale spatial features and the multi-scale temporal features and inputting them into the feature pyramid network to obtain multi-level spatiotemporal features;

[0067] Step S400: concatenating and fusing the multi-level spatiotemporal features and the noise segmentation mask to obtain a fused feature;

[0068] Step S500: perform inverse denoising on the fused features, output a predicted segmentation mask of the same dimension as the last polyp image frame, determine a predicted segmentation result based on the predicted segmentation mask, and determine the polyp position based on the predicted segmentation result.

[0069] The video polyp segmentation method based on the self-supervised task diffusion model includes: first, obtaining a polyp video, and performing a data set annotation process on the polyp video to obtain N polyp image frames and a segmentation mask corresponding to each polyp image frame, and encoding the segmentation mask corresponding to the last polyp image frame obtained by the acquisition module in a hidden state sequence or hidden space by using a conditional diffusion model and a Markov random process, and performing a noise addition process to obtain a noise segmentation mask, and at the same time extracting multi-scale features of the polyp image frame to obtain a segmentation mask of the same resolution for the current polyp image of the polyp video. The last polyp image frame is used as the input of the image encoder to generate multi-scale spatial features, and at the same time, the N polyp image frames are used as the input of the temporal encoder to generate multi-scale temporal features, and then the multi-scale spatial features and multi-scale temporal features are integrated and input into the feature pyramid network to obtain multi-level spatiotemporal features. The high-level semantic information of the objects in the polyp image frame is used to provide context and discriminant information for the segmentation task, thereby improving the discrimination and generalization capabilities of the multi-task diffusion model and the accuracy of polyp recognition. Afterwards, the MLP layer is used to splice and fuse the multi-level spatiotemporal features and the noise segmentation mask, and the fused features are denoised through the denoising decoder to obtain a predicted segmentation mask with the same dimension as the last polyp image frame. The predicted segmentation result is obtained through the predicted segmentation mask, and the polyp position is determined according to the predicted segmentation result. The timing information related to the previous and next frames in the polyp video is used to assist in locating the polyp position in a complex environment, thereby achieving accurate positioning of the polyp position, improving the accuracy of polyp diagnosis, and at the same time improving the accuracy of polyp segmentation.

[0070] Reference Figure 4 , Figure 4 is a flow chart of a method for reconstructing a target polyp image frame provided by an embodiment of the present application. The method for reconstructing a target polyp image frame includes but is not limited to steps S600 to S610. Specifically,

[0071] Step S600: averaging N polyp image frames along the time axis and summarizing them into multi-scale temporal features;

[0072] Step S610: input the multi-scale temporal features into the reconstruction decoder to predict the next polyp image frame to obtain a target polyp image frame.

[0073] In some embodiments of the present application, in order to capture the dynamic appearance and motion clues in the colonoscopy video, a temporal reasoning module is introduced in the video polyp segmentation system. It can be understood that, through the self-supervised strategy, from the previous N frames of polyp image frames {I i-δ ,…,I i-1}Reconstruct the target polyp image frame I i Then, the previous N polyp image frames are averaged along the time axis and summarized into multi-scale temporal features. The multi-scale temporal features are input into the CNN-based reconstruction decoder. The reconstruction decoder consists of an MLP layer. The reconstruction decoder receives and integrates the multi-scale temporal features to predict the next polyp image frame. Get the target polyp image frame I i The temporal reasoning module is used to reason about unseen target polyp image frames, explore temporal dependencies from polyp video sequences, and learn spatiotemporal discriminative features to provide guidance for discriminating and segmenting highly camouflaged polyps.

[0074] Reference Figure 5 , Figure 5 is a flowchart of a method for optimizing a discriminator provided in an embodiment of the present application. The method for optimizing a discriminator includes but is not limited to steps S700 to S720. Specifically,

[0075] Step S700: distinguishing the target polyp image frame from the real polyp image frame;

[0076] Step S710: determining the adversarial loss according to the target polyp image frame and the real polyp image frame;

[0077] Step S720: Optimize the discriminator by minimizing the adversarial loss.

[0078] The video polyp segmentation system includes an adversarial self-supervision module. By introducing the adversarial self-supervision module, more realistic polyp image frames can be generated. The adversarial self-supervision module includes a generator and a discriminator. The temporal reasoning module is used as the generator, and the Inception3 network pre-trained on ImageNet is used as the discriminator. The discriminator is used to distinguish between the target polyp image frames and the real polyp image frames. The adversarial loss is determined by the target polyp image frames and the real polyp image frames, and then the discriminator is optimized by minimizing the adversarial loss, thereby improving the discrimination and generalization capabilities of the video polyp segmentation system and the accuracy of polyp recognition.

[0079] It can be understood that in order to generate more realistic frame-by-frame polyp image frames, a generative adversarial self-supervision function is introduced in the process of reconstructing the current target polyp image frame from the temporal information of the previous polyp image frame.

[0080] The adversarial loss is determined based on the target polyp image frame and the real polyp image frame, and is expressed by the following formula:

[0081]

[0082] in, To combat losses, is the target polyp image frame, I i is the real polyp image frame, For the discriminator.

[0083] for With fixed parameters, the preset generator can fool the discriminator to generate images that are more like real polyp images:

[0084]

[0085] in, is the loss of the generator, is the mean square error loss between the target polyp image frame and the real polyp image frame, is the target polyp image frame, I i is the real polyp image frame, λ adv To balance the hyperparameters, we set λ adv Set to 0.001 to balance the mean square error reconstruction term and the adversarial term. It should be noted that the adversarial loss Only used to optimize the discriminator Not relevant to the multi-task diffusion model.

[0086] Reference Figure 6 , Figure 6 : is a flowchart of a method for determining the total loss of a video polyp segmentation system based on a self-supervised task diffusion model provided by an embodiment of the present application. The method for determining the total loss of a video polyp segmentation system based on a self-supervised task diffusion model includes but is not limited to steps S800 to S830. Specifically,

[0087] Step S800: performing inverse denoising processing on the fused features, and outputting classification scores and bounding boxes;

[0088] Step S810: determining the category of the target polyp image frame according to the classification score and the bounding box;

[0089] Step S820: determining the loss of the multi-task diffusion model based on the classification score and the bounding box;

[0090] Step S830: Determine the total loss of the video polyp segmentation system based on the self-supervised task diffusion model according to the loss of the task diffusion model and the loss of the adversarial self-supervision module.

[0091] After the multi-level spatiotemporal features and the noise segmentation mask are spliced ​​and fused to obtain the fused features, the following steps are also included: performing reverse denoising on the fused features and outputting the classification scores. and bounding box The classification score and bounding box Used for instance detection of target polyp image frames to determine the category of the target polyp image frames.

[0092] Apply a segmentation loss consisting of pixel cross entropy loss, mean squared error, and IoU loss The loss task of determining the multi-task diffusion model is optimized by cross entropy loss. Therefore, the loss objective function of the multi-task diffusion model can be expressed as:

[0093]

[0094] in, is the loss of the multi-task diffusion model, λ seg ,λ cls ,λ det is the balanced hyperparameter for the three tasks and sets λ seg =0.5,λ cls is 0.05, λ det is 0.2; is the classification score, Y cls is the target polyp image frame I i The categories of polyps include low-grade adenoma, high-grade adenoma, hyperplastic polyp, traditional serrated adenoma, sessile serrated lesion, and invasive carcinoma. is the predicted bounding box, Y box is the bounding box of the target lesion in the target polyp image frame (x c ,y c ,w,h), is the predicted pixel-level segmentation mask, Y seg is the segmentation mask of the target polyp image frame; is the pixel-level segmentation loss, Loss by Dice Loss Binary Cross Entropy Loss and mean square error loss composition, is the cross entropy loss for the category, is the square root loss for the bounding box.

[0095] By introducing the adversarial self-supervision module, more realistic polyp image frames can be generated. The adversarial self-supervision module greatly enhances the discrimination and generalization capabilities of the conditional diffusion model, alleviates the class imbalance caused by the scarcity of lesions in the pixel-level segmentation of polyps, and improves the accuracy of polyp recognition.

[0096] Then, the loss of the multi-task diffusion model is determined according to the classification scores and bounding boxes, and the total loss of the video polyp segmentation system based on the self-supervised task diffusion model is determined according to the loss of the task diffusion model and the loss of the adversarial self-supervised module.

[0097] The total loss of the video polyp segmentation system based on the self-supervised multi-task diffusion model is calculated by combining the loss of the multi-task diffusion model MDM and the loss of the temporal reasoning module TRM. as follows:

[0098]

[0099] in, is the total loss of the video polyp segmentation system based on the self-supervised multi-task diffusion model, is the loss of the multi-task diffusion model, is the loss of the temporal reasoning module, λ MDM and λ TRM To balance the hyperparameters, set λ MDM is 0.25, λ TRM is 0.25.

[0100] In some embodiments of the present application, a conditional diffusion model is introduced into the prediction segmentation mask process. The conditional diffusion model is a likelihood-based model inspired by non-equilibrium thermodynamics, which presupposes a forward noise process, performs forward noise processing on the prediction segmentation mask of the hidden state sequence, and obtains a noise segmentation mask, including:

[0101] First, Gaussian noise is introduced; the predicted segmentation mask of the hidden state sequence is forward noise processed by Gaussian noise to obtain the probability distribution of the noise segmentation mask;

[0102] Among them, the probability distribution of the noise segmentation mask is expressed by the following formula:

[0103]

[0104] Where I is the unit matrix, z0 is the segmentation mask corresponding to the input initial polyp image frame, and z t is the noise segmentation mask obtained after t steps of noise addition, is a Gaussian distribution, is the mean vector, is the covariance matrix, q(z t|z0) represents the state noise segmentation mask z at time t given the segmentation mask z0 corresponding to the initial polyp image frame t The conditional probability distribution of .

[0105] Then, in order to predict the predicted segmentation mask of the same dimension as the last polyp image frame from the noise segmentation mask, the present application introduces a CNN-based denoising head f θ , the denoising head is the mapping function f θ (z t ,x,t), through the denoising head f θ It is used to perform inverse denoising on the fused features and output a predicted segmentation mask with the same dimension as the last polyp image frame, including:

[0106] After introducing the CNN-based denoising head, under the guidance of condition x, the segmentation mask corresponding to the input initial polyp image frame is predicted from the noise segmentation mask obtained after t steps of denoising by minimizing the objective function to obtain the predicted segmentation mask; the predicted segmentation mask is reconstructed from the Gaussian noise by the denoising head conditioned on x to obtain a predicted segmentation mask of the same dimension as the last polyp image frame, and the predicted segmentation mask is output in a Markov manner. The predicted segmentation mask is expressed by the following formula:

[0107]

[0108] Among them, p θ is the parameterized probability distribution, θ is the denoising head f θ Parameters, is the estimated segmentation mask of the hidden state sequence from time 0 to T-1, is the estimated segmentation mask of the hidden state at time t, T is the length of the hidden state sequence, and p(z T ) is the initial probability of the hidden state at time T, is the noise segmentation mask z after given noise addition T The probability of the hidden state at time T-1 under condition x is, is the hidden state transition probability at each time point t from time 1 to T-1, given the hidden state and condition x at the previous time point t+1.

[0109] The conditional diffusion model is introduced in the video polyp segmentation task. The Markov random process is used to add noise and denoise the segmentation mask in the hidden state sequence. At the same time, the multi-scale features of the fused image are extracted to obtain the segmentation mask of the same resolution for the polyp image of the current frame of the polyp video.

[0110] In some embodiments of the present application, the video polyp segmentation method based on the self-supervised task diffusion model also includes preliminary work such as polyp video data collection, data set annotation, model training, model testing and ablation experiments, specifically:

[0111] The collection of polyp video data includes: professional physicians in the Department of Gastroenterology collect videos of polyps during the diagnosis process. The video collection objects are 10 different patients. During the operation of each patient, a video of about 1 hour is collected, and a total of 10 surgical videos of about 1 hour are collected. Then, professional surgeons cut out 10 short videos of about 15 to 20 seconds in length from each 1-hour video, and it is necessary to ensure that each frame in each short video contains the polyp area. Finally, a total of 100 surgical videos of 3 to 11 seconds are obtained.

[0112] The annotation of the dataset includes: interval sampling of all short videos of 3 to 11 seconds, extracting each frame in the video and saving it as an image in JPG format, a total of 100 case videos containing polyps were obtained, the real-time frame rate was 30 frames / second, and 49,136 images containing polyps were obtained after interval sampling, of which 14,176 frames were used for training the model and 24,736 frames were used for testing. Then, these images were annotated to mark the boundaries of the polyps and retained as images in PNG format.

[0113] Model training includes:

[0114] The model was trained using 14,176 polyp video frames from 100 cases. The segmentation masks of the current N polyp image frames and the last polyp image frame were input. The temporal reasoning module extracted multi-scale temporal features from the N polyp image frames, and the image encoder extracted the last polyp image frame to obtain multi-scale spatial features. The multi-scale temporal features were averaged in the time dimension and added pixel by pixel with the multi-scale spatial features at the corresponding scale to obtain multi-level spatiotemporal features. Then, the multi-level spatiotemporal features of different sizes were merged into a spatiotemporal feature of one scale as the condition for the denoising process of the multi-task diffusion model. The last polyp image frame was encoded into the hidden state sequence to obtain the segmentation mask of the hidden state sequence. After random t-step sampling and denoising in the forward process, the noise segmentation mask was obtained. The noise segmentation mask and the multi-level spatiotemporal features were spliced ​​and input into the denoising decoder. After denoising in the reverse process, the predicted segmentation mask of the same dimension as the last polyp image frame was output.

[0115] Model testing includes:

[0116] The test data set is divided into two sub-test sets, EASY and HARD, according to the distribution of polyp video data. Each sub-test set is further divided into two cases, Seen and Unseen. Seen represents the visible case, and Unseen represents the invisible case. Seen divides the polyp image frames in a case video into two non-intersecting parts, one for training and the other for testing. There are a total of 4 non-intersecting test sets, EASY-Seen, EASY-Unseen, HARD-Seen, and HARD-Unseen. For each test set, the current N frames of polyp image frames are input into the trained model, and the model outputs a predicted segmentation mask with the same dimension as the last frame of polyp image frame.

[0117] Ablation experiments include:

[0118] In order to verify the effectiveness of various models and modules in the video polyp segmentation system based on the self-supervised multi-task diffusion model, such as the processing module, conditional diffusion model, multi-task diffusion model, MLP layer, denoising decoder, etc., ablation experiments were carried out on each model and module to avoid parameter redundancy and improve the accuracy of polyp diagnosis in the video polyp segmentation system of the self-supervised multi-task diffusion model.

[0119] Reference Figure 7 As shown, Figure 7It is a structural diagram of a controller 1000 provided in an embodiment of the present application, including a processor 1001, which can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the method provided in the embodiment of the present application; a memory 1002, which can be implemented in the form of a read-only memory 1002 (Read Only Memory, ROM), a static storage device, a dynamic storage device or a random access memory 1002 (Random Access Memory, RAM), etc. The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1002 and are called by the processor 1001 to execute the embodiments of this application; the input / output interface 1003 is used to implement information input and output; the communication interface 1004 is used to implement communication interaction between this device and other devices, and communication can be achieved through wired methods (such as USB, network cables, etc.) or wireless methods (such as mobile networks, WIFI, Bluetooth, etc.); a bus transmits information between various components of the device (such as the processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004); wherein the processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004 are connected to each other within the device through the bus.

[0120] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer-readable storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0121] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present application. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A video polyp segmentation system based on a self-supervised multi-task diffusion model, characterized by: include: A processing module, used for acquiring a polyp video, and performing data set annotation processing on the polyp video to obtain N polyp image frames and a segmentation mask corresponding to each polyp image frame; The conditional diffusion model is used to encode the segmentation mask corresponding to the last polyp image frame to obtain the segmentation mask of the hidden state sequence, and perform forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask; A multi-task diffusion model, comprising an image encoder and a temporal encoder, wherein the multi-task diffusion model is used to use the last polyp image frame as an input of the image encoder to generate multi-scale spatial features, use the N polyp image frames as an input of the temporal encoder to generate multi-scale temporal features, and integrate the multi-scale spatial features and the multi-scale temporal features and input them into a feature pyramid network to obtain multi-level spatiotemporal features; The MLP layer is used to concatenate and fuse the multi-level spatiotemporal features and the noise segmentation mask to obtain a fused feature; A denoising decoder is used to perform inverse denoising on the fused features, output a predicted segmentation mask of the same dimension as the last polyp image frame, determine a predicted segmentation result based on the predicted segmentation mask, and determine the polyp position based on the predicted segmentation result.

2. The video polyp segmentation system based on the self-supervised multi-task diffusion model according to claim 1 is characterized in that: Also includes: The temporal reasoning module is used to average the N polyp image frames along the time axis and summarize them into the multi-scale temporal features, and input the multi-scale temporal features into the reconstruction decoder to predict the next polyp image frame to obtain the target polyp image frame.

3. The video polyp segmentation system based on the self-supervised multi-task diffusion model according to claim 2 is characterized in that: Also includes: An adversarial self-supervised module includes a generator and a discriminator, wherein the generator is the temporal reasoning module, and the discriminator is used to distinguish between the target polyp image frame and the real polyp image frame, and the adversarial loss is determined according to the target polyp image frame and the real polyp image frame; and the discriminator is optimized by minimizing the adversarial loss.

4. A video polyp segmentation method based on a self-supervised task diffusion model, characterized in that: Applied to the video polyp segmentation system according to any one of claims 1 to 3, the method comprises: Acquire a polyp video, and perform data set annotation processing on the polyp video to obtain N polyp image frames and a segmentation mask corresponding to each polyp image frame; Performing encoding processing on the segmentation mask corresponding to the last polyp image frame to obtain a segmentation mask of a hidden state sequence, and performing forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask; Using the last polyp image frame as an input of an image encoder to generate multi-scale spatial features, using the N polyp image frames as an input of a temporal encoder to generate multi-scale temporal features, integrating the multi-scale spatial features and the multi-scale temporal features and inputting them into a feature pyramid network to obtain multi-level spatiotemporal features; Concatenating and fusing the multi-level spatiotemporal features and the noise segmentation mask to obtain a fused feature; The fusion feature is subjected to inverse denoising processing, and a predicted segmentation mask having the same dimension as the last polyp image frame is outputted; a predicted segmentation result is determined according to the predicted segmentation mask; and a polyp position is determined according to the predicted segmentation result.

5. The video polyp segmentation method based on the self-supervised task diffusion model according to claim 4 is characterized in that: Also includes: Averaging the N polyp image frames along the time axis and summarizing them into the multi-scale temporal features; The multi-scale temporal features are input into a reconstruction decoder to predict a next polyp image frame, thereby obtaining a target polyp image frame.

6. The video polyp segmentation method based on the self-supervised task diffusion model according to claim 5 is characterized in that: The video polyp segmentation based on the self-supervised task diffusion model also includes an adversarial self-supervisory module, the adversarial self-supervisory module includes a generator and a discriminator, the generator is the temporal reasoning module, and the method also includes: distinguishing the target polyp image frame from the real polyp image frame, Determine an adversarial loss according to the target polyp image frame and the real polyp image frame; The discriminator is optimized by minimizing the adversarial loss.

7. The video polyp segmentation method based on the self-supervised task diffusion model according to claim 6 is characterized in that: After the multi-level spatiotemporal features and the noise segmentation mask are spliced ​​and fused to obtain fused features, the method further includes: Performing inverse denoising on the fused features, and outputting a classification score and a bounding box; Determining the category of the target polyp image frame according to the output classification score and the bounding box; determining a loss for a multi-task diffusion model based on the classification scores and the bounding boxes; The total loss of the video polyp segmentation system based on the self-supervised task diffusion model is determined according to the loss of the task diffusion model and the loss of the adversarial self-supervised module.

8. The video polyp segmentation method based on the self-supervised task diffusion model according to claim 4 is characterized in that: The forward noise processing is performed on the predicted segmentation mask of the hidden state sequence to obtain a noise segmentation mask, comprising: Introduce Gaussian noise; Performing forward noise processing on the predicted segmentation mask of the hidden state sequence by using the Gaussian noise to obtain a probability distribution of the noise segmentation mask; The probability distribution of the noise segmentation mask is expressed by the following formula: Where I is the unit matrix, z0 is the segmentation mask corresponding to the input initial polyp image frame, and z t is the noise segmentation mask obtained after t steps of noise addition, is a Gaussian distribution, is the mean vector, is the covariance matrix, q(z t |z0) represents the state noise segmentation mask z at time t given the segmentation mask z0 corresponding to the initial polyp image frame t The conditional probability distribution of .

9. The video polyp segmentation method based on the self-supervised task diffusion model according to claim 8, characterized in that: The performing inverse denoising processing on the fused features to output a predicted segmentation mask having the same dimension as the last polyp image frame includes: A CNN-based denoising head is introduced, which is a mapping function f θ (z t ,x,t); Under the guidance of condition x, the segmentation mask corresponding to the input initial polyp image frame is predicted from the noise segmentation mask obtained after t steps of noise addition by minimizing the objective function, thereby obtaining the predicted segmentation mask; The predicted segmentation mask is reconstructed from the Gaussian noise by using the denoising head conditioned on x to obtain a predicted segmentation mask of the same dimension as the last polyp image frame, and the predicted segmentation mask is output in a Markov manner. The predicted segmentation mask is represented by the following formula: Among them, p θ is the parameterized probability distribution, θ is the denoising head f θ Parameters, is the estimated segmentation mask of the hidden state sequence from time 0 to T-1, is the estimated segmentation mask of the hidden state at time t, T is the length of the hidden state sequence, and p(z T ) is the initial probability of the hidden state at time T, is the noise segmentation mask z after given noise addition T The probability of the hidden state at time T-1 under condition x is, is the hidden state transition probability at each time point t from time 1 to T-1, given the hidden state at the previous time point t+1 and the condition x.

10. A controller, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method according to any one of claims 4 to 9 when executing the computer program.

Images