Video polyp segmentation system and method based on self-supervised task diffusion model

Abstract

The application discloses a video polyp segmentation system and method based on a self-supervised task diffusion model. The video polyp segmentation system comprises a processing module, a conditional diffusion model, a multi-task diffusion model, an MLP layer, and a denoising decoder. The processing module is used for acquiring a polyp video, performing dataset labeling processing on the polyp video, and obtaining N polyp image frames and a segmentation mask corresponding to each polyp image frame. The conditional diffusion model is used for encoding and noise processing on the segmentation mask corresponding to the last polyp image frame, and obtaining a noise segmentation mask. The multi-task diffusion model is used for generating multi-scale spatial features and multi-scale temporal features, and integrating the input into a feature pyramid network to obtain multi-level spatio-temporal features. The MLP layer is used for splicing and fusing the multi-level spatio-temporal features and the noise segmentation mask to obtain fused features. The denoising decoder is used for reverse denoising processing on the fused features, outputting a predicted segmentation mask, and determining a predicted segmentation result according to the predicted segmentation mask. The application improves the accuracy of polyp diagnosis and polyp segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation processing, in particular to a video polyp segmentation system and method based on a self-supervised task diffusion model. BACKGROUND

[0002] Colorectal cancer has become the second leading cause of cancer-related deaths worldwide, accounting for about 10% of all cancer cases, and colonoscopy plays a crucial role in detecting polyps and reducing the incidence and mortality of colorectal cancer. The most effective way to prevent colorectal cancer is to regularly undergo endoscopy and timely polyp resection surgery. By taking video images with a colonoscope, doctors can assess the location and appearance of polyp tissue and remove it before it becomes cancerous. However, due to the complex and variable shape and color of polyps, the blurred boundary, and external shooting conditions such as low boundary contrast, mirror reflection, and artifacts, the high concealment of polyps and the high missed diagnosis rate in the diagnosis process.

[0003] In related technologies, most polyp segmentation work only trains and evaluates models on static images, ignoring the additional information for polyp positioning brought by the time sequence of videos and not fully utilizing the temporal information between endoscopic video frames. In the video polyp segmentation task, simply focusing on independent static images is obviously not enough. In the video polyp image tracking and segmentation task, pre-training is first performed on a large number of static images, and then fine-tuning is performed on video images. This training strategy requires a large amount of high-quality labeling, but the current video polyp data size is still very small. At the same time, due to the blurred boundary of polyps and the similarity to background tissue, even experienced clinicians may not be able to agree on the labeling of consecutive frames. Finally, the current open-source polyp dataset belongs to a sparse sequence, and the changes between some adjacent frames are large. Although the endoscopic video focuses on the same polyp tissue, due to the different camera angles or lighting, the context environment in which the polyp is located, such as the cavity, highlight, and mucosal tissue, may change, which may affect the prediction results of adjacent frames and thus affect the prediction results of polyp segmentation, reducing the accuracy of polyp segmentation diagnosis. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the embodiments of the present application provide a video polyp segmentation system and method based on a self-supervised multi-task diffusion model, which is beneficial to accurately identifying the position of polyps, improving the accuracy of polyp diagnosis, and improving the accuracy of polyp segmentation.

[0005] In a first aspect, the embodiments of the present application provide a video polyp segmentation system based on a self-supervised multi-task diffusion model, comprising:

[0006] The processing module is configured to obtain a polyp video, and perform dataset labeling processing on the polyp video to obtain N polyp image frames and a segmentation mask corresponding to each polyp image frame.

[0007] The conditional diffusion model is configured to perform encoding processing on the segmentation mask corresponding to the last polyp image frame to obtain a segmentation mask of a hidden state sequence, perform forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask.

[0008] The multi-task diffusion model includes an image encoder and a time sequence encoder. The multi-task diffusion model is configured to take the last polyp image frame as an input of the image encoder to generate multi-scale spatial features, take the N polyp image frames as an input of the time sequence encoder to generate multi-scale time features, and integrate and input the multi-scale spatial features and the multi-scale time features into a feature pyramid network to obtain multi-level spatio-temporal features.

[0009] The MLP layer is configured to splice and fuse the multi-level spatio-temporal features and the noise segmentation mask to obtain fused features.

[0010] The denoising decoder is configured to perform reverse denoising processing on the fused features to output a predicted segmentation mask of the same dimension as the last polyp image frame, determine a predicted segmentation result according to the predicted segmentation mask, and determine a polyp position according to the predicted segmentation result.

[0011] According to some embodiments of the present application, the video polyp segmentation system based on the self-supervised multi-task diffusion model further includes a time reasoning module configured to average and aggregate the N polyp image frames along a time axis into the multi-scale time features, and input the multi-scale time features into a reconstruction decoder to predict a next polyp image frame to obtain a target polyp image frame.

[0012] According to some embodiments of the present application, the video polyp segmentation system based on the self-supervised multi-task diffusion model further includes an adversarial self-supervised module including a generator and a discriminator. The generator is the time reasoning module, and the discriminator is configured to distinguish the target polyp image frame and a real polyp image frame, determine an adversarial loss according to the target polyp image frame and the real polyp image frame, and optimize the discriminator by minimizing the adversarial loss.

[0013] In a second aspect, the embodiments of the present application provide a video polyp segmentation method based on a self-supervised multi-task diffusion model, applied to the video polyp segmentation system of the technical solution of the first aspect. The method includes:

[0014] Obtaining a polyp video, and performing dataset labeling processing on the polyp video to obtain N polyp image frames and a segmentation mask corresponding to each of the polyp image frames;

[0015] Encoding the segmentation mask corresponding to the last polyp image frame to obtain a segmentation mask of a hidden state sequence, and performing forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask;

[0016] Taking the last polyp image frame as an input of an image encoder to generate multi-scale spatial features, taking the N polyp image frames as an input of a time sequence encoder to generate multi-scale temporal features, and integrating and inputting the multi-scale spatial features and the multi-scale temporal features into a feature pyramid network to obtain multi-level spatio-temporal features;

[0017] Splicing and fusing the multi-level spatio-temporal features and the noise segmentation mask to obtain fused features;

[0018] Performing reverse denoising processing on the fused features to output a predicted segmentation mask of the same dimension as the last polyp image frame, determining a predicted segmentation result according to the predicted segmentation mask, and determining a polyp position according to the predicted segmentation result.

[0019] According to some embodiments of the present application, the N polyp image frames are averaged and summarized along a time axis into the multi-scale temporal features;

[0020] Inputting the multi-scale temporal features into a reconstruction decoder to predict a next polyp image frame to obtain a target polyp image frame.

[0021] According to some embodiments of the present application, the video polyp segmentation based on a self-supervised task diffusion model further includes an adversarial self-supervised module, the adversarial self-supervised module includes a generator and a discriminator, the generator is the time reasoning module, and the method further includes:

[0022] Distinguishing the target polyp image frame from a real frame polyp image frame,

[0023] Determining an adversarial loss according to the target polyp image frame and the real frame polyp image frame;

[0024] Optimizing the discriminator by minimizing the adversarial loss.

[0025] According to some embodiments of the present application, after the splicing and fusing of the multi-level spatio-temporal features and the noise segmentation mask to obtain the fused features, the method further includes:

[0026] Performing reverse denoising processing on the fused features to output a classification score and a bounding box;

[0027] determine a class of the target polyp image frame according to the output classification score and the bounding box;

[0028] determine a loss of the multi-task diffusion model according to the classification score and the bounding box;

[0029] determine a total loss of the self-supervised task diffusion model based video polyp segmentation system according to the loss of the task diffusion model and the loss of the adversarial self-supervised module.

[0030] According to some embodiments of the present application, the forward noise processing on the predicted segmentation mask of the hidden state sequence comprises:

[0031] introducing Gaussian noise;

[0032] performing forward noise processing on the predicted segmentation mask of the hidden state sequence by the Gaussian noise to obtain a probability distribution of the noise segmentation mask;

[0033] wherein the probability distribution of the noise segmentation mask is represented by the following formula:

[0034]

[0035] wherein I is an identity matrix, z0 is a segmentation mask corresponding to an initial polyp image frame of the input, z t is a noise segmentation mask obtained after t-step noise addition, is a Gaussian distribution, is a mean vector, is a covariance matrix, q(z t |z0) represents a conditional probability distribution of the state noise segmentation mask z t at time t under the condition that the segmentation mask z0 corresponding to the initial polyp image frame of the input is given.

[0036] According to some embodiments of the present application, the reverse denoising processing on the fused feature comprises:

[0037] introducing a CNN-based denoising head, wherein the denoising head is a mapping function f θ (z t , x, t);

[0038] under the guidance of the condition x, the segmentation mask corresponding to the initial polyp image frame of the input is predicted from the noise segmentation mask obtained after t-step noise addition by minimizing the objective function, to obtain a predicted segmentation mask;

[0039] The predicted segmentation mask is reconstructed from the Gaussian noise by the denoising head conditioned on x to obtain a predicted segmentation mask with the same dimension as the last frame polyp image frame, and the predicted segmentation mask is output in a Markov manner, and the predicted segmentation mask is represented by the following formula:

[0040]

[0041] wherein p θ is a parameterized probability distribution, and θ is a parameter of the denoising head f θ , is a segmentation mask estimated value of the hidden state sequence from time 0 to T-1, is a segmentation mask estimated value of the hidden state at time t, T is the length of the hidden state sequence, and p(z T ) is an initial probability of the hidden state at time T, is a probability of the hidden state at time T-1 given the noisy noise segmentation mask z T and the condition x, is a hidden state transition probability at each time point t from time 1 to T-1, and the condition is given the hidden state at the previous time point t+1 and the condition x.

[0042] In a third aspect, the embodiments of the present application provide a controller, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to perform the method of the technical solution of the second aspect.

[0043] The video polyp segmentation system and method of the self-supervised multi-task diffusion model provided by the embodiments of the present application at least have one of the following advantages or beneficial effects: the conditional diffusion model is applied in the video polyp segmentation system, the last polyp image frame corresponding to the segmentation mask obtained by the acquisition module is encoded and noise processed by using a Markov random process in a hidden state sequence or a hidden space to obtain a noise segmentation mask, and multi-scale features of the polyp image frame are extracted to obtain a segmentation mask of the same resolution for the current frame polyp image of the polyp video; the multi-task diffusion model is introduced into the video polyp segmentation system, and the polyp classification and polyp detection tasks are performed on the frame polyp image frame by the multi-task diffusion model, so that the high-level semantic information of the object in the polyp image frame can be effectively utilized to provide context and discrimination information for the segmentation task, improve the discrimination and generalization ability of the multi-task diffusion model, and improve the accuracy of polyp recognition. The last polyp image frame is taken as the input of the image encoder to generate multi-scale spatial features, and N polyp image frames are taken as the input of the time sequence encoder to generate multi-scale temporal features. The multi-level spatio-temporal features and the noise segmentation mask are spliced and fused by using the MLP layer, and the fused features are denoised by the denoising decoder to obtain a predicted segmentation mask of the same dimension as the last polyp image frame. The predicted segmentation result is obtained through the predicted segmentation mask, and the polyp position is determined according to the predicted segmentation result. The temporal information related to the front and rear frames in the polyp video is utilized to assist in positioning the polyp position in a complex environment, realize accurate positioning of the polyp position, improve the accuracy of polyp diagnosis, and improve the accuracy of polyp segmentation.

[0044] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and attained by the structure particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a structural schematic diagram of a video polyp segmentation system based on a self-supervised multi-task diffusion model provided by the embodiments of the present application;

[0046] Figure 2 is a structural schematic diagram of another video polyp segmentation system based on a self-supervised multi-task diffusion model provided by the embodiments of the present application;

[0047] Figure 3 is a flowchart of a video polyp segmentation method based on a self-supervised multi-task diffusion model provided by the embodiments of the present application;

[0048] Figure 4is a flowchart of a method for reconstructing a target polyp image frame provided by an embodiment of the present application;

[0049] Figure 5 is a flowchart of a method for optimizing a discriminator provided by an embodiment of the present application;

[0050] Figure 6 is a flowchart of a method for determining the total loss of a video polyp segmentation system based on a self-supervised task diffusion model provided by an embodiment of the present application;

[0051] Figure 7 is a structural schematic diagram of a controller provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various embodiments. At the same time, the steps or actions in the method description can also be sequentially changed or adjusted in a manner obvious to those skilled in the art. Therefore, the order in the specification and drawings is only for clear description of a certain embodiment and does not mean a necessary order, unless otherwise stated that a certain order must be followed.

[0053] In the description of the present application, one or more is meant to be one or more, more than two is meant to be two or more, greater than, less than, more than, etc. are understood to not include the number, above, below, etc. are understood to include the number. If it is described as first, second, etc., it is only used to distinguish the technical features for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the sequence of indicated technical features.

[0054] In this paper, the serial numbers of components themselves, such as "first", "second", etc., are only used to distinguish the described objects, and do not have any order or technical meaning. Unless otherwise specified, the "connection" and "coupling" in the present application include direct and indirect connection (coupling).

[0055] Colorectal cancer has become the second leading cause of cancer-related deaths worldwide, accounting for about 10% of all cancer cases, and colonoscopy plays a crucial role in detecting polyps and reducing the incidence and mortality of colorectal cancer. In the related art, most polyp segmentation work only trains and evaluates the model on static images, ignoring the additional information for polyp positioning brought by the time sequence of videos, and not fully utilizing the time sequence information between endoscopic video frames. In the video polyp segmentation task, it is obviously not enough to only focus on independent static images. In the video polyp image tracking and segmentation task, it is necessary to first pre-train on a large number of static images and then fine-tune on video images. This training strategy requires a large amount of high-quality labeling, but the current video polyp data size is still small. At the same time, due to the fuzzy boundary of polyps and the similarity with the background tissue, even skilled clinicians may not be able to agree on the labeling of consecutive frames. Finally, the current open-source polyp dataset belongs to a sparse sequence, and the changes between some adjacent frames are large. Although the endoscopic video focuses on the same polyp tissue, due to the different camera angles or light, the context environment in which the polyp is located, such as the cavity, the highlight, the mucosal tissue, etc. will change, which may affect the prediction results of adjacent frames, and thus affect the prediction results of polyp segmentation, reducing the accuracy of polyp segmentation diagnosis.

[0056] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0057] Referring to Figure 1 , as shown in Figure 1The application provides a video polyp segmentation system based on a self-supervised multi-task diffusion model, and a structure diagram of the video polyp segmentation system based on the self-supervised multi-task diffusion model is shown in the figure. The video polyp segmentation system based on the self-supervised multi-task diffusion model comprises a processing module, a conditional diffusion model, a multi-task diffusion model, an MLP layer and a denoising decoder. The processing module is used for acquiring a polyp video, performing dataset labeling processing on the polyp video, and obtaining N polyp image frames and a segmentation mask corresponding to each polyp image frame. The conditional diffusion model is used for performing encoding processing on the segmentation mask corresponding to the last polyp image frame, obtaining a segmentation mask of a hidden state sequence, performing forward noise processing on the segmentation mask of the hidden state sequence, and obtaining a noise segmentation mask. The multi-task diffusion model comprises an image encoder and a time sequence encoder. The multi-task diffusion model is used for taking the last polyp image frame as an input of the image encoder to generate multi-scale spatial features, taking the N polyp image frames as an input of the time sequence encoder to generate multi-scale time features, and further used for integrating the multi-scale spatial features and the multi-scale time features and inputting the multi-scale spatial features and the multi-scale time features to a feature pyramid network to obtain multi-level space-time features. The MLP layer is used for splicing and fusing the multi-level space-time features and the noise segmentation mask to obtain fused features. The denoising decoder is used for performing reverse denoising processing on the fused features, outputting a predicted segmentation mask with the same dimension as the last polyp image frame, determining a predicted segmentation result according to the predicted segmentation mask, and determining a polyp position according to the predicted segmentation result.

[0058] The acquisition module obtains N polyp image frames and the segmentation mask corresponding to each polyp image frame. In this embodiment, the video polyp segmentation system applies a conditional diffusion model. Using a Markov random process, the segmentation mask corresponding to the last polyp image frame obtained by the acquisition module is encoded in the hidden state sequence or hidden space, and noise is added to obtain a noisy segmentation mask. At the same time, multi-scale features of the polyp image frames are extracted and used to obtain a segmentation mask of the same resolution for the current polyp image in the polyp video. A multi-task diffusion model is introduced into the video polyp segmentation system. By performing polyp classification and polyp detection tasks on the polyp image frames through the multi-task diffusion model, the high-level semantic information of objects in the polyp image frames can be effectively utilized, thereby providing context and discriminative information for the segmentation task, improving the discrimination and generalization ability of the multi-task diffusion model, and improving the accuracy of polyp identification. The last polyp image frame is used as input to an image encoder to generate multi-scale spatial features. Simultaneously, N polyp image frames are used as input to a temporal encoder to generate multi-scale temporal features. These multi-scale spatial and temporal features are then integrated and input into a feature pyramid network to obtain multi-level spatiotemporal features. An MLP layer is used to concatenate and fuse the multi-level spatiotemporal features and a noise segmentation mask. A denoising decoder is then used to denoise the fused features, resulting in a predicted segmentation mask with the same dimensions as the last polyp image frame. This predicted segmentation mask is used to obtain the predicted segmentation result, which is then used to determine the polyp location. By utilizing the temporal information related to preceding and following frames in the polyp video, the polyp location can be accurately located in complex environments, improving the accuracy of polyp diagnosis and segmentation.

[0059] Reference Figure 1 and Figure 2 As shown, in some embodiments of this application, the video polyp segmentation system based on the self-supervised multi-task diffusion model further includes a temporal inference module. The temporal inference module is used to average N polyp image frames along the time axis and summarize them into multi-scale temporal features. The multi-scale temporal features are input into the reconstruction decoder to predict the next polyp image frame and obtain the target polyp image frame.

[0060] The introduction of a time reasoning module in the video polyp segmentation system can capture dynamic appearance and maintain temporal continuity when segmenting polyps. A generative adversarial self-supervised strategy is used to reconstruct the current target polyp image frame from the temporal information of previous polyp image frames. The polyp image frames are averaged along the time axis and summarized into multi-scale temporal features. At the same time, the temporal multi-scale features extracted during the reconstruction process are fused with the spatial multi-scale features of the conditional diffusion model and input into the reconstruction decoder to predict the next polyp image frame and obtain the target polyp image frame. The time reasoning module explores the temporal dependence in the polyp video sequence by reasoning about unseen target polyp image frames and learns spatiotemporal discriminative features to guide the discrimination of polyps with high camouflage.

[0061] In some embodiments of the present application, the video polyp segmentation system based on a self-supervised multi-task diffusion model further comprises an adversarial self-supervised module. The adversarial self-supervised module comprises a generator and a discriminator. The generator is a time reasoning module. The discriminator is used to distinguish between target polyp image frames and real polyp image frames to determine an adversarial loss according to the target polyp image frames and the real polyp image frames. The discriminator is optimized by minimizing the adversarial loss.

[0062] The introduction of the adversarial self-supervised module in the video polyp segmentation system can generate more realistic polyp image frames. In the process of reconstructing the current target polyp image frame from the temporal information of previous polyp image frames, a generative adversarial self-supervised function is introduced. The time reasoning module is used as the generator. The Inception3 network pre-trained on ImageNet is used as the discriminator. The discriminator is used to distinguish between target polyp image frames and real polyp image frames. An adversarial loss is determined by the target polyp image frames and the real polyp image frames. The discriminator is then optimized by minimizing the adversarial loss, thereby improving the discrimination and generalization ability of the video polyp segmentation system and the accuracy of polyp recognition.

[0063] Reference Figure 3 , Figure 3 is a flowchart of a video polyp segmentation method based on a self-supervised task diffusion model provided by an embodiment of the present application. The video polyp segmentation method based on a self-supervised task diffusion model includes but is not limited to steps S100 to S500. Specifically,

[0064] Step S100: Obtain a polyp video and perform dataset annotation processing on the polyp video to obtain N polyp image frames and a segmentation mask corresponding to each polyp image frame.

[0065] Step S200: Encode the segmentation mask corresponding to the last polyp image frame to obtain a segmentation mask of a hidden state sequence. Perform forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask.

[0066] Step S300: taking the last frame of polyp image frame as the input of the image encoder to generate the multi-scale spatial features, taking the N frames of polyp image frames as the input of the temporal encoder to generate the multi-scale temporal features, integrating the multi-scale spatial features and the multi-scale temporal features and inputting them into the feature pyramid network to obtain the multi-level spatio-temporal features;

[0067] Step S400: splicing and fusing the multi-level spatio-temporal features and the noise segmentation mask to obtain the fused features;

[0068] Step S500: performing reverse denoising processing on the fused features to output a predicted segmentation mask of the same dimension as the last frame of polyp image frame, determining a predicted segmentation result according to the predicted segmentation mask, and determining the polyp position according to the predicted segmentation result.

[0069] The video polyp segmentation method based on the self-supervised task diffusion model comprises the following steps: first, obtaining a polyp video, and performing dataset labeling processing on the polyp video to obtain N frames of polyp image frames and a segmentation mask corresponding to each frame of polyp image frame; using a conditional diffusion model, using a Markov random process to encode and process the segmentation mask corresponding to the last frame of polyp image frame obtained by the acquisition module in a hidden state sequence or a hidden space, and performing noise processing to obtain a noise segmentation mask; meanwhile, multi-scale features of the polyp image frames are extracted to obtain a segmentation mask of the same resolution for the current frame of polyp image of the polyp video. The last frame of polyp image frame is taken as the input of the image encoder to generate multi-scale spatial features, and the N frames of polyp image frames are taken as the input of the temporal encoder to generate multi-scale temporal features. Then, the multi-scale spatial features and the multi-scale temporal features are integrated and input into the feature pyramid network to obtain multi-level spatio-temporal features. The high-level semantic information of the objects in the polyp image frames is used to provide context and discriminative information for the segmentation task, improve the discrimination and generalization ability of the multi-task diffusion model, and improve the accuracy of polyp recognition. Then, the multi-level spatio-temporal features and the noise segmentation mask are spliced and fused by using an MLP layer, and the fused features are denoised by a denoising decoder to obtain a predicted segmentation mask of the same dimension as the last frame of polyp image frame. The predicted segmentation result is obtained through the predicted segmentation mask, and the polyp position is determined according to the predicted segmentation result. The temporal information related to the front and rear frames in the polyp video is used to assist in positioning the polyp position in a complex environment, realize accurate positioning of the polyp position, improve the accuracy of polyp diagnosis, and improve the accuracy of polyp segmentation.

[0070] Reference Figure 4 , Figure 4 is a flowchart of a method for reconstructing a target polyp image frame provided by an embodiment of the present application. The method for reconstructing a target polyp image frame comprises but is not limited to steps S600 to S610, specifically,

[0071] Step S600: average the N frames of polyp image frames along the time axis and aggregate into multi-scale temporal features;

[0072] Step S610: input the multi-scale temporal features into the reconstruction decoder to predict the next frame of polyp image frame, to obtain the target polyp image frame.

[0073] In some embodiments of the present application, in order to capture the dynamic appearance and motion cues in the colonoscopy video, a time reasoning module is introduced in the video polyp segmentation system. It can be understood that through a self-supervised strategy, the target polyp image frame I i-δ is reconstructed from the previous N frames of polyp image frames {I i-1 , …, I i , then the previous N frames of polyp image frames are averaged along the time axis and aggregated into multi-scale temporal features, and the multi-scale temporal features are input into the CNN-based reconstruction decoder composed of MLP layers. The reconstruction decoder receives and integrates the multi-scale temporal features to predict the next frame of polyp image frame to obtain the target polyp image frame I i . Through the time reasoning module, the unseen target polyp image frame is reasoned, the temporal dependency is explored from the polyp video sequence, and the spatio-temporal discriminative features are learned, which provides guidance for discriminating and segmenting highly camouflaged polyps.

[0074] Referring to Figure 5 , Figure 5 is a flowchart of a method for optimizing a discriminator provided by an embodiment of the present application. The method for optimizing the discriminator includes but is not limited to steps S700 to S720, specifically,

[0075] Step S700: distinguish the target polyp image frame and the real frame polyp image frame;

[0076] Step S710: determine the adversarial loss according to the target polyp image frame and the real frame polyp image frame;

[0077] Step S720: optimize the discriminator by minimizing the adversarial loss.

[0078] The video polyp segmentation system includes an adversarial self-supervised module. By introducing the adversarial self-supervised module, more realistic frame polyp image frames can be generated. The adversarial self-supervised module includes a generator and a discriminator. The time reasoning module is used as the generator, and the Inception3 network pre-trained on ImageNet is used as the discriminator. The discriminator is used to distinguish the target polyp image frame and the real frame polyp image frame. The adversarial loss is determined through the target polyp image frame and the real frame polyp image frame, and then the discriminator is optimized by minimizing the adversarial loss, thereby improving the discrimination and generalization ability of the video polyp segmentation system, and improving the accuracy of polyp recognition.

[0079] It can be understood that, in order to generate more realistic frame polyp image frames, a generative adversarial self-supervised function is introduced in the process of reconstructing the current target polyp image frame from the time sequence information of the previous polyp image frame.

[0080] The adversarial loss is determined according to the target polyp image frame and the real frame polyp image frame, and is expressed by the following formula:

[0081]

[0082] Wherein, is the adversarial loss, is the target polyp image frame, I i is the real frame polyp image frame, is the discriminator.

[0083] For fixed parameters, the preset generator can deceive the discriminator to generate images more like real frame polyp image frames:

[0084]

[0085] Wherein, is the loss of the generator, is the mean square error loss of the target polyp image frame and the real frame polyp image frame, is the target polyp image frame, I i is the real frame polyp image frame, λ adv is a balance hyperparameter, and λ adv is set to 0.001 to balance the mean square error reconstruction term and the adversarial term, and it should be noted that the adversarial loss is only used to optimize the discriminator and is irrelevant to the multi-task diffusion model.

[0086] Referring to Figure 6 , Figure 6 is a flowchart of a method for determining the total loss of a video polyp segmentation system based on a self-supervised task diffusion model provided by the embodiments of the application. The method for determining the total loss of the video polyp segmentation system based on the self-supervised task diffusion model includes but is not limited to steps S800 to S830, and specifically,

[0087] Step S800: performing reverse denoising processing on the fused features to output classification scores and bounding boxes;

[0088] Step S810: determining the category of the target polyp image frame according to the classification scores and the bounding boxes;

[0089] Step S820: determining the loss of the multi-task diffusion model according to the classification scores and the bounding boxes;

[0090] Step S830: determining the total loss of the video polyp segmentation system based on the self-supervised task diffusion model according to the loss of the task diffusion model and the loss of the adversarial self-supervised module.

[0091] After splicing and fusing the multi-level spatio-temporal features and the noise segmentation mask, the method further comprises: performing reverse denoising processing on the fused features to output classification scores and a bounding box The classification scores and the bounding box are used for instance detection of the target polyp image frame to determine the category of the target polyp image frame.

[0092] The segmentation loss composed of the pixel cross-entropy loss, the mean square error and the IoU loss is applied The loss of the multi-task diffusion model is optimized by the cross-entropy loss. Therefore, the loss objective function of the multi-task diffusion model can be expressed as:

[0093]

[0094] wherein, is the loss of the multi-task diffusion model, λ seg ,λ cls ,λ det are balance hyperparameters of the three tasks, and λ seg is set to 0.5, λ cls is set to 0.05, and λ det is set to 0.2. is the classification score, Y cls is the category of the polyp in the target polyp image frame I i , including low-grade adenoma, high-grade adenoma, hyperplastic polyp, traditional serrated adenoma, sessile serrated lesion and invasive carcinoma, is the predicted bounding box, Y box is the bounding box (x c , y c , w, h) of the target lesion of the target polyp image frame, is the predicted pixel-level segmentation mask, Y seg is the segmentation mask of the target polyp image frame; is the pixel-level segmentation loss, the pixel-level segmentation loss is composed of the Dice loss loss the binary cross-entropy loss and the mean square error loss , is the cross-entropy loss for the category, is the square root loss for the bounding box.

[0095] By introducing the adversarial self-supervised module, more realistic frame polyp image frames can be generated. The adversarial self-supervised module greatly enhances the discriminability and generalization ability of the conditional diffusion model, reduces the class imbalance caused by the scarcity of lesions in polyp pixel-level segmentation, and improves the accuracy of polyp recognition.

[0096] Then, the loss of the multi-task diffusion model is determined according to the classification score and the bounding box, and the total loss of the video polyp segmentation system based on the self-supervised task diffusion model is determined according to the loss of the task diffusion model and the loss of the adversarial self-supervised module.

[0097] By merging the loss of the multi-task diffusion model MDM and the loss of the temporal reasoning module TRM, the total loss of the video polyp segmentation system based on the self-supervised multi-task diffusion model is calculated As follows:

[0098]

[0099] Wherein, is the total loss of the video polyp segmentation system based on the self-supervised multi-task diffusion model, is the loss of the multi-task diffusion model, is the loss of the temporal reasoning module, and λ MDM and λ TRM are balance hyperparameters, and λ MDM is set to 0.25, and λ TRM is set to 0.25.

[0100] In some embodiments of the present application, a conditional diffusion model is introduced to the predicted segmentation mask process. The conditional diffusion model belongs to a likelihood-based model inspired by non-equilibrium thermodynamics, which presets a forward noise process to perform forward noise processing on the predicted segmentation mask of the hidden state sequence to obtain a noise segmentation mask, including:

[0101] First, Gaussian noise is introduced; the predicted segmentation mask of the hidden state sequence is processed by Gaussian noise to obtain the probability distribution of the noise segmentation mask;

[0102] Wherein, the probability distribution of the noise segmentation mask is represented by the following formula:

[0103]

[0104] Wherein, I is an identity matrix, z0 is the segmentation mask corresponding to the input initial polyp image frame, and z t is the noise segmentation mask obtained after t-step noise addition, is a Gaussian distribution, is a mean vector, is a covariance matrix, and q(z t|z0) represents the state noise segmentation mask z at time t, given the segmentation mask z0 corresponding to the initial polyp image frame. t The conditional probability distribution.

[0105] Then, in order to predict a segmentation mask of the same dimension as the last polyp image frame from the noisy segmentation mask, this application introduces a CNN-based denoising head f θ The noise reduction head is a mapping function f θ (z t (x,t), through the noise reduction head f θ Used for inverse denoising of the fused features, outputting a predicted segmentation mask with the same dimensions as the last polyp image frame, including:

[0106] After introducing a CNN-based denoising head, guided by condition x, the segmentation mask corresponding to the initial polyp image frame is predicted from the noise segmentation mask obtained after t steps of noise addition by minimizing the objective function, thus obtaining the predicted segmentation mask. The predicted segmentation mask is then reconstructed from Gaussian noise using the denoising head with x as the condition, resulting in a predicted segmentation mask with the same dimensions as the last polyp image frame. The predicted segmentation mask is then output in a Markov manner and is expressed by the following formula:

[0107]

[0108] Where, p θ Let f be a parameterized probability distribution, and θ be the noise reduction head. θ The parameters, This is the estimated segmentation mask value for the hidden state sequence from time 0 to T-1. p(z) is the segmentation mask estimate of the hidden state at time t, where T is the length of the hidden state sequence. T Let be the initial probability of the hidden state at time T. For a given noise segmentation mask z after adding noise T Given condition x, what is the probability of the hidden state at time T-1? Let x be the hidden state transition probability at each time point t from time 1 to T-1, given the hidden state and condition x at the previous time point t+1.

[0109] In the video polyp segmentation task, a conditional diffusion model is introduced. Using a Markov random process, the segmentation mask is denoised and denoised in the hidden state sequence. At the same time, multi-scale features of the fused image are extracted and used to obtain a segmentation mask of the same resolution for the polyp image of the current frame of the polyp video.

[0110] In some embodiments of the present application, the video polyp segmentation method based on the self-supervised task diffusion model further comprises the following pre-work: polyp video data collection, data set labeling, model training, model testing and ablation experiment, etc. Specifically:

[0111] The polyp video data collection includes: by a professional physician of the department of gastroenterology, video collection of polyps in the diagnosis process, the video collection object is 10 different patients, and a video with a length of about 1 hour is collected in the process of surgery of each patient, and a total of 10 videos with a length of about 1 hour are collected. Then, a professional surgeon cuts out 10 short videos with a length of 15 to 20 seconds from each video with a length of 1 hour, and needs to ensure that each frame in each short video contains a polyp region, and finally a total of 100 surgical videos with a length of 3 to 11 seconds are obtained.

[0112] The data set labeling includes: interval sampling of all short videos with a length of 3 to 11 seconds, extracting each frame in the video and saving it as a JPG format image, a total of 100 case videos containing polyps are obtained, with a real-time frame frequency of 30 frames / second, and after interval sampling, 49136 images containing polyps are obtained, of which 14176 frames are used for training the model and 24736 frames are used for testing. Then, the images are labeled to mark the boundaries of the polyps and saved as PNG format images.

[0113] The model training includes:

[0114] The model is trained using 14176 frames of polyp video frames in 100 cases. The input is the segmentation mask of the current N frame polyp image frame and the last frame polyp image frame, the time reasoning module extracts multi-scale temporal features from the N frame polyp image frame, and the image encoder extracts multi-scale spatial features from the last frame polyp image frame. The multi-scale temporal features are averaged in the time dimension, and the multi-scale spatial features are added pixel by pixel in the corresponding scale to obtain multi-level spatio-temporal features. Then, the multi-level spatio-temporal features of different sizes are combined into a scale of spatio-temporal features as the condition of the multi-task diffusion model denoising process. The last frame polyp image frame is encoded into a hidden state sequence to obtain a segmentation mask of the hidden state sequence, which is randomly sampled and added noise after the forward process to obtain a noise segmentation mask. The noise segmentation mask and the multi-level spatio-temporal features are spliced and input into the denoising decoder, and the output after the reverse process denoising is a predicted segmentation mask with the same dimension as the last frame polyp image frame.

[0115] The model testing includes:

[0116] The test data set is divided into two sub-test sets, EASY and HARD, according to the distribution of polyp video data. Each sub-test set is further divided into Seen and Unseen cases. Seen represents visible cases, and Unseen represents invisible cases. Seen divides the polyp image frames in a case video into two mutually exclusive parts, one for training and the other for testing. There are a total of four mutually exclusive test sets, EASY-Seen, EASY-Unseen, HARD-Seen, and HARD-Unseen. For each test set, the current N frames of polyp image frames are input into the trained model, and the model outputs a predicted segmentation mask of the same dimension as the last frame of polyp image frame.

[0117] The ablation experiment includes:

[0118] To verify the effectiveness of each model and module in the video polyp segmentation system based on the self-supervised multi-task diffusion model, such as the processing module, the conditional diffusion model, the multi-task diffusion model, the MLP layer, and the denoising decoder, an ablation experiment is conducted on each model and module to avoid parameter redundancy and improve the accuracy of polyp diagnosis of the self-supervised multi-task diffusion model video polyp segmentation system.

[0119] Referring to Figure 7 as shown, Figure 7Fig. 1 is a structural schematic diagram of a controller 1000 provided by an embodiment of the present application, which comprises a processor 1001, which can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the method provided by the embodiment of the present application; a memory 1002, which can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 1002 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the relevant program codes are saved in the memory 1002 and are called and executed by the processor 1001 to implement the embodiments of the present application; an input / output interface 1003, which is used to implement information input and output; a communication interface 1004, which is used to implement the communication interaction between the device and other devices, and can realize communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.); and a bus, which is used to transmit information between various components (for example, the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004) of the device. The processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 are communicatively connected to each other inside the device through the bus.

[0120] As will be appreciated by one of ordinary skill in the art, all or some of the steps, systems, etc. in the above-disclosed methods can be embodied in software, firmware, hardware, and / or suitable combinations thereof. Some or all of the physical components can be implemented with software executed by a processor, such as a central processing unit, a digital signal processor, or microprocessor, or can be implemented with hardware, or can be implemented with an integrated circuit, such as an application- specific integrated circuit. Such software can be distributed on computer readable media, which can comprise computer-readable storage media (or non-transitory media), and communication media (or transitory media). Computer-readable storage media, as used herein, includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, as will be appreciated by one skilled in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves or other transport mechanisms, and includes any information delivery media.

[0121] The above description is that of the preferred embodiments of the present application. Various equivalents substitutions and modifications can be conceived by those skilled in the art without departing from the spirit and scope of the present application, and these equivalents substitutions and modifications are included in the scope of the claims.

Claims

1. A video polyp segmentation system based on self-supervised multi-task diffusion model, characterized in that, Comprise: A processing module for obtaining a polyp video and performing dataset annotation processing on the polyp video to obtain N frames of polyp image frames and a segmentation mask corresponding to each frame of the polyp image frames; wherein N is a positive integer; A conditional diffusion model for performing encoding processing on the segmentation mask corresponding to the last frame of polyp image frames to obtain a segmentation mask of a hidden state sequence, and performing forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask; A multi-task diffusion model comprising an image encoder and a time sequence encoder, the multi-task diffusion model being configured to take the last frame of polyp image frames as input of the image encoder to generate multi-scale spatial features, take the N frames of polyp image frames as input of the time sequence encoder to generate multi-scale temporal features, and integrate and input the multi-scale spatial features and the multi-scale temporal features into a feature pyramid network to obtain multi-level spatio-temporal features; An MLP layer for splicing and fusing the multi-level spatio-temporal features and the noise segmentation mask to obtain fused features; A denoising decoder for performing reverse denoising processing on the fused features to output a predicted segmentation mask of the same dimension as the last frame of polyp image frames, determining a predicted segmentation result according to the predicted segmentation mask, and determining a polyp position according to the predicted segmentation result; Wherein, the forward noise processing on the segmentation mask of the hidden state sequence to obtain the noise segmentation mask comprises: Introducing Gaussian noise; Performing forward noise processing on the segmentation mask of the hidden state sequence through the Gaussian noise to obtain a probability distribution of the noise segmentation mask; Wherein, the probability distribution of the noise segmentation mask is represented by the following formula: ， wherein, is an identity matrix, is a segmentation mask corresponding to the input initial polyp image frame, is a segmentation mask obtained after adding noise to the segmentation mask, is a noise segmentation mask, is a Gaussian distribution, is a mean vector, is a covariance matrix, denotes the conditional probability distribution of the state noise segmentation mask at time given the segmentation mask corresponding to the input initial polyp image frame.

2. The video polyp segmentation system based on self-supervised multi-task diffusion model according to claim 1, wherein, Further comprising: A time reasoning module for averaging and summarizing the N frames of polyp image frames along a time axis into the multi-scale temporal features, and inputting the multi-scale temporal features into a reconstruction decoder to predict a next frame of polyp image frames to obtain a target polyp image frame.

3. The video polyp segmentation system based on self-supervised multi-task diffusion model according to claim 2, wherein, Further comprising: An adversarial self-supervised module comprising the time reasoning module and a discriminator, the discriminator being configured to distinguish between the target polyp image frame and a real frame of polyp image frame, and determine an adversarial loss according to the target polyp image frame and the real frame of polyp image frame; Optimizing the discriminator by minimizing the adversarial loss.

4. The method for video polyp segmentation based on a self-supervised task diffusion model, characterized in that, The method is applied to the video polyp segmentation system of any one of claims 1 to 3, and the method comprises: Obtaining a polyp video and performing dataset annotation processing on the polyp video to obtain N frames of polyp image frames and a segmentation mask corresponding to each frame of the polyp image frames; wherein N is a positive integer; Performing encoding processing on the segmentation mask corresponding to the last frame of polyp image frames to obtain a segmentation mask of a hidden state sequence, and performing forward noise processing on the segmentation mask of the hidden state sequence to obtain a noise segmentation mask; Taking the last frame of polyp image frames as input of the image encoder to generate multi-scale spatial features, taking the N frames of polyp image frames as input of the time sequence encoder to generate multi-scale temporal features, and integrating and inputting the multi-scale spatial features and the multi-scale temporal features into a feature pyramid network to obtain multi-level spatio-temporal features; The multi-level spatiotemporal features and the noise segmentation mask are spliced and fused to obtain fused features; The fused features are reversely denoised to output a predicted segmentation mask of the same dimension as the last frame of polyp image frames, a predicted segmentation result is determined according to the predicted segmentation mask, and a polyp position is determined according to the predicted segmentation result; The segmentation mask of the hidden state sequence is forwardly denoised to obtain a noise segmentation mask, including: Gaussian noise is introduced; The segmentation mask of the hidden state sequence is forwardly denoised through the Gaussian noise to obtain a probability distribution of the noise segmentation mask; The probability distribution of the noise segmentation mask is represented by the following formula: ， wherein, is the identity matrix, is the input initial polyp image frame corresponding segmentation mask, is the segmentation mask after step added noise, is the Gaussian distribution, is the mean vector, is the covariance matrix, denotes the conditional probability distribution of the state noise segmentation mask at time given the initial polyp image frame corresponding segmentation mask .

5. The method of claim 4, wherein the self-supervised task diffusion model-based video polyp segmentation method is characterized by, Further comprising: The N frames of polyp image frames are averaged along the time axis and summarized into the multi-scale temporal features; The multi-scale temporal features are input into a reconstruction decoder to predict a next frame of polyp image frames to obtain a target polyp image frame.

6. The method of claim 5, wherein the self-supervised task diffusion model-based video polyp segmentation method is characterized by, The video polyp segmentation system based on a self-supervised task diffusion model further comprises an adversarial self-supervised module, the adversarial self-supervised module comprises a temporal reasoning module and a discriminator, and the method further comprises: distinguishing the target polyp image frame from a real frame polyp image frame, determining an adversarial loss according to the target polyp image frame and the real frame polyp image frame; optimizing the discriminator by minimizing the adversarial loss.

7. The method of claim 6, wherein the self-supervised task diffusion model-based video polyp segmentation method is characterized by, After the multi-level spatiotemporal features and the noise segmentation mask are spliced and fused to obtain fused features, further comprising: The fused features are reversely denoised to output a classification score and a bounding box; determining the category of the target polyp image frame according to the output classification score and the bounding box; determining a loss of a multi-task diffusion model according to the classification score and the bounding box; determining a total loss of the video polyp segmentation system based on the self-supervised task diffusion model according to the loss of the task diffusion model and the loss of the adversarial self-supervised module.

8. The method of claim 4, wherein the self-supervised task diffusion model-based video polyp segmentation method is characterized by, The fused features are reversely denoised to output a predicted segmentation mask of the same dimension as the last frame of polyp image frames, including: A CNN-based denoising head is introduced, which is a mapping function ; Under conditions Under the guidance of this method, the goal is to achieve the desired outcome by minimizing the objective function. The segmentation mask corresponding to the initial polyp image frame is predicted from the noise segmentation mask obtained after the noise addition step, and the predicted segmentation mask is obtained. The predicted segmentation mask is used as A conditional denoising head reconstructs from the Gaussian noise to obtain a predicted segmentation mask with the same dimensions as the last polyp image frame, and outputs the predicted segmentation mask in a Markovian manner. The predicted segmentation mask is expressed by the following formula: ； in, For parameterized probability distributions, For noise reduction head The parameters, From time 0 to The estimated segmentation mask value of the hidden state sequence. In time The estimated value of the segmentation mask for the hidden state. The length of the hidden state sequence. Let be the initial probability of the hidden state at time T. For a given noise segmentation mask after adding noise and conditions In this case, time The probability of the hidden state. From time 1 to Each time point The hidden state transition probability, the condition being given a previous time point. Hidden states and conditions .

9. A controller characterized by comprising: Further comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the computer program to perform the method of any one of claims 4 to 8.

Images