An unsupervised medical video smoke removal method and program product

Abstract

The present invention belongs to the technical field of image enhancement or restoration, and more specifically, to an unsupervised medical video desmog method and program product, the method comprising: first, preprocessing the data; second, inputting the data into a smoke detector to distinguish foggy frame images from non-fog frame images; then, building a CycleGAN model framework, using a set of unpaired foggy videos and foggy videos combined with a loss function to train the model, and obtaining a trained CycleGAN model; finally, using the CycleGAN model to generate foggy frame images from foggy frames in the video. The present invention solves the problem that unified processing will seriously affect the processing efficiency and the quality of non-fog images, and that smoke may block the field of view during medical surgery, affecting the smooth progress of the surgery and the difficulty in obtaining data sets.

Description

Technical Field

[0001] The present invention belongs to the technical field of image enhancement or restoration, and more specifically, relates to an unsupervised medical video smoke removal method and a program product. Background Art

[0002] With the continuous advancement of medical technology, robotic surgery is increasingly used in surgical operations. However, during robotic surgery, especially when using tools such as electroscalpels or lasers to cut, cauterize and coagulate tissues, a large amount of smoke is often produced. These smoke will not only blur the surgical field of view and affect the surgeon's observation and judgment of the surgical site, but may also interfere with the normal operation of the robot's visual system, thereby affecting the safety and effectiveness of the operation. Traditional smoke removal methods mainly rely on physical means, such as the use of smoke aspirators or smoke exhaust systems, but these methods often have limited effects and may increase the complexity and difficulty of operation of surgical equipment. In recent years, with the rapid development of deep learning technology, deep learning-based smoke removal methods have gradually become a research hotspot, but traditional deep learning-based smoke removal methods usually rely on supervised learning technology and require a large amount of labeled data to train the model. However, in the medical field, due to the fact that the operation is only performed once, it is extremely difficult to collect gold standard images of smoke-free and smoke-containing images simultaneously during the operation. It is understood that the current unsupervised video defogging methods are usually designed for natural scenes and are suitable for scenes where the entire video is foggy, and there is no design to distinguish between fog-free frames and foggy frames. During the operation, there is no fog for most of the time. Smoke only appears for a short time when the tissue is burned. If the entire video is dehazed, it will not only affect the processing efficiency, but also have an adverse effect on the fog-free frames, thus affecting the overall quality of the video. Therefore, it is of great practical significance to design an unsupervised medical video dehazing method that can autonomously distinguish foggy and fog-free frames and process them separately.

[0003] Chinese patent document CN117649362B discloses a laparoscopic image smoke removal method based on a conditional diffusion model. The method divides the video of laparoscopic surgery by frame number to form a data set in the form of pictures, performs smoke rendering on the obtained laparoscopic smoke-free image, synthesizes paired smoke-containing images, and obtains a synthetic data set containing smoke-free images and smoke-containing images; the smoke-free image is input into the conditional diffusion model for forward noise addition, and noise is continuously added until the smoke-free image is completely noised to obtain a series of noisy images; the smoke-containing image is input into the smoke perception module to obtain the concentration and position information of the smoke, and the neural network is trained, and the trained neural network is continuously used to reversely denoise the completely noisy image until a clear smoke-free image is output; the smoke removal model is optimized through a multi-loss function fusion strategy. This method relies on paired data and requires the existence of pairs of frames with and without smoke. It is extremely difficult to obtain such paired data during surgery. According to analysis of the surgical process, smoke will only appear for a short period of time during operations such as burning. The desmog method designed for natural scenes often cannot effectively distinguish between foggy frames and non-fog frames, resulting in the same degree of processing for all frames during the processing process. It is inefficient and ineffective, and is not suitable for medical video desmog tasks.

[0004] Chinese patent document CN116362999A discloses an image defogging method and an assisted driving system based on an unsupervised framework. The method uses a network framework of a regularized cyclic generative adversarial network, and realizes a new calculation method of a comprehensive loss function through adversarial loss, cyclic consistency loss, identity loss and color reconstruction loss, so that the fuzzy samples in the real foggy image can be fully utilized without relying on the real foggy image, and the fog-free image can be directly generated end-to-end in a cyclic consistent method, and it is conducive to improving the generalization ability of the real foggy image. However, the above patent is designed for natural scenes, and the natural scene is characterized by a long existence of haze. In surgery, the haze often appears suddenly and only exists for a few seconds. Therefore, if the above patent is applied to surgery, all frames of the entire video are uniformly processed without distinction, which will be inefficient and affect the quality of the fog-free frame, thereby affecting the quality of the entire video. Therefore, it is important to design an unsupervised medical defogging method that is specifically used for defogging during the surgical process, which can well distinguish between foggy frames and fog-free frames in the video, and process them separately, thereby improving the processing efficiency and quality of the entire video.

[0005] In view of this, the present invention designs an unsupervised medical video dehazing method, which can autonomously distinguish foggy frames and non-fog frames and process them separately. Summary of the invention

[0006] The present invention aims to overcome the defects of the above-mentioned prior art and provide an unsupervised medical video smoke removal method;

[0007] The invention also discloses a program product for realizing the smoke removal method.

[0008] The detailed technical scheme of the present invention is as follows:

[0009] An unsupervised medical video smoke removal method, characterized in that the method comprises the following steps:

[0010] S1, data preprocessing of medical video dataset;

[0011] S2, inputting the preprocessed data into the smoke detector to distinguish foggy images from non-fog images;

[0012] S3. Build a CycleGAN model framework, use a set of unpaired fog-free videos and foggy videos combined with a loss function to train the model, and obtain a trained CycleGAN model.

[0013] The model includes a generator G, a generator F, and a discriminator; the generator G is used to convert a foggy image into a fog-free image, the generator F is used to convert a fog-free image into a foggy image, the discriminator includes a discriminator Dx and a discriminator Dy, the discriminator Dx is used to distinguish between a real fog-free image and a generated fog-free image, and the discriminator Dy is used to distinguish between a real foggy image and a generated foggy image;

[0014] The loss function is a total loss function calculated based on adversarial loss, cycle consistency loss, smoke concentration coefficient supervision loss and time loss;

[0015] S4. Use the CycleGAN model to generate fog-free images from foggy images in the video.

[0016] Preferably, according to the present invention, the data preprocessing comprises the following steps:

[0017] S11. Data preparation: Decompose the medical video dataset containing foggy frames and non-fog frames into single frame images for frame-by-frame processing;

[0018] S12, data annotation: annotate all single-frame images to distinguish foggy frames from non-fog frames;

[0019] S13, data enhancement: rotate, flip, and scale the labeled data to form preprocessed data.

[0020] Preferably, according to the present invention, the smoke detector uses a pre-trained ResNet18 model to perform fine-tuning on pre-processed data to train a binary classification model to distinguish whether there is fog in the current frame.

[0021] Preferably, according to the present invention, the goal of the generator G is to generate the corresponding fog-free image from the foggy image and predict the smoke concentration of the foggy image for use by the subsequent discriminator; the generator G includes an encoder, a bottleneck layer and a decoder: the encoder gradually extracts image features through multi-layer convolution blocks and compresses the spatial information of the image into a feature map; the bottleneck layer contains multiple residual blocks for further extracting and enhancing features; the decoder gradually restores the image size through deconvolution and upsampling, and performs detail recovery in combination with the feature map of the encoder, and finally uses the convolution layer to generate the image after desmoking, and calculates the feature map of the multi-layer convolution block in the encoder through the adaptive average pooling layer, and obtains the smoke concentration coefficient to ensure the desmoking effect; the overall defogging process is expressed as:

[0022] C=G(H)(1)

[0023] In formula (1), C represents the generated haze-free image, G represents the generator G, and H represents the haze-free image.

[0024] Preferably according to the present invention, the method of calculating the feature map of the multi-layer convolution block in the encoder through the adaptive average pooling layer to obtain the smoke concentration coefficient specifically refers to calculating the feature map of the multi-layer convolution layer in the encoder through the adaptive average pooling layer, aggregating the global information of each feature map into a scalar, and then concatenating the pooled results in the channel dimension to obtain a tensor containing multi-layer features, and obtaining the output features through 1x1 convolution, and finally obtaining the final smoke concentration coefficient through activation function processing and linear transformation of the output features.

[0025] Preferably, according to the present invention, the goal of the generator F is to generate a corresponding foggy image from a given fog-free image. The generator F includes a smoke generator and an image refiner. The generator F first randomly samples the smoke concentration, and then inputs the fog-free image and the smoke concentration into the smoke generator to generate a rough foggy image with different smoke concentrations. The image refiner is then used to refine the rough foggy image to make it more consistent with the actual smoke distribution. Specifically, the smoke generator and the image refiner have the same structure, including an encoder, a bottleneck layer and a decoder. The encoder gradually extracts image features through multi-layer convolution blocks and compresses the spatial information of the image into a feature map. The bottleneck layer includes multiple residual blocks for further extracting and enhancing features. The decoder gradually restores the image size through deconvolution and upsampling, and restores details in combination with the feature map of the encoder, and finally uses the convolution layer to generate a foggy image and a refined image. The entire fog generation process is expressed as follows:

[0026] H=F(C)(2)

[0027] In formula (2), H represents the hazy image, F represents the generator F, and C represents the haze-free image.

[0028] Preferably, according to the present invention, the discriminator adopts a combination of a local discriminator guided by depth information and a global discriminator. After obtaining the smoke concentration coefficient in the generator G, a threshold is given. During the training process, the local discriminator only selects an area centered on a pixel with a threshold depth as the top for identification;

[0029] When training the generator G, the adversarial loss of the global discriminator and the local discriminator is used for optimization, while when training the generator F, only the adversarial loss of the global discriminator is used for optimization.

[0030] Preferably, according to the present invention, the loss function used in the training process is as follows:

[0031] There are two groups of adversarial losses, namely the adversarial loss between the generator G and the discriminator Dx and the adversarial loss between the generator F and the discriminator Dy; the adversarial loss ensures that the image generated by the generator can deceive the discriminator, making the generated image as realistic as possible;

[0032] The adversarial loss of the generator G and the discriminator Dx makes the generated haze-free image look like a real haze-free image, and the formula is as follows:

[0033] (3)

[0034] (4)

[0035] In formula (3) and (4), represents the adversarial loss of the discriminator Dx, represents the adversarial loss of the generator G, represents the average value of the respective errors, is a discriminator for judging whether a fog-free image is real or fake, Y is a real fog-free image, X is a real foggy image, and G(X) is the generated fog-free image;

[0036] The adversarial loss of the generator F and the discriminator Dy makes the generated foggy image look like a real foggy image, and the formula is as follows:

[0037] (5)

[0038] (6)

[0039] In formula (5) and (6), represents the adversarial loss of the discriminator Dy, represents the adversarial loss of the generator F, represents the average value of the respective errors, is a discriminator for determining whether a foggy image is real or fake. is the generated foggy image;

[0040] The cycle consistency loss restores an image to the original image after passing through two generators;

[0041] The cycle consistency loss formula is as follows:

[0042] (7)

[0043] In formula (7), represents the cycle consistency loss, X represents the real foggy image, Y represents the real fog-free image, G represents the generator G, and F represents the generator F;

[0044] The smoke concentration coefficient supervision loss formula is:

[0045] (8)

[0046] In formula (8), Represents the smoke concentration coefficient obtained by random sampling in the fog generation module, It represents the smoke concentration coefficient predicted in the defogging module. It is necessary to establish a constraint between the smoke concentration coefficient randomly sampled by the fogging module and the smoke concentration coefficient predicted by the defogging module, so that the defogging module can correctly predict the smoke concentration coefficient;

[0047] The time loss formula is:

[0048] (9)

[0049] In formula (9), the subscript t+1 represents the next frame of foggy or fog-free image generated by self-motion synthesis. and It indicates the pixel mask that guarantees that the pixel boundary will not be exceeded in the process of generating the next frame. Since the processing of video data needs to consider the problem of temporal consistency, the use of temporal loss for constraint can make the video smoother.

[0050] The total loss function formula is:

[0051] (10)

[0052] In formula (10), , and is the importance weight used to balance each loss term.

[0053] In another aspect of the present invention, a computer program product is provided, comprising computer instructions, which are used to implement the steps of the above-mentioned unsupervised medical video smoke removal method when executed by a processor.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention adopts a deep learning-based approach to design an unsupervised image dehazing method specifically for medical surgery. The method adopts an unsupervised learning approach to generate dehazed images by training CycleGAN. In particular, the present invention introduces a smoke detector to process foggy and non-fog images separately before the image enters the generator, solving the problem that unified processing will seriously affect processing efficiency and quality of non-fog images. Such a design is more in line with the needs of the surgical process, and at the same time solves the problem that smoke may block the field of view during medical surgery, affecting the smooth progress of the operation and the difficulty in obtaining data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of the cyclic adversarial network training method of the present invention.

[0057] Figure 2 It is a flow chart of an unsupervised medical video smoke removal method described in the present invention. DETAILED DESCRIPTION

[0058] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0059] Embodiment 1,

[0060] like Figure 2 As shown, an unsupervised medical video smoke removal method is characterized in that the method comprises:

[0061] S1, data preprocessing;

[0062] S2, input smoke detector to distinguish foggy frame images and non-fog frame images;

[0063] S3. Build a CycleGAN model framework, use a set of unpaired fog-free videos and foggy videos combined with a loss function to train the model, and obtain a trained CycleGAN model.

[0064] like Figure 1 As shown, the model includes a generator G, a generator F, a discriminator Dx and a discriminator Dy, the generator G is used to convert a foggy image into a fog-free image, the generator F is used to convert a fog-free image into a foggy image, the discriminator Dx is used to distinguish between a real fog-free image and a generated fog-free image, and the discriminator Dy is used to distinguish between a real foggy image and a generated foggy image;

[0065] The loss function is a total loss function calculated based on adversarial loss, cycle consistency damage, smoke concentration coefficient supervision loss and time loss;

[0066] S4. Use the CycleGAN model to generate fog-free frame images from the foggy frame images in the video.

[0067] The data preprocessing comprises the following steps:

[0068] S11. Data preparation: First, you need to prepare a medical video dataset containing frames with and without smoke, decompose the video into single-frame images for frame-by-frame processing, and divide it into training set, validation set, and test set;

[0069] S12, data labeling: label all frames and distinguish between frames with smoke and frames without smoke;

[0070] S13. Data enhancement: Perform data enhancement on the labeled data, such as rotation, flipping, and scaling.

[0071] The smoke detector uses a pre-trained ResNet model or a VGG model to perform fine-tuning on a labeled dataset to train a binary classification model to distinguish whether the current frame has fog for subsequent different processing. Specifically, the cross entropy loss function and the Adam optimizer are used to train the model.

[0072] The goal of the generator G is to generate the corresponding fog-free image from the foggy image and predict the smoke density of the foggy image for subsequent discriminator use. Specifically, the generator G consists of three parts: encoder, bottleneck layer and decoder. In the encoder part, the image features are gradually extracted through multi-layer convolution blocks, and the spatial information of the image is compressed into a feature map. The bottleneck layer contains multiple residual blocks for further extracting and enhancing features. The decoder part gradually restores the image size through deconvolution and upsampling, and combines the feature map of the encoder part to restore details. Finally, the convolution layer is used to generate the de-smoked image, and the feature maps of different layers are calculated through the adaptive average pooling layer to obtain the smoke density parameter to ensure the de-smoking effect. Specifically, the feature maps of the multi-layer convolution layers in the encoder are calculated through the adaptive average pooling layer, and the global information of each feature map is aggregated into a scalar. Then the pooled results are spliced ​​in the channel dimension to obtain a tensor containing multi-layer features, and the output features are obtained through 1x1 convolution. Finally, the output features are processed by activation function and linear transformation to obtain the final smoke density coefficient; the overall defogging process can be expressed as:

[0073] C=G(H)(1)

[0074] In formula (1), C represents the generated haze-free image, G represents the generator G, and H represents the haze-free image.

[0075] The goal of generator F is to generate the corresponding foggy image from a given fog-free image. Generator F includes a smoke generator and an image refiner. Generator F first randomly samples the smoke concentration, and then inputs the fog-free image and the smoke concentration into the smoke generator to generate rough foggy images with different smoke concentrations. The refiner is then used to refine the rough foggy image to make it more consistent with the actual smoke distribution. Specifically, the smoke generator and the image refiner have the same structure, including an encoder, a bottleneck layer, and a decoder. The encoder gradually extracts image features through multiple layers of convolution blocks and compresses the spatial information of the image into a feature map; the bottleneck layer contains multiple residual blocks for further extracting and enhancing features; the decoder gradually restores the image size through deconvolution and upsampling, and combines the encoder's feature map to restore details, and finally uses the convolution layer to generate a foggy image and a refined image; the entire fog generation process is expressed as:

[0076] H=F(C)(2)

[0077] In formula (2), H represents the hazy image, F represents the generator F, and C represents the haze-free image.

[0078] Discriminator: The defogging ability of most unsupervised defogging methods based on generative adversarial networks comes from the adversarial loss, but a single global discriminator is usually difficult to effectively identify tiny foggy areas in the defogging image, resulting in poor defogging effect. Therefore, a combination of a local discriminator guided by depth information and a global discriminator is used to solve this problem. The local discriminator guided by depth information is based on the observation that the farther away from the camera, the greater the smoke concentration. According to the depth map predicted in the defogging module, the discriminator pays more attention to distant areas where the smoke concentration may be greater. Specifically, after obtaining the depth map, a threshold is given. During the training process, the local discriminator will only select the area centered on the pixel with the threshold depth as the top for identification. Specifically, when training the generator G, the adversarial loss of the global and local discriminators is used for optimization, and when training the generator F, only the adversarial loss of the global discriminator is used for optimization.

[0079] The loss function used in the training process is introduced as follows:

[0080] Adversarial Loss: Adversarial loss ensures that the images generated by the generator can fool the discriminator and make the generated images as realistic as possible. There are two generators and discriminators in the network, so there are two sets of adversarial losses.

[0081] Adversarial loss of the generator G and the discriminator Dx: makes the generated haze-free images look like real haze-free images;

[0082] (3)

[0083] (4)

[0084] In formula (3) and (4), represents the adversarial loss of the discriminator Dx, represents the adversarial loss of the generator G, represents the average value of the respective errors, It is a discriminator for judging whether the fog-free image is real or fake, Y is the real fog-free image, X is the real foggy image, and G(X) is the generated fog-free image.

[0085] The adversarial loss between the generator F and the discriminator Dy: makes the generated foggy images look like real foggy images;

[0086] (5)

[0087] (6)

[0088] In formula (5) and (6), represents the adversarial loss of the discriminator Dy, represents the adversarial loss of the generator F, represents the average value of the respective errors, is a discriminator for determining whether a foggy image is real or fake. is the generated foggy image.

[0089] Cycle Consistency Loss: Cycle consistency loss can ensure that an image can be restored to the original image after passing through two generators. The cycle consistency loss formula is:

[0090] (7)

[0091] In formula (7), represents the cycle consistency loss, X represents the real hazy image, Y represents the real haze-free image, G represents the generator G, and F represents the generator F.

[0092] The smoke concentration coefficient monitoring loss formula is:

[0093] (8)

[0094] In formula (8), since the smoke concentration coefficient is randomly sampled in the above fog generation module The defogging module estimates the smoke concentration coefficient through the network Therefore, it is necessary to establish a constraint between the smoke concentration coefficient randomly sampled by the fog generating module and the smoke concentration coefficient estimated by the defogging module, so that the defogging module can correctly estimate the smoke concentration coefficient.

[0095] Time loss: Since we are processing video data, we need to consider the time consistency problem. We use time loss to constrain the video to make it smoother. The time loss formula is:

[0096] (9)

[0097] In formula (9), the subscript t+1 represents the next frame of foggy or fog-free image generated by self-motion synthesis. and Represents a pixel mask that ensures that pixel boundaries are not crossed when generating the next frame.

[0098] Total loss function:

[0099] (10)

[0100] In formula (10), , and is the importance weight used to balance each loss term.

Claims

1. An unsupervised medical video smoke removal method, characterized in that: The method comprises the following steps: S1, data preprocessing of medical video dataset; S2, inputting the preprocessed data into the smoke detector to distinguish foggy images from non-fog images; S3. Build a CycleGAN model framework, use a set of unpaired fog-free videos and foggy videos combined with a loss function to train the model, and obtain a trained CycleGAN model. The model includes a generator G, a generator F, and a discriminator: the generator G is used to convert a foggy image into a fog-free image; the generator F is used to convert a fog-free image into a foggy image; the discriminator includes a discriminator Dx and a discriminator Dy, the discriminator Dx is used to distinguish between a real fog-free image and a generated fog-free image, and the discriminator Dy is used to distinguish between a real foggy image and a generated foggy image; The loss function used in the training process is a total loss function calculated based on adversarial loss, cycle consistency loss, smoke concentration coefficient supervision loss and time loss, as follows: There are two groups of adversarial losses, namely the adversarial loss between the generator G and the discriminator Dx and the adversarial loss between the generator F and the discriminator Dy; The adversarial loss of the generator G and the discriminator Dx makes the generated haze-free image look like a real haze-free image, and the formula is as follows: （1） （2） In formula (1) and (2), represents the adversarial loss of the discriminator Dx, represents the adversarial loss of the generator G, represents the average value of the respective errors, is a discriminator for judging whether a fog-free image is real or fake, Y is a real fog-free image, X is a real foggy image, and G(X) is the generated fog-free image; The adversarial loss formula of the generator F and the discriminator Dy is as follows: （3） （4） In formula (3) and (4), represents the adversarial loss of the discriminator Dy, represents the adversarial loss of the generator F, represents the average value of the respective errors, is a discriminator for determining whether a foggy image is real or fake. is the generated foggy image; The cycle consistency loss formula is as follows: （5） In formula (5), represents the cycle consistency loss, X represents the real foggy image, Y represents the real fog-free image, G represents the generator G, and F represents the generator F; The smoke concentration coefficient supervision loss formula is: （6） In formula (6), Represents the smoke concentration coefficient obtained by random sampling in the fog generation module, Represents the smoke concentration coefficient predicted in the defogging module; The time loss formula is: （7） In formula (7), the subscript t+1 represents the next frame of foggy or fog-free image generated by self-motion synthesis. and Represents a pixel mask that ensures that the pixel boundaries are not exceeded during the generation of the next frame; The total loss function formula is: （8） In formula (8), , and It is the importance weight used to balance each loss item; S4. Use the CycleGAN model to generate fog-free images from foggy images in the video.

2. The unsupervised medical video smoke removal method according to claim 1, characterized in that: The data preprocessing comprises the following steps: S11. Data preparation: Decompose the medical video dataset containing foggy frames and non-fog frames into single frame images; S12, data annotation: annotate all single-frame images to distinguish foggy frames from non-fog frames; S13, data enhancement: rotate, flip, and scale the labeled data to form preprocessed data.

3. The unsupervised medical video smoke removal method according to claim 1, characterized in that: The smoke detector uses a pre-trained ResNet18 model to perform fine-tuning on the pre-processed data and trains a binary classification model to distinguish whether there is fog in the current frame.

4. The unsupervised medical video smoke removal method according to claim 1, characterized in that: The generator G includes an encoder, a bottleneck layer and a decoder: the encoder gradually extracts image features through multi-layer convolution blocks and compresses the spatial information of the image into a feature map; the bottleneck layer contains multiple residual blocks for further extracting and enhancing features; The decoder gradually restores the image size through deconvolution and upsampling, and restores details in combination with the feature map of the encoder. Finally, the convolution layer is used to generate the desmoked image, and the feature map of the multi-layer convolution block in the encoder is calculated through the adaptive average pooling layer to obtain the smoke concentration coefficient. The overall defogging process is expressed as: C=G(H)(9) In formula (9), C represents the generated haze-free image, G represents the generator G, and H represents the haze-free image.

5. The unsupervised medical video smoke removal method according to claim 4, characterized in that: The method of calculating the feature map of the multi-layer convolution block in the encoder through the adaptive average pooling layer to obtain the smoke concentration coefficient specifically refers to calculating the feature map of the multi-layer convolution layer in the encoder through the adaptive average pooling layer, aggregating the global information of each feature map into a scalar, and then concatenating the pooled results in the channel dimension to obtain a tensor containing multi-layer features, and obtaining the output features through 1x1 convolution, and finally obtaining the final smoke concentration coefficient through activation function processing and linear transformation of the output features.

6. The unsupervised medical video smoke removal method according to claim 1, characterized in that: The generator F comprises a smoke generator and an image refiner. The generator F first randomly samples the smoke density, then inputs the fog-free image and the smoke density into the smoke generator to generate rough foggy images with different smoke densities, and then uses the image refiner to refine the rough foggy images. Specifically, the smoke generator and the image refiner have the same structure, including an encoder, a bottleneck layer and a decoder. The encoder gradually extracts image features through multi-layer convolution blocks and compresses the spatial information of the image into a feature map. The bottleneck layer comprises multiple residual blocks for further extracting and enhancing features. The decoder gradually restores the image size through deconvolution and upsampling, and combines the encoder's feature map to restore details. Finally, the convolution layer is used to generate the foggy image and the refined image. The entire fog generation process is expressed as: H=F(C)(10) In formula (10), H represents the hazy image, F represents the generator F, and C represents the haze-free image.

7. The unsupervised medical video smoke removal method according to claim 1, characterized in that: The discriminator adopts a combination of a local discriminator guided by depth information and a global discriminator. After obtaining the smoke concentration coefficient in the generator G, a threshold is given. During the training process, the local discriminator only selects the area centered on the pixel with the threshold depth as the top for identification; When training the generator G, the adversarial loss of the global discriminator and the local discriminator is used for optimization, while when training the generator F, only the adversarial loss of the global discriminator is used for optimization.

8. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, they are used to implement the steps of the method according to any one of claims 1 to 7.

Images