Domain-adaptive endoscope image restoration method and device and storage medium

Abstract

The invention provides a domain self-adaptive endoscopic image restoration method, which comprises the following steps: aiming at aerosol noise in an operation image, respectively generating an operation smoke simulation image and a track simulation image which are used for simulating tissue fragments which are not vaporized and float in front of a lens in an operation, performing gray scale and binaryzation on the operation smoke and the track simulation image respectively; the operation video is intercepted frame by frame, the intercepted clear operation image is subjected to color filtering through the operation smoke simulation image subjected to gray processing, then the binarized track simulation image is superposed, and a composite image with aerosol noise is obtained; training an endoscopic image restoration model based on domain self-adaption by using the synthesized image and the intercepted surgical image; and detecting and binarizing the trajectory in the to-be-repaired surgical image, superposing the binarized trajectory image and the to-be-repaired surgical image, and inputting the superposed image into the endoscopic image repair model to obtain a repaired surgical image. The training data required by the method is easy to obtain, and the repairing effect is good.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and in particular relates to a method, device and storage medium for adaptive endoscopic surgical image restoration in a field containing aerosol noise. Background Art

[0002] Laparoscopic surgery is a minimally invasive surgical procedure. Compared to open surgery, laparoscopic surgery requires a smaller incision. By inserting a specialized camera into the incision, the doctor can observe the patient's internal conditions on a screen and then perform the corresponding surgery. However, before laparoscopic surgery, the patient's abdomen needs to be expanded to accommodate surgical instruments, so gases such as CO2 must be injected into the patient's body. During laparoscopic surgery, cutting instruments such as electrocautery and ultrasonic scalpels, or laser ablation, can cause smoke to form inside the patient's body. Therefore, under the influence of CO2 and the cutting or burning of surgical instruments, the quality of surgical images is reduced, which can cause temporary visual impairment for the doctor and affect the efficiency of the operation. Therefore, smoke noise needs to be repaired. Currently, there are three main methods for endoscopic surgical image restoration: image restoration methods based on prior knowledge, image restoration methods based on deep learning, and methods that combine the two. Early research, influenced by weather defogging algorithms, assumed that surgical smoke resembled fog in natural environments. Consequently, most of these studies relied on atmospheric scattering models as prior knowledge. However, observations of foggy images from real surgical environments and subsequent research have revealed certain differences between surgical smoke and natural fog. With the development of deep learning, data-driven image restoration methods that do not rely on prior knowledge have been proposed. However, paired images of real endoscopic surgical procedures with and without fog are difficult to obtain. Therefore, some studies have used synthetic surgical smoke to generate paired training sets of foggy and fog-free surgical images. Deep learning-based surgical image restoration methods can be categorized as supervised, unsupervised, and self-supervised, depending on the type of training set used. Supervised methods rely on a large amount of labeled data; that is, paired foggy and fog-free surgical images are required for training. Unsupervised methods do not require labeled data; the training images can be unpaired. Self-supervised methods fall between supervised and unsupervised approaches and can alleviate the problem of scarce labeled data to some extent. In addition to the above two methods, there are also some studies that combine the two to better play the advantages of both.

[0003] The aforementioned methods all assume that endoscopic surgery produces smoke. However, literature research reveals that endoscopic surgery produces aerosols. Real surgical aerosols are complex and varied, and their distribution is difficult to describe using a universal formula. Even if simulated surgical aerosols can roughly simulate the basic form and diffusion trend of aerosols, subtle details such as color are often not fully captured. Furthermore, because real surgical aerosols are random, there are differences in the distribution of simulated and real surgical aerosols. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0005] To this end, the present invention provides a domain-adaptive endoscopic image restoration method, device, and storage medium, the required training data of which are easy to obtain and can accurately remove aerosol noise in surgical images.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides a domain-adaptive endoscopic image restoration method, comprising:

[0008] To address aerosol noise in surgical images, a surgical smoke simulation image and a trajectory simulation image are generated. The trajectory simulation image is used to simulate tissue fragments that are not vaporized and float in front of the camera during surgery. The surgical smoke simulation image and the trajectory simulation image are respectively subjected to grayscale and binarization processing. The surgical video is intercepted frame by frame, and the intercepted surgical image without aerosol noise is used as the original image. The grayscale-processed surgical smoke simulation image is used to filter the original image, and then the binarized trajectory simulation image is superimposed to obtain a composite image with aerosol noise.

[0009] Using the synthesized image and the intercepted surgical image to train an endoscopic image restoration model based on domain adaptation, so that the model has the ability to remove aerosol noise in the image;

[0010] The trajectory in the surgical image to be repaired is detected and binarized to obtain a binarized trajectory image, which is superimposed with the surgical image to be repaired to obtain an enhanced image, and the enhanced image is input into the endoscopic image repair model to obtain a repaired surgical image.

[0011] In some embodiments, when generating the surgical smoke simulation image, winds from different directions are added to simulate surgical smoke in different situations;

[0012] When producing the trajectory simulation image, particles of different shapes are used to enrich the types of simulated trajectories.

[0013] In some embodiments, each type of image is the same size.

[0014] In some embodiments, the endoscopic image restoration model includes a first encoder, a noise removal network, and a second decoder connected in sequence;

[0015] The first encoder adopts a first encoder in a source domain encoding network, wherein the source domain encoding network is used to encode the synthetic image as a source domain and the surgical image with aerosol noise as a target domain into the same latent space;

[0016] The noise removal network is used to perform denoising on the encoding result of the aerosol noise image output by the first encoder and output the result as the result after the aerosol noise is removed in the latent space;

[0017] The second decoder uses a second decoder in a target domain decoding network, and the target domain decoding network is used to decode the result after removing aerosol noise in the latent space into an image without aerosol noise.

[0018] In some embodiments, the source domain coding network adopts a first variational autoencoder, including a first encoder, a first decoder and a first discriminator; the source domain coding network is trained using a first training data set consisting of multiple synthetic images and multiple surgical images with aerosol noise, and an image is randomly selected from the first training data set as input during training, and the loss function used is set to L VAE1 (x), the expression is as follows:

[0019]

[0020] Where x is the image input to the source domain encoding network, and each input image is regarded as a sample; is the objective function of conditional GAN, V(D,G) is the core value function of GAN, D is the first discriminator, G is the first decoder, P z (z) is the prior distribution of the noise z input to the first decoder, P data (x) is the probability distribution of x; E is the expected calculation function; KL() is the KL divergence function; E VAE1 (x) is the encoding result after x is input to the first encoder; is a Gaussian distribution, I represents a covariance matrix where all dimension variances are 1 and there is no correlation between dimensions; L FM is the multi-scale discriminator loss; L VGG For perception loss.

[0021] In some embodiments, the target domain decoding network adopts a second variational autoencoder, including a second encoder, a second decoder and a second discriminator; the target domain decoding network is trained using a second training data set consisting of a plurality of surgical images without aerosol noise, and the training function used is the same as that of the source domain encoding network.

[0022] In some embodiments, the noise removal network includes a first residual module, a defogging module, and a second residual module connected in sequence, the defogging module includes a multi-scale parallel large convolution kernel module and an enhanced parallel attention module; the loss function used when training the noise removal network is L(s), which is expressed as follows:

[0023]

[0024] Where, is the objective function of the conditional GAN, V(D,G) is the core value function of GAN, D is the second discriminator in the target domain decoding network, G is the generator in the defogging module; E is the expected calculation function; M is the mapping of the noise removal network; E VAE1 (s) is the encoding result after the synthetic image s with aerosol noise is input to the first encoder, E VAE2 (g) is the decoding result after the surgical image g without aerosol noise is input to the second decoder; g' represents the output of the synthetic image s with aerosol noise after the endoscopic image restoration model, that is, g'=G VAE2 (M(E VAE1 (s)));L FM is the multi-scale discriminator loss; L VGG is the perceptual loss; L ID is the identity mapping loss.

[0025] In some embodiments, a trajectory detection network is used to detect the trajectory in the surgical image to be repaired. The trajectory detection network is trained using trajectory training data. The step of obtaining the trajectory training data includes:

[0026] The surgical image without aerosol noise is used as the original image, and the surgical smoke simulation image is used to perform color filtering on the original image to obtain a preliminary result image. The preliminary result image is then color filtered using the trajectory simulation image to obtain a trajectory composite image, which is used as the trajectory training data.

[0027] A second aspect of the present invention provides a field-adaptive endoscopic image restoration device, comprising:

[0028] The synthetic image acquisition module is configured to generate a surgical smoke simulation image and a trajectory simulation image, respectively, based on aerosol noise in the surgical image. The trajectory simulation image is used to simulate tissue fragments that are not vaporized and float in front of the camera during surgery, and the surgical smoke simulation image and the trajectory simulation image are respectively grayscaled and binarized. The surgical video is intercepted frame by frame, and the intercepted surgical image without aerosol noise is used as the original image. The grayscale-processed surgical smoke simulation image is used to filter the original image, and then the binarized trajectory simulation image is superimposed to obtain a synthetic image with aerosol noise.

[0029] a training module configured to train an endoscopic image restoration model based on domain adaptation using the synthetic image and the intercepted surgical image, so as to remove aerosol noise from the image;

[0030] The repair module is configured to detect and binarize the trajectory in the surgical image to be repaired to obtain a binarized trajectory image, superimpose the binarized trajectory image with the surgical image to be repaired to obtain an enhanced image, and input the enhanced image into the endoscopic image repair model to obtain a repaired surgical image.

[0031] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the domain-adaptive endoscopic image restoration method according to any embodiment of the first aspect of the present invention.

[0032] Compared with the prior art, the present invention has the following characteristics and beneficial effects:

[0033] 1. Domain adaptation is introduced based on image translation methods to enhance the generalization ability of the endoscopic image restoration model on real degraded surgical images. At the same time, the aerosol noise in surgical images is divided into two categories: smoke and trajectory. The two categories are simulated separately during the simulation process. This divide-and-conquer approach makes the simulation of aerosol noise easier and facilitates training. Randomly superimposing the two categories during training can also increase the randomness and diversity of the training samples.

[0034] 2. Ability to detect and restore aerosol noise in endoscopic surgical images. Because the color of the tracks in aerosol noise is difficult to fully simulate, in order to better restore the tracks, it is necessary to first detect the tracks in the image. The detection results are then superimposed on the surgical image containing aerosol noise and used as input for the endoscopic image restoration model. Ultimately, a clean surgical image is obtained after removing the aerosol noise.

[0035] 3. It also achieves good restoration results for real surgical images containing aerosol noise. Using domain adaptation, the training process not only uses a noise-free surgical image dataset but also integrates real surgical smoke images. Therefore, even when faced with real surgical images containing aerosol noise, the endoscopic image restoration model can accurately remove this noise and restore a clear surgical field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is an overall flow chart of a domain-adaptive endoscopic image restoration method provided by an embodiment of the first aspect of the present disclosure;

[0037] Figure 2 yes Figure 1 A schematic diagram of the specific implementation process of the method;

[0038] Figure 3 This is a real surgical image, in which the aerosol noise is divided into two types: surgical smoke and trajectory;

[0039] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the third aspect of the present disclosure. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0041] On the contrary, this application covers any alternatives, modifications, equivalents, and solutions made within the spirit and scope of this application as defined by the claims. Furthermore, to facilitate a better understanding of this application, certain specific details are described in detail below in the detailed description of this application. Those skilled in the art will be able to fully understand this application without these details.

[0042] See also Figure 1 、 Figure 2 The first embodiment of the present disclosure provides a domain-adaptive endoscopic image restoration method, comprising the following steps:

[0043] Step S100: Generate a surgical smoke simulation image and a trajectory simulation image based on the aerosol noise in the surgical image. The surgical smoke simulation image is used to simulate the vaporized tissue protein and fat caused by high temperature when the surgical instrument cuts the tissue, and the trajectory simulation image is used to simulate the non-vaporized tissue fragments floating in front of the camera. The surgical smoke simulation image is grayscale processed, and the trajectory simulation image is binarized. The surgical video is captured frame by frame, and the captured surgical images are divided into surgical images without aerosol noise and surgical images with aerosol noise. The surgical image without aerosol noise is used as the original image, and the grayscale-processed surgical smoke simulation image is used to perform color filtering on the original image. The binarized trajectory simulation image is then superimposed to obtain a composite image with aerosol noise.

[0044] Step S200: Using the synthesized image in step S100 and the intercepted surgical image, a domain-adaptive endoscopic image restoration model is trained to enable it to have the ability to remove aerosol noise in the image;

[0045] Step S300: Detect and binarize the trajectory in the surgical image to be repaired, superimpose the obtained binary trajectory image with the surgical image to be repaired to obtain an enhanced image, and input the enhanced image into the above-mentioned endoscopic image repair model to obtain the repaired surgical image.

[0046] In order to solve the problem of inconsistent distribution between simulated aerosols and aerosols generated in real surgery, the embodiment of the present invention introduces a domain adaptation method. The goal of domain adaptation is to apply the training results on one domain to another domain, where the trained domain is called the source domain and the domain to be applied is called the target domain. In the network, an implicit alignment domain adaptation method is used, that is, by training a network, the source domain and the target domain are encoded into a latent space to achieve the purpose of neighborhood alignment. After the training is completed, the overall network has a good restoration effect on real surgical aerosol noise.

[0047] In some embodiments, step S100 specifically includes:

[0048] Step S110: Generate aerosol noise simulation image:

[0049] First, based on the different geometric forms of aerosol noise in real surgical images, the aerosol noise in surgical images is divided into two categories: surgical smoke and tracks. Surgical smoke is the vaporization of tissue protein and fat caused by high temperature when instruments such as scalpels cut tissue, while the unvaporized tissue fragments float in front of the camera to form tracks. Figure 3 As shown, the trajectory is the white substance in the upper right line.

[0050] Subsequently, based on the above definition, a surgical smoke simulation image and a trajectory simulation image are generated for simulating surgical smoke. When simulating surgical smoke, it is necessary to add wind from different directions to the smoke to simulate surgical smoke in different situations; when simulating trajectories, it is necessary to utilize a particle system and add particles of different shapes to enrich the types of simulated trajectories. In order to shorten the subsequent training time of the endoscopic image restoration model and reduce the amount of calculation, the size of the simulation image is 256*256, the same size as the surgical image. The generated surgical smoke simulation image is grayscale processed. In order to avoid the problem of poor trajectory restoration due to the distribution difference between the simulated trajectory and the real trajectory, the generated trajectory simulation image needs to be converted into a binary image.

[0051] In a specific embodiment of the present application, the open source 3D rendering software - Blender is used to complete the simulation of surgical smoke and trajectory respectively.

[0052] Step S120: surgical image classification:

[0053] The surgical video is captured in frames, and the captured surgical images are divided into surgical images without aerosol noise and surgical images with aerosol noise. Considering that most surgical videos have black edges, which are not conducive to the training of the subsequent endoscopic image restoration model, it is necessary to crop the black edges in the surgical images as invalid areas to ensure that the obtained images are surgical images without black edges. At the same time, because surgical aerosols are unevenly distributed, in a surgical image containing aerosol noise, there may be a part with aerosol noise and another part without aerosol noise. Therefore, it is necessary to further crop the surgical image containing aerosol noise to ensure that most or all of the cropped surgical images contain aerosol noise, and finally obtain a surgical image t with aerosol noise and a surgical image g without aerosol noise. In a specific embodiment of the present application, the size of the obtained surgical image t with aerosol noise and the surgical image g without aerosol noise are both 256*256.

[0054] Step S130: Image synthesis

[0055] The surgical image g without aerosol noise is used as the original image. The grayscale processed surgical smoke simulation image is first used to filter the original image, and then the binary trajectory simulation image is superimposed to obtain the synthetic image s with aerosol noise.

[0056] The first training dataset was constructed using multiple different surgical images with aerosol noise and multiple different synthetic images with aerosol noise, and the second training dataset was constructed using multiple surgical images without aerosol, which were used for the subsequent training of the endoscopic image restoration model.

[0057] In a specific embodiment of the present application, the color filter principle of Adobe Photoshop is used, and the specific formula is:

[0058]

[0059] Among them, the base color is the surgical image without aerosol noise, and the mixed color is the simulated image with aerosol noise;

[0060] A surgical smoke simulation image and a trajectory simulation image are randomly selected to form an aerosol noise simulation image. Then, the above formula is used to superimpose them with the surgical image g without aerosol noise to obtain about 20,000 synthetic images s.

[0061] In some embodiments, see Figure 2 Step S200 includes: constructing and training an endoscopic image restoration model based on domain adaptation, wherein the endoscopic image restoration model includes a first encoder, a noise removal network, and a second decoder connected in sequence; wherein the first encoder adopts the first encoder in the source domain encoding network, and the source domain encoding network is used to encode the synthetic image with aerosol noise as the source domain and the surgical image with aerosol noise as the target domain into the same latent space; the noise removal network is used to denoise the encoding result of the aerosol-noised image output by the first encoder and output it as a result after the aerosol noise is removed in the latent space; the second decoder adopts the second decoder in the target domain decoding network, and the target domain decoding network is used to complete the decoding of the aerosol-noise-free image, that is, to decode the result after the aerosol noise is removed in the latent space output by the noise removal network into an image without aerosol noise.

[0062] Furthermore, step S200 specifically includes:

[0063] Step S210: The first encoder obtains:

[0064] In order to obtain the first encoder in the domain-adaptive endoscopic image restoration model, it is necessary to complete the construction and training of the source domain encoding network. Specifically:

[0065] See also Figure 2The source encoding network in the upper half of the image processing model aims to achieve domain adaptation in the endoscopic image restoration model. This involves encoding the source and target domains into a latent space. In this process, the synthetic image s with aerosol noise belongs to the source domain, while the real surgical image t with aerosol noise belongs to the target domain. The source encoding network can be trained using a variety of networks. In this embodiment, a Pix2Pix network is used for training. The source encoding network in this embodiment employs a variational autoencoder (VAE1), comprising a first encoder, a first decoder, and a first discriminator. The first decoder is equivalent to the generator in the Pix2Pix model. During the Pix2Pix network training process, adversarial training is performed between the generator and the discriminator. The generator attempts to generate more realistic images to deceive the first discriminator, while the first discriminator strives to improve its discrimination ability to identify the images generated by the generator. This adversarial training process improves the performance of both the generator and the discriminator. The discriminator improves its ability to discriminate generated images by optimizing its own discrimination results, thereby encouraging the generator to produce higher-quality images. During training, one image is randomly selected from the first training dataset constructed in step S130 each time (the image can be a surgical image with aerosol noise or a synthetic image with aerosol noise) and input into the source domain encoding network. Assume that the loss function used in training is L VAE1 (x), the expression is as follows:

[0066]

[0067] Where:

[0068] x is the image input to the source domain encoding network, and each image input to the source domain encoding network is regarded as a sample;

[0069] Item 1 is the objective function of the conditional GAN, V(D,G) is the core value function of GAN, which defines the adversarial target between the discriminator and the generator, D is the first discriminator, G is the first decoder (also known as the generator), and P z (z) is the prior distribution of the noise z input to the generator, P data (x) is the probability distribution of x; E is the expected calculation function; To sample all samples x from the true data distribution, calculate the logarithm of the probability that the first discriminator judges it as true, and then take the average; To sample all noise z from the noise prior distribution, calculate the logarithm of the probability that the generated sample G(z) is judged as false data by the first discriminator, and then take the average;

[0070] Item 2 is the KL divergence, which aims to measure the distribution in the potential space, namely EVAE1 (x) and Gaussian distribution The difference between them makes the two closer and closer, where KL() is the KL divergence function, E VAE1 Represents the first encoder, E VAE1 (x) represents the encoding result after x is input to the first encoder, Gaussian distribution The I in represents a covariance matrix where all dimension variances are 1 and there is no correlation between dimensions;

[0071] The third item L FM It is a multi-scale discriminator loss, which uses the feature matching loss in pix2pixHD to better train the discriminator;

[0072] The last item L VGG The perceptual loss aims to retain more texture information in real surgical images with aerosol noise.

[0073] The initial learning rate is set to 0.0002. When the number of training rounds reaches 100, the learning rate begins to decrease. Training is continued for 100 rounds until the learning rate reaches 0. The training of the source domain encoding network is terminated, and the trained first encoder is obtained.

[0074] Step S220: The second decoder obtains:

[0075] In order to obtain the second decoder in the domain-adaptive endoscopic image restoration model, it is necessary to complete the construction and training of the target domain decoding network. Specifically:

[0076] See also Figure 2 In the lower half of the figure, the purpose of the target domain decoding network is to complete the decoding of the image without aerosol noise, that is, to decode the latent vector of the repaired surgical image in the latent space into the repaired surgical image. A variety of networks can be used to complete the training of the target domain decoding network, and the pix2pix network is also used here. The target domain decoding network of this embodiment adopts a variational autoencoder VAE2, which includes a second encoder, a second decoder and a second discriminator. The training data used is the surgical image g without aerosol noise in the second training data set. The loss function and termination condition used in training the target domain decoding network are the same as those of the source domain encoding network, and will not be repeated here.

[0077] Step S230: noise removal network acquisition:

[0078] See also Figure 2In the middle part, the noise removal network aims to remove aerosol noise from the input image. The input of the noise removal network is the encoding result of the aerosol-noised image output by the first encoder, and the output of the noise removal network is the result of restoring the degraded image after encoding. This restored result is input into the second decoder to obtain the restored surgical image. The noise removal network of this embodiment includes a first residual module ResNet Block, a dehazing module Mix Structure Block, and a second residual module ResNet Block, all of which are connected in sequence. The Mix Structure Block is a dehazing module that includes a multi-scale parallel large convolution kernel module (MSPLCK) and an enhanced parallel attention module (EPA). The MSPLCK has a multi-scale large receptive field and can obtain multi-scale features of the image, thereby restoring images containing larger smoke areas while preserving local texture. The EPA can handle unevenly distributed smoke by extracting shared global information and position-dependent local information of the original features in parallel. Compared with the previous single large kernel dehazing module, the MSPLCK pays more attention to local texture. In addition, the EPA can better handle uneven smoke. The loss function used in this embodiment when training the noise removal network is L(s), which is expressed as follows:

[0079]

[0080] Where:

[0081] Item 1 is the objective function of the conditional GAN, V(D,G) is the core value function of the GAN, which defines the adversarial goal between the discriminator and the generator, D is the second discriminator in the target domain decoding network, and G is the generator in the dehazing module (i.e., the output layer of the dehazing module);

[0082] The second term E||M(E VAE1 (s))-E VAE2 (g)||1 represents the average value of the potential coding difference calculated for all samples (s, g) in the input data distribution, where s is a synthetic image with aerosol noise, g is a surgical image without aerosol noise, M() represents the mapping of the noise removal network, and E VAE1 (s) is the encoding result after s is input to the first encoder, E VAE2 (g) is the decoding result after g is input to the second decoder. The purpose of the second term is to make the distribution M(E) of the perceptually restored image generated in the latent space VAE1 (s)) The loss of the distribution of the surgical image g close to the aerosol noise free image after being decoded by the target domain decoding network;

[0083] The third term E||g'-g||1 represents the average value of the difference between g' and g, where g' represents the output of the synthetic image s with aerosol noise after the endoscopic image restoration model, i.e., g'=G VAE2 (M(E VAE1 (s))), which is the L1 loss between the generated dehazed image and the true value;

[0084] The last item L ID The identity mapping loss is added to make the restored image retain the color of the original image (i.e., the image with aerosol noise to be restored) as much as possible;

[0085] The meanings of the remaining items are the same as those of L VAE1 The corresponding items in (x) have the same meanings and are not repeated here.

[0086] The initial learning rate is set to 0.00025. When the number of training rounds reaches 150, the learning rate begins to decrease. Training is continued for 100 rounds until the learning rate reaches 0. The training of the noise removal network is terminated, and the trained noise removal network and the trained endoscopic image restoration model are obtained.

[0087] As can be seen, step S200 first obtains the trained first and second encoders, respectively, to provide a low-dimensional representation space for the noise removal network. Next, the weights of the first encoder and second decoder are fixed, and a synthetic image with aerosol noise is input as training data into the first encoder to obtain a representation of the synthetic image in a low-dimensional latent space. The noise removal network is then used to restore the image, and finally, the result of the noise removal network restoration is input into the second decoder for decoding to obtain the final restored image, thus completing the training of the entire endoscopic image restoration model, including the noise removal network.

[0088] In some embodiments, step S300 specifically includes:

[0089] S310: Building and training the trajectory detection network:

[0090] The purpose of the trajectory detection network is to complete the detection of trajectories in aerosol noise. The output trajectory image is a binary image of the same size as the input image. In this embodiment, the Unet network is used. Because it is difficult to obtain the ground truth corresponding to the surgical image that actually contains the trajectory, the training data used is still synthetic data. Specifically, the surgical image without aerosol noise intercepted in step S120 is used as the original image. The surgical smoke simulation image generated in step S110 is first used to filter the original image to obtain a preliminary result image. The preliminary result image is then filtered using the trajectory simulation image generated in step S110 to obtain a trajectory synthetic image, which is used as the trajectory training data. The loss function used in training the trajectory detection network in this embodiment is L Unet , the expression is as follows:

[0091]

[0092] Among them, i represents the pixel position in the image input to the trajectory detection network, N represents the total number of pixels in the image input to the trajectory detection network, and outputs i Represents the predicted pixel value at pixel position i, targets i Represents the true value of the pixel at pixel position i, weight i Represents the weight value at pixel position i, and the pixel value at pixel position i is used as its weight value.

[0093] Training was stopped when the trajectory detection network performed well on both synthetic images with aerosol noise and real surgical images containing aerosol noise.

[0094] Step S320: Image restoration:

[0095] First, the surgical image to be inpainted is fed into a trained trajectory detection network to generate a corresponding trajectory image. The resulting trajectory image is then superimposed with the surgical image to generate an enhanced image. This enhanced image is then fed into a trained endoscopic image inpainting model to obtain the final inpainted image—the clean surgical image after aerosol noise removal.

[0096] It can be understood that the embodiment of the present disclosure distinguishes between smoke and tracks for the aerosol noise in the surgical image, and divides the aerosol noise in the surgical image into two categories: smoke and tracks. The two categories are simulated separately during the simulation process. This divide-and-conquer approach can make the simulation of aerosol noise easier and easier to train. Randomly superimposing the two categories during training can also increase the randomness and diversity of the training samples. The entire endoscopic image restoration model realizes the conversion from aerosol images to clear images by combining the encoding-decoding capabilities of the variational autoencoder and the noise removal network. The introduction of the first encoder in the source domain encoding network enhances the generalization ability of the endoscopic image restoration model for different types of degraded images, while the introduction of the second encoder in the target domain decoding network ensures high-quality reconstruction of the image after de-aerosolization. This architecture makes full use of synthetic data and real data, improving the practicality and effectiveness of the overall endoscopic image restoration model.

[0097] A second aspect of the present disclosure provides a domain-adaptive endoscopic image restoration device, comprising:

[0098] The synthetic image acquisition module is configured to generate a surgical smoke simulation image and a trajectory simulation image, respectively, based on aerosol noise in the surgical image. The trajectory simulation image is used to simulate tissue fragments that are not vaporized and float in front of the camera during surgery, and the surgical smoke simulation image and the trajectory simulation image are respectively grayscaled and binarized. The surgical video is intercepted frame by frame, and the intercepted surgical image without aerosol noise is used as the original image. The grayscale-processed surgical smoke simulation image is used to filter the original image, and then the binarized trajectory simulation image is superimposed to obtain a synthetic image with aerosol noise.

[0099] a training module configured to train an endoscopic image restoration model based on domain adaptation using the synthetic image and the intercepted surgical image, so as to remove aerosol noise from the image;

[0100] The repair module is configured to detect and binarize the trajectory in the surgical image to be repaired to obtain a binarized trajectory image, superimpose the binarized trajectory image with the surgical image to be repaired to obtain an enhanced image, and input the enhanced image into the endoscopic image repair model to obtain a repaired surgical image.

[0101] It should be noted that the aforementioned explanation of the embodiment of the domain-adaptive endoscopic image restoration method is also applicable to the domain-adaptive endoscopic image restoration device of this embodiment, and will not be repeated here.

[0102] In order to implement the above embodiments, the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to execute the domain-adaptive endoscopic image restoration method of the above embodiments.

[0103] Reference below Figure 4 , which shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. It should be noted that the electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs, desktop computers, and servers. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0104] like Figure 4 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 into a random access memory (RAM) 103. Various programs and data required for the operation of the electronic device are also stored in the RAM 103. The processing device 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0105] Typically, the following devices may be connected to the I / O interface 105: an input device 106 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, etc.; an output device 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 109. The communication device 109 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 4 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0106] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, this embodiment includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 109, or installed from the storage device 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0107] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0108] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0109] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the domain-adaptive endoscopic image restoration method.

[0110] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C-" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0113] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0114] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.

[0115] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0116] Those skilled in the art will understand that all or part of the steps carried out in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the developed program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0117] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0118] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A domain-adaptive endoscopic image restoration method, characterized in that: include: In response to aerosol noise in surgical images, a surgical smoke simulation image and a trajectory simulation image are generated. The trajectory simulation image is used to simulate tissue fragments that are not vaporized and float in front of the camera during surgery. The surgical smoke simulation image and the trajectory simulation image are respectively grayscaled and binarized. The surgical video is captured frame by frame, and the captured surgical image without aerosol noise is used as the original image. The original image is subjected to color filtering using the grayscale-processed surgical smoke simulation image, and then the binary-processed trajectory simulation image is superimposed to obtain a synthetic image with aerosol noise. Using the synthesized image and the intercepted surgical image to train an endoscopic image restoration model based on domain adaptation, so that the model has the ability to remove aerosol noise in the image; The trajectory in the surgical image to be repaired is detected and binarized to obtain a binarized trajectory image, which is superimposed with the surgical image to be repaired to obtain an enhanced image, and the enhanced image is input into the endoscopic image repair model to obtain a repaired surgical image.

2. The domain-adaptive endoscopic image restoration method according to claim 1, characterized in that: When generating the surgical smoke simulation image, winds in different directions are added to simulate surgical smoke in different situations; When producing the trajectory simulation image, particles of different shapes are used to enrich the types of simulated trajectories.

3. The domain-adaptive endoscopic image restoration method according to claim 1, characterized in that: All images are the same size.

4. The domain-adaptive endoscopic image restoration method according to claim 1, characterized in that: The endoscopic image restoration model includes a first encoder, a noise removal network, and a second decoder connected in sequence; The first encoder adopts a first encoder in a source domain encoding network, wherein the source domain encoding network is used to encode the synthetic image as a source domain and the surgical image with aerosol noise as a target domain into the same latent space; The noise removal network is used to perform denoising on the encoding result of the aerosol noise image output by the first encoder and output the result as the result after the aerosol noise is removed in the latent space; The second decoder uses a second decoder in a target domain decoding network, and the target domain decoding network is used to decode the result after removing aerosol noise in the latent space into an image without aerosol noise.

5. The domain-adaptive endoscopic image restoration method according to claim 4, characterized in that: The source domain coding network adopts a first variational autoencoder, including a first encoder, a first decoder and a first discriminator; the source domain coding network is trained using a first training data set consisting of multiple synthetic images and multiple surgical images with aerosol noise. During training, an image is randomly selected from the first training data set as input, and the loss function used is set to L VAE1 (x), the expression is as follows: Where x is the image input to the source domain encoding network, and each input image is regarded as a sample; is the objective function of conditional GAN, V(D,G) is the core value function of GAN, D is the first discriminator, G is the first decoder, P z (z) is the prior distribution of the noise z input to the first decoder, P data (x) is the probability distribution of x; E is the expected calculation function; KL( ) is the KL divergence function; E VAE1 (x) is the encoding result after x is input to the first encoder; is a Gaussian distribution, I represents a covariance matrix where all dimension variances are 1 and there is no correlation between dimensions; L FM is the multi-scale discriminator loss; L VGG For perception loss.

6. The domain-adaptive endoscopic image restoration method according to claim 5, characterized in that: The target domain decoding network adopts a second variational autoencoder, including a second encoder, a second decoder and a second discriminator; the target domain decoding network is trained using a second training data set consisting of multiple surgical images without aerosol noise, and the training function used is the same as that of the source domain encoding network.

7. The domain-adaptive endoscopic image restoration method according to claim 4, characterized in that: The noise removal network includes a first residual module, a defogging module, and a second residual module connected in sequence. The defogging module includes a multi-scale parallel large convolution kernel module and an enhanced parallel attention module. The loss function used in training the noise removal network is L(s), which is expressed as follows: Where, is the objective function of the conditional GAN, V(D,G) is the core value function of GAN, D is the second discriminator in the target domain decoding network, G is the generator in the defogging module; E is the expected calculation function; M is the mapping of the noise removal network; E VAE1 (s) is the encoding result after the synthetic image s with aerosol noise is input to the first encoder, E VAE2 (g) is the decoding result after the surgical image g without aerosol noise is input to the second decoder; g' represents the output of the synthetic image s with aerosol noise after the endoscopic image restoration model, that is, g'=G VAE2 (M(E VAE1 (s)));L FM is the multi-scale discriminator loss; L VGG is the perceptual loss; L ID is the identity mapping loss.

8. The domain-adaptive endoscopic image restoration method according to claim 1, characterized in that: A trajectory detection network is used to detect the trajectory in the surgical image to be repaired. The trajectory detection network is trained using trajectory training data. The steps of obtaining the trajectory training data include: The surgical image without aerosol noise is used as the original image, and the surgical smoke simulation image is used to perform color filtering on the original image to obtain a preliminary result image. The preliminary result image is then color filtered using the trajectory simulation image to obtain a trajectory composite image, which is used as the trajectory training data.

9. A domain-adaptive endoscopic image restoration device, characterized in that: include: a synthetic image acquisition module configured to generate a surgical smoke simulation image and a trajectory simulation image based on aerosol noise in the surgical image, wherein the trajectory simulation image is used to simulate tissue fragments that are not vaporized and float in front of the camera during surgery, and to perform grayscale and binarization processing on the surgical smoke simulation image and the trajectory simulation image, respectively; The surgical video is captured frame by frame, and the captured surgical image without aerosol noise is used as the original image. The original image is subjected to color filtering using the grayscale-processed surgical smoke simulation image, and then the binary-processed trajectory simulation image is superimposed to obtain a synthetic image with aerosol noise. a training module configured to train an endoscopic image restoration model based on domain adaptation using the synthetic image and the intercepted surgical image, so as to remove aerosol noise from the image; The repair module is configured to detect and binarize the trajectory in the surgical image to be repaired to obtain a binarized trajectory image, superimpose the binarized trajectory image with the surgical image to be repaired to obtain an enhanced image, and input the enhanced image into the endoscopic image repair model to obtain a repaired surgical image.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the domain-adaptive endoscopic image restoration method according to any one of claims 1 to 8.

Images