A domain adaptive endoscopic image inpainting method and device and storage medium

Abstract

This invention provides a domain-adaptive endoscopic image restoration method, comprising: generating a simulated surgical smoke image and a simulated trajectory image to address aerosol noise in surgical images, the latter simulating unvaporized tissue fragments floating in front of the lens during surgery; performing grayscale and binarization on the simulated surgical smoke and trajectory images respectively; extracting surgical video frame by frame, filtering the extracted clear surgical images using the grayscale-processed simulated surgical smoke image, and then superimposing the binarized trajectory simulation image to obtain a synthesized image with aerosol noise; training a domain-adaptive endoscopic image restoration model using the synthesized image and the extracted surgical images; detecting and binarizing the trajectory in the surgical image to be restored, superimposing the binarized trajectory image with the surgical image to be restored, and inputting the result into the endoscopic image restoration model to obtain the restored surgical image. This method requires readily available training data and achieves good restoration results.

Description

Technical Field

[0001] This disclosure belongs to the field of image processing technology, and specifically relates to a method, apparatus and storage medium for domain-adaptive endoscopic surgical image recovery containing aerosol noise. Background Technology

[0002] Laparoscopic surgery is a minimally invasive surgical method. Compared with open surgery, laparoscopic surgery has a smaller incision. By inserting a special camera into the incision, the doctor can observe the patient's internal condition on a screen and then perform the corresponding surgery. However, before laparoscopic surgery, the patient's abdomen needs to be enlarged to accommodate surgical instruments, so CO2 or other gases need to be introduced into the patient's body. Furthermore, during laparoscopic surgery, the cutting of surgical instruments such as electrocautery and ultrasonic scalpels, or the ablation of lasers, can generate smoke inside the patient's body. Therefore, the quality of the surgical images is reduced due to the influence of CO2 and the cutting or ablation of surgical instruments, which can cause temporary visual impairment for the doctor and affect the efficiency of the surgery. Thus, smoke noise restoration is necessary. Currently, methods for endoscopic surgical image restoration mainly fall into three categories: image restoration methods based on prior knowledge, image restoration methods based on deep learning, and methods combining both. Early research was influenced by weather defogging algorithms, assuming surgical smoke was similar to fog in the natural environment. Therefore, most of these studies relied on atmospheric scattering models as prior knowledge. However, observations of foggy images in real surgical environments and subsequent research revealed differences between surgical smoke and natural fog. Later, with the development of deep learning, data-driven image inpainting methods independent of prior knowledge were proposed. However, obtaining paired foggy and fog-free surgical images from real endoscopic surgical environments is difficult. Therefore, some studies used synthetic surgical smoke to obtain paired foggy-fog-free surgical image training sets. Deep learning-based surgical image inpainting methods can be further categorized into supervised, unsupervised, and self-supervised methods, depending on the type of training set used. Supervised methods rely on a large amount of labeled data, requiring paired foggy-fog-free surgical images during training. Unsupervised methods do not require labeled data; the surgical images used during training can be unpaired. Self-supervised methods fall between supervised and unsupervised methods, alleviating the problem of scarce labeled data to some extent. In addition to the two methods mentioned above, some studies combine the two to better leverage their respective advantages.

[0003] The methods described above all assume that the aerosols generated during endoscopic surgery are smoke. However, literature review reveals that the aerosols generated during endoscopic surgery are actually aerosols. Furthermore, real surgical aerosols are complex and variable, and their distribution is difficult to express using a single, universal formula. Even if simulated surgical aerosols can roughly mimic the basic morphology and diffusion trends, they often fail to fully depict subtle details such as color. Moreover, because real surgical aerosols exhibit randomness, there are differences in distribution between simulated and real surgical aerosols. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0005] Therefore, the present invention provides a domain-adaptive endoscopic image restoration method, device and storage medium, the required training data is easy to obtain, and it can accurately remove aerosol noise in surgical images.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The first aspect of this invention provides a domain-adaptive endoscopic image restoration method, comprising:

[0008] To address aerosol noise in surgical images, simulated surgical smoke images and simulated trajectory images are generated. The trajectory simulation image is used to simulate tissue fragments that are not vaporized and float in front of the camera during surgery. The simulated surgical smoke images and the trajectory simulation images are processed by grayscale and binarization, respectively. The surgical video is extracted frame by frame, and the extracted surgical images without aerosol noise are used as the original images. The original images are then color-filtered using the grayscale-processed simulated surgical smoke images, and then the binarized trajectory simulation images are superimposed to obtain a composite image with aerosol noise.

[0009] The synthesized image and the cropped surgical image are used to train a domain-adaptive endoscopic image restoration model, enabling it to remove aerosol noise from the image.

[0010] The trajectory in the surgical image to be repaired is detected and binarized to obtain a binarized trajectory image. This binarized trajectory image is then superimposed on the surgical image to be repaired to obtain an enhanced image. The enhanced image is then input into the endoscopic image repair model to obtain the repaired surgical image.

[0011] In some embodiments, when generating the surgical smoke simulation image, winds from different directions are added to simulate surgical smoke under different conditions;

[0012] When producing the trajectory simulation images, particles of different shapes are used to enrich the types of simulated trajectories.

[0013] In some embodiments, all types of images are the same size.

[0014] In some embodiments, the endoscopic image restoration model includes a first encoder, a noise removal network, and a second decoder connected in sequence;

[0015] The first encoder employs a first encoder in a source domain coding network, which is used to encode the synthetic image, which is the source domain, and the surgical image with aerosol noise, which is the target domain, into the same latent space.

[0016] The noise removal network is used to denoise the encoding result of the image with aerosol noise output by the first encoder and output the result after removing aerosol noise in the latent space.

[0017] The second decoder employs a second decoder in a target domain decoding network, which is used to decode the result after removing aerosol noise in the latent space into an image without aerosol noise.

[0018] In some embodiments, the source domain coding network employs a first variational autoencoder, including a first encoder, a first decoder, and a first discriminator; the source domain coding network is trained using a first training dataset consisting of multiple synthetic images and multiple surgical images with aerosol noise, wherein during training, an image is randomly selected from the first training dataset as input, and the loss function used is L. VAE1 (x), expressed as follows:

[0019]

[0020] In the formula, x is the image input to the source domain coding network, and each input image is treated as a sample; Let V(D,G) be the objective function of the conditional GAN, and let D be the first discriminator, G be the first decoder, and P be the core value function of the GAN. z (z) represents the prior distribution of the noise z input to the first decoder, P data (x) is the probability distribution of x; E is the expectation function; KL() is the KL divergence function; E VAE1 (x) is the encoding result of x after it is input into the first encoder; The distribution is Gaussian, where I represents a covariance matrix with variance of 1 in all dimensions and no correlation between dimensions; L FM For multi-scale discriminator loss; L VGG To perceive loss.

[0021] In some embodiments, the target domain decoding network employs a second variational autoencoder, including a second encoder, a second decoder, and a second discriminator; the target domain decoding network is trained using a second training dataset consisting of multiple surgical images without aerosol noise, and the training function used is the same as that of the source domain encoding network.

[0022] In some embodiments, the noise removal network includes a first residual module, a dehazing module, and a second residual module connected in sequence. The dehazing module includes a multi-scale parallel large convolutional kernel module and an enhanced parallel attention module. The loss function used when training the noise removal network is L(s), expressed as follows:

[0023]

[0024] In the formula, Let V(D,G) be the objective function of the conditional GAN, where D is the second discriminator in the target domain decoding network, G is the generator in the dehazing module, E is the expectation calculation function, and M is the mapping of the noise removal network. VAE1 (s) is the encoded result of the synthetic image s with aerosol noise after being input into the first encoder, E VAE2 (g) represents the decoding result of the surgical image g without aerosol noise after being input into the second decoder; g' represents the output of the synthetic image s with aerosol noise after passing through the endoscopic image restoration model, i.e., g' = G. VAE2 (M(E VAE1 (s)));L FM For multi-scale discriminator loss; L VGG For perceived loss; L ID This is the loss from the identity mapping.

[0025] In some embodiments, a trajectory detection network is used to detect trajectories in the surgical image to be repaired. The trajectory detection network is trained using trajectory training data, and the step of obtaining the trajectory training data includes:

[0026] The surgical image without aerosol noise is used as the original image. The original image is then color-filtered using the surgical smoke simulation image to obtain a preliminary result image. The preliminary result image is then color-filtered using the trajectory simulation image to obtain a trajectory composite image, which is used as the trajectory training data.

[0027] A second aspect of the present invention provides a domain-adaptive endoscopic image restoration apparatus, comprising:

[0028] The composite image acquisition module is configured to generate a surgical smoke simulation image and a trajectory simulation image to address aerosol noise in surgical images. The trajectory simulation image is used to simulate tissue fragments that are not vaporized and float in front of the camera during surgery. The surgical smoke simulation image and the trajectory simulation image are processed by grayscale and binarization, respectively. The surgical video is extracted frame by frame, and the extracted surgical images without aerosol noise are used as the original images. The original images are then subjected to color filtering using the grayscale processed surgical smoke simulation image, and then the binarized trajectory simulation image is superimposed to obtain a composite image with aerosol noise.

[0029] The training module is configured to train a domain-adaptive endoscopic image restoration model using the synthesized image and the cropped surgical image to remove aerosol noise from the image.

[0030] The repair module is configured to detect and binarize the trajectory in the surgical image to be repaired, obtain a binarized trajectory image, overlay it with the surgical image to be repaired to obtain an enhanced image, and input the enhanced image into the endoscopic image repair model to obtain the repaired surgical image.

[0031] A third aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to perform the domain-adaptive endoscopic image restoration method according to any embodiment of the first aspect of the present invention.

[0032] Compared with the prior art, the present invention has the following characteristics and beneficial effects:

[0033] 1. Based on the image translation method, domain adaptation is introduced to enhance the generalization ability of the endoscopic image restoration model on real surgical degraded images. At the same time, aerosol noise in surgical images is divided into two categories: smoke and trajectory. The two categories are simulated separately during the simulation. This divide-and-conquer method makes the simulation of aerosol noise easier and easier to train. Randomly superimposing the two categories during training can also increase the randomness and diversity of training samples.

[0034] 2. It can detect and restore aerosol noise in endoscopic surgical images. In this process, since it is difficult to completely simulate the color of the trajectory in aerosol noise, in order to better restore the trajectory, it is necessary to first detect the trajectory in the image, and then superimpose the detection result with the surgical image containing aerosol noise as the input to the endoscopic image restoration model, and finally obtain a clean surgical image after removing aerosol noise.

[0035] 3. It also shows good restoration results for real surgical images containing aerosol noise. When using domain adaptation, the training process not only uses noise-free surgical image datasets, but also integrates real surgical smoke images. Therefore, even when faced with real surgical images containing aerosol noise, the endoscopic image restoration model can accurately remove this noise and restore a clear surgical field of view. Attached Figure Description

[0036] Figure 1 This is an overall flowchart of a domain-adaptive endoscopic image restoration method provided in the first aspect of this disclosure;

[0037] Figure 2 yes Figure 1 A schematic diagram illustrating the specific implementation process of the method;

[0038] Figure 3 It is a real surgical image, and the aerosol noise in it is divided into two types: surgical smoke and trajectory.

[0039] Figure 4 This is a schematic diagram of the structure of an electronic device provided in a third aspect embodiment of the present disclosure. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining this application and are not intended to limit this application.

[0041] Conversely, this application covers any alternatives, modifications, equivalent methods, and schemes made within the spirit and scope of this application as defined by the claims. Furthermore, to provide the public with a better understanding of this application, certain specific details are described in detail below. However, this application can be fully understood by those skilled in the art even without these detailed descriptions.

[0042] See Figure 1 , Figure 2 The first aspect of this disclosure provides a domain-adaptive endoscopic image restoration method, comprising the following steps:

[0043] Step S100: For aerosol noise in the surgical images, generate surgical smoke simulation images and trajectory simulation images respectively. The surgical smoke simulation image is used to simulate the vaporization of tissue proteins and fats caused by high temperature when surgical instruments cut tissue. The trajectory simulation image is used to simulate tissue fragments that are not vaporized and float in front of the lens. The surgical smoke simulation image is processed in grayscale, and the trajectory simulation image is processed in binarization. The surgical video is captured frame by frame, and the captured surgical images are divided into surgical images without aerosol noise and surgical images with aerosol noise. The surgical image without aerosol noise is used as the original image. The original image is color filtered using the grayscale processed surgical smoke simulation image. Then, the binarized trajectory simulation image is superimposed to obtain the synthetic image with aerosol noise.

[0044] Step S200: The domain-adaptive endoscopic image restoration model is trained using the synthesized image and the cropped surgical image from step S100, so that it has the ability to remove aerosol noise from the image.

[0045] Step S300: Detect and binarize the trajectory in the surgical image to be repaired, and superimpose the obtained binarized trajectory image with the surgical image to be repaired to obtain an enhanced image. Input the enhanced image into the endoscopic image repair model to obtain the repaired surgical image.

[0046] To address the inconsistency in aerosol distribution between simulated aerosols and those generated during real surgery, this invention introduces a domain adaptation method. The goal of domain adaptation is to apply training results from one domain to another, where the training domain is called the source domain and the domain to be applied is called the target domain. In the network, an implicit alignment-based domain adaptation method is used, whereby a network is trained to encode the source and target domains into a latent space to achieve neighborhood alignment. After training, the overall network demonstrates good recovery performance for real surgical aerosol noise.

[0047] In some embodiments, step S100 specifically includes:

[0048] Step S110: Generation of aerosol noise simulation image:

[0049] First, based on the different geometric shapes of aerosol noise in real surgical images, aerosol noise in surgical images is divided into two categories: surgical smoke and trails. Surgical smoke is caused by the vaporization of tissue proteins and fats due to high temperatures when surgical instruments such as scalpels cut tissue, while unvaporized tissue fragments float in front of the lens, forming trails. For example... Figure 3 As shown, the trajectory is a white line-shaped substance in the upper right corner.

[0050] Subsequently, based on the above definition, simulated surgical smoke images and trajectory simulation images are generated to simulate surgical smoke. When simulating surgical smoke, wind from different directions is added to the smoke to simulate different surgical smoke conditions. When simulating the trajectory, a particle system is used, and particles of different shapes are added to enrich the simulated trajectory types. To shorten the training time of the subsequent endoscopic image restoration model and reduce the computational load, the simulation image size is 256*256, the same as the size of the surgical image. The generated surgical smoke simulation image is then processed into grayscale. To avoid poor trajectory restoration results due to differences in the distribution of simulated and real trajectories, the generated trajectory simulation image needs to be converted into a binary image.

[0051] In one specific embodiment of this application, the open-source 3D rendering software Blender is used to simulate surgical smoke and trajectory.

[0052] Step S120, Surgical image classification:

[0053] The surgical video is segmented frame by frame, and the segmented surgical images are divided into surgical images without aerosol noise and surgical images with aerosol noise. Considering that most surgical videos have black borders, which are not conducive to the training of subsequent endoscopic image restoration models, it is necessary to crop the black borders in the surgical images as invalid regions to ensure that the obtained images are all surgical images without black borders. At the same time, because surgical aerosols are unevenly distributed, a surgical image containing aerosol noise may contain some aerosol noise and others not. Therefore, it is necessary to further crop the surgical images containing aerosol noise to ensure that the cropped surgical images contain most or all of the aerosol noise, and finally obtain surgical images t with aerosol noise and surgical images g without aerosol noise. In a specific embodiment of this application, the size of the obtained surgical images t with aerosol noise and surgical images g without aerosol noise is 256*256.

[0054] Step S130: Image synthesis

[0055] Using the surgical image g without aerosol noise as the original image, the original image is first filtered by the surgical smoke simulation image after grayscale processing, and then the trajectory simulation image after binarization is superimposed to obtain the synthetic image s with aerosol noise.

[0056] A first training dataset was constructed using multiple different surgical images with aerosol noise and multiple different synthetic images with aerosol noise. A second training dataset was constructed using multiple surgical images without aerosol noise, which were then used to train the endoscopic image repair model.

[0057] In one specific embodiment of this application, the color filtering principle of Adobe Photoshop is used, and the specific formula is as follows:

[0058]

[0059] The base color represents the surgical image without aerosol noise, while the mixed color represents the simulated aerosol noise image.

[0060] A surgical smoke simulation image and a trajectory simulation image are randomly selected to form an aerosol noise simulation image. Then, the above formula is used to superimpose the aerosol noise simulation image with a surgical image g without aerosol noise to obtain approximately 20,000 composite images s.

[0061] In some embodiments, see Figure 2 Step S200 includes: constructing and training a domain-adaptive endoscopic image restoration model, which includes a first encoder, a noise removal network, and a second decoder connected in sequence; wherein, the first encoder is a first encoder in the source domain encoding network, which encodes the synthetic image with aerosol noise as the source domain and the surgical image with aerosol noise as the target domain into the same latent space; the noise removal network is used to denoise the encoded result of the aerosol-noise image output by the first encoder and output the result after removing aerosol noise in the latent space; the second decoder is a second decoder in the target domain decoding network, which is used to complete the decoding of the aerosol-noise-free image, that is, to decode the result after removing aerosol noise in the latent space output by the noise removal network into an aerosol-noise-free image.

[0062] Furthermore, step S200 specifically includes:

[0063] Step S210, First encoder acquisition:

[0064] To obtain the first encoder in the domain-adaptive endoscopic image restoration model, it is necessary to construct and train the source domain encoding network. Specifically:

[0065] See Figure 2The upper part of the model, the source domain encoding network, aims to achieve domain adaptation for the endoscopic image restoration model. This involves encoding the source and target domains into a latent space. In this process, the synthetic image *s* with aerosol noise belongs to the source domain, while the real surgical image *t* with aerosol noise belongs to the target domain. Various networks can be used to train the source domain encoding network; this embodiment uses a Pix2Pix network. The source domain encoding network in this embodiment employs a variational autoencoder (VAE1), comprising a first encoder, a first decoder, and a first discriminator. The first decoder is also equivalent to the generator in the Pix2Pix model. During the training of the Pix2Pix network, the generator and discriminator engage in adversarial training. The generator attempts to generate more realistic images to deceive the first discriminator, while the first discriminator strives to improve its discrimination ability to identify the images generated by the generator. This adversarial training process improves the performance of both the generator and the discriminator. By optimizing its discrimination results, the discriminator improves its ability to recognize the generated images, thereby prompting the generator to generate higher-quality images. During training, each time an image is randomly selected from the first training dataset constructed in step S130 (this image can be a surgical image with aerosol noise or a synthetic image with aerosol noise) and input into the source domain encoding network. Let the loss function used during training be L. VAE1 (x), expressed as follows:

[0066]

[0067] In the formula:

[0068] x is the image input to the source domain coding network, and each image input to the source domain coding network is treated as a sample.

[0069] First item Let V(D,G) be the objective function of the conditional GAN, and let P be the core value function of the GAN. This defines the adversarial objectives between the discriminator and the generator, where D is the first discriminator, G is the first decoder (i.e., the generator), and P is the second most important function. z (z) represents the prior distribution of the noise z input to the generator, P data (x) represents the probability distribution of x; E is the expectation function. To sample all samples x from the real data distribution, calculate the logarithm of the probability that the first discriminator classifies it as real, and then take the average. To sample all noise z from the noise prior distribution, calculate the logarithm of the probability that the generated sample G(z) is judged as fake data by the first discriminator, and then take the average;

[0070] Second item The KL divergence is used to measure the distribution in the latent space, i.e., E.VAE1 (x) and Gaussian distribution The differences between them bring them closer together, where KL() is the KL divergence function, and E VAE1 E represents the first encoder. VAE1 (x) represents the encoding result of x after it is input into the first encoder, which follows a Gaussian distribution. In this context, I represents a covariance matrix in which all dimensions have a variance of 1 and there is no correlation between the dimensions.

[0071] The third item L FM For the multi-scale discriminator loss, the feature matching loss from pix2pixHD is adopted to better train the discriminator;

[0072] The last item L VGG The goal of perceiving loss is to preserve more texture information in real surgical images with aerosol noise.

[0073] The initial learning rate is set to 0.0002. When the number of training rounds reaches 100, the learning rate begins to decrease. Training continues for another 100 rounds until the learning rate is 0, at which point the training of the source domain encoding network is terminated, and the first encoder is obtained after training.

[0074] Step S220, Second decoder acquisition:

[0075] To obtain the second decoder in the domain-adaptive endoscopic image restoration model, it is necessary to construct and train the target domain decoding network. Specifically:

[0076] See Figure 2 The lower half of the diagram describes the target domain decoding network, which aims to decode the aerosol-free image, specifically by decoding the latent vectors of the restored surgical image in the latent space. Various networks can be used to train the target domain decoding network; here, the pix2pix network is used as well. This embodiment employs a variational autoencoder (VAE2), which includes a second encoder, a second decoder, and a second discriminator. The training data used is the aerosol-free surgical image g from the second training dataset. The loss function and termination condition used when training the target domain decoding network are the same as those used in the source domain encoding network, and will not be repeated here.

[0077] Step S230: Obtaining the noise removal network:

[0078] See Figure 2In the middle part of the image, the noise removal network aims to remove aerosol noise from the input image. The input to this network is the encoded image containing aerosol noise, output by the first encoder. The output is the restored image after the degradation is removed. This restored image is then input to the second decoder to obtain the repaired surgical image. The noise removal network in this embodiment includes a first residual module (ResNet Block), a dehazing module (Mix Structure Block), and a second residual module (ResNet Block) connected in sequence; both are existing modules. The Mix Structure Block is a dehazing module containing a multi-scale parallel large convolutional kernel module (MSPLCK) and an enhanced parallel attention module (EPA). MSPLCK has a large receptive field across multiple scales, enabling it to acquire multi-scale features of the image, thus allowing for the restoration of images containing larger smoke areas while preserving local texture. EPA can handle unevenly distributed smoke by extracting shared global information and location-dependent local information of the original features in parallel. Compared to previous single large kernel dehazing modules, MSPLCK pays more attention to local texture, and EPA can better handle unevenly distributed smoke. In this embodiment, the loss function used when training the noise removal network is L(s), and its expression is as follows:

[0079]

[0080] In the formula:

[0081] First item Let V(D,G) be the objective function of the conditional GAN, and let V(D,G) be the core value function of the GAN. Let D be the second discriminator in the target domain decoding network, and G be the generator in the dehazing module (i.e., the output layer of the dehazing module).

[0082] The second term E||M(E) VAE1 (s))-E VAE2 (g)||1 represents the average of the latent coding differences calculated for all samples (s,g) in the input data distribution, where s is a synthetic image with aerosol noise, g is a surgical image without aerosol noise, M() represents the mapping of the noise removal network, and E VAE1 (s) is the encoded result of s after it is input into the first encoder, E VAE2 (g) is the decoding result after g is input into the second decoder. The purpose of the second term is to make the distribution M(E) of the perceptually repaired image generated in the latent space... VAE1 (s)) The loss of the distribution of a surgical image g that is close to aerosol noise-free after being decoded by the target domain decoding network;

[0083] The third term, E||g'-g||1, represents the average value of the difference between g' and g, where g' represents the output of the synthetic image s with aerosol noise after passing through the endoscopic image restoration model, i.e., g' = G. VAE2 (M(E VAE1 (s))), this term is the L1 loss between the generated dehazed image and the ground truth;

[0084] The last item L ID This is the identity mapping loss. The purpose of adding this loss is to ensure that the restored image retains as much color as possible the original image (i.e., the image with aerosol noise to be restored).

[0085] The meanings of the remaining terms are all the same as L. VAE1 The meanings of the corresponding terms in (x) are the same, and will not be repeated here.

[0086] The initial learning rate is set to 0.00025. When the number of training rounds reaches 150, the learning rate begins to decrease. Training continues for another 100 rounds until the learning rate is 0, at which point the training of the noise removal network is terminated, resulting in the trained noise removal network and the trained endoscopic image restoration model.

[0087] Therefore, in step S200, the first and second encoders, which have been trained, are first obtained to provide a low-dimensional representation space for the noise removal network. Then, the weights of the first encoder and the second decoder are fixed. The synthesized image with aerosol noise is input as training data into the first encoder to obtain the representation of the synthesized image in the low-dimensional latent space. The noise removal network is then used to restore the image, and finally, the result restored by the noise removal network is input into the second decoder for decoding to obtain the final restored image. This completes the training of the entire endoscopic image restoration model, including the noise removal network.

[0088] In some embodiments, step S300 specifically includes:

[0089] S310: Construction and training of trajectory detection network:

[0090] The purpose of the trajectory detection network is to detect trajectories in aerosol noise. The output trajectory image is a binary image of the same size as the input image. In this embodiment, the Unet network is used. Because it is difficult to obtain ground truth values ​​corresponding to real surgical images containing trajectories, the training data used is still synthetic data. Specifically, the surgical image without aerosol noise extracted in step S120 is used as the original image. First, the original image is color-filtered using the surgical smoke simulation image generated in step S110 to obtain a preliminary result image. Then, the preliminary result image is color-filtered using the trajectory simulation image generated in step S110 to obtain a synthetic trajectory image, which is used as the trajectory training data. The loss function used in this embodiment to train the trajectory detection network is L. Unet The expression is as follows:

[0091]

[0092] Where i represents the pixel position in the image input to the trajectory detection network, N represents the total number of pixels in the image input to the trajectory detection network, and outputs i The targets represent the predicted pixel value at pixel position i. i The weight represents the true value of the pixel at pixel position i. i This represents the weight value at pixel position i, using the pixel value at pixel position i as its weight value.

[0093] Training stops when the trajectory detection network performs well on both synthetic images with aerosol noise and real surgical images with aerosol noise.

[0094] Step S320, Image Restoration:

[0095] First, the surgical image to be repaired is input into the trained trajectory detection network to obtain the corresponding trajectory image. Then, the obtained trajectory image is superimposed on the surgical image to be repaired to obtain an enhanced surgical image to be repaired. This enhanced surgical image is then input into the trained endoscopic image repair model to obtain the repaired surgical image, which is a clean surgical image after aerosol noise removal.

[0096] It is understood that the embodiments of this disclosure distinguish between smoke and trajectory in aerosol noise in surgical images, classifying aerosol noise in surgical images into two categories: smoke and trajectory. During simulation, the two categories are simulated separately. This divide-and-conquer approach makes the simulation of aerosol noise easier and more facilitates training. Randomly superimposing the two categories during training also increases the randomness and diversity of the training samples. The entire endoscopic image restoration model achieves the conversion from aerosol-containing images to clear images by combining the encoder-decoder capability of a variational autoencoder and a noise removal network. The introduction of the first encoder in the source domain encoding network enhances the generalization ability of the endoscopic image restoration model to different types of degraded images, while the introduction of the second encoder in the target domain decoding network ensures high-quality reconstruction of the image after aerosol removal. This architecture fully utilizes synthetic and real data, improving the practicality and effectiveness of the overall endoscopic image restoration model.

[0097] A second aspect of this disclosure provides a domain-adaptive endoscopic image restoration apparatus, comprising:

[0098] The composite image acquisition module is configured to generate a surgical smoke simulation image and a trajectory simulation image to address aerosol noise in surgical images. The trajectory simulation image is used to simulate tissue fragments that are not vaporized and float in front of the camera during surgery. The surgical smoke simulation image and the trajectory simulation image are processed by grayscale and binarization, respectively. The surgical video is extracted frame by frame, and the extracted surgical images without aerosol noise are used as the original images. The original images are then subjected to color filtering using the grayscale processed surgical smoke simulation image, and then the binarized trajectory simulation image is superimposed to obtain a composite image with aerosol noise.

[0099] The training module is configured to train a domain-adaptive endoscopic image restoration model using the synthesized image and the cropped surgical image to remove aerosol noise from the image.

[0100] The repair module is configured to detect and binarize the trajectory in the surgical image to be repaired, obtain a binarized trajectory image, overlay it with the surgical image to be repaired to obtain an enhanced image, and input the enhanced image into the endoscopic image repair model to obtain the repaired surgical image.

[0101] It should be noted that the foregoing explanation of the embodiments of the domain-adaptive endoscopic image restoration method also applies to the domain-adaptive endoscopic image restoration device of this embodiment, and will not be repeated here.

[0102] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing a computer program thereon, which is executed by a processor to perform the domain-adaptive endoscopic image restoration method of the above embodiments.

[0103] The following is for reference. Figure 4 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. It should be noted that the electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs, desktop computers, and servers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0104] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 into a random access memory (RAM) 103. The RAM 103 also stores various programs and data required for the operation of the electronic device. The processing unit 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0105] Typically, the following devices can be connected to I / O interface 105: input devices 106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 108 including, for example, magnetic tapes, hard disks, etc.; and communication devices 109. Communication device 109 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0106] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via communication device 109, or installed from storage device 108, or installed from ROM 102. When the computer program is executed by processing device 101, it performs the functions defined above in the methods of embodiments of this disclosure.

[0107] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0108] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0109] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the aforementioned field-adaptive endoscopic image restoration method.

[0110] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and Python, as well as conventional procedural programming languages ​​such as the "C-" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0113] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0114] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0115] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0116] Those skilled in the art will understand that implementing all or part of the steps of the methods in the above embodiments can be accomplished by instructing related hardware through a program. The developed program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0118] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A domain-adaptive endoscopic image restoration method, characterized in that, include: To address aerosol noise in surgical images, simulated surgical smoke images and simulated trajectory images are generated. The simulated trajectory images are used to simulate tissue fragments that are not vaporized and float in front of the camera during surgery. The simulated surgical smoke images and simulated trajectory images are then processed for grayscale and binarization, respectively. The surgical video is extracted frame by frame. The extracted surgical images without aerosol noise are used as the original images. The original images are then subjected to color filtering using a grayscale simulated surgical smoke image. Finally, a binarized trajectory simulation image is superimposed to obtain a synthetic image with aerosol noise. The synthesized image and the cropped surgical image are used to train a domain-adaptive endoscopic image restoration model, enabling it to remove aerosol noise from the image. The trajectory in the surgical image to be repaired is detected and binarized to obtain a binarized trajectory image. This binarized trajectory image is then superimposed on the surgical image to be repaired to obtain an enhanced image. The enhanced image is then input into the endoscopic image repair model to obtain the repaired surgical image.

2. The domain-adaptive endoscopic image restoration method according to claim 1, characterized in that, When generating the surgical smoke simulation image, wind from different directions is added to simulate surgical smoke under different conditions; When producing the trajectory simulation images, particles of different shapes are used to enrich the types of simulated trajectories.

3. The domain-adaptive endoscopic image restoration method according to claim 1, characterized in that, All images are the same size.

4. The domain-adaptive endoscopic image restoration method according to claim 1, characterized in that, The endoscopic image restoration model includes a first encoder, a noise removal network, and a second decoder connected in sequence. The first encoder employs a first encoder in a source domain coding network, which is used to encode the synthetic image, which is the source domain, and the surgical image with aerosol noise, which is the target domain, into the same latent space. The noise removal network is used to denoise the encoding result of the image with aerosol noise output by the first encoder and output the result after removing aerosol noise in the latent space. The second decoder employs a second decoder in a target domain decoding network, which is used to decode the result after removing aerosol noise in the latent space into an image without aerosol noise.

5. The domain-adaptive endoscopic image restoration method according to claim 4, characterized in that, The source domain coding network employs a first variational autoencoder, comprising a first encoder, a first decoder, and a first discriminator. The source domain coding network is trained using a first training dataset consisting of multiple synthetic images and multiple surgical images with aerosol noise. During training, an image is randomly selected from the first training dataset as input. Let the loss function used be L. VAE1 (x), expressed as follows: In the formula, x is the image input to the source domain coding network, and each input image is treated as a sample; Let V(D,G) be the objective function of the conditional GAN, and let D be the first discriminator, G be the first decoder, and P be the core value function of the GAN. z (z) represents the prior distribution of the noise z input to the first decoder, P data (x) is the probability distribution of x; E is the expectation function; KL() is the KL divergence function; E VAE1 (x) is the encoding result of x after it is input into the first encoder; The distribution is Gaussian, where I represents a covariance matrix with variance of 1 in all dimensions and no correlation between dimensions; L FM For multi-scale discriminator loss; L VGG To perceive loss.

6. The domain-adaptive endoscopic image restoration method according to claim 5, characterized in that, The target domain decoding network employs a second variational autoencoder, including a second encoder, a second decoder, and a second discriminator; the target domain decoding network is trained using a second training dataset consisting of multiple surgical images without aerosol noise, and the training function used is the same as that of the source domain encoding network.

7. The domain-adaptive endoscopic image restoration method according to claim 4, characterized in that, The noise removal network comprises a first residual module, a dehazing module, and a second residual module connected in sequence. The dehazing module includes a multi-scale parallel large convolutional kernel module and an enhanced parallel attention module. The loss function used to train the noise removal network is L(s), expressed as follows: In the formula, Let V(D,G) be the objective function of the conditional GAN, where D is the second discriminator in the target domain decoding network, G is the generator in the dehazing module, E is the expectation calculation function, and M is the mapping of the noise removal network. VAE1 (s) is the encoded result of the synthetic image s with aerosol noise after being input into the first encoder, E VAE2 (g) represents the decoding result of the surgical image g without aerosol noise after being input into the second decoder; g' represents the output of the synthetic image s with aerosol noise after passing through the endoscopic image restoration model, i.e., g' = G. VAE2 (M(E VAE1 (s)));L FM For multi-scale discriminator loss; L VGG For perceived loss; L ID This is the loss from the identity mapping.

8. The domain-adaptive endoscopic image restoration method according to claim 1, characterized in that, A trajectory detection network is used to detect trajectories in the surgical image to be repaired. The trajectory detection network is trained using trajectory training data, and the steps for obtaining the trajectory training data include: The surgical image without aerosol noise is used as the original image. The original image is then color-filtered using the surgical smoke simulation image to obtain a preliminary result image. The preliminary result image is then color-filtered using the trajectory simulation image to obtain a trajectory composite image, which is used as the trajectory training data.

9. A domain-adaptive endoscopic image restoration device, characterized in that, include: The synthetic image acquisition module is configured to generate a surgical smoke simulation image and a trajectory simulation image respectively for aerosol noise in surgical images. The trajectory simulation image is used to simulate tissue fragments that are not vaporized and float in front of the lens during surgery. The surgical smoke simulation image and the trajectory simulation image are respectively processed by grayscale and binarization. The surgical video is extracted frame by frame. The extracted surgical images without aerosol noise are used as the original images. The original images are then subjected to color filtering using a grayscale simulated surgical smoke image. Finally, a binarized trajectory simulation image is superimposed to obtain a synthetic image with aerosol noise. The training module is configured to train a domain-adaptive endoscopic image restoration model using the synthesized image and the cropped surgical image to remove aerosol noise from the image. The repair module is configured to detect and binarize the trajectory in the surgical image to be repaired, obtain a binarized trajectory image, overlay it with the surgical image to be repaired to obtain an enhanced image, and input the enhanced image into the endoscopic image repair model to obtain the repaired surgical image.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the domain-adaptive endoscopic image restoration method according to any one of claims 1 to 8.

Images