Deep learning method for haze removal in surgical images

Abstract

The present invention proposes a deep learning method for surgical image smoke removal, which effectively improves the visibility of the surgical environment. The method proposes to divide the smoke removal task into synthetic data fully supervised smoke removal and real data self-supervised smoke removal. As shown in Figure 1, the flow chart of the deep learning method for surgical image smoke removal uses the U-Net structure to design a fully supervised smoke removal network and a self-supervised smoke removal network. First, a fully supervised learning model is constructed and pre-trained using a synthetic data set to predict the image after smoke removal. s , transmittance t s and smoke feature map F and obtain the pre-trained model. Then, the pre-trained model is used to perform self-supervised learning on real smoke images to achieve network optimization. This method can accurately estimate the smoke model, remove smoke from real surgical images, and fully restore image details.

Description

Technical Field

[0001] The present invention relates to a deep learning method for removing smoke from surgical images, and belongs to the fields of computer vision and medical image processing. Background Art

[0002] With the development of computer technology, the application of computer image processing technology in the medical field has been continuously developing, which provides a new technical support for clinical medicine. It is of great significance to use computer image processing methods to solve clinical medical research. In the process of clinical minimally invasive surgery, it is usually necessary to use a laparoscope to observe the lesion area and surrounding tissues. The images taken by the laparoscope are easily obscured by smoke due to the temperature difference between the patient's body and outside the body, the burning of the lesion tissue, etc. Therefore, we proposed a deep learning method for surgical image desmog. This algorithm can remove smoke from surgical images and effectively improve the visibility of the surgical environment.

[0003] At present, the methods for surgical image desmog removal mainly include atmospheric model-based and deep learning desmog removal methods. The atmospheric model-based desmog removal method uses the atmospheric scattering model to build a model, and applies the natural image desmog removal method to laparoscopic surgical image desmog. This type of method has a good recovery effect on the obstruction of surgical field caused by lens fogging, but it is not effective for the thick smoke produced by ultrasonic scalpels and electrosurgical equipment burning lesions. Deep learning methods have shown their advancement in medical image analysis, and solving the atmospheric scattering model through deep learning has also been proposed. This method improves the desmog effect of laparoscopic surgical images. However, due to the lack of surgical video resources and the cumbersome work of data set collation, the image processing results lack real labels and the desmog effect is poor.

[0004] Therefore, the present invention proposes a deep learning method for surgical image smoke removal, which mainly includes a fully supervised learning module and a self-supervised learning module. First, a fully supervised learning model is constructed and pre-trained using a synthetic dataset to learn the image J after smoke removal. s , transmittance t s and smoke feature map F, and obtain a pre-trained model. Then, the pre-trained model is used to perform self-supervised learning on real smoke images to achieve network fine-tuning and optimization. This method can accurately estimate the smoke model, remove smoke from real surgical images, and fully restore image details, which provides a relatively advanced clinical diagnosis and treatment auxiliary means for medical research. Summary of the invention

[0005] The purpose of this invention is to design a deep learning method for surgical image smoke removal to effectively improve the visibility of the surgical environment field of view.

[0006] In order to achieve the above objectives, this method first proposes a deep learning method for surgical image smoke removal, which mainly includes a fully supervised learning module and a self-supervised learning module. The fully supervised learning model is pre-trained with a synthetic dataset, and then the pre-trained model is used to perform self-supervised learning on real smoke images to achieve network fine-tuning and optimization.

[0007] The present invention comprises the following steps:

[0008] (1) Take X surgical video clips and divide them into clear smoke-free video sequences V 1_clear and smoke video sequence V 2_hazy , select N frames of images from each video sequence, and 1_clear The data set is processed by adding smoke, and the concentration of the added smoke is set to α = {0.25, 0.5, 0.75, 1}, and V 1_hazy The dataset contains 4 images with different smoke concentrations. 1_hazy Label V marked with "0", clear smoke-free 1_clear Marked as "1";

[0009] (2) Data set V 1_hazy and V 2_hazy Divide the training set and test set according to P:1-P(P∈(0,1));

[0010] (3) Construct a fully supervised smoke removal model and a self-supervised smoke removal model to train V 1_hazy and V 2_hazy ;

[0011] (4) Design the full supervision loss function L respectively fully-sup And the self-supervised loss function L self-sup , the formula is as follows:

[0012]

[0013]

[0014] According to the U-Net fully supervised smoke removal model Net-F and the self-supervised smoke removal model Net-S described in step (3), they respectively include net_g 0 , net_g 1 , net_g 2 Three sub-networks, through which feature images are extracted, including smoke removal image J s , transmittance t s and smoke screen characteristic diagram F, where A s ∈(0, 1) represents ambient light, Represents the output image with smoke, and the relationship between different feature maps is shown in Formula 3:

[0015]

[0016] The fully supervised loss function (Formula 1) described in step (4) consists of three parts: L con Represents the consistency loss function:

[0017]

[0018] Among them, They are the synthetic smoke image and the network output results, Respectively represent net_g 0 、net_g 1 、net_g 2 The output result of * indicates the matrix corresponding point multiplication operation, γ J , γ F are the weights of the loss function, MSE is the mean absolute error (MSE); L rec Represents the reconstruction loss function:

[0019]

[0020] Among them J gt Indicates a clear image without smoke; L TV Represents the smoke screen prior loss function:

[0021]

[0022] D x F=F(x+1,y,c)-F(x,y,c) (7)

[0023] D y F=F(x,y+1,c)-F(x,y,c) (8)

[0024] D c F=F(x,y,c+1)-F(x,y,c) (9)

[0025] Among them, x, y, and c represent the horizontal coordinate, vertical coordinate, and number of network channels in image F, respectively; i is the number of pixels in image F; D x , D y , D c represent the derivatives along x, y, and c respectively.

[0026] The self-supervised loss function (Formula 2) described in step (4) consists of three parts, where L t Defined as:

[0027]

[0028] Among them, ε 1 , ε 2 are constants, γ t is the weight, mean means the mean operation of all elements in the matrix; Defined as:

[0029]

[0030] where |·| is the absolute value operation, γ F is the weight.

[0031] The beneficial effects of the present invention are as follows: the present invention proposes a deep learning method for surgical image smoke removal, which effectively improves the visibility of the surgical environment. The present invention proposes to divide the smoke removal task into synthetic data fully supervised smoke removal and real data self-supervised smoke removal, and designs a fully supervised smoke removal network and a self-supervised smoke removal network based on the U-Net structure. First, a fully supervised learning model is constructed and pre-trained using a synthetic data set to learn the J image after smoke removal. s , transmittance t s The pre-trained model is obtained by using the smoke feature map F. Then, the pre-trained model is used to perform self-supervised learning on real smoke images to achieve network fine-tuning and optimization. This method can accurately estimate the smoke model, remove smoke from real surgical images, and fully restore image details. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 : Block diagram of the deep learning approach for surgical image haze removal. DETAILED DESCRIPTION

[0033] After obtaining the surgical video data, the videos are numbered in sequence, and X surgical video clips are cut from all the videos as the video data obtained by preliminary screening, and then divided into clear smoke-free video sequences V 1_clear and smoke video sequence V 2_hazy Each video sequence contains N frames of images, and the video sequence data files are numbered. 1_clear After backup, 1_clear The dataset is processed by adding smoke, and the concentration of the added smoke is set to α = {0.25, 0.5, 0.75, 1}. The larger the concentration value, the more serious the image is blocked. A synthetic dataset V containing 4 different smoke concentrations is obtained. 1_hazy , label the above data set, and V 1_hazy Marked as "0", clear and smoke-free label V 1_clear Marked as "1", after labeling, the dataset V 1_hazy and V 2_hazyThe training set and test set are divided according to P:1-P(P∈(0,1)).

[0034] Construct a deep desmog model for surgical images. This method is based on the U-Net structure to design a fully supervised desmog network Net-F and a self-supervised desmog network Net-S. The input of the Net-F neural network is the labeled V 1_hazy , that is, in the fully supervised desmog network, the input part of the data set is paired data V with labels 1_hazy and V 1_clear Since it is difficult to obtain a real smoke-free label in a real surgical environment, the input of the Net-S neural network is an unlabeled V 2_hazy , the above networks all contain net_g 0 , net_g 1 , net_g 2 Three sub-networks, the three sub-networks are consistent in network structure, and feature images are extracted through the three sub-networks, including the image after smoke removal. s , perspective rate t s and smoke screen characteristic diagram F, where A s ∈(0, 1) represents ambient light, Indicates output with smoke image.

[0035] This method divides the surgical image desmog task into synthetic data fully supervised desmog and real data self-supervised desmog, and designs different types of loss functions. The fully supervised loss function is shown in Formula 1, and the self-supervised loss function is shown in Formula 2. γ J , γ F , γ t Represents the weight values ​​corresponding to different types of loss functions. Different loss functions are designed using smoke prior knowledge. rec calculate With J gt The difference, L con Calculate the input smoke image I and the output smoke image The difference, L TV represents the smoke prior loss function, using smoke prior knowledge to ensure net_g 2 Output accurate smoke feature map F, for transmittance t s Using the loss function L t , are coefficients based on the inverse tangent function.

[0036] The loss functions used in the fully supervised haze removal model include L rec , L con and L TV , the loss functions used in the self-supervised smoke removal model include L con , LTV , L t and coefficient Loss function L rec and L con MSE loss is used to calculate the difference between the parameter estimate and the true value of the parameter. I represents the input image with smoke. represents the smoke image obtained through network calculation, * represents the matrix corresponding point multiplication operation, Indicates passing through subnet net_g 0 , net_g 1 , net_g 2 The obtained feature map, J gt Indicates V 1_clear The true label of the dataset, L rec calculate With J gt The difference, L con Calculate the input smoke image I and The difference, L TV is the smoke screen prior loss function, ε 1 =1×10 -5 , ε 2 =1×10 3 .

[0037] This method mainly evaluates the smoke removal performance by calculating the peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and mean square error (MSE). The formula is as follows:

[0038]

[0039]

[0040]

[0041] The above formulas represent different objective standards for image quality evaluation. PSNR (Peak Signal to Noise Ratio) represents the peak signal-to-noise ratio, as shown in formula (12). The calculation unit is dB, and n represents the number of bits of the sampled value. Generally, the larger the PSNR value, the less image distortion and the better the image quality. MSE (Mean Square Error) represents a measure of the difference between the parameter estimate and the true value of the parameter, as shown in formula (13). m represents the number of samples. Represents the predicted value, y i represents the target value. The smaller the MSE value, the smaller the difference between the two images. SSIM (Structural Similarity) represents the structural similarity index, which is an index to measure the similarity between two images, as shown in formula (14), where μ xis the mean value of x, μ y is the mean value of y, is the variance of x, is the variance of y, σ xy represents the covariance of y and y, c 1 =(k 1 L) 2 , c 2 =(k 2 L) 2 is a constant used to maintain stability, L is the numerical range of image pixel values, k 1 =0.01, k 2 =0.03, the range of structural similarity is [0, 1], and the closer the value is to 1, the greater the image similarity.

[0042] It should be noted that the above is only an embodiment of the present invention, which is only for explaining the present invention and does not limit the patent scope of the present invention. Any obvious changes that belong to the technical concept of the present invention are also within the protection scope of the present invention.

Claims

1. A deep learning method for removing smoke from surgical images, characterized in that: The steps include: (1) Take X surgical video clips and divide them into clear and smoke-free video sequences V 1_clcar and smoke video sequence V 2_hazy , select N frames of images from each video sequence, and 1_clear The data set is processed by adding smoke, and the concentration of the added smoke is set to α = {0.25, 0.5, 0.75, 1}, and V 1_hazy The dataset contains 4 images with different smoke concentrations. 1_hazy The clear and smoke-free label V1_clear is marked as "0" and the clear and smoke-free label V1_clear is marked as "1"; (2) Data set V 1_hazy and V 2_hazy Divide the training set and test set according to P:1-P(P∈(0,1)); (3) Construct a fully supervised smoke removal model and a self-supervised smoke removal model to train V 1_hazy and V 2_hazy ; (4) Design the full supervision loss function L respectively fully-sup And the self-supervised loss function L self-sup , the formula is as follows: The fully supervised desmog model and the self-supervised desmog model in step (3) are based on the U-Net fully supervised desmog model Net-F and the self-supervised desmog model Net-S, which respectively include three sub-networks, net_g0, net_g1, and net_g2. The feature images are extracted through the three sub-networks, including the desmog image J s , transmittance t s and smoke screen characteristic diagram F, where A s ∈(0, 1) represents ambient light, Represents the output image with smoke, and the relationship between different feature maps is shown in Formula 3: The formula (1) of the fully supervised loss function described in step (4) consists of three parts: L con Represents the consistency loss function: Among them, They are the synthetic smoke image and the network output results, Respectively represent the output results of net_g0, net_g1, and net_g2, * represents the matrix corresponding point multiplication operation, γ J , γ F are the weights of the loss function, MSE is the mean absolute error (MSE); L rec Represents the reconstruction loss function: Among them J gt Clear image without smoke; L TV Represents the smoke screen prior loss function: D x F=F(x+1,y,c)-F(x,y,c) (7) D y F=F(x,y+1,c)-F(x,y,c) (8) D c F=F(x,y,c+1)-F(x,y,c) (9) Among them, x, V, and c represent the horizontal coordinate, vertical coordinate, and number of network channels in image F, respectively; i is the number of pixels in image F; D x , D y , D c Respectively represent the derivative along x, y, and c; The formula (2) of the self-supervised loss function described in step (4) consists of three parts, where L t Defined as: Among them, ε1 and ε2 are constants, γ t is the weight, mean means the mean operation of all elements in the matrix; Defined as: where |·| is the absolute value operation, γ F is the weight.

Images