An endoscope video real-time smoke removing method based on smoke attention mechanism

Abstract

The present application is a kind of endoscope video real-time smoke removal method based on smoke attention mechanism.The present application relates to the field of artificial or robot-assisted minimally invasive surgery under endoscope field, and the present application is directed to a real-time algorithm for endoscope video smoke removal, which quickly and accurately detects the position of smoke in the image containing smoke and reasonably removes it.The present application generates a synthetic endoscope dataset and obtains a real endoscope dataset, and in the encoder, the endoscope video is down-sampled using image block embedding layer and image block merging layer;In the network, the texture prior information is used, the smoke posterior distribution is introduced in the residual term by guiding the residual, and the smoke removal network is optimized;The atmospheric scattering model is applied to the reconstruction process of smoke-free image by adopting the domain transformation of relaxation constraint, to realize smoke detection and removal.

Description

Technical Field

[0001] This invention relates to the field of endoscopic-guided, human- or robot-assisted minimally invasive surgery, and is a real-time smoke removal method for endoscopic video based on a smoke attention mechanism. Background Technology

[0002] With the rapid development of minimally invasive surgery, endoscopic minimally invasive surgery has become an indispensable treatment method in clinical practice. During endoscopic minimally invasive surgery, the smoke generated by medical instruments such as Gamma Knife, high-frequency electrocautery, laser scalpel, and ultrasonic scalpel during the vaporization of tissue proteins and fats can significantly affect the quality of endoscopic video, obstruct the surgeon's view, and thus pose a potential risk to the patient. The presence of smoke can also negatively impact the surgeon's surgical precision and may lead to serious surgical complications. This invention proposes a method for real-time smoke removal from endoscopic videos. Due to the heterogeneity of surgical smoke, this invention establishes a two-stage network to estimate the smoke distribution and uses hybrid embedding to guide the reconstruction of the corresponding smoke-free scene, including the estimated smoke image and the input image. The design of this network follows the classical atmospheric scattering model, making the network easier to interpret. Summary of the Invention

[0003] This invention relates to a real-time algorithm for smoke removal in endoscopic videos, specifically for minimally invasive endoscopic surgery. This algorithm rapidly and accurately detects the location of smoke in images containing smoke and removes it accordingly. Based on this, the invention proposes a real-time smoke removal method for endoscopic videos based on a smoke attention mechanism.

[0004] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0005] This invention provides a real-time smoke removal method for endoscopic videos based on a smoke attention mechanism. The invention provides the following technical solution:

[0006] A real-time smoke removal method for endoscopic videos based on a smoke attention mechanism, the method comprising the following steps:

[0007] Step 1: In the smoke attention evaluation module, the encoder uses an image block embedding layer and an image block merging layer to downsample the endoscopic video; the decoder uses a local image block expansion layer (LPE) to upsample the video.

[0008] Step 2: In the hybrid guided reconstruction module, texture prior information and guided residual terms are used as guides to reconstruct the fog-free image;

[0009] Step 3: During the reconstruction process, the atmospheric scattering model is applied to the dehazing process in the endoscopic environment using a domain transformation with relaxed constraints, and the weights of the texture prior information and guiding residual term in Step 2 are calculated.

[0010] Preferably, step 1 specifically comprises:

[0011] In the local image patch expansion layer, the input image is segmented into a series of image patches. Each image patch is flattened into a vector representation and processed through a linear layer to obtain depth features. The image patch merging layer connects the image patches and applies a linear layer to expand the feature dimension by 2 times while downsampling the resolution by half.

[0012] In the decoder, a Local Image Patch Extension (LPE) layer is used for upsampling. The LPE process is as follows:

[0013] LPE(x)=LN(PS(Conv 3×3 ((x))) (1)

[0014] Where LN is layer normalization and PS is pixel rearrangement operation;

[0015] Incorporate the predicted smoke information into the modeling network:

[0016] J(x)=I(x)+K(x)I(x)+B(x)I s (x)+ε (2)

[0017] Where J(x) is the scene radiation to be recovered, and I(x) is the input image, I s (x) is the smoke image, K(x) and B(x) are coefficients, K(x) = 1 / t(x)-1, B(x) = -1 / t(x), where t(x) is the transmittance of the medium, and ε represents the background intensity used to synthesize the smoke image;

[0018] It can be decomposed into

[0019] I s (x)=εt(x)+A(1-t(x)) (3)

[0020] Field transformation was used to avoid gradient explosion:

[0021]

[0022] Preferably, in the direct learning strategy, to avoid the vanishing gradient problem, a signal-dependent noise of the prior gradient difference is proposed, which guides the residual to introduce an additional posterior smoke distribution in the residual term, providing a gradient for optimizing the smoke removal network.

[0023] Preferably, the Hybrid Guided Reconstruction (HGE) module takes the smoke attention map t^(x) as input, calculates the residuals and biases according to formulas (4) and (3), and makes predictions for the smoke mask and the smoke-free image;

[0024] In the hybrid guided reconstruction module, the input image is integrated into the reconstruction layer through skip connections.

[0025] Preferably, step 3 specifically comprises:

[0026] Considering the exponential form of K(x) and B(x) after applying Equation (4), directly processing the field transformation may hinder the convergence of the model. Therefore, a relaxed constraint domain transformation (SFT) is adopted:

[0027]

[0028] Preferably, for smokeless pixels, when t(x) = 1, according to the definition of K(x), K(x) = 0, there is no smoke, the prior residual and the guiding residual will both be zero, and the smokeless pixel will be directly output as the final result without any risk of over-removal; the construction process of the SFT module is as follows:

[0029]

[0030] Where ^K and ^B are the estimated values ​​of K(x) and B(x), respectively;

[0031] The loss function of this algorithm can be defined as:

[0032] L total =L r +L g (7)

[0033] Wherein, reconstruction loss Lr is the mean square error between the predicted smoke-free image and the real smoke-free image, and guiding loss Lg is the mean square error between the predicted smoke image and the generated smoke image.

[0034] A real-time smoke removal system for endoscopic video based on a smoke attention mechanism, the system comprising:

[0035] The system consists of a Smoke Attention Evaluation (SAE) module and a Hybrid Guided Embedding (HGE) module. The Smoke Attention Evaluation (SAE) module is composed of a 5-level U-Net architecture, with the Swin-Transformer module as the basic unit. Two consecutive Swin-Transformer modules each employ a window-based multi-head self-attention module. The Swin-Transformer module at the end of the network integrates a W-MSA module. The Smoke Attention Evaluation (SAE) module estimates the attention map of smoke from the input image, and the Hybrid Guided Embedding (HGE) module converts the attention map into a smoke image and embeds it into the reconstruction process of the input image to achieve smoke detection and removal.

[0036] A computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a real-time smoke removal method for endoscopic video based on a smoke attention mechanism.

[0037] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement a real-time smoke removal method for endoscopic video based on a smoke attention mechanism.

[0038] The present invention has the following beneficial effects:

[0039] Compared with the prior art, the present invention:

[0040] To qualitatively and quantitatively evaluate this method, this invention generated a synthetic endoscopy dataset and obtained a real endoscopy dataset. Experimental results show that the peak signal-to-noise ratio (PSNR) of this method is 1.9% higher than existing methods, while the runtime is more than 40.9% shorter. By comparing a series of key parameters (including: PSNR, SSIM (Structural Similarity Index), LPIPS (Learned Perceptual Image Patch Similarity), BRISQUE (Blind / Unreferenced Image Spatial Quality Assessment), NIQE (Naturalness Image Quality Evaluator), and PIQE (Perception-Based Image Quality Evaluator), the method represented by this invention outperforms current mainstream methods, such as the ASM model and DCP model, in terms of design quality and computational cost.

[0041] This invention addresses the problem in minimally invasive surgery where smoke generation interferes with the accuracy of algorithms for tracking, identifying, and segmenting anatomical structures or surgical instruments in endoscopic videos. Experimental results show that the proposed method achieves a peak signal-to-noise ratio 2.79% higher than existing methods, while also significantly reducing runtime by 38.2%. Overall, compared to state-of-the-art methods, this invention offers comparable, or even superior, performance in terms of smoke removal quality and computational efficiency. Attached Figure Description

[0042] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is a network structure diagram of the present invention;

[0044] Figure 2 Setting the main parameters for the smoke attention assessment module;

[0045] Figure 3 To compare quantitative results on synthetic datasets and real datasets using various methods. Detailed Implementation

[0046] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0048] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0049] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0050] The present invention will be described in detail below with reference to specific embodiments. Specific Implementation Example 1:

[0052] according to Figures 1-3 As shown, the specific optimized technical solution adopted by the present invention to solve the above-mentioned technical problems is: The present invention relates to a real-time smoke removal method for endoscopic video based on a smoke attention mechanism.

[0053] Step 1: In the smoke attention evaluation module, the encoder uses an image block embedding layer and an image block merging layer to downsample the endoscopic video; the decoder uses a local image block expansion layer (LPE) to upsample the video.

[0054] Step 1 specifically involves:

[0055] In the image patch expansion layer, the input image is segmented into a series of image patches. Each image patch is flattened into a vector representation and processed by a linear layer to obtain depth features. The image patch merging layer connects the image patches and applies a linear layer to expand the feature dimension by 2 times while downsampling the resolution by half.

[0056] In the decoder, a Local Image Patch Extension (LPE) layer is used for upsampling. The LPE process is as follows:

[0057] LPE(x)=LN(PS(Conv 3×3 (x))) (1)

[0058] Where LN is layer normalization and PS is pixel rearrangement operation;

[0059] Incorporate the predicted smoke information into the modeling network:

[0060] J(x)=I(x)+K(x)I(x)+B(x)I s (x)+ε (2)

[0061] J(x) represents the scene radiation to be recovered, and I(x) is the input image. s (x) is the smoke image, K(x) and B(x) are coefficients, K(x) = 1 / t(x)-1, B(x) = -1 / t(x), where t(x) is the transmittance of the medium, and ε represents the background intensity used to synthesize the smoke image;

[0062] It can be decomposed into

[0063] I s (x)=εt(x)+A(1-t(x)) (3)

[0064] Field transformation was used to avoid gradient explosion:

[0065]

[0066] Step 2: In the hybrid guided reconstruction module, texture prior information and guided residual terms are used as guides to reconstruct the fog-free image;

[0067] In the direct learning strategy, to avoid the vanishing gradient problem, a signal-dependent noise of the prior gradient difference is proposed, which guides the residual to introduce an additional posterior smoke distribution in the residual term, providing a gradient for optimizing the smoke removal network.

[0068] The hybrid guided reconstruction module takes the smoke attention map t^(x) as input, calculates the residuals and biases according to formulas (4) and (3), and makes predictions for smoke masks and smoke-free images;

[0069] In the hybrid guided reconstruction module, the input image is integrated into the reconstruction layer through skip connections.

[0070] Step 3: Employ a relaxed constraint domain transform embedded in the reconstruction process of the input image to achieve smoke detection and removal.

[0071] Step 3 specifically involves:

[0072] Considering the exponential form of K(x) and B(x) after applying Equation (4), directly processing the field transformation may hinder the convergence of the model. Therefore, the domain transformation SFT with relaxed constraints is adopted:

[0073]

[0074] 6. The method according to claim 5, characterized in that:

[0075] For smokeless pixels, when t(x) = 1, according to the definition of K(x), K(x) = 0, there is no smoke, the prior residual and the guiding residual will both be zero, and the smokeless pixel will be directly output as the final result without any risk of over-removal;

[0076] The construction process of the SFT module is as follows:

[0077]

[0078] Where ^K and ^B are the estimated values ​​of K(x) and B(x), respectively;

[0079] The loss function of this algorithm can be defined as:

[0080] L total =L r +L g (7)

[0081] Wherein, the reconstruction loss Lr is the mean square error between the predicted smoke-free image and the real smoke-free image, and the guiding loss Lg is the mean square error between the predicted smoke image and the generated smoke image.

[0082] This invention provides a real-time smoke removal system for endoscopic video based on a smoke attention mechanism, the system comprising:

[0083] The system consists of a Smoke Attention Estimation (SAE) module and a Hybrid Guided Embedding and Hybrid Guided Reconstruction (HGE) module. The SAE is composed of a 5-level U-Net architecture with Swin-Transformer modules as the basic units. Two consecutive Swin-Transformer modules each employ window-based multi-head self-attention modules. The Swin-Transformer module at the end of the network integrates a W-MSA module. The SAE estimates the attention map of smoke from the input image, and the Hybrid Guided Reconstruction module converts the attention map into a smoke mask, which is then incorporated into the reconstruction process of the embedded input image to achieve smoke detection and removal. Specific Implementation Example 2:

[0085] The only difference between Embodiment 2 and Embodiment 1 of this application is that:

[0086] The purpose of this invention is to provide a real-time algorithm for smoke removal in endoscopic videos related to minimally invasive endoscopic surgery, enabling rapid and accurate detection and removal of smoke in images containing smoke. The proposed algorithm's architecture is as follows: Figure 1 As shown, this network comprises two modules: a Smoke Attention Estimator (SAE) module and a Hybrid Guided Embedding (HGE) module. The SAE consists of a 5-level U-Net architecture, using Swin-Transformer modules as basic units. Two consecutive Swin-Transformer modules each employ a window-based multi-head self-attention module. The Swin-Transformer module at the end of the network integrates a W-MSA module. For ease of convergence, the bottleneck layer consists of only two Swin-Transformer modules. The SAE estimates the smoke attention map from the input image. The Hybrid Guided Reconstruction module converts the attention map into a smoke image and then embeds it into the reconstruction process of the smoke-free image, achieving smoke detection and removal.

[0087] The objective of this invention is achieved through the following technical solution:

[0088] The smoke attention estimation module of this network

[0089] 1.1) In the encoder, downsampling is performed using an image patch embedding layer and an image patch merging layer. In the image patch expansion layer, the input image is segmented into a series of image patches, and each image patch is then flattened into a vector representation. These vectors are then processed through a linear layer to obtain depth features. The image patch merging layer concatenates the image patches and applies a linear layer to expand the feature dimension by a factor of 2 while downsampling the resolution by half. Skip connections are used to mitigate the vanishing gradient problem during training.

[0090] 1.2) In the decoder, a Local Image Patch Extension (LPE) layer is proposed for upsampling. LPE is a modified version of the Image Patch Extension (PE) layer. It aims to effectively suppress the loss of local relationships and achieves better results in the output compared to PE. The working process of LPE is as follows: Equation 1

[0091] LPE(x)=LN(PS(Conv 3×3 (x))) (1)

[0092] Where LN is layer normalization and PS is pixel rearrangement operation.

[0093] Incorporate the predicted smoke information into the modeling network: (Equation 2)

[0094] J(x)=I(x)+K(x)I(x)+B(x)I s (x)+ε (2)

[0095] Where J(x) is the scene radiation to be recovered, and I(x) is the input image, I s (x) is the smoke image, K(x) and B(x) are coefficients, K(x) = 1 / t(x)-1, B(x) = -1 / t(x), where t(x) is the transmittance of the medium, and ε represents the background intensity used to synthesize the smoke image;

[0096] It can be decomposed into (Formula 3): I s (x)=εt(x)+A(1-t(x)) (3)

[0097] Since there is no gradient of 1 / t(x) when t(x) = 0, gradient explosion will occur when t(x) approaches 0. Therefore, this invention employs field transformation to avoid the above problem: (Equation 4)

[0098] 2.1) To provide strong constraints, texture is added to the network. In the direct learning strategy, signal-related noise of the prior gradient difference is proposed to avoid the vanishing gradient problem. Guided residuals introduce an additional posterior smoke distribution into the residual term, providing gradients for optimizing the smoke removal network. The hybrid guided reconstruction module takes the smoke attention map t^(x) as input. It calculates the residuals and biases according to Equations 4 and 3, and makes predictions for the smoke mask and the smoke-free image. In the hybrid guided reconstruction module, the input image is not directly learned by applying the residuals, but is integrated into the reconstruction layer through skip connections, such as... Figure 1 As shown.

[0099] 3) Considering the exponential forms of K(x) and B(x) after applying Equation 4, directly processing the field transformation might hinder the convergence of the model. Therefore, a relaxed constraint domain transformation (SFT) is adopted: Equation 5

[0100]

[0101] 3.1) For smokeless pixels, when t(x) = 1, according to the definition of K(x), K(x) = 0. Therefore, since there is no smoke, both the prior residual and the guided residual will be zero. Thus, smokeless pixels will be directly output as the final result without any risk of over-removal.

[0102] 3.2) The construction process of the SFT module is as follows (Formula 6):

[0103]

[0104] Where ^K and ^B are the estimated values ​​of K(x) and B(x), respectively.

[0105] 4) The loss function of this algorithm can be defined as: (Equation 7)

[0106] L total =Lr+L g (7)

[0107] Wherein, the reconstruction loss Lr is the mean square error between the predicted smoke-free image and the smoke-free image, and the guiding loss Lg is the mean square error between the predicted smoke image and the generated smoke image.

[0108] The present invention will be further described in conjunction with the accompanying drawings and embodiments, and the specific process for obtaining the above-mentioned excellent results will be explained in detail:

[0109] Dataset

[0110] The dataset of this invention is generated using Blender. The synthesized image consists of a smoke-free image and a smoke image, with the smoke image generated by a fluid domain model in Blender. Therefore, each synthesized image corresponds to one smoke image and one smoke-free image. Given a synthesized image, the corresponding smoke and smoke-free images can be used as ground truth for training the network.

[0111] 1.1) In this invention, 18,140 smoke-free images were selected from the cholec80 dataset. Of the 18,140 smoke-free images selected from the dataset, 15,682 images from 21 videos were used to generate the training set, and 2,458 images from 3 videos were used to generate the test dataset.

[0112] 1.2) The training and test sets contain 47,046 and 7,374 samples, respectively. The real dataset consists of 158 images from 11 videos in the cholec80 dataset. These videos were not used in the synthetic dataset. The real dataset was used to evaluate the ability of the invention to remove smoke in real-world scenes. All images in both datasets were cropped to remove black backgrounds and resized to 256×256 pixels for efficient training and validation.

[0113] 2) Training Implementation Details

[0114] The network was implemented in PyTorch and trained on a computer equipped with two NVIDIA RTX 3090 GPUs. The training epochs were set to 300, with an initial learning rate of 7e-4. The learning rate was controlled by a cosine annealing strategy. AdamW was used as the optimizer.

[0115] 3) Quantitative assessment

[0116] This invention selects other methods for comparison, including DCP, AODNet, GridDehazeNet, and DehazeFormer. For a fair comparison, all methods were trained on synthetic datasets under the same conditions and with the same parameter settings. 3.1)

[0118] To evaluate the quality of smoke-removed images, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) were used as evaluation metrics.

[0119] In terms of PSNR and LPIPS, this method outperforms other methods, indicating its high efficiency. However, the SSIM of this algorithm is slightly lower than that of DehazeFormer-s. This can be explained by the possibility that the uncertainty in reconstruction may lead to the loss of high-frequency structural information. 3.2)

[0121] Experiments on real-world datasets involved three no-reference metrics: Blind / No-Reference Image Spatial Quality Evaluator (BRISQUE), Naturalness Image Quality Evaluator (NIQE), and Perception-Based Image Quality Evaluator (PIQE). Lower scores indicate more natural results, and vice versa. The proposed method outperformed other deep learning-based methods on all three metrics. While DCP achieved best results on BRISQUE and PIQE, its output images exhibited severe color distortion. This result demonstrates that, compared to other methods, the present invention generates more natural smoke-free images.

[0122] Experimental results demonstrate the feasibility and superior performance of the proposed network. That is, if a network trained on a synthetic endoscopy dataset can eliminate smoke in actual surgery, then the proposed network can also eliminate smoke in actual surgery. Specific Implementation Example 4:

[0124] The only difference between Embodiment 4 and Embodiment 3 of this application is that:

[0125] The present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a real-time smoke removal method for endoscopic video based on a smoke attention mechanism. Specific Implementation Example 5:

[0127] The only difference between Embodiment 5 and Embodiment 4 of this application is that:

[0128] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a real-time smoke removal method for endoscopic video based on a smoke attention mechanism.

[0129] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or N embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified. Any process or method described in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logical functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain. The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or N wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM).Furthermore, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory. It should be understood that various parts of the invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0130] The above description is merely a preferred embodiment of a real-time smoke removal method for endoscopic videos based on a smoke attention mechanism. The scope of protection for such a method is not limited to the above embodiments; all technical solutions falling within this conceptual framework are within the protection scope of this invention. It should be noted that for those skilled in the art, any improvements and variations made without departing from the principles of this invention should also be considered within the protection scope of this invention.

Claims

1. A real-time smoke removal method for endoscopic videos based on a smoke attention mechanism, characterized by: The method includes the following steps: Step 1: In the smoke attention evaluation module, the encoder part uses an image block embedding layer and an image block merging layer to downsample the endoscopic video; the decoder part uses a local image block expansion layer to upsample. Step 1 specifically involves: In the image patch embedding layer, the input image is segmented into a series of image patches. Each image patch is flattened into a vector representation and processed by a linear layer to obtain depth features. The image patch merging layer connects the image patches and applies a linear layer to expand the feature dimension by 2 times while downsampling the resolution by half. In the decoder, a local image patch expansion layer is used for upsampling, and its operation is as follows: （1） Where LN is layer normalization and PS is pixel rearrangement operation; Incorporate the predicted smoke information into the modeling network: （2） Where J(x) is the scene radiation to be recovered, and I(x) is the input image, I s (x) is the smoke image, K(x) and B(x) are coefficients, K(x)=1 / t(x)-1, B(x)=-1 / t(x), where t(x) is the transmittance of the medium, and ε represents the background intensity used to synthesize the smoke image; It can be decomposed into （3） A domain transformation was employed to avoid the gradient explosion problem: （4）； Step 2: In the hybrid guided reconstruction module, texture prior information and guided residual terms are used as guides to reconstruct the fog-free image; The hybrid guided reconstruction module takes the smoke attention map tˆ(x) as input, calculates the residuals and biases according to formulas (4) and (3), and makes predictions for smoke images and smoke-free images; In the hybrid guided reconstruction module, the input image is integrated into the reconstruction layer through skip connections; Step 3: During the reconstruction process, a relaxed constraint domain transformation is used to apply the traditional atmospheric scattering model to the dehazing process in the endoscopic environment, and the weights of the texture prior information and guiding residual term in Step 2 are calculated.

2. The method according to claim 1, characterized in that: In the direct learning strategy, to avoid the gradient vanishing problem, a prior gradient difference signal-related noise is proposed, and a posterior smoke distribution is introduced into the guiding residual term to provide a gradient for optimizing the smoke removal network.

3. The method according to claim 2, characterized in that: Step 3 specifically involves: Considering the exponential form of K(x) and B(x) after applying Equation (4), directly processing the domain transformation may hinder the convergence of the model. Therefore, a domain transformation SFT with relaxed constraints is adopted: （5）。 4. The method according to claim 3, characterized in that: For smokeless pixels, when t(x) = 1, according to the definition of K(x), K(x) = 0, there is no smoke, the prior residual and the guiding residual will both be zero, and the smokeless pixel will be directly output as the final result without any risk of over-removal; The construction process of the SFT module is as follows: （6）； Where ˆK and ˆB are the estimated values ​​of K(x) and B(x), respectively; The loss function of this algorithm can be defined as: （7） Wherein, reconstruction loss Lr is the mean square error between the predicted smoke-free image and the real smoke-free image, and guiding loss Lg is the mean square error between the predicted smoke image and the generated smoke image.

5. A real-time smoke removal system for endoscopic video based on a smoke attention mechanism, the system operating based on the real-time smoke removal method for endoscopic video based on a smoke attention mechanism as claimed in claim 1, characterized in that: The system includes: The system consists of a smoke attention evaluation module (SAE) and a hybrid guided embedding module (HGE). The SAE is composed of a 5-level U-Net architecture with Swin-Transformer modules as the basic units. The two consecutive Swin-Transformer modules employ a window-based multi-head self-attention module and a sliding window-based multi-head self-attention module, respectively. The SAE estimates the smoke attention map from the input image, and the hybrid guided reconstruction module converts the attention map into a smoke image, which is then used in the reconstruction process of the smoke-free image to achieve smoke detection and removal.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method as described in any one of claims 1-4.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the method according to any one of claims 1-4.

Images