Image desmog removal methods, devices, and electronic equipment based on smoke removal models

Abstract

This disclosure provides an image desmoking method, apparatus, and electronic device based on a smoke removal model, which can be applied to the field of image processing technology. The method includes: processing a smoky image using a smoke detector to obtain a smoke mask image; performing M cascaded upsampling stages and M cascaded downsampling stages on the smoky image and the smoke mask image to output a predicted smoke-free image; wherein any Kth downsampling stage includes downsampling feature extraction and downsampling feature fusion of the K-1th downsampling output and the smoky image, the downsampling feature extraction including: dividing the Kth preprocessed features according to the number of branches N to obtain N image branch features; for each image branch feature, performing a first feature extraction using a dilated convolution enhancement module to obtain local image features; and performing a second feature extraction using a visual state space feature extraction module to obtain global image features.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and specifically to an image desmoking method, apparatus, and electronic device based on a smoke removal model. Background Technology

[0002] Electrocautery is commonly used to heat tissue to reduce bleeding; however, its use during laparoscopic surgery generates smoke, severely impacting image observation of tissue sites. Existing desmearing methods rely on convolutional architectures, but the inherent local receptive field of convolutional operations limits their ability to capture long-range global information, resulting in low efficiency and accuracy in image restoration. Furthermore, improving model performance requires deeper backbones or more complex network architectures, significantly increasing model size and computational complexity, and also reducing the real-time performance of image desmearing. Summary of the Invention

[0003] In view of the above problems, this disclosure provides an image desmoking method, apparatus and electronic device based on a smoke removal model.

[0004] According to a first aspect of this disclosure, an image desmoking method based on a smoke removal model is provided, comprising: processing a smoky image using a smoke detector to obtain a smoke mask image, wherein the smoky image represents a frame image rendered with smoke in a laparoscopic surgical device, and the pixel values ​​of the smoke mask image represent the smoke concentration; performing M cascaded upsampling stages and M cascaded downsampling stages on the smoky image and the smoke mask image to output a predicted smoke-free image, wherein any Kth downsampling stage includes downsampling feature extraction and downsampling feature fusion of the K-1th downsampling output result and the smoky image, and the K-1th downsampling output result is a predicted smoke-free image. The K-1th downsampling fusion feature is obtained by processing the smoke image and the smoke mask image in the K-1th downsampling stage. The downsampling feature extraction includes: dividing the K-th preprocessing feature according to the number of branches N to obtain N image branch features, where the K-th preprocessing feature is obtained by preprocessing the K-1th downsampling fusion feature and the smoke image feature, where the smoke image feature is extracted from the smoke image; for each image branch feature, the first feature extraction is performed using the dilated convolution enhancement module to obtain the local image features; the second feature extraction is performed using the visual state space feature extraction module to obtain the global image features.

[0005] According to an embodiment of the present disclosure, preprocessing the (K-1)-th downsampling fusion feature and the smoky image feature includes: in the processing of the K=1-th downsampling stage, performing convolutional processing on the smoke mask image feature and the smoky image feature to obtain the first preprocessing feature, where the smoke mask image feature is extracted from the smoke mask image; in the processing of the K=2-th downsampling stage, performing downsampling processing on the first downsampling fusion feature and the smoky image feature to obtain the second preprocessing feature; in the processing of the K>2-th downsampling stage, performing processing on the (K-1)-th downsampling fusion feature, the (K-1)-th preprocessing feature, and the pooling feature to obtain the K-th preprocessing feature, where the pooling feature is obtained based on performing a pooling operation on the smoky image feature.

[0006] According to an embodiment of the present disclosure, any K-th upsampling stage processing includes: performing upsampling feature extraction and upsampling feature fusion on the (M-K+1)-th downsampling fusion feature and the upsampling feature of the (K-1)-th upsampling output result to obtain the K-th upsampling fusion feature; performing preprocessing on the K-th upsampling fusion feature to obtain the K-th upsampling feature, where the (K-1)-th upsampling output result is the (K-1)-th upsampling feature obtained by performing the (K-1)-th upsampling stage processing on the (M-K+2)-th downsampling fusion feature and the (K-2)-th upsampling output result.

[0007] According to an embodiment of the present disclosure, performing preprocessing on the K-th upsampling fusion feature to obtain the K-th upsampling feature includes: in the processing of the K<M-th upsampling stage, performing upsampling processing on the K-th upsampling fusion feature to obtain the K-th upsampling feature; in the processing of the K=M-th upsampling stage, performing convolutional processing on the M-th upsampling fusion feature to obtain the M-th upsampling feature.

[0008] According to an embodiment of the present disclosure, performing M cascaded upsampling stage processing and M cascaded downsampling stage processing on the smoky image and the smoke mask image, and outputting a predicted smokeless image further includes: fusing the M-th upsampling feature and the smoky image feature to obtain the predicted smokeless image.

[0009] According to an embodiment of the present disclosure, downsampling feature fusion includes: fusing N image global features to obtain a global fusion feature; performing pooling processing on the global fusion feature to obtain N weight vectors corresponding to N branches; respectively performing normalization processing on the N weight vectors to obtain N weight factors; weighting the N weight factors and the N image global features to obtain the K-th downsampling fusion feature.

[0010] According to embodiments of this disclosure, the smoke removal model is trained based on the following operations: acquiring training samples, including smoke-free images; rendering smoke on the smoke-free images to obtain smoke-containing images; processing the smoke-containing images using a smoke detector to obtain smoke mask images; processing the smoke mask images and smoke-containing images according to the smoke removal model to obtain predicted smoke-free images; and training the smoke removal model based on the predicted smoke-free images and smoke-free images to obtain the trained smoke removal model.

[0011] According to embodiments of this disclosure, training a smoke removal model based on a sample predicted smoke-free image and a sample smoke-free image to obtain the trained smoke removal model includes: calculating a first loss value between the sample predicted smoke-free image and the sample smoke-free image using a pixel-aware loss function; calculating a second loss value between the sample predicted smoke-free image and the sample smoke-free image using a feature space-aware loss function; calculating a third loss value between the sample predicted smoke-free image and the sample smoke-free image using a multi-scale structure loss function; calculating a fourth loss value between the sample predicted smoke-free image using a smoke-aware loss function; obtaining a target loss value based on the first loss value, the second loss value, the third loss value, and the fourth loss value; and training the smoke removal model based on the target loss value to obtain the trained smoke removal model.

[0012] The second aspect of this disclosure provides an image desmoking device based on a smoke removal model, comprising: a detection module for processing a smoky image using a smoke detector to obtain a smoke mask image, wherein the smoky image represents a frame image rendered with smoke in a laparoscopic surgical device, and the pixel values ​​of the smoke mask image represent the smoke concentration; and a desmoking module for performing M cascaded upsampling stages and M cascaded downsampling stages on the smoky image and the smoke mask image, outputting a predicted smoke-free image, wherein any Kth downsampling stage includes downsampling feature extraction and downsampling feature fusion of the K-1th downsampling output and the smoky image, and the K-1th downsampling output... The result is the (K-1)th downsampling fusion feature obtained by processing the smoke image and the smoke mask image in the (K-1)th downsampling stage. The downsampling feature extraction includes: dividing the Kth preprocessing feature according to the number of branches N to obtain N image branch features, where the Kth preprocessing feature is obtained by preprocessing the K-1th downsampling fusion feature and the smoke image feature, where the smoke image feature is extracted from the smoke image; for each image branch feature, the first feature extraction is performed using the dilated convolution enhancement module to obtain the local image features; the second feature extraction is performed using the visual state space feature extraction module to obtain the global image features.

[0013] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] According to the image desmoking method, apparatus, and electronic device based on the smoke removal model provided in this disclosure, a smoke mask image is obtained by processing a smoke image using a smoke detector. The smoke image and the smoke mask image are then processed through M cascaded upsampling stages and M cascaded downsampling stages to output a predicted smoke-free image. Since the smoke mask image output by the smoke detector can reflect the actual degree of smoke concentration in different regions, it provides an accurate reference for the smoke removal model, improving the accuracy of smoke removal. The smoke removal model adopts a multi-branch structure, utilizing the dilated convolution enhancement module and the visual state space feature extraction module in each branch to first perform point-by-point convolution to enhance local pixel correlation, and then extract global enhanced features, thereby improving feature inference speed and more efficiently removing non-uniformly distributed smoke. Furthermore, the multi-branch structure can reduce the number of channels in each branch, reducing the number of model parameters, thereby reducing the computational load of convolution operations and the computational load in the visual state space feature extraction module caused by the number of channels. This achieves a trade-off between model complexity and inference speed while ensuring good smoke removal performance. Attached Figure Description

[0015] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 A flowchart of an image desmoking method based on a smoke removal model according to an embodiment of the present disclosure is shown.

[0017] Figure 2 An example schematic diagram of a smoke removal model according to an embodiment of the present disclosure is shown.

[0018] Figure 3A An example schematic diagram of downsampling feature extraction according to an embodiment of the present disclosure is shown.

[0019] Figure 3B An example schematic diagram of downsampling feature fusion according to an embodiment of the present disclosure is shown.

[0020] Figure 4 A structural block diagram of an image desmoke device based on a smoke removal model according to an embodiment of the present disclosure is shown.

[0021] Figure 5A block diagram schematically illustrates an electronic device suitable for implementing an image desmog method based on a smoke removal model according to embodiments of the present disclosure. Detailed Implementation

[0022] The embodiments according to this disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of this disclosure.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] In developing this disclosure, it was discovered that electrocautery is commonly used to heat tissue to reduce bleeding. However, during laparoscopic surgery, electrocautery generates smoke, severely affecting image observation of tissue sites. Existing desmearing methods rely on convolutional architectures. The inherent local receptive field of convolutional operations leads to limitations in capturing long-range global information, resulting in low efficiency and accuracy in image restoration. Furthermore, improving model performance requires deeper backbones or more complex network architectures, significantly increasing model size and computational complexity, and also reducing the real-time performance of image desmearing.

[0027] Meanwhile, due to the memory limitations of laparoscopic equipment and the real-time requirement for smoke removal in laparoscopic surgical images, there is an urgent need to design a lightweight and effective smoke removal model that can be better embedded into laparoscopic surgical equipment to achieve efficient smoke removal.

[0028] In view of this, embodiments of the present disclosure provide an image desmoking method, apparatus, and electronic device based on a smoke removal model. The method includes: processing a smoky image using a smoke detector to obtain a smoke mask image; performing M cascaded upsampling stages and M cascaded downsampling stages on the smoky image and the smoke mask image to output a predicted smoke-free image; wherein any Kth downsampling stage includes downsampling feature extraction and downsampling feature fusion of the (K-1)th downsampling output and the smoky image, the downsampling feature extraction including: dividing the Kth preprocessed features according to the number of branches N to obtain N image branch features; for each image branch feature, performing a first feature extraction using a dilated convolution enhancement module to obtain local image features; and performing a second feature extraction using a visual state space feature extraction module to obtain global image features.

[0029] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0030] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0031] Figure 1 A flowchart of an image desmoking method based on a smoke removal model according to an embodiment of the present disclosure is shown.

[0032] like Figure 1 As shown, the method 100 includes operations S110 to S120.

[0033] In operation S110, the smoke detector is used to process the smoke image to obtain a smoke mask image.

[0034] According to embodiments of this disclosure, video from a laparoscopic surgical device is processed using Open Source Computer Vision Library (OpenCV) technology to extract individual, high-quality frame images. Smoke images represent frame images rendered with smoke within the laparoscopic surgical device.

[0035] According to embodiments of this disclosure, the smoke detector is constructed based on an image segmentation network. The smoke detector can employ a 6-layer image segmentation network (Convolutional Networks for Biomedical Image Segmentation, U-Net) structure, including an encoder, decoder, and skip connections. The number of channels in the image segmentation network is set to 8, 16, 24, 32, 48, and 64, respectively. The first three layers of the image segmentation network use convolutional modules to extract shallow features, while the last three layers each use a parallel visual state space module with four branches. The number of branches in each branch is one-quarter of the initial input image segmentation network features, used to reduce the number of model parameters.

[0036] According to embodiments of this disclosure, the skip connection path uses a Channel Attention Block (CAB) module and a Spatial Attention Block (SAB) module for feature fusion to finally obtain a smoke mask image.

[0037] According to embodiments of this disclosure, the pixel values ​​of the smoke mask image represent the smoke concentration, and the pixel values ​​range from 0 to 1. A higher pixel value indicates a greater smoke concentration.

[0038] In operation S120, the smoke image and the smoke mask image are processed through M cascaded upsampling stages and M cascaded downsampling stages to output the predicted smoke-free image.

[0039] According to embodiments of this disclosure, any Kth downsampling stage processing includes downsampling feature extraction and downsampling feature fusion of the K-1th downsampling output result and the smoke image. The K-1th downsampling output result is the K-1th downsampling fusion feature obtained by performing the K-1th downsampling stage processing on the smoke image and the smoke mask image. The downsampling feature extraction includes: dividing the Kth preprocessing feature according to the number of branches N to obtain N image branch features, wherein the Kth preprocessing feature is obtained by preprocessing the K-1th downsampling fusion feature and the smoke image feature, wherein the smoke image feature is extracted from the smoke image; for each image branch feature, a first feature extraction is performed using a dilated convolution enhancement module to obtain local image features; and a second feature extraction is performed using a visual state space feature extraction module to obtain global image features.

[0040] According to embodiments of this disclosure, the smoke removal model includes an encoder-decoder structure with M cascaded upsampling stages and an encoder-decoder structure with M cascaded downsampling stages.

[0041] According to embodiments of this disclosure, the (K-1)th downsampling output is the (K-1)th downsampling fusion feature obtained by processing the smoke image and the smoke mask image in the (K-1)th downsampling stage. Each (K)th downsampling stage processing includes downsampling feature extraction and downsampling feature fusion of the (K-1)th downsampling output and the smoke image. M and K are both positive integers.

[0042] According to embodiments of this disclosure, in the downsampling feature extraction process, N branches are set, where N is a positive integer. Each branch includes a dilated convolution enhancement module and a visual state space feature extraction module. The dilated convolution enhancement module is constructed based on a dilated convolutional neural network and uses pointwise convolution to enhance pixels. The visual state space feature extraction module is obtained by global modeling based on linear-time sequence modeling with selective state spaces (Mamba).

[0043] According to embodiments of this disclosure, the smoke image features obtained by feature extraction from a smoke image, and the Kth preprocessed features obtained by preprocessing the (K-1)th downsampling fusion features and smoke image features, may include: performing feature fusion, downsampling, and other processing on the (K-1)th downsampling fusion features and smoke image features to obtain the Kth preprocessed features.

[0044] According to an embodiment of this disclosure, the Kth preprocessing feature is divided into N equal parts according to the number of branches N, resulting in N image branch features, each of which is a local feature of the Kth preprocessing feature.

[0045] According to embodiments of this disclosure, N image branch features are each input into their corresponding branches. For each branch's image branch features, a first feature extraction is performed using a dilated convolution enhancement module to obtain local image features. Then, a second feature extraction is performed using a visual state space feature extraction module to obtain global image features. Finally, N global image features output from the N branches are obtained.

[0046] According to embodiments of this disclosure, the dilated convolution enhancement modules in each branch have different convolution dilation rate parameters to achieve different degrees of dilation in learning and enhancing local features.

[0047] According to embodiments of this disclosure, local image features characterize the enhanced inter-pixel correlation in local spatial regions, while global image features characterize the enhanced linear complex information of global pixels.

[0048] According to embodiments of this disclosure, feature fusion is performed on N global image features to obtain the Kth downsampling fusion feature, and then M cascaded upsampling stages are performed on the M downsampling fusion features obtained from the M downsampling stages to obtain the predicted smoke-free image.

[0049] According to embodiments of this disclosure, a smoke-free image is predicted to represent a high-resolution image after smoke removal.

[0050] According to embodiments of this disclosure, since the smoke mask image output by the smoke detector can reflect the actual degree of smoke concentration in different areas, it provides an accurate reference for the smoke removal process of the smoke removal model, thereby improving the smoke removal accuracy of the smoke removal model. The smoke removal model adopts a multi-branch structure, which utilizes the dilated convolution enhancement module and the visual state space feature extraction module in each branch to first perform point-by-point convolution to enhance local pixel correlation, and then extract global enhancement features, thereby improving the feature inference speed and removing non-uniformly distributed smoke more efficiently. In addition, the multi-branch structure can reduce the number of channels in each branch and reduce the number of model parameters, thereby reducing the convolution operation load and the computational load in the visual state space feature extraction module caused by the number of channels. Under the premise of ensuring good smoke removal performance, a trade-off between model complexity and inference speed is achieved.

[0051] According to embodiments of this disclosure, preprocessing the (K-1)th downsampling fusion feature and the smoke image feature includes: in the K=1th downsampling stage, convolution processing is performed on the smoke mask image feature and the smoke image feature to obtain the first preprocessed feature, wherein the smoke mask image feature is extracted from the smoke mask image; in the K=2th downsampling stage, downsampling processing is performed on the first downsampling fusion feature and the smoke image feature to obtain the second preprocessed feature; in the K>2th downsampling stage, the (K-1)th downsampling fusion feature, the (K-1)th preprocessed feature, and the pooling feature are combined to obtain the Kth preprocessed feature, wherein the pooling feature is obtained based on the pooling operation performed on the smoke image feature.

[0052] According to an embodiment of this disclosure, in the Kth downsampling stage, K=1, the smoke mask image features and the smoke image features are concatenated, and the concatenated features are convolved using a convolutional layer to obtain the first preprocessed features. Feature extraction and feature fusion are then performed on the first preprocessed features to obtain the first downsampling fused features output from the first downsampling stage.

[0053] According to an embodiment of this disclosure, in the Kth downsampling stage, K=2, the first downsampling fusion features and the smoke image features are concatenated. The concatenated features are then downsampled using a downsampling layer to obtain the second preprocessed features. Feature extraction and feature fusion are performed on the second preprocessed features to obtain the second downsampling fusion features output from the second downsampling stage.

[0054] According to embodiments of this disclosure, the downsampling layer can be constructed based on a downsampling algorithm (Pixel Unshuffle).

[0055] According to embodiments of this disclosure, in the Kth downsampling stage processing (K>2), pooling features are obtained by pooling the smoke image features. The (K-1)th downsampling fusion features, the (K-1)th preprocessed features, and the pooled features are then concatenated. A downsampling layer is used to downsample the concatenated features to obtain the Kth preprocessed features. Feature extraction and feature fusion are then performed on the Kth preprocessed features to obtain the Kth downsampling fusion features output from the Kth downsampling stage processing.

[0056] According to embodiments of this disclosure, the downsampling process in the downsampling stage is used to extract preprocessed features at multiple scales, which can also avoid generating grid artifacts. The smoke image features are passed through an average pooling layer to obtain pooling features of different resolutions, which are then input into different downsampling stages, thereby maximizing the use of the original scale information in the smoke image. At the same time, the preprocessed features from the previous downsampling stage are also input into the current downsampling stage for cross-stage connection, thereby reducing information loss.

[0057] According to embodiments of this disclosure, any Kth upsampling stage processing includes: extracting and fusing upsampling features from the M-K+1th downsampling fusion features and the K-1th upsampling output results to obtain the Kth upsampling fusion features; preprocessing the Kth upsampling fusion features to obtain the Kth upsampling features, wherein the K-1th upsampling output result is the K-1th upsampling feature obtained by performing the K-1th upsampling stage processing on the M-K+2th downsampling fusion features and the K-2th upsampling output results.

[0058] According to embodiments of this disclosure, the downsampled fusion features obtained in each downsampling stage need to be input into the corresponding upsampling stage. The output result of the (K-1)th upsampling is the (K-1)th upsampling feature obtained by performing the (M-K+2)th downsampling fusion feature and the output result of the (K-2)th upsampling stage on the (K-1)th upsampling.

[0059] For example, when M = 3, the third downsampling fusion feature is input into the first upsampling stage for processing, and the output of the first upsampling result is obtained; the second downsampling fusion feature and the output of the first upsampling result are input into the second upsampling stage for processing, and the output of the second upsampling result is obtained; the first downsampling fusion feature and the output of the second upsampling result are input into the third upsampling stage for processing, and the output of the third upsampling result is obtained.

[0060] According to an embodiment of the present disclosure, in each upsampling feature extraction process, N branches are set, and each branch includes an atrous convolution enhancement module and a visual state space feature extraction module. The atrous convolution enhancement module is constructed based on an atrous convolutional neural network, and the atrous convolution enhancement module uses pointwise convolution for pixel enhancement; the visual state space feature extraction module is globally modeled based on Linear-Time Sequence Modeling with Selective State Spaces (Mamba).

[0061] According to an embodiment of the present disclosure, the (M - K + 1)-th downsampling fusion feature and the output of the (K - 1)-th upsampling are subjected to feature splicing, and then the spliced features are subjected to upsampling feature extraction and upsampling feature fusion to obtain the K-th upsampling fusion feature.

[0062] According to an embodiment of the present disclosure, the spliced features are equally divided according to the number of branches N. For each branch, first, the atrous convolution enhancement module is used for first feature extraction, and then the visual state space feature extraction module is used for second feature extraction. Finally, the features output by the N branches are subjected to upsampling feature fusion to obtain the K-th upsampling fusion feature.

[0063] According to an embodiment of the present disclosure, preprocessing the K-th upsampling fusion feature to obtain the K-th upsampling feature may include: performing feature convolution or upsampling on the K-th upsampling fusion feature to obtain the K-th upsampling feature.

[0064] According to an embodiment of the present disclosure, preprocessing the K-th upsampling fusion feature to obtain the K-th upsampling feature includes: in the upsampling stage processing where K < M, performing upsampling on the K-th upsampling fusion feature to obtain the K-th upsampling feature; in the upsampling stage processing where K = M, performing convolution on the M-th upsampling fusion feature to obtain the M-th upsampling feature.

[0065] According to an embodiment of the present disclosure, in the upsampling stage processing of the K-th time, when K < M, the upsampling layer is used to perform upsampling on the K-th upsampling fusion feature to obtain the K-th upsampling feature. The K-th upsampling feature is the output result of the K-th upsampling.

[0066] According to embodiments of this disclosure, the upsampling layer can be constructed based on an upsampling algorithm (pixel-shuffle).

[0067] According to an embodiment of this disclosure, in the Kth upsampling stage, K=M, a convolutional layer is used to perform convolution processing on the Mth upsampling fused features to obtain the Mth upsampling features. The Mth upsampling features are the output results of the Mth upsampling.

[0068] According to embodiments of this disclosure, the process of performing M cascaded upsampling stages and M cascaded downsampling stages on the smoke image and the smoke mask image, and outputting the predicted smoke-free image, further includes fusing the Mth upsampling features and the smoke image features to obtain the predicted smoke-free image.

[0069] According to embodiments of this disclosure, the Mth upsampled features and the features of the smoke-containing image are stitched together to obtain a predicted smoke-free image.

[0070] Figure 2 An example schematic diagram of a smoke removal model according to an embodiment of the present disclosure is shown.

[0071] like Figure 2 As shown, the smoke removal model includes an encoder-decoder structure with four cascaded upsampling stages and an encoder-decoder structure with four cascaded downsampling stages. In the first downsampling stage (downsampling stage 1), the smoke mask image features and the smoke image features are concatenated, followed by convolution, feature extraction, and feature fusion to obtain the first downsampling fused features. In the second downsampling stage (downsampling stage 2), the first downsampling fused features and the smoke image features are concatenated, followed by downsampling, feature extraction, and feature fusion to obtain the second downsampling fused features. In the third downsampling stage (downsampling stage 3), the second downsampling fused features, the second preprocessed features, and the pooling features are concatenated, followed by downsampling, feature extraction, and feature fusion to obtain the third downsampling fused features. In the fourth downsampling stage (downsampling stage 4), the third downsampling fused features, the third preprocessed features, and the pooling features are concatenated, followed by downsampling, feature extraction, and feature fusion to obtain the fourth downsampling fused features. The number of output channels in the downsampling stages are 32, 64, 128, and 256, respectively.

[0072] According to the implementation of this disclosure, in the first upsampling stage (upsampling stage 1), feature extraction, feature fusion, and upsampling are performed on the fourth downsampling fusion features to obtain the first upsampling output result; in the second upsampling stage (upsampling stage 2), feature concatenation, feature extraction, feature fusion, and upsampling are performed on the third downsampling fusion features and the first upsampling output result to obtain the second upsampling output result; in the third upsampling stage (upsampling stage 3), feature concatenation, feature extraction, feature fusion, and upsampling are performed on the second downsampling fusion features and the second upsampling output result to obtain the third upsampling output result; in the fourth upsampling stage (upsampling stage 4), feature concatenation, feature extraction, feature fusion, and convolution are performed on the first downsampling fusion features and the third upsampling output result to obtain the fourth upsampling output result; feature concatenation is performed on the fourth upsampling output result and the smoke-containing image features to obtain the predicted smoke-free image.

[0073] Figure 3A An example schematic diagram of downsampling feature extraction according to an embodiment of the present disclosure is shown.

[0074] like Figure 3A As shown, in the downsampling feature extraction process during the Kth downsampling stage, four branches are set up. Each branch includes a dilated convolution enhancement module and a visual state space feature extraction module. The Kth preprocessed features are processed according to the number of branches. The image is divided into four equal parts to obtain four image branch features. The four image branch features are input into their respective branches. For each branch's image branch features, the first feature extraction is performed using the dilated convolution enhancement module, and the second feature extraction is performed using the visual state space feature extraction module to obtain the global image features. Finally, the four global image features output from the four branches are obtained. The dilation rate of the dilated convolution enhancement module in each branch can be set to 1, 3, 5, and 7, respectively.

[0075] According to embodiments of this disclosure, the network structures used in upsampling feature extraction and downsampling feature extraction are the same.

[0076] According to embodiments of this disclosure, downsampling feature fusion includes: fusing N global image features to obtain global fusion features; pooling the global fusion features to obtain N weight vectors corresponding to N branches; normalizing the N weight vectors to obtain N weight factors; and weighting the N weight factors with the N global image features to obtain the Kth downsampling fusion feature.

[0077] According to embodiments of this disclosure, N global image features are fused in a merging manner to obtain global fused features. Pooling layers are then used to pool these global fused features to obtain N weight vectors corresponding to the N branches. Each weight vector represents the importance of a channel in each branch.

[0078] According to embodiments of this disclosure, a normalization layer is used to normalize N weight vectors to obtain N weight factors. The normalization layer can be constructed based on the softmax function.

[0079] According to embodiments of this disclosure, after weighted summation of N weighting factors and N global image features, a convolutional layer is used to process the weighted features to restore the number of channels and obtain the Kth downsampling fusion feature.

[0080] In one embodiment, the Kth downsampling fusion feature As shown in formula (1):

[0081] (1)

[0082] in, Characterizing the i-th weight factor, Characterizing convolutional layers, Characterize the global features of the i-th image.

[0083] According to embodiments of this disclosure, feature fusion is performed on global image features from different branches, and adaptive weight vectors are learned for global image features of each branch in the channel dimension to reduce information redundancy and highlight important features, thereby ensuring the desmog performance of the model.

[0084] Figure 3B An example schematic diagram of downsampling feature fusion according to an embodiment of the present disclosure is shown.

[0085] like Figure 3B As shown, for the four global features of the image ( The four global image features are then fused together to obtain global fusion features. Pooling is applied to these global fusion features to obtain four weight vectors corresponding to the four branches. These four weight vectors are then normalized to obtain four weight factors. These four weight factors are weighted together with the four global image features, and finally, the result is passed through a convolutional layer to obtain the Kth downsampled fusion feature. .

[0086] According to embodiments of this disclosure, the network structures used in upsampling feature fusion and downsampling feature fusion are the same.

[0087] According to embodiments of this disclosure, the smoke removal model is trained based on the following operations: acquiring training samples, including smoke-free images; rendering smoke on the smoke-free images to obtain smoke-containing images; processing the smoke-containing images using a smoke detector to obtain smoke mask images; processing the smoke mask images and smoke-containing images according to the smoke removal model to obtain predicted smoke-free images; and training the smoke removal model based on the predicted smoke-free images and smoke-free images to obtain the trained smoke removal model.

[0088] According to embodiments of this disclosure, sample videos of laparoscopic surgical equipment are screened frame by frame to select clear, smoke-free sample images.

[0089] According to embodiments of this disclosure, smoke is rendered on a sample smokeless image using rendering software. By randomly adjusting parameters such as density, buoyancy, heat, vorticity, and position, the smoke effect in a real scene is simulated to generate a reference smoke mask image with different density distributions and shape characteristics. At the same time, the rendered smoke is superimposed on the sample smokeless image through transparency channel synthesis to obtain a sample smoke image paired with the sample smokeless image.

[0090] According to embodiments of this disclosure, a reference smoke mask image is used as label information, a sample smoke image is input into a smoke detector to obtain a sample smoke mask image, and the smoke detector is trained based on the loss value between the sample smoke mask image and the reference smoke mask image.

[0091] According to embodiments of this disclosure, a sample smoke mask image is used as prior information and is input into a smoke removal model along with a sample smoke image to assist in locating the smoke region, thereby obtaining a sample predicted smoke-free image, which is a predicted image after smoke removal.

[0092] According to embodiments of this disclosure, a loss function is used to calculate the loss value between the predicted smoke-free image of the sample and the sample smoke-free image. A smoke removal model is trained based on the loss value. If the loss value meets the preset model performance, training is stopped; otherwise, the model parameters are adjusted and training continues.

[0093] According to embodiments of this disclosure, training a smoke removal model based on a sample predicted smoke-free image and a sample smoke-free image to obtain the trained smoke removal model includes: calculating a first loss value between the sample predicted smoke-free image and the sample smoke-free image using a pixel-aware loss function; calculating a second loss value between the sample predicted smoke-free image and the sample smoke-free image using a feature space-aware loss function; calculating a third loss value between the sample predicted smoke-free image and the sample smoke-free image using a multi-scale structure loss function; calculating a fourth loss value between the sample predicted smoke-free image using a smoke-aware loss function; obtaining a target loss value based on the first loss value, the second loss value, the third loss value, and the fourth loss value; and training the smoke removal model based on the target loss value to obtain the trained smoke removal model.

[0094] According to embodiments of this disclosure, a first loss value characterizes the difference between the sample predicted smoke-free image and the sample smoke-free image constrained at the pixel level.

[0095] In one embodiment, the pixel-aware loss function As shown in formula (2):

[0096] (2)

[0097] in, Characterize samples to predict smoke-free images. Characterizing smoke-free images of samples, The first loss value is represented.

[0098] According to embodiments of this disclosure, the predicted smoke-free image and the sample smoke-free image are respectively input into a deep convolutional neural network model (Visual Geometry Group Network 19, VGG19). The similarity between the output features of each layer in the deep convolutional neural network model is calculated using a feature space perceptual loss function. A second loss value characterizes the perceptual similarity between the predicted smoke-free image and the sample smoke-free image in the feature space.

[0099] In one embodiment, the feature space-aware loss function is shown in Equation (3):

[0100] (3)

[0101] in, Characterize the output features of the i-th layer of the deep convolutional neural network model for predicting smoke-free images from the sample. Characterize the output features of the i-th layer of the deep convolutional neural network model for smoke-free sample images. Characterizes the second loss value.

[0102] According to embodiments of this disclosure, the third loss value characterizes the contrast between the sample predicted smoke-free image and the sample smoke-free image in the high-frequency region.

[0103] In one embodiment, the multi-scale structural loss function As shown in formula (4):

[0104] (4)

[0105] in, Characterizes the third loss value.

[0106] According to embodiments of this disclosure, the fourth loss value characterizes the probability that a sample predicts the presence of smoke in a smoke-free image. The fourth loss value is used to ensure that the smoke-free image recovered by the network does not contain detectable smoke.

[0107] In one embodiment, the feature space perception loss function is shown in Equation (5):

[0108] (5)

[0109] in, Characterizing smoke detectors, Characterizes the fourth loss value.

[0110] In one embodiment, the target loss value As shown in formula (6):

[0111] (6)

[0112] in, Characterizing the first loss value, Characterizing the second loss value, Characterizing the third loss value, Characterizes the fourth loss value.

[0113] According to embodiments of this disclosure, a target loss value is obtained by evaluating the loss from multiple dimensions. If the target loss value meets the preset model performance, training is stopped, and a trained smoke removal model is obtained; otherwise, the model parameters are adjusted, and training continues.

[0114] Based on the above-described image desmoking method based on a smoke removal model, this disclosure also provides an image desmoking apparatus based on a smoke removal model. The following will be combined with... Figure 4 The device is described in detail.

[0115] Figure 4 A structural block diagram of an image desmoke device based on a smoke removal model according to an embodiment of the present disclosure is shown.

[0116] like Figure 4As shown, the image smoke removal device 400 based on the smoke removal model in this embodiment includes a detection module 410 and a smoke removal module 420.

[0117] The detection module 410 is used to process the smoke image using a smoke detector to obtain a smoke mask image. The smoke image represents a frame image rendered with smoke in the laparoscopic surgical equipment, and the pixel values ​​of the smoke mask image represent the smoke concentration. In one embodiment, the detection module 410 can be used to perform the operation S110 described above, which will not be repeated here.

[0118] The smoke removal module 420 is used to perform M cascaded upsampling stages and M cascaded downsampling stages on the smoky image and the smoke mask image, outputting a predicted smoke-free image. Each K-th downsampling stage includes the extraction and fusion of downsampling features from the (K-1)-th downsampling output and the smoky image. The (K-1)-th downsampling output is the (K-1)-th downsampling fusion feature obtained from the (K-1)-th downsampling stage on the smoky image and the smoke mask image. The downsampling feature extraction includes: dividing the K-th preprocessing feature according to the number of branches N to obtain N image branch features, where the K-th preprocessing feature is obtained by preprocessing the (K-1)-th downsampling fusion feature and the smoky image feature, where the smoky image feature is extracted from the smoky image; for each image branch feature, a first feature extraction is performed using a dilated convolution enhancement module to obtain local image features; and a second feature extraction is performed using a visual state space feature extraction module to obtain global image features. In one embodiment, the smoke removal module 420 can be used to perform the operation S120 described above, which will not be repeated here.

[0119] According to embodiments of this disclosure, the smoke removal module 420 includes a first processing submodule, a second processing submodule, and a third processing submodule.

[0120] The first processing submodule is used to perform convolution processing on the smoke mask image features and the smoke image features in the K=1th downsampling stage to obtain the first preprocessed features, wherein the smoke mask image features are extracted from the smoke mask image.

[0121] The second processing submodule is used to perform downsampling processing on the first downsampling fusion features and smoke image features in the K=2th downsampling stage to obtain the second preprocessed features.

[0122] The third processing submodule is used in the K>2 downsampling stage to obtain the K-th preprocessed feature from the K-1 downsampling fusion feature, the K-1 preprocessed feature, and the pooling feature. The pooling feature is obtained by performing a pooling operation on the smoke image features.

[0123] According to an embodiment of the present disclosure, the smoke removal module 420 further includes a first up-sampling sub-module and a second up-sampling sub-module.

[0124] The first up-sampling sub-module is configured to perform up-sampling feature extraction and up-sampling feature fusion on the (M-K+1)-th down-sampled fusion feature and the up-sampling feature of the (K-1)-th up-sampling output result, so as to obtain the K-th up-sampled fusion feature.

[0125] The second up-sampling sub-module is configured to preprocess the K-th up-sampled fusion feature to obtain the K-th up-sampling feature, where the (K-1)-th up-sampling output result is the (K-1)-th up-sampling feature obtained by performing the (K-1)-th up-sampling stage processing on the (M-K+2)-th down-sampled fusion feature and the (K-2)-th up-sampling output result.

[0126] According to an embodiment of the present disclosure, the second up-sampling sub-module includes a first up-sampling unit and a second up-sampling unit.

[0127] The first up-sampling unit is configured to perform up-sampling processing on the K-th up-sampled fusion feature in the up-sampling stage processing when K<M, so as to obtain the K-th up-sampling feature.

[0128] The second up-sampling unit is configured to perform convolution processing on the M-th up-sampled fusion feature in the up-sampling stage processing when K = M, so as to obtain the M-th up-sampling feature.

[0129] According to an embodiment of the present disclosure, the smoke removal module 420 further includes a prediction sub-module.

[0130] The prediction sub-module is configured to fuse the M-th up-sampling feature and the smoky image feature to obtain a predicted smokeless image.

[0131] According to an embodiment of the present disclosure, the smoke removal module 420 further includes a first fusion sub-module, a second fusion sub-module, a third fusion sub-module, and a fourth fusion sub-module.

[0132] The first fusion sub-module is configured to fuse N image global features to obtain a global fusion feature.

[0133] The second fusion sub-module is configured to perform pooling processing on the global fusion feature to obtain N weight vectors corresponding to N branches.

[0134] The third fusion sub-module is configured to perform normalization processing on the N weight vectors respectively to obtain N weight factors.

[0135] The fourth fusion sub-module is configured to weight the N weight factors and the N image global features to obtain the K-th down-sampled fusion feature.

[0136] According to embodiments of this disclosure, the image desmoking device 400 further includes an acquisition module, a rendering module, a first processing module, a second processing module, and a training module.

[0137] The acquisition module is used to acquire training samples, which include smoke-free images of the samples.

[0138] The rendering module is used to render smoke from the smoke-free sample image to obtain the smoke-containing sample image.

[0139] The first processing module is used to process the smoke image of the sample using a smoke detector to obtain a smoke mask image of the sample.

[0140] The second processing module is used to process the sample smoke mask image and the sample smoke image according to the smoke removal model to obtain the sample predicted smoke-free image.

[0141] The training module is used to train the smoke removal model based on the predicted smoke-free image and the sample smoke-free image, resulting in the trained smoke removal model.

[0142] According to embodiments of this disclosure, the training module includes a first training submodule, a second training submodule, a third training submodule, a fourth training submodule, a fifth training submodule, and a sixth training submodule.

[0143] The first training submodule is used to calculate a first loss value between the sample predicted smokeless image and the sample smokeless image using a pixel-aware loss function.

[0144] The second training submodule is used to calculate a second loss value between the predicted smokeless image of the sample and the sample smokeless image using the feature space perception loss function.

[0145] The third training submodule is used to calculate a third loss value between the sample predicted smokeless image and the sample smokeless image using a multi-scale structural loss function.

[0146] The fourth training submodule is used to calculate the fourth loss value for predicting smoke-free images from samples using the smoke perception loss function.

[0147] The fifth training submodule is used to obtain the target loss value based on the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0148] The sixth training submodule is used to train the smoke removal model based on the target loss value, and obtain the trained smoke removal model.

[0149] According to embodiments of this disclosure, any plurality of modules, including modules, submodules, units, and subunits, can be combined into one module for implementation, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of modules, submodules, units, and subunits can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of modules, submodules, units, and subunits can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0150] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing an image desmog method based on a smoke removal model according to embodiments of the present disclosure.

[0151] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0152] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 502 and / or RAM 503. It should be noted that programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0153] According to embodiments of this disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0154] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the image desmog method according to the embodiments of this disclosure.

[0155] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0156] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the image dehazing method provided in the embodiments of this disclosure.

[0157] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0158] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0159] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0160] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0162] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0163] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An image desmoking method based on a smoke removal model, characterized in that, The method includes: Processing a smoky image by using a smoke detector to obtain a smoke mask image, where the smoky image represents a frame image with smoke rendering in a laparoscopic surgical device, and the pixel value of the smoke mask image represents the smoke concentration; Performing M cascaded upsampling stages and M cascaded downsampling stages on the smoky image and the smoke mask image, and outputting a predicted smokeless image. Among them, any K-th downsampling stage includes downsampling feature extraction and downsampling feature fusion of the output result of the (K - 1)-th downsampling and the smoky image. The output result of the (K - 1)-th downsampling is the (K - 1)-th downsampling fusion feature obtained by performing the (K - 1)-th downsampling stage on the smoky image and the smoke mask image. The downsampling feature extraction includes: In the first downsampling stage (K = 1), performing convolution processing on the smoke mask image feature and the smoky image feature to obtain the first preprocessing feature, where the smoke mask image feature is extracted from the smoke mask image; In the second downsampling stage (K = 2), performing downsampling on the first downsampling fusion feature and the smoky image feature to obtain the second preprocessing feature; In the downsampling stage where K > 2, obtaining the K-th preprocessing feature from the (K - 1)-th downsampling fusion feature, the (K - 1)-th preprocessing feature, and the pooling feature, where the pooling feature is obtained based on a pooling operation on the smoky image feature; Dividing the K-th preprocessing feature according to the number of branches N to obtain N image branch features, where the smoky image feature is extracted from the smoky image; For each of the image branch features, using an atrous convolution enhancement module to perform first feature extraction to obtain image local features; Using a visual state space feature extraction module to perform second feature extraction on the image local features to obtain image global features.

2. The method according to claim 1, characterized in that, Any K-th upsampling stage includes: Performing upsampling feature extraction and upsampling feature fusion on the output result of the (M - K + 1)-th downsampling fusion feature and the (K - 1)-th upsampling to obtain the K-th upsampling fusion feature; Preprocessing the K-th upsampling fusion feature to obtain the K-th upsampling feature, where the output result of the (K - 1)-th upsampling is the (K - 1)-th upsampling feature obtained by performing the (K - 1)-th upsampling stage on the output result of the (M - K + 2)-th downsampling fusion feature and the (K - 2)-th upsampling output result.

3. The method according to claim 2, characterized in that, Preprocessing the K-th upsampling fusion feature to obtain the K-th upsampling feature includes: In the upsampling stage where K < M, performing upsampling on the K-th upsampling fusion feature to obtain the K-th upsampling feature; In the upsampling stage where K = M, performing convolution processing on the M-th upsampling fusion feature to obtain the M-th upsampling feature.

4. The method according to claim 3, characterized in that, Performing M cascaded upsampling stages and M cascaded downsampling stages on the smoky image and the smoke mask image, and outputting a predicted smokeless image further includes: The Mth upsampling feature and the smoke-containing image feature are fused to obtain the predicted smoke-free image.

5. The method according to claim 1, characterized in that, The downsampling feature fusion includes: The N global features of the image are fused to obtain the global fused features; The global fusion features are pooled to obtain N weight vectors corresponding to N channels; Normalize each of the N weight vectors to obtain N weight factors; The N weighting factors and N global image features are weighted to obtain the Kth downsampling fusion feature.

6. The method according to claim 1, characterized in that, The smoke removal model was trained based on the following operations: Acquire training samples, which include smoke-free images of samples; The smokeless sample image is rendered with smoke to obtain a sample image with smoke. The smoke image of the sample is processed using a smoke detector to obtain a smoke mask image of the sample; The sample smoke mask image and the sample smoke image are processed according to the smoke removal model to obtain the sample predicted smoke-free image; Based on the predicted smoke-free image from the sample and the sample smoke-free image, the smoke removal model is trained to obtain the trained smoke removal model.

7. The method according to claim 6, characterized in that, Based on the predicted smoke-free image from the sample and the sample smoke-free image, a smoke removal model is trained to obtain the trained smoke removal model, which includes: The first loss value between the predicted smokeless image of the sample and the smokeless image of the sample is calculated using the pixel-perceptual loss function; A second loss value is calculated between the predicted smoke-free image of the sample and the smoke-free image of the sample using the feature space perception loss function; A third loss value is calculated between the predicted smoke-free image of the sample and the smoke-free image of the sample using a multi-scale structural loss function; The fourth loss value for predicting smoke-free images of the sample is calculated using the smoke perception loss function; The target loss value is obtained based on the first loss value, the second loss value, the third loss value, and the fourth loss value; Based on the target loss value, the smoke removal model is trained to obtain the trained smoke removal model.

8. An image desmoking device based on a smoke removal model, the device comprising: The detection module is used to process the smoke image using a smoke detector to obtain a smoke mask image, wherein the smoke image represents a frame image rendered with smoke in the laparoscopic surgical equipment, and the pixel value of the smoke mask image represents the smoke concentration; The smoke removal module is used to perform M cascaded upsampling stages and M cascaded downsampling stages on the smoky image and the smoke mask image, outputting a predicted smoke-free image. Each Kth downsampling stage includes the (K-1)th downsampling output result and the downsampling feature extraction and fusion of the smoky image. The (K-1)th downsampling output result is the (K-1)th downsampling fusion feature obtained from the (K-1)th downsampling stage processing of the smoky image and the smoke mask image. The downsampling feature extraction includes: in the K=1th downsampling stage processing, convolution processing is performed on the smoke mask image features and the smoky image features to obtain the first preprocessed features, where the smoke mask image features are extracted from the smoke mask image; in the K=2th downsampling stage processing... In the sampling stage processing, the first downsampling fusion feature and the smoke image feature are downsampled to obtain the second preprocessed feature; in the K>2 downsampling stage processing, the (K-1)th downsampling fusion feature, the (K-1)th preprocessed feature, and the pooling feature are processed to obtain the Kth preprocessed feature, wherein the pooling feature is obtained based on the pooling operation of the smoke image feature; the Kth preprocessed feature is divided according to the number of branches N to obtain N image branch features, wherein the smoke image feature is extracted from the smoke image; for each image branch feature, the first feature extraction is performed using the dilated convolution enhancement module to obtain the local image feature; the second feature extraction is performed using the visual state space feature extraction module to obtain the global image feature.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

Images