Image smog removal method and device based on smog removal model, and electronic equipment

Abstract

The invention provides an image smog removal method and device based on a smog removal model, and electronic equipment, and can be applied to the technical field of image processing. The method comprises the following steps: processing a smoke image by using a smoke detector to obtain a smoke mask image; m cascaded up-sampling stage processing and M cascaded down-sampling stage processing are carried out on the smoke image and the smoke mask image, and a predicted smokeless image is output; wherein any Kth downsampling stage processing comprises downsampling feature extraction and downsampling feature fusion of a (K-1) th downsampling output result and a smoky image, and the downsampling feature extraction comprises the steps of dividing Kth preprocessing features according to the number N of branches to obtain N image branch features; for each image branch feature, performing first feature extraction by using an expansion convolution enhancement module to obtain an image local feature; and performing second feature extraction on the image local features by using a visual state space feature extraction module to obtain image global features.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image smoke removal method, device and electronic device based on a smoke removal model. Background Art

[0002] Electrocautery is often used to heat tissue to reduce bleeding. However, the use of electrocautery during laparoscopic surgery produces smoke, which seriously affects the image observation of tissue sites. Existing desmog methods rely on convolutional architectures. Due to the inherent local receptive field of view of convolutional operations, they face defects in capturing remote global information, resulting in low efficiency and accuracy of image recovery. In addition, in order to improve model performance, a deeper backbone or more complex network architecture is required, which greatly increases the model size and computational complexity, and also reduces the real-time performance of image desmog. Summary of the invention

[0003] In view of the above problems, the present disclosure provides an image desmog method, device and electronic device based on a smoke removal model.

[0004] According to a first aspect of the present disclosure, a method for removing smoke from an image based on a smoke removal model is provided, comprising: using a smoke detector to process a smoke image to obtain a smoke mask image, wherein the smoke image represents a frame image rendered with smoke in a laparoscopic surgical device, and the pixel value of the smoke mask image represents the smoke concentration; performing M cascaded upsampling stage processing and M cascaded downsampling stage processing on the smoke image and the smoke mask image, and outputting a predicted smoke-free image, wherein any K-th downsampling stage processing includes extracting downsampling features and fusion of downsampling features of the K-1-th downsampling output result and the smoke image, and the K-1-th downsampling output result is the smoke image. The smoke image and the smoke mask image are processed in the K-1th downsampling stage to obtain the K-1th downsampling fusion features. The downsampling feature extraction includes: dividing the Kth preprocessing features according to the number of branches N to obtain N image branch features, wherein the Kth preprocessing features are obtained by preprocessing the K-1th downsampling fusion features and the smoke image features, wherein the smoke image features are extracted from the smoke image; for each image branch feature, a first feature extraction is performed using a dilated convolution enhancement module to obtain an image local feature; and a second feature extraction is performed on the image local feature using a visual state space feature extraction module to obtain an image global feature.

[0005] According to an embodiment of the present disclosure, preprocessing the (K - 1)-th downsampled fusion feature and the smoky image feature includes: in the processing of the K = 1-th downsampling stage, performing convolution processing on the smoke mask image feature and the smoky image feature to obtain the first preprocessed feature, where the smoke mask image feature is extracted from the smoke mask image; in the processing of the K = 2-th downsampling stage, performing downsampling processing on the first downsampled fusion feature and the smoky image feature to obtain the second preprocessed feature; in the processing of the K > 2-th downsampling stage, performing processing on the (K - 1)-th downsampled fusion feature, the (K - 1)-th preprocessed feature, and the pooling feature to obtain the K-th preprocessed feature, where the pooling feature is obtained based on a pooling operation on the smoky image feature.

[0006] According to an embodiment of the present disclosure, any K-th upsampling stage processing includes: performing upsampling feature extraction and upsampling feature fusion on the (M - K + 1)-th downsampled fusion feature and the upsampling feature of the (K - 1)-th upsampling output result to obtain the K-th upsampled fusion feature; performing preprocessing on the K-th upsampled fusion feature to obtain the K-th upsampled feature, where the (K - 1)-th upsampling output result is the (K - 1)-th upsampled feature obtained by performing the (K - 1)-th upsampling stage processing on the (M - K + 2)-th downsampled fusion feature and the (K - 2)-th upsampling output result.

[0007] According to an embodiment of the present disclosure, performing preprocessing on the K-th upsampled fusion feature to obtain the K-th upsampled feature includes: in the processing of the K < M-th upsampling stage, performing upsampling processing on the K-th upsampled fusion feature to obtain the K-th upsampled feature; in the processing of the K = M-th upsampling stage, performing convolution processing on the M-th upsampled fusion feature to obtain the M-th upsampled feature.

[0008] According to an embodiment of the present disclosure, performing M cascaded upsampling stage processing and M cascaded downsampling stage processing on the smoky image and the smoke mask image, and outputting a predicted non-smoky image further includes: fusing the M-th upsampled feature and the smoky image feature to obtain the predicted non-smoky image.

[0009] According to an embodiment of the present disclosure, downsampled feature fusion includes: fusing N image global features to obtain a global fusion feature; performing pooling processing on the global fusion feature to obtain N weight vectors corresponding to N branches; respectively performing normalization processing on the N weight vectors to obtain N weight factors; weighting the N weight factors and the N image global features to obtain the K-th downsampled fusion feature.

[0010] According to an embodiment of the present disclosure, the smoke removal model is trained based on the following operations: obtaining training samples, the training samples include sample smoke-free images; performing smoke rendering on the sample smoke-free images to obtain sample smoke images; using a smoke detector to process the sample smoke images to obtain sample smoke mask images; processing the sample smoke mask images and the sample smoke images according to the smoke removal model to obtain sample predicted smoke-free images; training the smoke removal model based on the sample predicted smoke-free images and the sample smoke-free images to obtain a trained smoke removal model.

[0011] According to an embodiment of the present disclosure, a smoke removal model is trained based on a sample predicted smoke-free image and a sample smoke-free image to obtain a trained smoke removal model, including: calculating a first loss value between a sample predicted smoke-free image and a sample smoke-free image using a pixel perception loss function; calculating a second loss value between a sample predicted smoke-free image and a sample smoke-free image using a feature space perception loss function; calculating a third loss value between a sample predicted smoke-free image and a sample smoke-free image using a multi-scale structure loss function; calculating a fourth loss value of a sample predicted smoke-free image using a smoke perception loss function; obtaining a target loss value based on the first loss value, the second loss value, the third loss value and the fourth loss value; and training the smoke removal model based on the target loss value to obtain a trained smoke removal model.

[0012] The second aspect of the present disclosure provides an image desmoking device based on a smoke removal model, comprising: a detection module, used to process a smoke image using a smoke detector to obtain a smoke mask image, wherein the smoke image represents a frame image rendered with smoke in a laparoscopic surgical device, and the pixel value of the smoke mask image represents the smoke concentration; a smoke removal module, used to perform M cascaded upsampling stage processing and M cascaded downsampling stage processing on the smoke image and the smoke mask image, and output a predicted smoke-free image, wherein any K-th downsampling stage processing includes extracting downsampling features and fusion of downsampling features of the K-1-th downsampling output result and the smoke image, and the K-1-th downsampling output The result is the K-1th downsampling fusion feature obtained by processing the smoke image and the smoke mask image in the K-1th downsampling stage. The downsampling feature extraction includes: dividing the Kth preprocessing feature according to the number of branches N to obtain N image branch features, wherein the Kth preprocessing feature is obtained by preprocessing the K-1th downsampling fusion feature and the smoke image feature, wherein the smoke image feature is extracted from the smoke image; for each image branch feature, the first feature extraction is performed using the dilated convolution enhancement module to obtain the local feature of the image; the second feature extraction is performed on the local feature of the image using the visual state space feature extraction module to obtain the global feature of the image.

[0013] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] According to the image desmog method, device and electronic device based on the smoke removal model provided by the present disclosure, a smoke mask image is obtained by processing a smoke image with a smoke detector, and the smoke image and the smoke mask image are processed in M ​​cascade upsampling stages and M cascade downsampling stages, and a predicted smoke-free image is output. Since the smoke mask image output by the smoke detector can reflect the actual degree of smoke concentration in different regions, it provides an accurate reference for the smoke removal processing of the smoke removal model, and improves the smoke removal accuracy of the smoke removal model; the smoke removal model adopts a multi-branch structure, and uses the dilated convolution enhancement module and the visual state space feature extraction module in each branch to first perform point-by-point convolution to enhance the local pixel correlation, and then extract the global enhancement feature, thereby improving the feature inference speed and more efficiently removing non-uniformly distributed smoke; in addition, the multi-branch structure can reduce the number of channels of each branch, reduce the model parameter amount, thereby reducing the convolution operation load and the calculation load caused by the number of channels in the visual state space feature extraction module, and achieves a trade-off between model complexity and inference speed under the premise of ensuring good smoke removal performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0016] Figure 1 A flowchart of an image desmog method based on a smoke removal model according to an embodiment of the present disclosure is shown.

[0017] Figure 2 An example schematic diagram of a smoke removal model according to an embodiment of the present disclosure is shown.

[0018] Figure 3A An example schematic diagram of downsampling feature extraction according to an embodiment of the present disclosure is shown.

[0019] Figure 3B An example schematic diagram showing down-sampling feature fusion according to an embodiment of the present disclosure

[0020] Figure 4 A structural block diagram of an image desmog device based on a smoke removal model according to an embodiment of the present disclosure is shown.

[0021] Figure 5A block diagram of an electronic device suitable for implementing an image desmog method based on a smoke removal model according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments according to the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0023] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0024] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0025] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0026] In the process of implementing the present disclosure, it was found that electrocautery is often used to heat tissue to reduce bleeding. However, the use of electrocautery during the use of laparoscopic surgical equipment will produce smoke, which seriously affects the image observation of tissue parts. Existing smoke removal methods rely on convolutional architectures. Due to the inherent local receptive field of view of convolutional operations, they face defects in capturing remote global information, resulting in low efficiency and accuracy of image recovery. In addition, in order to improve model performance, it is necessary to use a deeper backbone or a more complex network architecture, which greatly increases the model scale and computational complexity, and also reduces the real-time performance of image smoke removal.

[0027] At the same time, due to the memory limitations of laparoscopic equipment and the real-time requirement of smoke removal in laparoscopic surgical images, there is an urgent need to design a lightweight and effective smoke removal model so that it can be better embedded in laparoscopic surgical equipment to achieve efficient smoke removal.

[0028] In view of this, according to the embodiments of the present disclosure, a method, device and electronic device for removing smoke from an image based on a smoke removal model are provided. The method includes: using a smoke detector to process a smoke image to obtain a smoke mask image; performing M cascade upsampling stage processing and M cascade downsampling stage processing on the smoke image and the smoke mask image, and outputting a predicted smoke-free image; wherein any K-th downsampling stage processing includes downsampling feature extraction and downsampling feature fusion of the K-1-th downsampling output result and the smoke image, and the downsampling feature extraction includes: dividing the K-th preprocessing feature according to the number of branches N to obtain N image branch features; for each image branch feature, using a dilated convolution enhancement module to perform a first feature extraction to obtain an image local feature; using a visual state space feature extraction module to perform a second feature extraction on the image local feature to obtain an image global feature.

[0029] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0031] Figure 1 A flowchart of an image desmog method based on a smoke removal model according to an embodiment of the present disclosure is shown.

[0032] like Figure 1 As shown, the method 100 includes operations S110 to S120.

[0033] In operation S110, the smoke image is processed using a smoke detector to obtain a smoke mask image.

[0034] According to an embodiment of the present disclosure, the video in the laparoscopic surgical device is processed using Open Source Computer Vision Library (OpenCV) technology to extract a single, high-quality frame image from the video. The smoke image represents a frame image rendered with smoke in the laparoscopic surgical device.

[0035] According to an embodiment of the present disclosure, a smoke detector is constructed based on an image segmentation network. The smoke detector can adopt a 6-layer image segmentation network (Convolutional Networks for Biomedical Image Segmentation, U-Net) structure, including an encoder, a decoder, and a jump connection path, and the number of channels of the image segmentation network is set to 8, 16, 24, 32, 48, and 64 respectively. The first three layers of the image segmentation network use convolution modules to extract shallow features, and the last three layers of the image segmentation network are all processed using a 4-branch parallel visual state space module. The number of branches of each branch is 1 / 4 of the initial input image segmentation network features, which is used to reduce the number of model parameters.

[0036] According to an embodiment of the present disclosure, the skip connection path uses a channel attention block (CAB) module and a spatial attention block (SAB) module to perform feature fusion, and finally obtains a smoke mask image.

[0037] According to an embodiment of the present disclosure, the pixel value of the smoke mask image represents the smoke density, and the pixel value ranges from 0 to 1. The higher the pixel value, the greater the smoke density.

[0038] In operation S120, the smoke image and the smoke mask image are processed in M ​​cascaded upsampling stages and M cascaded downsampling stages to output a predicted smoke-free image.

[0039] According to the embodiments of the present disclosure, any K-th downsampling stage processing includes downsampling feature extraction and downsampling feature fusion of the K-1-th downsampling output result and the smoke image, and the K-1-th downsampling output result is the K-1-th downsampling fusion feature obtained by performing the K-1-th downsampling stage processing on the smoke image and the smoke mask image, and the downsampling feature extraction includes: dividing the K-th preprocessing feature according to the number of branches N to obtain N image branch features, wherein the K-th preprocessing feature is obtained by preprocessing the K-1-th downsampling fusion feature and the smoke image feature, wherein the smoke image feature is extracted from the smoke image; for each image branch feature, using the dilated convolution enhancement module to perform a first feature extraction to obtain a local image feature; using the visual state space feature extraction module to perform a second feature extraction on the local image feature to obtain a global image feature.

[0040] According to an embodiment of the present disclosure, the smoke removal model includes M cascaded encoder-decoder structures of upsampling stages and M cascaded encoder-decoder structures of downsampling stages.

[0041] According to an embodiment of the present disclosure, the K-1th downsampling output result is the K-1th downsampling fusion feature obtained by performing the K-1th downsampling stage processing on the smoke image and the smoke mask image, and any K-th downsampling stage processing includes the downsampling feature extraction and downsampling feature fusion of the K-1th downsampling output result and the smoke image. M and K are both positive integers.

[0042] According to an embodiment of the present disclosure, in the downsampling feature extraction process, N branches are set, N is a positive integer, each branch includes a dilated convolution enhancement module and a visual state space feature extraction module, the dilated convolution enhancement module is constructed based on a dilated convolutional neural network, and the dilated convolution enhancement module uses point-by-point convolution to perform pixel enhancement; the visual state space feature extraction module is obtained by global modeling based on Linear-Time Sequence Modeling with Selective State Spaces (Mamba).

[0043] According to an embodiment of the present disclosure, extracting features of a smoke image to obtain a smoke image feature, and preprocessing the K-1th downsampled fusion feature and the smoke image feature to obtain the Kth preprocessed feature may include: performing feature fusion, downsampling, etc. on the K-1th downsampled fusion feature and the smoke image feature to obtain the Kth preprocessed feature.

[0044] According to an embodiment of the present disclosure, the K-th preprocessing feature is divided into equal parts according to the number of branches N to obtain N image branch features, each of which is a local feature of the K-th preprocessing feature.

[0045] According to the embodiment of the present disclosure, N image branch features are respectively input into the corresponding branches. For the image branch features in each branch, the image branch features are firstly extracted using the dilated convolution enhancement module to obtain the local features of the image. Then, the local features of the image are extracted using the visual state space feature extraction module to obtain the global features of the image. Finally, the N global features of the image output by the N branches are obtained.

[0046] According to an embodiment of the present disclosure, the convolution dilation rate parameters of the dilated convolution enhancement module in each branch are different to achieve learning and enhancing local features at different dilation degrees.

[0047] According to an embodiment of the present disclosure, the local feature of the image represents the feature of enhancing the correlation between pixels located in the local spatial area, and the global feature of the image represents the feature of enhancing the linear complex information of global pixels.

[0048] According to an embodiment of the present disclosure, feature fusion is performed on N global features of the image to obtain the Kth downsampling fusion feature, and then the M downsampling fusion features obtained by processing the M downsampling stages are processed by M cascaded upsampling stages to obtain a predicted smoke-free image.

[0049] According to an embodiment of the present disclosure, the predicted smoke-free image represents a high-definition image after smoke is removed.

[0050] According to the embodiments of the present disclosure, since the smoke mask image output by the smoke detector can reflect the actual degree of smoke concentration in different areas, it can provide an accurate reference for the smoke removal processing of the smoke removal model, thereby improving the smoke removal accuracy of the smoke removal model; the smoke removal model adopts a multi-branch structure, and uses the dilated convolution enhancement module and the visual state space feature extraction module in each branch to first perform point-by-point convolution to enhance local pixel correlation, and then extract global enhancement features, thereby improving the feature inference speed and more efficiently removing non-uniformly distributed smoke; in addition, the multi-branch structure can reduce the number of channels in each branch and the amount of model parameters, thereby reducing the convolution operation load and the computational load caused by the number of channels in the visual state space feature extraction module, and achieves a trade-off between model complexity and inference speed while ensuring good smoke removal performance.

[0051] According to an embodiment of the present disclosure, preprocessing the K-1th down-sampling fusion features and smoke image features includes: in the K=1th down-sampling stage processing, convolution processing is performed on the smoke mask image features and the smoke image features to obtain the first pre-processing features, wherein the smoke mask image features are obtained by extracting the smoke mask image; in the K=2th down-sampling stage processing, down-sampling processing is performed on the first down-sampling fusion features and the smoke image features to obtain the second pre-processing features; in the K>2th down-sampling stage processing, the K-1th down-sampling fusion features, the K-1th pre-processing features and the pooling features are performed to obtain the Kth pre-processing features, wherein the pooling features are obtained based on the pooling operation performed on the smoke image features.

[0052] According to the embodiment of the present disclosure, in the Kth downsampling stage, K=1, the smoke mask image features and the smoke image features are spliced, and the spliced ​​features are convoluted using a convolution layer to obtain the first preprocessing features. The first preprocessing features are feature extracted and fused to obtain the first downsampling fusion features output by the first downsampling stage.

[0053] According to the embodiment of the present disclosure, in the Kth downsampling stage processing, K=2, the first downsampling fusion feature and the smoke image feature are feature spliced, and the spliced ​​features are downsampled using the downsampling layer to obtain the second preprocessing feature. Feature extraction and feature fusion are performed on the second preprocessing feature to obtain the second downsampling fusion feature output by the second downsampling stage processing.

[0054] According to an embodiment of the present disclosure, the downsampling layer may be constructed based on a downsampling algorithm (Pixel Unshuffle).

[0055] According to the embodiments of the present disclosure, in the Kth downsampling stage, K>2, the pooling feature obtained by the pooling operation on the smoke image feature is feature spliced ​​on the K-1th downsampling fusion feature, the K-1th preprocessing feature and the pooling feature, and the spliced ​​feature is downsampled using the downsampling layer to obtain the Kth preprocessing feature. Feature extraction and feature fusion are performed on the Kth preprocessing feature to obtain the Kth downsampling fusion feature output by the Kth downsampling stage.

[0056] According to the embodiments of the present disclosure, the downsampling process in the downsampling stage is used to extract multi-scale preprocessing features, which can also avoid the generation of grid artifacts; the smoke image features are subjected to the average pooling layer to obtain pooling features of different resolutions and are respectively input into different downsampling stage processes, thereby maximizing the use of the original scale information in the smoke image. At the same time, the preprocessing features in the previous downsampling stage process are also input into the current downsampling stage process for cross-stage connection, thereby reducing information loss.

[0057] According to an embodiment of the present disclosure, any K-th upsampling stage processing includes: extracting and fusing upsampling features of the M-K+1-th downsampling fusion features and the K-1-th upsampling output results to obtain the K-th upsampling fusion features; preprocessing the K-th upsampling fusion features to obtain the K-th upsampling features, wherein the K-1-th upsampling output result is the K-1-th upsampling feature obtained by performing the K-1-th upsampling stage processing on the M-K+2-th downsampling fusion features and the K-2-th upsampling output results.

[0058] According to the embodiments of the present disclosure, the down-sampled fusion features obtained in each down-sampling stage processing need to be input into the corresponding up-sampling stage processing. The K-1th up-sampling output result is the K-1th up-sampling feature obtained by performing the K-1th up-sampling stage processing on the M-K+2th down-sampled fusion features and the K-2th up-sampling output result.

[0059] For example, when M = 3, the fused feature of the third downsampling is input into the first upsampling stage for processing, and the output result of the first upsampling is obtained; the fused feature of the second downsampling and the output result of the first upsampling are input into the second upsampling stage for processing, and the output result of the second upsampling is obtained; the fused feature of the first downsampling and the output result of the second upsampling are input into the third upsampling stage for processing, and the output result of the third upsampling is obtained.

[0060] According to an embodiment of the present disclosure, in each upsampling feature extraction process, N branches are set, and each branch includes an atrous convolution enhancement module and a visual state space feature extraction module. The atrous convolution enhancement module is constructed based on an atrous convolutional neural network, and the atrous convolution enhancement module uses pointwise convolution for pixel enhancement; the visual state space feature extraction module is globally modeled based on Linear-Time Sequence Modeling with Selective State Spaces (Mamba).

[0061] According to an embodiment of the present disclosure, the fused feature of the (M - K + 1)-th downsampling and the output result of the (K - 1)-th upsampling are subjected to feature splicing, and then the spliced features are subjected to upsampling feature extraction and upsampling feature fusion to obtain the fused feature of the K-th upsampling.

[0062] According to an embodiment of the present disclosure, the spliced features are equally divided according to the number of branches N. For each branch, first, the atrous convolution enhancement module is used for first feature extraction, and then the visual state space feature extraction module is used for second feature extraction. Finally, the features output by the N branches are subjected to upsampling feature fusion to obtain the fused feature of the K-th upsampling.

[0063] According to an embodiment of the present disclosure, preprocessing the fused feature of the K-th upsampling to obtain the K-th upsampling feature may include: performing feature convolution or upsampling on the fused feature of the K-th upsampling to obtain the K-th upsampling feature.

[0064] According to an embodiment of the present disclosure, preprocessing the fused feature of the K-th upsampling to obtain the K-th upsampling feature includes: in the upsampling stage processing when K < M, performing upsampling on the fused feature of the K-th upsampling to obtain the K-th upsampling feature; in the upsampling stage processing when K = M, performing convolution on the fused feature of the M-th upsampling to obtain the M-th upsampling feature.

[0065] According to an embodiment of the present disclosure, in the upsampling stage processing of the K-th upsampling, when K < M, the upsampling layer is used to perform upsampling on the fused feature of the K-th upsampling to obtain the K-th upsampling feature. The K-th upsampling feature is the output result of the K-th upsampling.

[0066] According to an embodiment of the present disclosure, the upsampling layer may be constructed based on an upsampling algorithm (pixel-shuffle).

[0067] According to an embodiment of the present disclosure, in the Kth upsampling stage, K=M, a convolution layer is used to perform convolution processing on the Mth upsampling fusion feature to obtain the Mth upsampling feature. The Mth upsampling feature is the Mth upsampling output result.

[0068] According to an embodiment of the present disclosure, the smoke image and the smoke mask image are processed in M ​​cascaded upsampling stages and M cascaded downsampling stages, and outputting a predicted smoke-free image further includes: fusing the Mth upsampling features with the smoke image features to obtain a predicted smoke-free image.

[0069] According to an embodiment of the present disclosure, feature concatenation is performed on the M-th up-sampled features and the smoke image features to obtain a predicted smoke-free image.

[0070] Figure 2 An example schematic diagram of a smoke removal model according to an embodiment of the present disclosure is shown.

[0071] like Figure 2 As shown, the smoke removal model includes an encoder-decoder structure of four cascaded upsampling stages and an encoder-decoder structure of four cascaded downsampling stages. In the first downsampling stage (downsampling stage 1), the smoke mask image features and the smoke image features are concatenated, and then convolution processing, feature extraction and feature fusion are performed to obtain the first downsampling fusion features; in the second downsampling stage (downsampling stage 2), the first downsampling fusion features and the smoke image features are concatenated, and then downsampling processing, feature extraction and feature fusion are performed to obtain the second downsampling fusion features; in the third downsampling stage (downsampling stage 3), the second downsampling fusion features, the second preprocessing features and the pooling features are concatenated, and then downsampling processing, feature extraction and feature fusion are performed to obtain the third downsampling fusion features; in the fourth downsampling stage (downsampling stage 4), the third downsampling fusion features, the third preprocessing features and the pooling features are concatenated, and then downsampling processing, feature extraction and feature fusion are performed to obtain the fourth downsampling fusion features. The number of output channels in the downsampling stage is 32, 64, 128, and 256, respectively.

[0072] According to the implementation of the present disclosure, in the first upsampling stage processing (upsampling stage 1), the fourth downsampling fusion features are subjected to feature extraction, feature fusion and upsampling processing to obtain the first upsampling output result; in the second upsampling stage processing (upsampling stage 2), the third downsampling fusion features and the first upsampling output result are subjected to feature splicing, feature extraction, feature fusion and upsampling processing to obtain the second upsampling output result; in the third upsampling stage processing (upsampling stage 3), the second downsampling fusion features and the second upsampling output result are subjected to feature splicing, feature extraction, feature fusion and upsampling processing to obtain the third upsampling output result; in the fourth upsampling stage processing (upsampling stage 4), the first downsampling fusion features and the third upsampling output result are subjected to feature splicing, feature extraction, feature fusion and convolution processing to obtain the fourth upsampling output result; feature splicing is performed on the fourth upsampling output result and the smoke image features to obtain a predicted smoke-free image.

[0073] Figure 3A An example schematic diagram of downsampling feature extraction according to an embodiment of the present disclosure is shown.

[0074] like Figure 3A As shown in the figure, in the downsampling feature extraction process in the K-th downsampling stage, four branches are set, each of which includes a dilated convolution enhancement module and a visual state space feature extraction module. Divide into four equal parts and obtain 4 image branch features ( ), the four image branch features are input into the corresponding branches respectively. For the image branch features in each branch, the first feature is extracted by using the dilated convolution enhancement module, and then the second feature is extracted by using the visual state space feature extraction module to obtain the global features of the image. Finally, the four global features of the image output by the four branches are obtained ( ). The dilated convolution expansion rate of the dilated convolution enhancement module in each branch can be set to 1, 3, 5, and 7 respectively.

[0075] According to an embodiment of the present disclosure, the network structure used in upsampling feature extraction and downsampling feature extraction is the same.

[0076] According to an embodiment of the present disclosure, the downsampling feature fusion includes: fusing N global features of the image to obtain a global fusion feature; performing pooling processing on the global fusion feature to obtain N weight vectors corresponding to the N branches; normalizing the N weight vectors respectively to obtain N weight factors; weighting the N weight factors and the N global features of the image to obtain the Kth downsampling fusion feature.

[0077] According to an embodiment of the present disclosure, N image global features are fused in a merged manner to obtain a global fused feature, and the global fused feature is pooled using a pooling layer to obtain N weight vectors corresponding to the N branches. The weight vector represents the importance of the channel on each branch.

[0078] According to an embodiment of the present disclosure, a normalization layer is used to perform normalization processing on N weight vectors respectively to obtain N weight factors. The normalization layer can be constructed based on a softmax function.

[0079] According to an embodiment of the present disclosure, after weighted summation of N weight factors and N global features of the image, the weighted features are processed using a convolutional layer to restore the number of channels and obtain the Kth down-sampling fusion features.

[0080] In one embodiment, the K-th downsampling fusion feature As shown in formula (1):

[0081] (1)

[0082] in, represents the i-th weight factor, Representation convolutional layer, Characterize the global features of the i-th image.

[0083] According to an embodiment of the present disclosure, feature fusion is performed on the global features of images from different branches, and an adaptive weight vector is learned for the global features of the image of each branch in the channel dimension to reduce information redundancy and highlight important features, thereby ensuring the desmog performance of the model.

[0084] Figure 3B An example schematic diagram of down-sampling feature fusion according to an embodiment of the present disclosure is shown.

[0085] like Figure 3B As shown, for the four image global features ( ) to obtain the global fusion feature; perform pooling on the global fusion feature to obtain 4 weight vectors corresponding to the 4 branches; normalize the 4 weight vectors respectively to obtain 4 weight factors; weight the 4 weight factors and the 4 image global features, and finally obtain the Kth down-sampling fusion feature through the convolution layer .

[0086] According to an embodiment of the present disclosure, the network structure used in upsampling feature fusion and downsampling feature fusion is the same.

[0087] According to an embodiment of the present disclosure, the smoke removal model is trained based on the following operations: obtaining training samples, the training samples include sample smoke-free images; performing smoke rendering on the sample smoke-free images to obtain sample smoke images; using a smoke detector to process the sample smoke images to obtain sample smoke mask images; processing the sample smoke mask images and the sample smoke images according to the smoke removal model to obtain sample predicted smoke-free images; training the smoke removal model based on the sample predicted smoke-free images and the sample smoke-free images to obtain a trained smoke removal model.

[0088] According to an embodiment of the present disclosure, sample videos of laparoscopic surgical equipment are screened frame by frame to select clear and smoke-free sample smoke-free images.

[0089] According to an embodiment of the present disclosure, rendering software is used to perform smoke rendering on the sample smoke-free image, and the smoke effect in the real scene is simulated by randomly adjusting parameters such as density, buoyancy, heat, vorticity and position, so as to generate a reference smoke mask image with different density distribution and shape characteristics. At the same time, the smoke generated by the rendering is superimposed on the sample smoke-free image through transparent channel synthesis, so as to obtain a sample smoke image paired with the sample smoke-free image.

[0090] According to an embodiment of the present disclosure, a reference smoke mask image is used as label information, a sample smoke image is input into a smoke detector to obtain a sample smoke mask image, and the smoke detector is trained based on a loss value between the sample smoke mask image and the reference smoke mask image.

[0091] According to an embodiment of the present disclosure, a sample smoke mask image is used as prior information and input into a smoke removal model together with a sample smoke image to assist in locating the smoke area, thereby obtaining a sample predicted smoke-free image, which is a predicted image after smoke removal.

[0092] According to an embodiment of the present disclosure, a loss function is used to calculate the loss value between a sample predicted smoke-free image and a sample smoke-free image, and a smoke removal model is trained according to the loss value. When the loss value satisfies the preset model performance, the training is stopped; otherwise, the model parameters are adjusted and the training continues.

[0093] According to an embodiment of the present disclosure, a smoke removal model is trained based on a sample predicted smoke-free image and a sample smoke-free image to obtain a trained smoke removal model, including: calculating a first loss value between a sample predicted smoke-free image and a sample smoke-free image using a pixel perception loss function; calculating a second loss value between a sample predicted smoke-free image and a sample smoke-free image using a feature space perception loss function; calculating a third loss value between a sample predicted smoke-free image and a sample smoke-free image using a multi-scale structure loss function; calculating a fourth loss value of a sample predicted smoke-free image using a smoke perception loss function; obtaining a target loss value based on the first loss value, the second loss value, the third loss value and the fourth loss value; and training the smoke removal model based on the target loss value to obtain a trained smoke removal model.

[0094] According to an embodiment of the present disclosure, the first loss value represents the difference between the sample predicted smoke-free image and the sample smoke-free image constrained at the pixel level.

[0095] In one embodiment, the pixel-perceptual loss function As shown in formula (2):

[0096] (2)

[0097] in, Characterize the sample prediction of smoke-free image, Characterize the sample smoke-free image, Characterizes the first loss value.

[0098] According to an embodiment of the present disclosure, the sample predicted smoke-free image and the sample smoke-free image are respectively input into a deep convolutional neural network model (Visual Geometry Group Network 19, VGG19), and the similarity between the sample predicted smoke-free image and the sample smoke-free image in each layer of the deep convolutional neural network model is calculated using a feature space perceptual loss function. The second loss value represents the perceptual similarity between the sample predicted smoke-free image and the sample smoke-free image in the feature space.

[0099] In one embodiment, the feature space perception loss function is shown in formula (3):

[0100] (3)

[0101] in, Characterize the output features of the i-th layer network in the deep convolutional neural network model for sample prediction of smoke-free images. Characterize the output features of the i-th layer network in the deep convolutional neural network model of the sample smoke-free image, Characterizes the second loss value.

[0102] According to an embodiment of the present disclosure, the third loss value represents the contrast between the sample predicted smoke-free image and the sample smoke-free image in the high-frequency region.

[0103] In one embodiment, the multi-scale structure loss function As shown in formula (4):

[0104] (4)

[0105] in, Characterizes the third loss value.

[0106] According to an embodiment of the present disclosure, the fourth loss value represents the probability that there is smoke in the sample prediction smoke-free image. The fourth loss value is used to ensure that the smoke-free image restored by the network does not contain detectable smoke.

[0107] In one embodiment, the feature space perception loss function is shown in formula (5):

[0108] (5)

[0109] in, Characterizing smoke detectors, Characterizes the fourth loss value.

[0110] In one embodiment, the target loss value As shown in formula (6):

[0111] (6)

[0112] in, Characterize the first loss value, Characterize the second loss value, Characterize the third loss value, Characterizes the fourth loss value.

[0113] According to an embodiment of the present disclosure, a target loss value is obtained by evaluating the loss in multiple dimensions. When the target loss value satisfies the preset model performance, training is stopped to obtain a trained smoke removal model; otherwise, the model parameters are adjusted and training is continued.

[0114] Based on the above-mentioned image desmog method based on the smoke removal model, the present disclosure also provides an image desmog device based on the smoke removal model. Figure 4 The device is described in detail.

[0115] Figure 4 A structural block diagram of an image desmog device based on a smoke removal model according to an embodiment of the present disclosure is shown.

[0116] like Figure 4As shown, the image smoke removal device 400 based on the smoke removal model of this embodiment includes a detection module 410 and a smoke removal module 420 .

[0117] The detection module 410 is used to process the smoke image using the smoke detector to obtain a smoke mask image, wherein the smoke image represents a frame image rendered with smoke in the laparoscopic surgical device, and the pixel value of the smoke mask image represents the smoke concentration. In one embodiment, the detection module 410 can be used to perform the operation S110 described above, which will not be repeated here.

[0118] The smoke removal module 420 is used to perform M cascade upsampling stage processing and M cascade downsampling stage processing on the smoke image and the smoke mask image, and output a predicted smoke-free image, wherein any K-th downsampling stage processing includes downsampling feature extraction and downsampling feature fusion of the K-1-th downsampling output result and the smoke image, and the K-1-th downsampling output result is the K-1-th downsampling fusion feature obtained by performing the K-1-th downsampling stage processing on the smoke image and the smoke mask image, and the downsampling feature extraction includes: dividing the K-th preprocessing feature according to the number of branches N to obtain N image branch features, wherein the K-th preprocessing feature is obtained by preprocessing the K-1-th downsampling fusion feature and the smoke image feature, and the smoke image feature is extracted from the smoke image; for each image branch feature, using the dilated convolution enhancement module to perform a first feature extraction to obtain an image local feature; using the visual state space feature extraction module to perform a second feature extraction on the image local feature to obtain an image global feature. In one embodiment, the smoke removal module 420 may be used to perform the operation S120 described above, which will not be described in detail herein.

[0119] According to an embodiment of the present disclosure, the smoke removal module 420 includes a first processing submodule, a second processing submodule and a third processing submodule.

[0120] The first processing submodule is used to perform convolution processing on the smoke mask image features and the smoke image features in the K=1th downsampling stage processing to obtain the first preprocessing features, wherein the smoke mask image features are extracted from the smoke mask image.

[0121] The second processing submodule is used to downsample the first downsampled fusion features and the smoke image features in the K=2th downsampling stage to obtain the second preprocessing features.

[0122] The third processing submodule is used to fuse the K-1th downsampling features, the K-1th preprocessing features and the pooling features in the K>2th downsampling stage to obtain the Kth preprocessing features, wherein the pooling features are obtained based on the pooling operation on the smoke image features.

[0123] According to an embodiment of the present disclosure, the smoke removal module 420 further includes a first upsampling sub-module and a second upsampling sub-module.

[0124] The first upsampling sub-module is configured to perform upsampling feature extraction and upsampling feature fusion on the (M - K + 1)-th downsampling fusion feature and the upsampling feature of the (K - 1)-th upsampling output result, so as to obtain the K-th upsampling fusion feature.

[0125] The second upsampling sub-module is configured to preprocess the K-th upsampling fusion feature to obtain the K-th upsampling feature, where the (K - 1)-th upsampling output result is the (K - 1)-th upsampling feature obtained by performing the (K - 1)-th upsampling stage processing on the (M - K + 2)-th downsampling fusion feature and the (K - 2)-th upsampling output result.

[0126] According to an embodiment of the present disclosure, the second upsampling sub-module includes a first upsampling unit and a second upsampling unit.

[0127] The first upsampling unit is configured to perform upsampling processing on the K-th upsampling fusion feature in the upsampling stage processing where K < M, so as to obtain the K-th upsampling feature.

[0128] The second upsampling unit is configured to perform convolution processing on the M-th upsampling fusion feature in the upsampling stage processing where K = M, so as to obtain the M-th upsampling feature.

[0129] According to an embodiment of the present disclosure, the smoke removal module 420 further includes a prediction sub-module.

[0130] The prediction sub-module is configured to fuse the M-th upsampling feature and the smoky image feature to obtain a predicted smokeless image.

[0131] According to an embodiment of the present disclosure, the smoke removal module 420 further includes a first fusion sub-module, a second fusion sub-module, a third fusion sub-module, and a fourth fusion sub-module.

[0132] The first fusion sub-module is configured to fuse N image global features to obtain a global fusion feature.

[0133] The second fusion sub-module is configured to perform pooling processing on the global fusion feature to obtain N weight vectors corresponding to N branches.

[0134] The third fusion sub-module is configured to perform normalization processing on the N weight vectors respectively to obtain N weight factors.

[0135] The fourth fusion sub-module is configured to weight the N weight factors and the N image global features to obtain the K-th downsampling fusion feature.

[0136] According to an embodiment of the present disclosure, the image desmog device 400 further includes an acquisition module, a rendering module, a first processing module, a second processing module and a training module.

[0137] The acquisition module is used to acquire training samples, and the training samples include sample smoke-free images.

[0138] The rendering module is used to perform smoke rendering on the sample smoke-free image to obtain a sample smoke-containing image.

[0139] The first processing module is used to process the sample smoke image using the smoke detector to obtain a sample smoke mask image;

[0140] The second processing module is used to process the sample smoke mask image and the sample smoke image according to the smoke removal model to obtain a sample predicted smoke-free image;

[0141] The training module is used to train the smoke removal model according to the sample predicted smoke-free image and the sample smoke-free image to obtain the trained smoke removal model.

[0142] According to an embodiment of the present disclosure, the training module includes a first training submodule, a second training submodule, a third training submodule, a fourth training submodule, a fifth training submodule and a sixth training submodule.

[0143] The first training submodule is used to calculate a first loss value between a sample predicted smoke-free image and a sample smoke-free image by using a pixel-aware loss function.

[0144] The second training submodule is used to calculate a second loss value between the sample predicted smoke-free image and the sample smoke-free image by using a feature space perception loss function.

[0145] The third training submodule is used to calculate a third loss value between the sample predicted smoke-free image and the sample smoke-free image by using a multi-scale structure loss function.

[0146] The fourth training submodule is used to calculate a fourth loss value of the sample prediction of the smoke-free image by using the smoke perception loss function.

[0147] The fifth training submodule is used to obtain a target loss value according to the first loss value, the second loss value, the third loss value and the fourth loss value.

[0148] The sixth training submodule is used to train the smoke removal model based on the target loss value to obtain the trained smoke removal model.

[0149] According to an embodiment of the present disclosure, any multiple modules among the modules, submodules, units, and subunits can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the modules, submodules, units, and subunits can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the modules, submodules, units, and subunits can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0150] Figure 5 A block diagram of an electronic device suitable for implementing an image desmog method based on a smoke removal model according to an embodiment of the present disclosure is schematically shown.

[0151] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0152] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0153] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0154] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the image desmog method according to the embodiment of the present disclosure is implemented.

[0155] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0156] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the image desmog method provided by the embodiment of the present disclosure.

[0157] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0158] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0159] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0160] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0161] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0163] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An image smoke removal method based on a smoke removal model, characterized in that: The method includes: Processing a smoky image by using a smoke detector to obtain a smoke mask image, where the smoky image represents a frame image with smoke rendering in a laparoscopic surgical device, and the pixel value of the smoke mask image represents the smoke concentration; Performing M cascaded upsampling stage processes and M cascaded downsampling stage processes on the smoky image and the smoke mask image, and outputting a predicted smokeless image, where any K-th downsampling stage process includes downsampling feature extraction and downsampling feature fusion of the output result of the (K - 1)-th downsampling and the smoky image. The output result of the (K - 1)-th downsampling is the (K - 1)-th downsampling fusion feature obtained by performing the (K - 1)-th downsampling stage process on the smoky image and the smoke mask image. The downsampling feature extraction includes: Dividing the K-th preprocessing feature according to the number of branches N to obtain N image branch features, where the K-th preprocessing feature is obtained by preprocessing the (K - 1)-th downsampling fusion feature and the smoky image feature, and the smoky image feature is obtained by extracting the smoky image; For each of the image branch features, performing first feature extraction by using an atrous convolution enhancement module to obtain image local features; Performing second feature extraction on the image local features by using a visual state space feature extraction module to obtain image global features.

2. The method according to claim 1, characterized in that Preprocessing the (K - 1)-th downsampling fusion feature and the smoky image feature includes: In the first downsampling stage process (K = 1), performing convolution processing on the smoke mask image feature and the smoky image feature to obtain the first preprocessing feature, where the smoke mask image feature is obtained by extracting the smoke mask image; In the second downsampling stage process (K = 2), performing downsampling processing on the first downsampling fusion feature and the smoky image feature to obtain the second preprocessing feature; In the downsampling stage process where K > 2, performing the (K - 1)-th downsampling fusion feature, the (K - 1)-th preprocessing feature, and pooling features to obtain the K-th preprocessing feature, where the pooling features are obtained based on a pooling operation on the smoky image feature.

3. The method according to claim 1, characterized in that Any K-th upsampling stage process includes: Performing upsampling feature extraction and upsampling feature fusion on the (M - K + 1)-th downsampling fusion feature and the upsampling output result of the (K - 1)-th time to obtain the K-th upsampling fusion feature; Preprocessing the K-th upsampling fusion feature to obtain the K-th upsampling feature, where the upsampling output result of the (K - 1)-th time is the (K - 1)-th upsampling feature obtained by performing the (K - 1)-th upsampling stage process on the (M - K + 2)-th downsampling fusion feature and the upsampling output result of the (K - 2)-th time.

4. The method according to claim 3, characterized in that Preprocessing the K-th upsampling fusion feature to obtain the K-th upsampling feature includes: In the upsampling stage process where K < M, performing upsampling processing on the K-th upsampling fusion feature to obtain the K-th upsampling feature; In the upsampling stage process where K = M, performing convolution processing on the M-th upsampling fusion feature to obtain the M-th upsampling feature.

5. The method according to claim 4, characterized in that The smoke image and the smoke mask image are processed in M ​​cascade upsampling stages and M cascade downsampling stages, and outputting a predicted smoke-free image further includes: The M-th up-sampling feature and the smoke image feature are fused to obtain a predicted smoke-free image.

6. The method according to claim 1, characterized in that The down-sampling feature fusion includes: Fusing the N global features of the image to obtain a global fusion feature; Performing pooling processing on the global fusion features to obtain N weight vectors corresponding to the N channels; Normalizing the N weight vectors respectively to obtain N weight factors; The N weight factors and the N global features of the image are weighted to obtain a K-th down-sampling fusion feature.

7. The method according to claim 1, characterized in that The smoke removal model is trained based on the following operations: Acquire a training sample, wherein the training sample includes a sample smoke-free image; Performing smoke rendering on the sample smoke-free image to obtain a sample smoke-containing image; Processing the sample smoke image using a smoke detector to obtain a sample smoke mask image; Processing the sample smoke mask image and the sample smoke image according to the smoke removal model to obtain a sample predicted smoke-free image; The smoke removal model is trained according to the sample predicted smoke-free image and the sample smoke-free image to obtain a trained smoke removal model.

8. The method according to claim 7, characterized in that According to the sample predicted smoke-free image and the sample smoke-free image, a smoke removal model is trained, and the trained smoke removal model is obtained, comprising: Calculating a first loss value between the sample predicted smoke-free image and the sample smoke-free image using a pixel-perceptual loss function; Calculating a second loss value between the sample predicted smoke-free image and the sample smoke-free image using a feature space perception loss function; Calculating a third loss value between the sample predicted smoke-free image and the sample smoke-free image using a multi-scale structure loss function; Calculating a fourth loss value of the sample predicting a smoke-free image using a smoke perception loss function; Obtaining a target loss value according to the first loss value, the second loss value, the third loss value, and the fourth loss value; Based on the target loss value, the smoke removal model is trained to obtain a trained smoke removal model.

9. An image smoke removal device based on a smoke removal model, the device comprising: A detection module, used for processing a smoke image using a smoke detector to obtain a smoke mask image, wherein the smoke image represents a frame image rendered with smoke in a laparoscopic surgical device, and a pixel value of the smoke mask image represents smoke concentration; The smoke removal module is used to perform M cascade upsampling stage processing and M cascade downsampling stage processing on the smoke image and the smoke mask image, and output a predicted smoke-free image, wherein any K-th downsampling stage processing includes downsampling feature extraction and downsampling feature fusion of the K-1-th downsampling output result and the smoke image, and the K-1-th downsampling output result is the K-1-th downsampling fusion feature obtained by performing the K-1-th downsampling stage processing on the smoke image and the smoke mask image, and the downsampling feature extraction includes: dividing the K-th preprocessing feature according to the number of branches N to obtain N image branch features, wherein the K-th preprocessing feature is obtained by preprocessing the K-1-th downsampling fusion feature and the smoke image feature, and wherein the smoke image feature is obtained by extracting the smoke image; for each of the image branch features, using the dilated convolution enhancement module to perform a first feature extraction to obtain an image local feature; using the visual state space feature extraction module to perform a second feature extraction on the image local feature to obtain an image global feature.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images