Endoscope image deblurring method based on sampling split convolution and model training method

Abstract

The present disclosure provides an endoscopic image deblurring method based on sampling split convolution and a model training method. The endoscopic image deblurring method based on sampling split convolution comprises: obtaining a plurality of to-be-processed images with different scales; using a convolution layer to respectively extract features of the plurality of to-be-processed images to obtain a plurality of first feature maps; using a down-sampling layer to respectively fuse features of a plurality of first feature map sets to obtain a plurality of encoding feature maps; using a feature interaction fusion layer to respectively fuse the plurality of encoding feature maps and the to-be-processed images corresponding to the encoding feature maps to obtain a plurality of second feature maps; and using an up-sampling layer to reconstruct images based on the plurality of second feature maps to obtain a plurality of target images with different scales. In addition, the present disclosure also provides an endoscopic image deblurring model training method and device based on sampling split convolution.

Description

Technical Field

[0001] This disclosure relates to the fields of medical image processing technology and computer vision technology, and more specifically, to an endoscope image deblurring method and model training method based on sampling segmentation convolution. Background Technology

[0002] In clinical medicine, endoscopic invasive examinations can effectively increase the detection rate of related diseases, thereby improving the cure rate. However, due to the small size of the endoscope, the images acquired during endoscopic examinations inevitably suffer from blurring.

[0003] In realizing the present invention, the inventors discovered that the image deblurring methods in the related art have the problem of insufficient image clarity when applied to the deblurring of endoscopic images. Summary of the Invention

[0004] In view of this, this disclosure provides a method for deblurring endoscopic images based on sampling and segmentation convolution, as well as a model training method.

[0005] One aspect of this disclosure provides an endoscopic image deblurring method based on sampling and segmentation convolution, comprising: acquiring multiple images to be processed at different scales, wherein the multiple images to be processed are obtained by sampling a blurred endoscopic image at different magnifications; extracting features from the multiple images to be processed using convolutional layers to obtain multiple first feature maps; fusing features from the multiple sets of first feature maps using downsampling layers to obtain multiple encoded feature maps, wherein each set of first feature maps includes at least one of the first feature maps, and the multiple sets of first feature maps correspond one-to-one with the multiple images to be processed; fusing the multiple encoded feature maps and the images to be processed corresponding to the encoded feature maps using feature interaction fusion layers to obtain multiple second feature maps; and reconstructing images based on the multiple second feature maps using upsampling layers to obtain multiple target images at different scales.

[0006] Another aspect of this disclosure provides a training method for an endoscopic image deblurring model based on sampling and segmentation convolution, comprising: acquiring multiple sample images of different scales, wherein the multiple sample images are obtained by sampling an initial sample image at different magnifications; extracting features from the multiple sample images using convolutional layers to obtain multiple first feature maps; and fusing features from the multiple sets of first feature maps using downsampling layers to obtain multiple encoded feature maps, wherein each set of first feature maps includes at least one of the first feature maps, and the multiple sets of first feature maps correspond one-to-one with the multiple sample images; The feature interaction fusion layer fuses multiple encoded feature maps and corresponding sample images to obtain multiple second feature maps; the upsampling layer reconstructs images based on the multiple second feature maps to obtain multiple reconstructed images with different scales; and the model parameters of the convolutional layer, the downsampling layer, the feature interaction fusion layer, and the upsampling layer are adjusted based on the multiple reconstructed images and the label images of the multiple sample images to obtain an endoscopy image deblurring model, wherein the label images of the multiple sample images include those obtained by downsampling the label images of the initial sample images based on the different magnifications.

[0007] Another aspect of this disclosure provides an endoscopic image descrambling device based on sampling and segmentation convolution, comprising: a first acquisition module for acquiring multiple images to be processed at different scales, wherein the multiple images to be processed are obtained by sampling a blurred endoscopic image at different magnifications; a first extraction module for extracting features from the multiple images to be processed using convolutional layers to obtain multiple first feature maps; a first fusion module for fusing features from the multiple sets of first feature maps using downsampling layers to obtain multiple coded feature maps, wherein each set of first feature maps includes at least one of the first feature maps, and the multiple sets of first feature maps correspond one-to-one with the multiple images to be processed; a second fusion module for fusing the multiple coded feature maps and the images to be processed corresponding to the coded feature maps using feature interaction fusion layers to obtain multiple second feature maps; and a first reconstruction module for reconstructing images based on the multiple second feature maps using upsampling layers to obtain multiple target images at different scales.

[0008] Another aspect of this disclosure provides a training apparatus for an endoscopic image deblurring model based on sampling and segmentation convolution, comprising: a second acquisition module for acquiring multiple sample images with different scales, wherein the multiple sample images are obtained by sampling an initial sample image at different magnifications; a second extraction module for extracting features from the multiple sample images using convolutional layers to obtain multiple first feature maps; and a third fusion module for fusing features from the multiple sets of first feature maps using downsampling layers to obtain multiple encoded feature maps, wherein each set of first feature maps includes at least one of the first feature maps, and the multiple sets of first feature maps correspond one-to-one with the multiple sample images. The fourth fusion module is used to fuse multiple encoded feature maps and corresponding sample images using a feature interaction fusion layer to obtain multiple second feature maps; the second reconstruction module is used to reconstruct images based on multiple second feature maps using an upsampling layer to obtain multiple reconstructed images with different scales; and the training module is used to adjust the model parameters of the convolutional layer, the downsampling layer, the feature interaction fusion layer, and the upsampling layer based on the multiple reconstructed images and the label images of the multiple sample images to obtain an endoscopy image deblurring model, wherein the label images of the multiple sample images include those obtained by downsampling the label images of the initial sample images based on the different magnifications.

[0009] According to embodiments of this disclosure, by using features from blurred endoscopic images at different scales for image reconstruction, the input image can possess richer hierarchical features. Feature fusion at the downsampling layer preserves information from features at different scales. Through the processing of the feature interaction fusion layer, interactive fusion and complementary advantages of detailed features and semantic features can be achieved, improving the network's reconstruction capability. Furthermore, feature fusion at the downsampling layer avoids the blurring and sparsity of semantic information in the image, thereby improving the expressive power of the reconstructed image. These technical means can at least partially overcome the technical problem of insufficient image sharpness in reconstructed images when applied to endoscopic image deblurring in related technologies, thus effectively improving the signal-to-noise ratio of the reconstructed target image and enhancing its sharpness. Attached Figure Description

[0010] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0011] Figure 1 The flowchart of an endoscopic image deblurring method based on sampling segmentation convolution according to an embodiment of the present disclosure is illustrated schematically.

[0012] Figure 2The schematic diagram illustrates the processing flow of the sampling segmentation convolution downsampling module according to an embodiment of the present disclosure.

[0013] Figure 3 The schematic diagram illustrates the processing flow of the feature interaction fusion module according to an embodiment of the present disclosure.

[0014] Figure 4 The schematic diagram illustrates the processing flow of the sampling segmentation convolution upsampling module according to an embodiment of the present disclosure.

[0015] Figure 5 The illustration shows a schematic diagram of an endoscope image deblurring model based on sampling segmentation convolution according to an embodiment of the present disclosure.

[0016] Figure 6 The illustration shows a schematic diagram of reconstructed images obtained by applying an endoscope image deblurring model based on sampling segmentation convolution and a correlation model, respectively, according to embodiments of the present disclosure.

[0017] Figure 7 The flowchart illustrates a method for training an endoscopic image deblurring model based on sampling segmentation convolution according to an embodiment of the present disclosure.

[0018] Figure 8 A schematic diagram of the dark channel image, intermediate channel image, and bright channel image of a reconstructed image according to an embodiment of the present disclosure is shown.

[0019] Figure 9 A block diagram of an endoscopic image deblurring apparatus based on sampling segmentation convolution according to an embodiment of the present disclosure is illustrated schematically.

[0020] Figure 10 A block diagram of a training apparatus for an endoscope image deblurring model based on sampling segmentation convolution according to an embodiment of the present disclosure is illustrated. Detailed Implementation

[0021] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0024] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0025] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0026] In clinical medicine, endoscopic examinations can effectively increase the detection rate of related diseases, thereby improving their cure rate. However, due to the small size of the endoscope, images acquired during endoscopic examinations inevitably suffer from blurriness. Blurred endoscopic images will inevitably affect the accuracy of subsequent operations such as recognition, detection, and segmentation.

[0027] In related technologies, methods based on Convolutional Neural Networks (CNNs) have been widely applied in image deblurring research and have achieved good results. Among these, "end-to-end" image deblurring methods, i.e., methods for deblurring scenarios where both input and output are a single image, have been extensively studied. For example, Kupyn et al. proposed the image deblurring algorithm DeblurGAN-v2 (DeblurringGenerative Adversarial Networks-v2), which has the advantage of restoring detail information well, but the disadvantage of risking the generation of false structures or ghosting. Another example is Nah et al., who proposed a novel CNN-based image deblurring network (Deep multi-scale CNN for dynamic scene deblurring, DeepDeblur), which involves inputting a sequence of blurred images at different resolutions into the network and reconstructing the full-resolution image. This network has high computational complexity and memory overhead. Inspired by DeepDeblur, various multi-scale image deblurring algorithms based on CNNs have been proposed, such as SRN-DeblurNet (Scale-Recurrent Network-DeblurNet), PSS-NSC (parameter selective sharing and nested skip connections), MT-RNN (Multi-Temporal Recurrent Neural Networks), and MIMO-UNet (multi-input multi-output U-net). The continuous emergence of new deblurring algorithms is constantly improving the performance of image deblurring.

[0028] Although the above methods have achieved good results in the field of image deblurring, when these methods are applied to the processing of medical images, such as endoscopic images, the sample size of medical images is relatively small and the semantic information they contain is relatively simple. Therefore, the above methods usually cannot make full use of the characteristics of endoscopic images themselves, resulting in unsatisfactory deblurring effects.

[0029] On the other hand, for end-to-end image processing tasks, the U-shaped network is a commonly used network structure that can effectively balance the extraction and utilization of high-level semantic information and low-level detail features of images.

[0030] In view of this, the embodiments of this disclosure adopt a U-shaped convolutional neural network as the basic framework, utilize the multi-scale concept and combine the characteristics of rich details and relatively little semantic information in endoscopic images to provide an end-to-end endoscopic image deblurring model, and provide a method for endoscopic image reconstruction using this model.

[0031] Specifically, embodiments of this disclosure provide an endoscopic image deblurring method and model training method based on sampling and segmentation convolution. This endoscopic image deblurring method based on sampling and segmentation convolution includes: acquiring multiple images to be processed at different scales, wherein the multiple images to be processed are obtained by sampling a blurred endoscopic image at different magnifications; extracting features from the multiple images to be processed using convolutional layers to obtain multiple first feature maps; fusing the multiple sets of first feature maps using downsampling layers to obtain multiple encoded feature maps, wherein each set of first feature maps includes at least one first feature map, and the multiple sets of first feature maps correspond one-to-one with the multiple images to be processed; fusing the multiple encoded feature maps and the images to be processed corresponding to the encoded feature maps using feature interaction fusion layers to obtain multiple second feature maps; and reconstructing the image based on the multiple second feature maps using upsampling layers to obtain multiple target images at different scales.

[0032] Figure 1 The flowchart of an endoscopic image deblurring method based on sampling segmentation convolution according to an embodiment of the present disclosure is illustrated schematically.

[0033] like Figure 1 As shown, the method includes operations S101 to S105.

[0034] In operation S101, multiple images to be processed with different scales are acquired, wherein the multiple images to be processed are obtained by sampling and processing blurred endoscopic images based on different magnifications.

[0035] In operation S102, convolutional layers are used to extract features from multiple images to be processed, resulting in multiple first feature maps.

[0036] In operation S103, the downsampling layer is used to perform feature fusion on multiple sets of first feature maps to obtain multiple encoded feature maps. Each set of first feature maps includes at least one first feature map, and the multiple sets of first feature maps correspond one-to-one with multiple images to be processed.

[0037] In operation S104, the feature interaction fusion layer is used to fuse multiple encoded feature maps and the images to be processed corresponding to the encoded feature maps to obtain multiple second feature maps.

[0038] In operation S105, image reconstruction is performed using an upsampling layer based on multiple second feature maps to obtain multiple target images with different scales.

[0039] According to embodiments of this disclosure, multiple images to be processed at different scales can refer to multiple images to be processed having different data dimensions in the width and height dimensions, that is, multiple images to be processed having different scales.

[0040] According to embodiments of this disclosure, sampling processing may include upsampling and downsampling, both of which can use multiple magnifications. Taking downsampling as an example, a blurred endoscopic image with a scale of H×W can be downsampled at magnifications of 1 / 2 and 1 / 4 to obtain two images to be processed with scales of H / 2×W / 2 and H / 4×W / 4, respectively.

[0041] According to embodiments of this disclosure, a convolutional layer may include multiple convolutional modules. For different images to be processed, a single convolutional module configured in the convolutional layer can be used to extract image features for each image to be processed. Alternatively, a separate convolutional module can be configured for each scale of the image to be processed, i.e., the convolutional module associated with that scale can be used to extract features from the image to be processed at that scale.

[0042] According to embodiments of this disclosure, at least one first feature map can be selected from a plurality of first feature maps to form a set of first feature maps. The set of first feature maps obtained through selection can be of various types. For example, there can be three first feature maps, represented as a, b, and c, respectively. Therefore, the set of first feature maps obtained through selection can have seven elements, and the first feature maps included can be represented as a, b, c, ab, ac, bc, and abc, respectively.

[0043] According to an embodiment of the present disclosure, alternatively, the number of the first feature map set may be the same as the number of images to be processed, each first feature map set may correspond to one image to be processed, and the first feature map set may contain a first feature map extracted from the image to be processed.

[0044] According to embodiments of this disclosure, the downsampling layer may include multiple downsampling modules. Alternatively, the number of downsampling modules may be the same as the number of images to be processed, and each downsampling module may be used for feature fusion processing of a first feature map set.

[0045] According to embodiments of this disclosure, feature fusion can refer to fusing the input feature maps along the channel dimension, or fusing them along dimensions other than width, height, and channels. The fused feature map retains the same width and height dimensions as the input feature map.

[0046] According to embodiments of this disclosure, the feature interaction fusion layer can achieve the interaction fusion of semantic features and detail features. Specifically, the encoded feature map may contain semantic features. During fusion, the semantic features in the encoded feature map can be used to activate the image to be processed, and then convolution is used to fuse the activated image to be processed and the encoded feature map to achieve the interaction fusion of detail features and semantic features.

[0047] According to embodiments of this disclosure, the feature interaction and fusion layer may include multiple feature interaction and fusion (FIF) modules. Alternatively, the number of feature interaction and fusion modules may be consistent with the number of images to be processed, and each feature interaction and fusion module may be used for the fusion processing of an encoded feature map and the corresponding image to be processed.

[0048] According to embodiments of this disclosure, the upsampling layer may include multiple upsampling modules. Alternatively, the number of upsampling modules may be the same as the number of images to be processed, and each upsampling module may be used for image reconstruction of a second feature map.

[0049] According to embodiments of this disclosure, convolutional layers, downsampling layers, feature interaction fusion layers, and upsampling layers can constitute an endoscope image deblurring model. When applying the method provided in this embodiment, the endoscope image deblurring model can be a model that has been trained using sample images.

[0050] According to embodiments of this disclosure, the reconstructed target image can have the same scale as the image to be processed, i.e., the image to be processed with a scale of H×W is processed sequentially through a convolutional layer, a downsampling layer, a feature interaction fusion layer, and an upsampling layer, resulting in a target image with a scale of H×W.

[0051] According to embodiments of this disclosure, by using features from blurred endoscopic images at different scales for image reconstruction, the input image can possess richer hierarchical features. Feature fusion at the downsampling layer preserves information from features at different scales. Through the processing of the feature interaction fusion layer, interactive fusion and complementary advantages of detailed features and semantic features can be achieved, improving the network's reconstruction capability. Furthermore, feature fusion at the downsampling layer avoids the blurring and sparsity of semantic information in the image, thereby improving the expressive power of the reconstructed image. These technical means can at least partially overcome the technical problem of insufficient image sharpness in reconstructed images when applied to endoscopic image deblurring in related technologies, thus effectively improving the signal-to-noise ratio of the reconstructed target image and enhancing its sharpness.

[0052] The following is for reference. Figures 2-6 In conjunction with specific embodiments, Figure 1 The method shown will be further explained.

[0053] According to embodiments of this disclosure, the downsampling layer may include multiple sampling slice convolution down-sampling (SSCD) modules and multiple first residual modules.

[0054] According to embodiments of this disclosure, operation S103 may include the following operations:

[0055] For each set of first feature maps, the first feature map set is fused using the sampling, segmentation, convolution, and downsampling module corresponding to the first feature map set to obtain a third feature map; and the third feature map is processed using the first residual module corresponding to the first feature map set to obtain an encoded feature map.

[0056] According to embodiments of this disclosure, the number of sampling segmentation convolutional downsampling modules in the downsampling layer can be less than the number of images to be processed. Specifically, when the first feature map set corresponding to the image to be processed contains only one first feature map, the first feature map set can be processed without using the sampling segmentation convolutional downsampling modules, that is, the sampling segmentation convolutional downsampling modules corresponding to the first feature map set can be omitted. When processing the first feature map set using the downsampling layer, the first feature map in the first feature map set can be processed directly using the first residual module to obtain the encoded feature map.

[0057] For example, after feature extraction, the three first feature maps obtained from the three images B1, B2, and B3 to be processed are O1, O2, and O3, respectively. After selecting the first feature map set, the first feature map set 1 corresponding to the image B1 contains the first feature map O1, the first feature map set 2 corresponding to the image B2 contains the first feature maps O1 and O2, and the first feature map set 3 corresponding to the image B3 contains the first feature maps O2 and O3. When processing using the method of operation S103, the first feature map O1 contained in the first feature map set 1 can be directly processed using the residual module 1 to obtain the encoded feature E1; the first feature maps O1 and O2 contained in the first feature map set 2 can be fused using the sampling segmentation convolution downsampling module 1, and then processed through the residual module 2 to obtain the encoded feature E2; the first feature maps O2 and O3 contained in the first feature map set 3 can be fused using the sampling segmentation convolution downsampling module 2, and then processed through the residual module 3 to obtain the encoded feature E3.

[0058] According to embodiments of this disclosure, the first residual module may consist of one or more residual blocks. Each residual block may contain a convolutional network, and the output of the residual block may be the sum of the input and output of the convolutional network.

[0059] According to embodiments of this disclosure, performing feature fusion on the first feature map set using a sampling segmentation convolution downsampling module corresponding to the first feature map set to obtain a third feature map may include the following operations:

[0060] Based on the scale of the feature maps, at least one first feature map in the first feature map set is divided into a first target feature map and a first feature map to be processed; the first feature map to be processed is sampled and downsampled to obtain a first intermediate feature map; the first intermediate feature map and the first target feature map are concatenated based on the channel dimension to obtain a second intermediate feature map; the second intermediate feature map is fused in the channel dimension using batch convolution to obtain a third intermediate feature map; and the third intermediate feature map is subjected to maximum value filtering to obtain a third feature map.

[0061] According to embodiments of this disclosure, the first target feature map may refer to the smallest first feature map among at least one first feature map. The first feature map to be processed may refer to a first feature map other than the first target feature map among at least one first feature map.

[0062] Figure 2 The schematic diagram illustrates the processing flow of the sampling segmentation convolution downsampling module according to an embodiment of the present disclosure.

[0063] like Figure 2 As shown, taking an example where there are at least two first feature maps, the scale of the first target feature map is H / 2×W / 2×2C, and the scale of the first feature map to be processed is H×W×C, the processing flow of the sampling, segmentation, convolution, and downsampling module can include:

[0064] First, the first feature map to be processed, with a scale of H×W×C, is sampled and downsampled to obtain four first intermediate feature maps with a scale of H / 2×W / 2×C. Specifically, the sampling and downsampling is performed by sampling the first feature map to be processed at intervals of one pixel, and the sampling results are then segmented to obtain four first intermediate feature maps.

[0065] Then, the four first intermediate feature maps with a scale of H / 2×W / 2×C are concatenated with the first target feature map with a scale of H / 2×W / 2×2C based on the channel dimension to obtain four second intermediate feature maps with a scale of H / 2×W / 2×3C.

[0066] Then, the four second intermediate feature maps with a scale of H / 2×W / 2×3C can be batch convolved to change the number of feature channels and achieve feature fusion, resulting in four third intermediate feature maps with a scale of H / 2×W / 2×2C. The specific calculation process is shown in formula (1):

[0067] P′ i =f conv(P i (1)

[0068] In equation (1), P′ i f represents the i-th third intermediate feature map; conv Represents the convolution operation; P i Let i represent the i-th second intermediate feature map; i = 1, 2, 3, 4.

[0069] Then, the maximum value filtering can be applied to the four third intermediate feature maps with a scale of H / 2×W / 2×2C, that is, the maximum value is taken according to the corresponding position, and finally the third feature map with a scale of H / 2×W / 2×2C is obtained. The specific calculation process is shown in formula (2):

[0070] O′=f max (P′1, P′2, P′3, P′4) (2)

[0071] In equation (2), O′ represents the third feature map; f conv This indicates a maximum value filtering operation.

[0072] According to embodiments of this disclosure, when using the sampling segmentation convolution downsampling module to perform feature fusion of the first target feature map and the first feature map to be processed, all points in the feature map are parameterized by the feature fusion process, and maximum value filtering is used for feature selection, so that the sampling and feature fusion processes are interdependent, avoiding the loss of information in the large-scale feature map during the downsampling process.

[0073] According to embodiments of this disclosure, the feature interaction fusion layer may include multiple feature interaction fusion modules.

[0074] Figure 3 The schematic diagram illustrates the processing flow of the feature interaction fusion module according to an embodiment of the present disclosure.

[0075] like Figure 3 As shown, B1 represents the image to be processed with a scale of H×W×3, and E1 represents the encoded feature map with a scale of H×W×C. This encoded feature map E1 can be obtained by processing the image to be processed B1 using methods S102 to S103. The processing flow of the feature interaction fusion module may include:

[0076] First, the encoded feature map E1 is input into the Channel Attention Module (CAM) to determine the channel weights. These channel weights can be multiplied by the image B1 to obtain the image B1′ to be processed, which is activated in the channel dimension.

[0077] Next, the image to be processed, B1′, can be input into the Spatial Attention Module (SAM) to determine the spatial weights. These spatial weights can be multiplied by the image to be processed, B1′, to obtain the image to be processed, B1″, which is activated in the spatial dimension. This completes the activation of the image to be processed, B1.

[0078] Then, the spatially activated image B1″ to be processed is concatenated with the encoded feature map E1, and features are fused by convolution to obtain the second feature map E1′ with a scale of H×W×C.

[0079] According to embodiments of this disclosure, the processing flow of the feature interaction fusion module can be as shown in formula (3):

[0080] E′1=f conv {f CAT {f SAM [f CAM (E1)×B1]×B1},E1} (3)

[0081] In equation (3), f CAT Indicates channel splicing operation; f SAM This indicates that spatial attention is used for processing; f CAM This indicates that the channel attention module is used for processing.

[0082] According to embodiments of this disclosure, by using semantic information from the encoded feature map to activate the image to be processed before decoding, and then using convolution to fuse the activated image to be processed and the encoded feature map, the interactive fusion and complementary advantages of detailed features and semantic features are achieved, thereby improving the reconstruction capability of the network.

[0083] According to embodiments of this disclosure, the upsampling layer may include multiple second residual modules and multiple sampling slice convolution up-sampling (SSCU) modules.

[0084] According to embodiments of this disclosure, operation S105 may include the following operations:

[0085] Multiple second residual modules are used to process the second feature maps corresponding to the second residual modules to obtain multiple decoded feature maps; multiple sampling, segmentation, convolution, and upsampling modules are used to perform feature fusion on the sets of second feature maps corresponding to the sampling, segmentation, convolution, and upsampling modules to obtain multiple fourth feature maps, wherein each set of second feature maps includes at least one decoded feature map, and the multiple sets of second feature maps correspond one-to-one with multiple images to be processed; and multiple fourth feature maps are used to perform convolution reconstruction on the images to be processed corresponding to the fourth feature maps to obtain multiple target images.

[0086] According to embodiments of this disclosure, the second residual module may consist of one or more residual blocks. Each residual block may contain a convolutional network, and the output of the residual block may be the sum of the input and output of the convolutional network.

[0087] According to embodiments of this disclosure, the number of sampling segmentation convolutional upsampling modules in the upsampling layer can be less than the number of images to be processed. Specifically, when the second feature map set corresponding to the image to be processed contains only one decoded feature map, the second feature map set can be processed without using sampling segmentation convolutional upsampling modules, that is, a sampling segmentation convolutional upsampling module corresponding to the second feature map set can be omitted. When processing the second feature map set using the upsampling layer, convolutional reconstruction can be directly performed using the decoded feature map in the second feature map set and the image to be processed corresponding to the decoded feature map to obtain the target image.

[0088] For example, if there are three images to be processed, denoted as B1, B2, and B3, after processing by a convolutional layer, a downsampling layer, a feature interaction fusion layer, and a second residual module, three decoded feature maps, denoted as D1, D2, and D3, can be obtained. After selecting the second feature map set, the second feature map set 1 corresponding to image B1 contains decoded feature maps D1 and D2; the second feature map set 2 corresponding to image B2 contains decoded feature maps D2 and D3; and the second feature map set 3 corresponding to image B3 contains decoded feature map D3. In subsequent processing using the method of operation S105, the sampling, segmentation, convolution, and upsampling module 1 can be used to fuse the decoded feature maps D1 and D2 contained in the second feature map set 1, and then added to the image B1 through a convolution module to obtain the target image. The sampling segmentation convolution upsampling module 2 is used to fuse the decoded feature maps D2 and D3 contained in the second feature map set 2, and then the fusion is performed by the convolution module and added to the image B2 to obtain the target image S2; for the decoded feature map D3 contained in the second feature map set 3, the decoded feature map D3 can be directly passed through the convolution module and added to the image B3 to obtain the target image S3.

[0089] According to embodiments of this disclosure, performing feature fusion on the second feature map set corresponding to the sampling segmentation convolution upsampling module using multiple sampling segmentation convolution upsampling modules to obtain multiple fourth feature maps may include the following operations:

[0090] For each set of second feature maps, based on the scale of the feature maps, at least one decoded feature map in the set of second feature maps is divided into a second target feature map and a second feature map to be processed; the second feature map to be processed is sampled and downsampled to obtain a fourth intermediate feature map; the fourth intermediate feature map and the second target feature map are concatenated based on the channel dimension to obtain a fifth intermediate feature map; the fifth intermediate feature map is fused in the channel dimension using batch convolution to obtain a sixth intermediate feature map; and the sixth intermediate feature map is restored based on the segmentation index position of the sampled and downsampled feature map to obtain the fourth feature map.

[0091] According to embodiments of this disclosure, the second target feature map may refer to the smallest-scale decoded feature map among at least one decoded feature map. The second feature map to be processed may refer to any decoded feature map other than the second target feature map among at least one decoded feature map.

[0092] Figure 4 The schematic diagram illustrates the processing flow of the sampling segmentation convolution upsampling module according to an embodiment of the present disclosure.

[0093] like Figure 4 As shown, taking an example where the number of at least one decoded feature map is 2, the scale of the second target feature map is H / 2×W / 2×2C, and the scale of the second feature map to be processed is H×W×C, the processing flow of the sampling, segmentation, convolution, and upsampling module can include:

[0094] First, the second feature map to be processed, with a scale of H×W×C, is sampled and downsampled to obtain four fourth intermediate feature maps with a scale of H / 2×W / 2×C. Specifically, the sampling and downsampling is performed on the second feature map to be processed at intervals of one pixel, and the sampling results are then segmented to obtain four fourth intermediate feature maps.

[0095] Then, the four fourth intermediate feature maps with a scale of H / 2×W / 2×C are concatenated with the second target feature map with a scale of H / 2×W / 2×2C based on the channel dimension to obtain four fifth intermediate feature maps with a scale of H / 2×W / 2×3C.

[0096] Then, the four fifth intermediate feature maps with a scale of H / 2×W / 2×3C can be batch convolved to change the number of feature channels and achieve feature fusion, resulting in four sixth intermediate feature maps with a scale of H / 2×W / 2×2C. The specific calculation process is shown in the above formula (1), where P′ in formula (1) is... i Change to represent the i-th sixth intermediate feature map, P i It can be changed to represent the i-th fifth intermediate feature map.

[0097] Then, the four sixth intermediate feature maps with a scale of H / 2×W / 2×2C can be restored according to the segmentation index position to finally obtain the fourth feature map with a scale of H / 2×W / 2×2C.

[0098] According to embodiments of this disclosure, the segmentation index position may refer to the position recorded when the second feature map to be processed is sampled and segmented, and segmented based on the sampling point.

[0099] According to embodiments of this disclosure, when using the sampling segmentation convolution upsampling module to perform feature fusion of the second target feature map and the second feature map to be processed, since the small-scale feature map, i.e. the second target feature map, is not directly upsampled, the ambiguity and sparsity of semantic information are avoided, thereby effectively improving the utilization rate of semantic information.

[0100] Figure 5 The illustration shows a schematic diagram of an endoscope image deblurring model based on sampling segmentation convolution according to an embodiment of the present disclosure.

[0101] like Figure 5 As shown, the number of input channels for this endoscopic image deblurring model based on sampling segmentation convolution can be 3.

[0102] First, in the encoding stage, three images B1, B2, and B3 at different scales can be simultaneously input into the model. Convolutional layers are then used to extract features from these images at different scales, resulting in first feature maps O1, O2, and O3. Next, a sampling-segmentation convolutional downsampling module is used to fuse these first feature maps at different scales, which are then processed by a first residual module to obtain encoded feature maps E1, E2, and E3. Then, a feature interaction fusion module is used to fuse the images to be processed and the encoded feature maps. The output of the feature interaction fusion module is processed by a second residual module to obtain decoded feature maps D1, D2, and D3. Finally, in the decoding stage, a sampling-segmentation convolutional upsampling module is used to fuse the decoded feature maps at different scales, and the decoded output consists of three target images at different scales.

[0103] According to embodiments of this disclosure, the comparative experimental results of an endoscope image deblurring model based on sampling segmentation convolution and other models in related technologies, such as DeepDeblur, DeblurGAN-v2, SRN-DeblurNet, PSS-NSC, DMPHN, MT-RNN, SAPHN, MPRNet, MIMO-UNet, etc., based on public datasets such as EAD (Endoscope Artificial Detection) and Kvasir-SEG, are shown in Table 1. As shown in Table 1, the endoscopic image deblurring model based on sampling segmentation convolution achieves improvements in both PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity). On the EAD and Kvasir-SEG datasets, the endoscopic image deblurring model based on sampling segmentation convolution provided in this disclosure improves PSNR by up to 3.80 dB and 3.92 dB, respectively, and SSIM by up to 5.0% and 5.2%, respectively, compared to other models in related technologies.

[0104] Table 1

[0105]

[0106] Figure 6 The illustration shows a schematic diagram of reconstructed images obtained by applying an endoscope image deblurring model based on sampling segmentation convolution and a correlation model, respectively, according to embodiments of the present disclosure.

[0107] like Figure 6 As shown, the blurred endoscopic images input to the above model can be derived from the EAD dataset. Figure 6 It can be seen that in the reconstructed image output by the endoscopic image deblurring model based on sampling segmentation convolution, the lesion boundary and lesion texture are clearer than those obtained by the relevant model, and no artifacts are produced.

[0108] According to embodiments of this disclosure, the ablation experiment results for the endoscopic image deblurring model based on sampling segmentation convolution on the Kvasir-SEG dataset are shown in Table 2. In Table 2, for the SSCD or SSCU column, "√" indicates the use of the sampling segmentation convolution downsampling module or the sampling segmentation convolution upsampling module, and "-" indicates the use of the traditional feature fusion module; for the FIF column, "√" indicates the use of the feature interaction fusion module, and "-" indicates that the feature interaction fusion module is not used. As can be seen from the results shown in Table 2, using both the sampling segmentation convolution downsampling module and the sampling segmentation convolution upsampling module simultaneously improves PSNR by 3.08 dB and SSIM by 4.30% compared to the initial model. Inserting the feature interaction fusion module improves PSNR by 1.99 dB and SSIM by 3.00% compared to the initial model. By combining the sampling segmentation convolution downsampling module, the sampling segmentation convolution upsampling module, and the feature interaction fusion module, the PSNR was improved by 3.98dB and the SSIM by 5.3% compared to the initial model. This demonstrates that the combined use of the downsampling layer, the feature interaction fusion layer, and the upsampling layer can achieve significant improvements in both PSNR and SSIM, resulting in target images with higher clarity.

[0109] Table 2

[0110]

[0111] Figure 7 The flowchart illustrates a method for training an endoscopic image deblurring model based on sampling segmentation convolution according to an embodiment of the present disclosure.

[0112] like Figure 7 As shown, the method includes operations S701 to S706.

[0113] In operation S701, multiple sample images with different scales are acquired, wherein the multiple sample images are obtained by sampling an initial sample image based on different magnifications.

[0114] In operation S702, convolutional layers are used to extract features from multiple sample images to obtain multiple first feature maps.

[0115] In operation S703, the downsampling layer is used to perform feature fusion on multiple sets of first feature maps to obtain multiple encoded feature maps. Each set of first feature maps includes at least one first feature map, and the multiple sets of first feature maps correspond one-to-one with multiple sample images.

[0116] In operation S704, the feature interaction fusion layer is used to fuse multiple encoded feature maps and sample images corresponding to the encoded feature maps to obtain multiple second feature maps.

[0117] In operation S705, image reconstruction is performed based on multiple second feature maps using an upsampling layer, resulting in multiple reconstructed images with different scales.

[0118] In operation S706, based on multiple reconstructed images and label images of multiple sample images, the model parameters of the convolutional layer, downsampling layer, feature interaction fusion layer and upsampling layer are adjusted to obtain an endoscope image deblurring model. The label images of multiple sample images are obtained by downsampling the label images of the initial sample images based on different magnifications.

[0119] According to embodiments of this disclosure, the operation performed to process a sample image to obtain a reconstructed image can be the same as the operation performed to process an image to be processed to obtain a target image. The operation performed to process a sample image to obtain a reconstructed image can be referred to the description of operation S101 to S105, which will not be repeated here.

[0120] According to embodiments of this disclosure, based on the RGB color space, the dark channel image, intermediate channel image, and bright channel image of each reconstructed image in a plurality of reconstructed images, as well as the dark channel image, intermediate channel image, and bright channel image of each label image in a plurality of label images, can be constructed respectively, as shown in formulas (4) to (6):

[0121]

[0122]

[0123]

[0124] In equations (4) to (6), This represents the dark channel image at the k-th scale; This represents the intermediate channel image at the k-th scale; This represents the bright channel image at the k-th scale; The R channel represents the reconstructed or labeled image at the k-th scale; The G channel represents the reconstructed or labeled image at the k-th scale; The B channel represents the reconstructed or labeled image at the k-th scale; f min f represents the operation of finding the minimum value; medi This indicates the operation of finding intermediate values; f max This indicates the operation of finding the maximum value.

[0125] Figure 8 A schematic diagram of the dark channel image, intermediate channel image, and bright channel image of a reconstructed image according to an embodiment of the present disclosure is shown.

[0126] likeFigure 8 As shown, the bright channel image can provide brightness information of the reconstructed image, while the intermediate channel image and the dark channel image can provide more detailed information of the reconstructed image.

[0127] According to embodiments of this disclosure, operation S706 may include the following operations:

[0128] For each reconstructed image, a first loss value is calculated based on the reconstructed image and the target label image, where the target label image is related to the target sample image, the target sample image belongs to multiple sample images, and the reconstructed image and the target sample image have the same scale; a second loss value is calculated based on the dark channel image of the reconstructed image and the dark channel image of the target label image; a third loss value is calculated based on the middle channel image of the reconstructed image and the middle channel image of the target label image; a fourth loss value is calculated based on the bright channel image of the reconstructed image and the bright channel image of the target label image; and, based on the first, second, third, and fourth loss values, the model parameters of the convolutional layer, downsampling layer, feature interaction fusion layer, and upsampling layer are adjusted to finally obtain the endoscope image deblurring model.

[0129] According to embodiments of this disclosure, calculating the first loss value based on the reconstructed image and the target label image can be performed by calculating the reconstructed image... With target label image S k The L1 loss between them is calculated as shown in formula (7):

[0130]

[0131] In equation (7), k represents the k-th scale; N k Represents the number of pixels in the image at the k-th scale; ||·||1 represents the 1-norm.

[0132] According to embodiments of this disclosure, the second loss value can be calculated based on the dark channel image of the reconstructed image and the dark channel image of the target label image by first calculating the gradients of the dark channel image of the reconstructed image and the dark channel image of the target label image respectively, and then minimizing the L1 loss of both. The calculation process is shown in formula (8):

[0133]

[0134] In equation (8), f grad This indicates the gradient calculation operation; This represents the dark channel image of the reconstructed image at the k-th scale. This represents the dark channel image of the target label image at the k-th scale.

[0135] According to embodiments of this disclosure, the third loss value can be calculated by first taking the gradients of the intermediate channel images of the reconstructed image and the intermediate channel images of the target label image, and then minimizing the L1 loss of both. The calculation process is shown in formula (9):

[0136]

[0137] In equation (9), This represents the intermediate channel image of the reconstructed image at the k-th scale; This represents the middle channel image of the target label image at the k-th scale.

[0138] According to embodiments of this disclosure, the fourth loss value can be calculated based on the bright channel image of the reconstructed image and the bright channel image of the target label image by first converting the bright channel image of the reconstructed image and the bright channel image of the target label image to the frequency domain, and then minimizing the L1 loss of both. The calculation process is shown in formula (10):

[0139]

[0140] In equation (10), f fft This indicates the Fourier transform operation; Represents the bright channel image of the reconstructed image at the k-th scale; This represents the bright channel image of the target label image at the k-th scale; This indicates the number of pixels after the Fourier transform.

[0141] According to embodiments of this disclosure, the total loss value can be determined based on formulas (7) to (10), as shown in formula (11):

[0142] L total =L cont +λL grad1 +λL grad2 +μL freq (11)

[0143] In equation (11), L total λ represents the total loss value; λ and μ represent the weight hyperparameters, whose values ​​can be adjusted before training, for example, they can be set to 0.25 and 0.5 respectively.

[0144] According to embodiments of this disclosure, the parameter configuration of the endoscopic image deblurring model based on sampling and segmentation convolution during training can be as shown in Table 3. The learning rate is updated using a step-down approach, decaying after a certain number of training epochs. A total of 5000 training epochs are conducted, with an initial learning rate of 0.001. The learning rate decays at epochs 500, 1000, 2000, and 3000 at a decay rate of 0.5, resulting in a final learning rate of 0.0000625.

[0145] Table 3

[0146]

[0147] According to embodiments of this disclosure, the bright channel image mainly contains brightness information and high-frequency components, while the dark channel image and intermediate channel image mainly contain texture details. By minimizing the distance between the bright channel images of the reconstructed image and the target label image in the frequency domain space, and minimizing the distance between the dark channel images and intermediate channel images of the reconstructed image and the target label image in the gradient space, the texture details of the image can be sharpened.

[0148] According to embodiments of this disclosure, the results of ablation experiments for the loss functions shown in formulas (7) to (10) are shown in Table 4. Table 4 shows that, compared to using only L... cont At the same time, use L cont L grad1 L grad2 and L freq The PSNR improved by 0.27 dB, and the SSIM improved by 0.3%. By setting the loss function as described above, the model training can be effectively guided, improving the model's image reconstruction performance.

[0149] Table 4

[0150]

[0151] Figure 9 A block diagram of an endoscopic image deblurring apparatus based on sampling segmentation convolution according to an embodiment of the present disclosure is illustrated schematically.

[0152] like Figure 9 As shown, the endoscope image deblurring device 900 based on sampling segmentation convolution includes a first acquisition module 910, a first extraction module 920, a first fusion module 930, a second fusion module 940, and a first reconstruction module 950.

[0153] The first acquisition module 910 is used to acquire multiple images to be processed with different scales, wherein the multiple images to be processed include those obtained by sampling and processing blurred endoscopic images based on different magnifications.

[0154] The first extraction module 920 is used to extract features from multiple images to be processed using convolutional layers to obtain multiple first feature maps.

[0155] The first fusion module 930 is used to perform feature fusion on multiple first feature map sets using the downsampling layer to obtain multiple coded feature maps. Each first feature map set includes at least one first feature map, and the multiple first feature map sets correspond one-to-one with multiple images to be processed.

[0156] The second fusion module 940 is used to fuse multiple encoded feature maps and the image to be processed corresponding to the encoded feature maps using the feature interaction fusion layer to obtain multiple second feature maps.

[0157] The first reconstruction module 950 is used to reconstruct images based on multiple second feature maps using an upsampling layer to obtain multiple target images with different scales.

[0158] According to embodiments of this disclosure, by using features from blurred endoscopic images at different scales for image reconstruction, the input image can possess richer hierarchical features. Feature fusion at the downsampling layer preserves information from features at different scales. Through the processing of the feature interaction fusion layer, interactive fusion and complementary advantages of detailed features and semantic features can be achieved, improving the network's reconstruction capability. Furthermore, feature fusion at the downsampling layer avoids the blurring and sparsity of semantic information in the image, thereby improving the expressive power of the reconstructed image. These technical means can at least partially overcome the technical problem of insufficient image sharpness in reconstructed images when applied to endoscopic image deblurring in related technologies, thus effectively improving the signal-to-noise ratio of the reconstructed target image and enhancing its sharpness.

[0159] According to embodiments of this disclosure, the first fusion module 930 includes a first fusion unit and a first processing unit.

[0160] The first fusion unit is used to perform feature fusion on each first feature map set using the sampling, segmentation, convolution, and downsampling module corresponding to the first feature map set, to obtain a third feature map.

[0161] The first processing unit is used to process the third feature map using the first residual module corresponding to the first feature map set to obtain the encoded feature map.

[0162] According to embodiments of this disclosure, the first fusion unit includes a first splitting subunit, a first sampling subunit, a first splicing subunit, a first fusion subunit, and a filtering subunit.

[0163] The first splitting subunit is used to divide at least one first feature map in the first feature map set into a first target feature map and a first feature map to be processed, based on the scale of the feature map.

[0164] The first sampling subunit is used to sample and downsample the first feature map to be processed to obtain the first intermediate feature map.

[0165] The first splicing subunit is used to splice the first intermediate feature map and the first target feature map based on the channel dimension to obtain the second intermediate feature map.

[0166] The first fusion subunit is used to perform feature fusion on the second intermediate feature map in the channel dimension using batch convolution to obtain the third intermediate feature map.

[0167] The filtering subunit is used to perform maximum value filtering on the third intermediate feature map to obtain the third feature map.

[0168] According to embodiments of this disclosure, the first reconstruction module 950 includes a second processing unit, a second fusion unit, and a first reconstruction unit.

[0169] The second processing unit is used to process the second feature map corresponding to the second residual module using multiple second residual modules respectively, so as to obtain multiple decoded feature maps.

[0170] The second fusion unit is used to perform feature fusion on the second feature map set corresponding to the sampling segmentation convolution upsampling module using multiple sampling segmentation convolution upsampling modules respectively, to obtain multiple fourth feature maps. Each second feature map set includes at least one decoded feature map, and the multiple second feature map sets correspond one-to-one with multiple images to be processed.

[0171] The first reconstruction unit is used to perform convolutional reconstruction on the image to be processed corresponding to the fourth feature map using multiple fourth feature maps respectively, so as to obtain multiple target images.

[0172] According to embodiments of this disclosure, the second fusion unit includes a second splitting subunit, a second sampling subunit, a second splicing subunit, a second fusion subunit, and a restoration subunit.

[0173] The second splitting subunit is used to, for each second feature map set, divide at least one decoded feature map in the second feature map set into a second target feature map and a second feature map to be processed based on the scale of the feature map.

[0174] The second sampling subunit is used to sample and downsample the second feature map to be processed to obtain the fourth intermediate feature map.

[0175] The second splicing subunit is used to splice the fourth intermediate feature map and the second target feature map based on the channel dimension to obtain the fifth intermediate feature map.

[0176] The second fusion subunit is used to perform feature fusion on the fifth intermediate feature map in the channel dimension using batch convolution to obtain the sixth intermediate feature map.

[0177] The restoration subunit is used to restore the sixth intermediate feature map based on the segmentation index position of the sampling segmentation downsampling to obtain the fourth feature map.

[0178] Figure 10 A block diagram of a training apparatus for an endoscope image deblurring model based on sampling segmentation convolution according to an embodiment of the present disclosure is illustrated.

[0179] like Figure 10 As shown, the training device 1000 for an endoscope image deblurring model based on sampling segmentation convolution includes a second acquisition module 1010, a second extraction module 1020, a third fusion module 1030, a fourth fusion module 1040, a second reconstruction module 1050, and a training module 1060.

[0180] The second acquisition module 1010 is used to acquire multiple sample images with different scales, wherein the multiple sample images are obtained by sampling an initial sample image based on different magnifications.

[0181] The second extraction module 1020 is used to extract features from multiple sample images using convolutional layers to obtain multiple first feature maps.

[0182] The third fusion module 1030 is used to perform feature fusion on multiple first feature map sets using the downsampling layer to obtain multiple encoded feature maps. Each first feature map set includes at least one first feature map, and the multiple first feature map sets correspond one-to-one with multiple sample images.

[0183] The fourth fusion module 1040 is used to fuse multiple encoded feature maps and sample images corresponding to the encoded feature maps using the feature interaction fusion layer to obtain multiple second feature maps.

[0184] The second reconstruction module 1050 is used to reconstruct the image based on multiple second feature maps using the upsampling layer, so as to obtain multiple reconstructed images with different scales.

[0185] The training module 1060 is used to adjust the model parameters of the convolutional layer, downsampling layer, feature interaction fusion layer and upsampling layer based on multiple reconstructed images and label images of multiple sample images to obtain an endoscope image deblurring model. The label images of the multiple sample images are obtained by downsampling the label images of the initial sample images based on different magnifications.

[0186] According to embodiments of this disclosure, the training module 1060 includes a first calculation unit, a second calculation unit, a third calculation unit, a fourth calculation unit, and an adjustment unit.

[0187] The first computing unit is used to calculate a first loss value for each reconstructed image based on the reconstructed image and the target label image, wherein the target label image is related to the target sample image, the target sample image belongs to multiple sample images, and the reconstructed image and the target sample image have the same scale.

[0188] The second calculation unit is used to calculate the second loss value based on the dark channel image of the reconstructed image and the dark channel image of the target label image.

[0189] The third calculation unit is used to calculate the third loss value based on the intermediate channel images of the reconstructed image and the intermediate channel images of the target label image.

[0190] The fourth calculation unit is used to calculate the fourth loss value based on the bright channel image of the reconstructed image and the bright channel image of the target label image.

[0191] The adjustment unit is used to adjust the model parameters of the convolutional layer, downsampling layer, feature interaction fusion layer and upsampling layer based on the first loss value, the second loss value, the third loss value and the fourth loss value, so as to finally obtain the endoscope image deblurring model.

[0192] According to embodiments of this disclosure, the endoscopic image deblurring model training device 1000 based on sampling segmentation convolution further includes a construction module.

[0193] The building module is used to construct the dark channel image, intermediate channel image, and bright channel image of each reconstructed image in multiple reconstructed images, as well as the dark channel image, intermediate channel image, and bright channel image of each label image in multiple label images, based on the RGB color space.

[0194] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0195] For example, any and multiple of the following modules can be implemented in a single module / unit / subunit: the first acquisition module 910, the first extraction module 920, the first fusion module 930, the second fusion module 940, and the first reconstruction module 950; or, any and multiple of the following modules / units can be implemented in a single module / unit / subunit: the second acquisition module 1010, the second extraction module 1020, the third fusion module 1030, the fourth fusion module 1040, the second reconstruction module 1050, and the training module 1060. Alternatively, any one of these modules / units / subunits can be split into multiple modules / units / subunits. Or, at least some of the functionality of one or more of these modules / units / subunits can be combined with at least some of the functionality of other modules / units / subunits and implemented in a single module / unit / subunit. According to embodiments of this disclosure, at least one of the first acquisition module 910, the first extraction module 920, the first fusion module 930, the second fusion module 940, and the first reconstruction module 950, or the second acquisition module 1010, the second extraction module 1020, the third fusion module 1030, the fourth fusion module 1040, the second reconstruction module 1050, and the training module 1060, can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first acquisition module 910, the first extraction module 920, the first fusion module 930, the second fusion module 940, and the first reconstruction module 950, or the second acquisition module 1010, the second extraction module 1020, the third fusion module 1030, the fourth fusion module 1040, the second reconstruction module 1050, and the training module 1060, can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0196] It should be noted that the endoscopic image deblurring device based on sampling segmentation convolution in the embodiments of this disclosure corresponds to the endoscopic image deblurring method based on sampling segmentation convolution in the embodiments of this disclosure. Similarly, the endoscopic image deblurring model training device based on sampling segmentation convolution in the embodiments of this disclosure corresponds to the endoscopic image deblurring model training method based on sampling segmentation convolution in the embodiments of this disclosure. For a detailed description of the endoscopic image deblurring device based on sampling segmentation convolution, please refer to the endoscopic image deblurring method based on sampling segmentation convolution. For a detailed description of the endoscopic image deblurring model training device based on sampling segmentation convolution, please refer to the endoscopic image deblurring model training method based on sampling segmentation convolution. These details will not be repeated here.

[0197] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0198] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for deblurring endoscopic images based on sampling segmentation convolution, comprising: Multiple images to be processed with different scales are acquired, wherein the multiple images to be processed are obtained by sampling a blurred endoscopic image based on different magnifications; Multiple first feature maps are obtained by extracting features from multiple images to be processed using convolutional layers; The downsampling layer is used to fuse features of multiple first feature map sets to obtain multiple encoded feature maps. Each first feature map set includes at least one first feature map, and the multiple first feature map sets correspond one-to-one with the multiple images to be processed. A feature interaction fusion layer is used to fuse multiple encoded feature maps and the corresponding images to be processed to obtain multiple second feature maps; and Image reconstruction is performed using an upsampling layer based on multiple second feature maps, resulting in multiple target images with different scales.

2. The method according to claim 1, wherein, The downsampling layer includes multiple sampling segmentation convolutional downsampling modules and multiple first residual modules; The step of fusing features from multiple first feature map sets using a downsampling layer to obtain multiple encoded feature maps includes: For each of the first feature map sets, the first feature map set is fused using the sampling, segmentation, convolution, and downsampling module corresponding to the first feature map set to obtain a third feature map; and The third feature map is processed using the first residual module corresponding to the first feature map set to obtain the encoded feature map.

3. The method according to claim 2, wherein, The step of using the sampling, segmentation, convolution, and downsampling module corresponding to the first feature map set to perform feature fusion on the first feature map set to obtain a third feature map includes: Based on the scale of the feature maps, at least one first feature map in the first feature map set is divided into a first target feature map and a first feature map to be processed; The first feature map to be processed is sampled, segmented, and downsampled to obtain the first intermediate feature map; The first intermediate feature map and the first target feature map are concatenated based on the channel dimension to obtain the second intermediate feature map; Batch convolution is used to fuse features of the second intermediate feature map along the channel dimension to obtain the third intermediate feature map; and The third intermediate feature map is obtained by performing maximum value filtering on the third intermediate feature map.

4. The method according to claim 1, wherein, The upsampling layer includes multiple second residual modules and multiple sampling segmentation convolutional upsampling modules; The method of using an upsampling layer to reconstruct images based on multiple second feature maps to obtain multiple target images with different scales includes: The second feature map corresponding to the second residual module is processed by multiple second residual modules respectively to obtain multiple decoded feature maps; The second feature map sets corresponding to the sampling, segmentation, convolution, and upsampling modules are respectively fused using multiple sampling, segmentation, convolution, and upsampling modules to obtain multiple fourth feature maps. Each second feature map set includes at least one of the decoded feature maps, and the multiple second feature map sets correspond one-to-one with the multiple images to be processed. Multiple target images are obtained by performing convolutional reconstruction on the images to be processed corresponding to the fourth feature maps using multiple fourth feature maps respectively.

5. The method according to claim 4, wherein, The second feature map set corresponding to each sampling segmentation convolution upsampling module is fused using multiple sampling segmentation convolution upsampling modules to obtain multiple fourth feature maps, including: For each of the second feature map sets, based on the scale of the feature map, at least one decoded feature map in the second feature map set is divided into a second target feature map and a second feature map to be processed; The second feature map to be processed is sampled and downsampled to obtain the fourth intermediate feature map; The fourth intermediate feature map and the second target feature map are concatenated based on the channel dimension to obtain the fifth intermediate feature map; The fifth intermediate feature map is fused along the channel dimension using batch convolution to obtain the sixth intermediate feature map; and The sixth intermediate feature map is restored based on the segmentation index position of the sampling segmentation downsampling to obtain the fourth feature map.

6. A method for training an endoscopic image deblurring model based on sampling segmentation convolution, comprising: Acquire multiple sample images with different scales, wherein the multiple sample images are obtained by sampling an initial sample image based on different magnifications; Multiple sample images are subjected to feature extraction using convolutional layers to obtain multiple first feature maps; The downsampling layer is used to fuse features of multiple first feature map sets to obtain multiple encoded feature maps. Each first feature map set includes at least one first feature map, and the multiple first feature map sets correspond one-to-one with the multiple sample images. Multiple second feature maps are obtained by fusing multiple encoded feature maps and sample images corresponding to the encoded feature maps using a feature interaction fusion layer; Image reconstruction is performed using an upsampling layer based on multiple second feature maps, resulting in multiple reconstructed images with different scales; and Based on the reconstructed images and the label images of the sample images, the model parameters of the convolutional layer, the downsampling layer, the feature interaction fusion layer, and the upsampling layer are adjusted to obtain an endoscopic image deblurring model. The label images of the sample images are obtained by downsampling the label images of the initial sample images based on the different magnifications.

7. The method according to claim 6, wherein, The method involves adjusting the model parameters of the convolutional layer, the downsampling layer, the feature interaction fusion layer, and the upsampling layer based on multiple reconstructed images and multiple sample images to obtain an endoscopic image deblurring model, including: For each reconstructed image, a first loss value is calculated based on the reconstructed image and the target label image, wherein the target label image is related to the target sample image, the target sample image belongs to the plurality of sample images, and the reconstructed image has the same scale as the target sample image; A second loss value is calculated based on the dark channel image of the reconstructed image and the dark channel image of the target label image; A third loss value is calculated based on the intermediate channel image of the reconstructed image and the intermediate channel image of the target label image; Based on the bright channel image of the reconstructed image and the bright channel image of the target label image, a fourth loss value is calculated; and Based on the first loss value, the second loss value, the third loss value, and the fourth loss value, the model parameters of the convolutional layer, the downsampling layer, the feature interaction fusion layer, and the upsampling layer are adjusted to finally obtain the endoscope image deblurring model.

8. The method according to claim 7, further comprising: Based on the RGB color space, the dark channel image, the intermediate channel image, and the bright channel image of each of the multiple reconstructed images, as well as the dark channel image, the intermediate channel image, and the bright channel image of each of the multiple label images, are constructed respectively.

9. An endoscopic image deblurring device based on sampling segmentation convolution, comprising: The first acquisition module is used to acquire multiple images to be processed with different scales, wherein the multiple images to be processed are obtained by sampling a blurred endoscopic image based on different magnifications; The first extraction module is used to extract features from multiple images to be processed using convolutional layers to obtain multiple first feature maps. The first fusion module is used to perform feature fusion on multiple first feature map sets using a downsampling layer to obtain multiple encoded feature maps, wherein each first feature map set includes at least one first feature map, and the multiple first feature map sets correspond one-to-one with the multiple images to be processed; The second fusion module is used to fuse multiple encoded feature maps and the corresponding images to be processed using a feature interaction fusion layer to obtain multiple second feature maps; and The first reconstruction module is used to reconstruct images based on multiple second feature maps using an upsampling layer, resulting in multiple target images with different scales.

10. An endoscopic image deblurring training device based on sampling segmentation convolution, comprising: The second acquisition module is used to acquire multiple sample images with different scales, wherein the multiple sample images are obtained by sampling an initial sample image based on different magnifications; The second extraction module is used to extract features from multiple sample images using convolutional layers to obtain multiple first feature maps. The third fusion module is used to perform feature fusion on multiple first feature map sets using the downsampling layer to obtain multiple encoded feature maps, wherein each first feature map set includes at least one first feature map, and the multiple first feature map sets correspond one-to-one with the multiple sample images; The fourth fusion module is used to fuse multiple encoded feature maps and sample images corresponding to the encoded feature maps using a feature interaction fusion layer to obtain multiple second feature maps. The second reconstruction module is used to reconstruct the image based on multiple second feature maps using the upsampling layer, resulting in multiple reconstructed images with different scales; and The training module is used to adjust the model parameters of the convolutional layer, the downsampling layer, the feature interaction fusion layer, and the upsampling layer based on multiple reconstructed images and multiple label images of the sample images to obtain an endoscope image deblurring model. The label images of the multiple sample images are obtained by downsampling the label images of the initial sample images based on the different magnifications.

Images