Fog removal method and training method of endoscope image based on deep learning model

Abstract

A method and a training method for removing fog from an endoscope image based on a deep learning model, the deep learning model comprising a feature extraction module, a self-attention transformation module and a feature restoration module; the method comprising: inputting a foggy endoscope image into the feature extraction module to obtain a shallow feature map; inputting the shallow feature map comprising at least one shallow feature submap into the self-attention transformation module to obtain a deep feature map; fusing the shallow feature map and the deep feature map to obtain a fused feature map; and inputting the fused feature map into the feature restoration module to obtain a fog-free endoscope image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to an endoscope image defogging method and a training method based on a deep learning model. BACKGROUND

[0002] With the development of surgical techniques, minimally invasive surgery has been more and more widely used. Endoscope is a kind of medical instrument widely used in minimally invasive surgery. During the operation, the surgeon can observe the tissues and organs in the human body through the endoscope to perform surgical operations. Therefore, whether the endoscope image is clear or not plays an important role in the completion of the operation. However, fog generated by temperature difference, laser ablation and burning during the operation will seriously block the doctor's view and increase the risk of the doctor's operation. At the same time, the smoke also affects the subsequent visual task processing, such as instrument segmentation, three-dimensional scene reconstruction, automatic operation, etc. Therefore, removing the smoke in the images taken by the endoscope during the operation can effectively improve the image quality of the operation and provide important reference information for the surgeon to perform the operation, so as to improve the accuracy, efficiency and safety of minimally invasive surgery.

[0003] At present, the methods for removing fog in endoscopic surgery mainly include two kinds. One is based on mechanical method, including endoscope lens heating strategy, anti-fogging material and endoscope lens equipment modification scheme. These mechanical design based methods increase the cost of endoscope research and development, and need to use special endoscope devices, which are not suitable for all image-guided endoscopic surgeries. The other is a visual-based defogging method, which removes the smoke in the endoscope output image through computer algorithm. However, the current defogging algorithm is mainly used for removing smoke in outdoor scenes, and there is no ideal defogging algorithm for endoscope images. SUMMARY

[0004] In view of the above technical problems, the present application provides an endoscope image defogging method and a training method based on a deep learning model.

[0005] In order to achieve the above purpose, as one aspect of the present application, an endoscope image defogging method based on a deep learning model is provided, wherein the deep learning model comprises a feature extraction module, a self-attention transformation module and a feature restoration module.

[0006] The method comprises:

[0007] inputting the foggy endoscope image into the feature extraction module to obtain a shallow feature map;

[0008] inputting the shallow feature map comprising at least one shallow feature sub-map into the self-attention transformation module to obtain a deep feature map;

[0009] The shallow feature map and the deep feature map are fused to obtain a fused feature map; and

[0010] The fused feature map is input into the feature restoration module to obtain a fog-free endoscope image.

[0011] According to an embodiment of the present invention, the self-attention transformation module includes a first convolutional layer and M transformation sub-modules, the m-th transformation sub-module includes a second convolutional layer and N transformation units, the n-th transformation unit includes P transformation sub-units, and the p-th transformation sub-unit includes a self-attention layer, where M, N, and P are all integers greater than or equal to 1, m is an integer greater than or equal to 1 and less than or equal to M, n is an integer greater than or equal to 1 and less than or equal to N, and p is an integer greater than or equal to 1 and less than or equal to P.

[0012] Wherein, when M>1 and N>1, the step of inputting the shallow feature map, which includes at least one shallow feature sub-map, into the self-attention transformation module to obtain the deep feature map includes:

[0013] The response is given by 1 < m ≤ M and 1 < n ≤ N.

[0014] When p=1

[0015] The mth n-1,P Intermediate feature map input at the m-th position n,p Transform the subunit to obtain the m-th unit. n,p Intermediate feature map;

[0016] In the case that 1 < p ≤ P

[0017] The mth n,p-1 Intermediate feature map input at the m-th position n,p Transform the subunit to obtain the m-th unit. n,p Intermediate feature map;

[0018] The mth n-1,P Intermediate feature map and m-th n,P The intermediate feature maps are fused to obtain the m-th feature map. n,P Fusion feature maps;

[0019] The mth n,P The fused feature map is input into the m-th second convolutional layer to obtain the m-th convolutional layer. n,P Intermediate feature map;

[0020] The Mth N，P The intermediate feature map is input into the first convolutional layer to obtain the deep feature map;

[0021] Wherein, the mth n,p-1 Intermediate feature map input at the m-th position n,p Transform the subunit to obtain the m-th unit. n,pIntermediate feature maps, including:

[0022] The mth n,p-1 The intermediate feature map is divided into at least one m-th... n,p-1 Intermediate feature subgraph;

[0023] The at least mth n,p-1 Intermediate feature subgraph input first m n,p Self-attention layer, obtain the first m n,p Self-attention feature map; and

[0024] According to the first m n,p The self-attention feature map is used to obtain the m-th... n,p Intermediate feature map.

[0025] According to an embodiment of the present invention, based on the first m n,p The self-attention feature map is used to obtain the m-th... n,p Intermediate feature maps include:

[0026] According to the first m n,p Self-attention feature map and m-th n,p-1 Intermediate feature map, to obtain the first m n,p Fusion feature maps;

[0027] The first m n,p By performing cyclic displacement on the fused feature map, the second m is obtained. n,p Fusion feature maps;

[0028] The second m n,p The fused feature map is input to the second m n,p Self-attention layer, to obtain the second m n,p Self-attention feature map;

[0029] According to the first m n,p Self-attention feature map and second m n,p The self-attention feature map is used to obtain the m-th... n,p Intermediate feature map.

[0030] According to an embodiment of the present invention, the first m n,p By performing cyclic displacement on the fused feature map, the second m is obtained. n,p The fused feature maps include:

[0031] The first m n,p The fused feature map is cyclically shifted along the first and second directions according to a preset length, wherein the preset length is less than the m-th... n,p-1 The size of the intermediate feature sub-image division is such that the first direction and the second direction are perpendicular.

[0032] According to an embodiment of the present invention, when the at least mth... n,p-1 Intermediate feature subgraph input first m n,p Before the self-attention layer, the m-th n,p-1 Intermediate feature map input at the m-th position n,p Transform the subunit to obtain the m-th unit. n,p The intermediate feature map also includes:

[0033] For the at least mth n,p-1 The intermediate feature subgraph is input into the first m n,p A normalized layer is created to perform normalization processing.

[0034] According to an embodiment of the present invention, the step based on the first m n,p Self-attention feature map and m-th n,p-1 Intermediate feature map, to obtain the first m n,p The fused feature maps include:

[0035] The first m n,p Self-attention feature map and m-th n,p-1 The intermediate feature maps are fused to obtain the third m. n,p Fusion feature maps;

[0036] The third m n,p The fused feature maps are sequentially input into the second m n,p Layer normalization layer and first m n,p The fully connected layer processes the feature map, and the processed feature map is then compared with the third m. n,p The feature maps are fused to obtain the first m. n,p Feature maps are fused.

[0037] According to an embodiment of the present invention, when the first m n,p By performing cyclic displacement on the fused feature map, the second m is obtained. n,p Before fusing feature maps, the first m n,p The self-attention feature map is used to obtain the m-th... n,p The intermediate feature map also includes:

[0038] The first m n,p The third m-th feature map input is fused. n,p A normalized layer is used for normalization processing.

[0039] According to an embodiment of the present invention, the step based on the first m n,p Self-attention feature map and second m n,p The self-attention feature map is used to obtain the m-th... n,p Intermediate feature maps include:

[0040] According to the first m n,pSelf-attention feature map and second m n,p The self-attention feature maps are fused to obtain the fourth m. n,p Fusion feature maps;

[0041] The fourth m n,p The fused feature maps are sequentially input into the fourth m n,p Layer normalization layer and second m n,p The fully connected layer is used for processing to obtain the m-th... n,p Intermediate feature map.

[0042] As a second aspect of the present invention, a dehazing training method for endoscopic images based on a deep learning model is also provided, wherein the deep learning model includes a feature extraction module, a self-attention transformation module, and a feature restoration module.

[0043] The training method includes:

[0044] The original fog-free endoscope sample image is input into the 3D rendering engine, and a foggy endoscope sample image corresponding to the original fog-free endoscope sample image is obtained through a smoke generation algorithm.

[0045] The fogged endoscope sample image is input into the feature extraction module to obtain a shallow feature sample image;

[0046] The self-attention transformation module inputs the shallow feature sample map, which includes at least one shallow feature sub-sample map, into the deep feature sample map to obtain the deep feature sample map.

[0047] The shallow feature sample map and the deep feature sample map are fused to obtain a fused feature sample map;

[0048] The fused feature sample image is input into the feature restoration module to obtain a fog-free endoscope sample image;

[0049] Based on the original fog-free endoscope sample image and the fog-free endoscope sample image, the network parameters of the deep learning model are adjusted to obtain the trained deep learning model.

[0050] As a third aspect of the present invention, an electronic device is also provided, comprising:

[0051] One or more processors;

[0052] Storage device for storing one or more programs.

[0053] When the one or more programs are executed by the one or more processors, the one or more processors perform the above-described method.

[0054] According to an embodiment of the present invention, by inputting the shallow feature map of at least one shallow feature sub-map into the self-attention transformation module, the self-attention transformation module performs parallel processing on each shallow feature sub-map, thereby reducing the amount of computation and accelerating the image processing speed. Attached Figure Description

[0055] Figure 1 A block diagram of a deep learning model provided according to an embodiment of the present invention is shown;

[0056] Figure 2 A block diagram of a transformation subunit provided according to an embodiment of the present invention is shown;

[0057] Figure 3 A schematic diagram illustrating window division of an image is provided according to an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0059] Figure 1 A block diagram of a deep learning model provided according to an embodiment of the present invention is shown.

[0060] like Figure 1 As shown, the deep learning model includes a feature extraction module, a self-attention transformation module, and a feature restoration module.

[0061] The dehazing method for endoscopic images based on deep learning models includes operations S101-S104.

[0062] During operation S101, a foggy endoscope image is input into the feature extraction module to obtain a shallow feature map;

[0063] In operation S102, a shallow feature map, including at least one shallow feature sub-map, is input into the self-attention transformation module to obtain a deep feature map.

[0064] In operation S103, the shallow feature map and the deep feature map are fused to obtain a fused feature map; and

[0065] In operation S104, the fused feature map is input into the feature restoration module to obtain a fog-free endoscope image.

[0066] According to an embodiment of the present invention, by inputting the shallow feature map of at least one shallow feature sub-map into the self-attention transformation module, the self-attention transformation module performs parallel processing on each shallow feature sub-map, thereby reducing the amount of computation and accelerating the image processing speed.

[0067] According to an embodiment of the present invention, in the feature extraction module, a convolution operation is performed on the input foggy endoscope image to extract different shallow features. The extracted shallow features mainly include color features, texture features, shape features, and spatial relationship features. The convolutional layer of the feature extraction module uses a 3*3 convolutional kernel. The convolutional kernel is used to traverse the entire foggy endoscope image to extract features. The three-layer convolution operation expands the receptive field of the network, thereby more effectively extracting the shallow features of the image. Specifically, inputting the foggy endoscope image into the feature extraction module includes: first, inputting it into the first convolutional layer of the feature extraction module with a convolutional kernel size of 3*3; then, inputting it into the second convolutional layer of the feature extraction module with a convolutional kernel size of 3*3; and finally, inputting it into the third convolutional layer of the feature extraction module with a convolutional kernel size of 3*3. The final output is the shallow feature map of the image, and the specific formula is as follows:

[0068] F0 = H s (x)

[0069] Where x is the fogged endoscope image, H s F0 is the feature extraction module, and F0 is the output shallow feature image. Convolutional layers are good at shallow visual processing, so they can obtain better results.

[0070] According to an embodiment of the present invention, the deep image features extracted by the deep self-attention module and the shallow image features extracted by the feature extraction module are fused, and then the fused information is input into the feature restoration module. The feature restoration module mainly consists of three convolutional layers, using 3*3 convolutional kernels. The convolutional kernels traverse the entire image, thereby enabling better recovery and restoration of image features.

[0071] According to an embodiment of the present invention, the self-attention transformation module includes a first convolutional layer and M transformation sub-modules, the m-th transformation sub-module includes a second convolutional layer and N transformation units, the n-th transformation unit includes P transformation sub-units, and the p-th transformation sub-unit includes a self-attention layer, where M, N, and P are all integers greater than or equal to 1, m is an integer greater than or equal to 1 and less than or equal to M, n is an integer greater than or equal to 1 and less than or equal to N, and p is an integer greater than or equal to 1 and less than or equal to P.

[0072] To better integrate the extracted deep image features with the shallow image features, a first convolutional layer is set in the deep self-attention transformation module, and a second convolutional layer is set in each transformation submodule, which can better express the deep features.

[0073] According to an embodiment of the present invention, when M > 1 and N > 1, a shallow feature map including at least one shallow feature sub-map is input into a self-attention transformation module to obtain a deep feature map, including:

[0074] The response is given by 1 < m ≤ M and 1 < n ≤ N.

[0075] When p=1

[0076] The mth n-1,P Intermediate feature map input at the m-th position n,p Transform the subunit to obtain the m-th unit. n,p Intermediate feature map;

[0077] In the case that 1 < p ≤ P

[0078] The mth n,p-1 Intermediate feature map input at the m-th position n,p Transform the subunit to obtain the m-th unit. n,p Intermediate feature map;

[0079] The mth n-1,P Intermediate feature map and m-th n,P The intermediate feature maps are fused to obtain the m-th feature map. n,P Fusion feature maps;

[0080] The mth n,P The fused feature map is input into the m-th second convolutional layer to obtain the m-th convolutional layer. n,P Intermediate feature map;

[0081] The Mth N，P The intermediate feature map is input into the first convolutional layer to obtain the deep feature map.

[0082] Figure 2 A block diagram of a transformation subunit provided according to an embodiment of the present invention is shown.

[0083] like Figure 2 As shown, the m-th n,p-1 Intermediate feature map input at the m-th position n,p Transform the subunit to obtain the m-th unit. n,p The intermediate feature map includes operations S201-S203.

[0084] In operation S201, the m-th... n,p-1 The intermediate feature map is divided into at least one m-th... n,p-1 Intermediate feature subgraph;

[0085] In operation S202, at least the m-th... n,p-1 Intermediate feature subgraph input first m n,p Self-attention layer, obtain the first m n,p Self-attention feature map; and

[0086] In operation S203, according to the first m n,p Self-attention feature map, to obtain the m-th n,p Intermediate feature map.

[0087] According to an embodiment of the present invention, in operation S202, the first mn,p self-attention layer can also be called a window-based multi-head self-attention layer.

[0088] The principle of image processing using window-based multi-head self-attention layers is as follows:

[0089] Figure 3 A schematic diagram illustrating window division of an image is provided according to an embodiment of the present invention.

[0090] like Figure 3 As shown in sections (a) and (b), section (b) is obtained by dividing the image in section (a) into windows, and a self-attention mechanism is applied to each window. The goal of the attention mechanism is to apply human perception and attention behavior to the machine. Therefore, the attention mechanism filters out the most important information from a large amount of data, ignoring most of the less important information. Larger weights focus more on their corresponding values, thus representing the importance of the information. The self-attention mechanism mainly improves model performance and reduces computational cost by assigning different weights to different positions in the image and learning from data with different weights during training. When encoding long input images, parallel computing can improve computational efficiency. Furthermore, window partitioning greatly reduces the computational load.

[0091] According to an embodiment of the present invention, when at least the mth... n,p-1 Intermediate feature subgraph input first m n,p Before the self-attention layer, the m-th... n,p-1 Intermediate feature map input at the m-th position n,p Transform the subunit to obtain the m-th unit. n,p The intermediate feature map also includes:

[0092] For at least the mth n,p-1 The intermediate feature subgraph is input into the first m n,p The normalization layer is used for normalization processing and is denoted by the method LN.

[0093] According to an embodiment of the present invention, based on the first m n,p Self-attention feature map and m-th n,p-1 Intermediate feature map, to obtain the first m n,p The fused feature maps include:

[0094] The first m n,p Self-attention feature map and m-th n,p-1 The intermediate feature maps are fused to obtain the third m. n,p Fusion feature maps;

[0095] The third m n,pThe fused feature maps are sequentially input into the second m n,p Normalized layer and first m n,p The fully connected layer processes the feature map, and the processed feature map is then compared with the third m-th layer. n,p The feature maps are fused to obtain the first m. n,p Feature maps are fused.

[0096] According to an embodiment of the present invention, based on the first m n,p Self-attention feature map, to obtain the m-th n,p The intermediate feature map includes operations S301-S304.

[0097] In operation S301, according to the first m n,p Self-attention feature map and m-th n,p-1 Intermediate feature map, to obtain the first m n,p Fusion feature maps;

[0098] In operation S302, the first m n,p By performing cyclic displacement on the fused feature map, the second m is obtained. n,p Fusion feature maps;

[0099] In operation S303, the second m n,p The fused feature map is input to the second m n,p Self-attention layer, to obtain the second m n,p Self-attention feature map;

[0100] In operation S304, according to the first m n,p Self-attention feature map and second m n,p Self-attention feature map, to obtain the m-th n,p Intermediate feature map.

[0101] According to an embodiment of the present invention, in operation S303, the second m n,p The self-attention layer is also called the multi-head self-attention layer based on moving windows.

[0102] According to an embodiment of the present invention, in operation S202, although the computational load is reduced after window division, information cannot be shared between each window. Therefore, content interaction between windows is achieved by moving the windows. Specifically, as follows... Figure 3As shown in parts (c), (d), (e), and (f), the moving window specifically involves cyclically shifting the pixels of the input image, using half the size of the window as the displacement length, thus changing the pixel positions of the input image (as shown in parts (c), (d), and (e)). Then, through window partitioning (as shown in part (f), the shifted input image is divided into different windows. This ensures that the shifted window and the original window respectively contain feature information from different regions of the input image, thereby achieving information interaction between different windows. Simultaneously, the moving window leverages the feature extraction advantages of a multi-head self-attention mechanism to model the global dependencies between input and output. The moving window multi-head self-attention module can effectively extract feature information from the preprocessed image, thereby obtaining long-term dependencies in the image and achieving global modeling capabilities.

[0103] According to an embodiment of the present invention, the first m n,p By performing cyclic displacement on the fused feature map, the second m is obtained. n,p The fused feature maps include:

[0104] The first m n,p The fused feature map is cyclically shifted along the first and second directions according to a preset length, wherein the preset length is less than the m-th... n,p-1 The size of the intermediate feature sub-image division is such that the first and second directions are perpendicular.

[0105] According to an embodiment of the present invention, based on the first m n,p Self-attention feature map and m-th n,p-1 Intermediate feature map, to obtain the first m n,p The fused feature maps include:

[0106] The first m n,p Self-attention feature map and m-th n,p-1 The intermediate feature maps are fused to obtain the third m. n,p Fusion feature maps;

[0107] The third m n,p The fused feature maps are sequentially input into the second m n,p Layer normalization layer and first m n,p The fully connected layer processes the feature map, and the processed feature map is then compared with the third m-th layer. n,p The feature maps are fused to obtain the first m. n,p Feature maps are fused.

[0108] According to an embodiment of the present invention, when the first m n,p By performing cyclic displacement on the fused feature map, the second m is obtained. n,p Before fusing feature maps, based on the first m n,p Self-attention feature map, to obtain the m-th n,pThe intermediate feature map also includes:

[0109] The first m n,p The third m-th feature map input is fused. n,p A normalization layer is used for normalization processing.

[0110] According to an embodiment of the present invention, based on the first m n,p Self-attention feature map and second m n,p Self-attention feature map, to obtain the m-th n,p Intermediate feature maps include:

[0111] According to the first m n,p Self-attention feature map and second m n,p The self-attention feature maps are fused to obtain the fourth m. n,p Fusion feature maps;

[0112] The fourth m n,p The fused feature maps are sequentially input into the fourth m n,p Normalized layer and second m n,p The fully connected layer is used for processing to obtain the m-th... n,p Intermediate feature map.

[0113] According to an embodiment of the present invention, based on the first m n,p Self-attention feature map and second m n,p Self-attention feature map, to obtain the m-th n,p Intermediate feature maps include:

[0114] According to the first m n,p Self-attention feature map and second m n,p The self-attention feature maps are fused to obtain the fourth m. n,p Fusion feature maps;

[0115] The fourth m n,p The fused feature maps are sequentially input into the fourth m n,p Normalized layer and second m n,p The fully connected layer is used for processing to obtain the m-th... n,p Intermediate feature map.

[0116] According to an embodiment of the present invention, the normalization layer (including the first m) n,p Normalization layer, second m n,p Normalization layer, third m n,p Normalized layer and fourth m n,pA normalization layer (Nullification Layer) normalizes the input data of a single training data point across all neurons in that layer. This process is performed simultaneously during training and testing. By summing the input data for the same layer and calculating the mean and variance, the input data for each layer is normalized. This results in a relatively stable distribution of input data across each layer of the network, accelerating model learning.

[0117] According to an embodiment of the present invention, the first m n,p Normalization layer, second m n,p Normalization layer, third m n,p Normalized layer and fourth m n,p The normalization layers have the same structure.

[0118] According to an embodiment of the present invention, the fully connected layer (including the first m) n,p Fully connected layer and second m n,p The fully connected layer contains a fully connected feedforward network, which mainly consists of two layers of fully connected linear units, with Gaussian error linear units used as activation functions in between.

[0119] According to an embodiment of the present invention, the first m n,p Fully connected layer and second m n,p The structure of fully connected layers is the same.

[0120] According to an embodiment of the present invention, in order to better fuse the extracted deep image features with the shallow image features, a convolutional neural network layer is added to the deep self-attention transformation module and its sub-modules, which can better express the deep features.

[0121] The following specific embodiments illustrate the defogging method of the present invention in detail. The defogging method includes operations A-C:

[0122] Operation A involves inputting a fogged endoscope image into the feature extraction module for convolution operations. First, the image is input into the first convolutional layer of the feature extraction module with a 3x3 kernel. Then, it is input into the second convolutional layer with a 3x3 kernel. Finally, it is input into the third convolutional layer with a 3x3 kernel. The final output is the shallow feature information of the image, as shown in the specific formula:

[0123] F0 = H s (x)(1)

[0124] Where x is the fogged endoscope image, H s F0 is the feature extraction module, and F0 is the output shallow feature map. Convolutional layers are good at shallow visual processing, so they can obtain better results.

[0125] Operation B involves inputting the shallow feature map into the depth self-attention transformation module. Through a series of transformation modules, calculations are performed, ultimately extracting the deep features of the image via a self-attention mechanism. This step specifically includes operations B1-B5.

[0126] Operation B1 first requires inputting the extracted shallow feature map into the depth self-attention transformation module. The transformation subunit of the self-attention transformation module extracts the features. The transformation subunit mainly consists of two parts: a first transformer and a second transformer. The first transformer includes: a first m... n,p Normalized layer, first m n,p Self-attention layer, second m n,p Normalized layer and first m n,p Fully connected layer. The transformer includes the third m n,p Normalization layer, second m n,p Self-attention layer, fourth m n,p Normalized layer and second m n,p Fully connected layer. Wherein, the first m of the first converter... n,p Self-attention layer calculation includes the following operations B11-B12:

[0127] Operation B11 first divides the feature map into multiple heads, then converts the input features of each head into an input vector, and multiplies them by three different weight matrices to obtain:

[0128] The specific calculation process of self-attention can be summarized into three stages: The first stage is to calculate the similarity or relevance between the query matrix Q and the key matrix K; the second stage is to normalize the original scores from the first stage to obtain weight coefficients; and the third stage is to perform a weighted summation of the value matrix V based on the weight coefficients. Therefore, the calculation formula for self-attention is:

[0129]

[0130] Where, d k Let K represent the dimension of matrix K, softmax represent the softmax function, and Q, K, and V represent the query matrix, key matrix, and value matrix, respectively. Multi-head self-attention computation refers to the computation of the self-attention value h times, where h represents the number of heads. During each computation, the heads do not share parameters; that is, the parameters for the linear transformation of Q, K, and V in the i-th computation are different. The results of the h computations of self-attention value are then concatenated, and a final linear transformation is performed to obtain the final value of the multi-head self-attention. This allows the model to learn relevant information in different representation subspaces.

[0131] Therefore, the calculation result of the i-th head is head. i Then the input feature vector F is multiplied by different weights. Obtain the Q, K, and V vectors, and then calculate the self-attention head for the i-th head. i :

[0132]

[0133] Where Concat represents a connection operation, W O This represents the weights of the fully connected layer.

[0134] Operation B12 is performed, and the results from the h multi-heads are then concatenated to obtain the final result of the multi-head self-attention.

[0135] MSA(Q,K,V)=Concat(head1,...,head h W O (4)

[0136] Where Concat represents a connection operation, W O This represents the weights of the fully connected layer. Therefore, the final result of self-attention, MSA, represents the weighted sum of the final value vectors, where the weights are the distribution of the correlation or similarity between each sample and each piece of information.

[0137] Operation B2 inputs the shallow feature map into the second transformer, which mainly consists of the following four steps:

[0138] Operation B21 first inputs the shallow feature map into the third m n,p Normalization layer, denoted by method LN;

[0139] Operation B22, then the third m n,p The intermediate feature map output from the normalization layer is input into a window-based multi-head self-attention layer, denoted by the method W-MSA;

[0140] Operation B23, then input the intermediate feature map output from the window-based multi-head self-attention layer into the third m n,p Normalization layer, denoted by method LN;

[0141] Operation B24 is performed, and then the features are input into the fully connected layer, represented by the method MLP.

[0142] This can be expressed by the following formula:

[0143] F′ W-MSA =W-SA(LN(F in ))+F in (5)

[0144] F out =MLP(LN(F′) W-MSA ))+F′ W-MSA (6)

[0145] Where F in F represents the characteristics input to the module. out F′ represents the feature output from the module. W-MSA To compute the intermediate value of window-based multi-head self-attention.

[0146] Operation B3, and finally inputting the features into the multi-head self-attention submodule based on the moving window, mainly consists of the following four steps:

[0147] Operation B31 first normalizes the feature input layer, denoted by method LN;

[0148] Operation B32 is performed, and then the features are input into a multi-head self-attention layer based on a moving window, represented by the method SW-MSA;

[0149] Operation B33, then normalize the feature input layer, denoted by method LN;

[0150] Operate B34, then input the features into the second m. n,p A fully connected layer, represented by the method MLP, can be expressed by the formula:

[0151] F S ′ W-MSA =SW-MSA(LN(F in ))+F in (7)

[0152] F out =MLP(LN(F S ′ W-MSA ))+F S ′ W-MSA (8)

[0153] Where F in F represents the intermediate feature map input to the second transformer. out F represents the intermediate feature map output from the second converter. S ′ W-MSA To compute intermediate feature maps from the output of a multi-head self-attention layer based on a moving window.

[0154] For operation B4, to simplify the above steps, the formula is reduced to method H. MSA H MSA The deep self-attention submodule can be represented by the formula:

[0155] F i =H MSA (F i-1 (9)

[0156] Where i represents the i-th layer, F i-1F represents the input of the previous layer. i This represents the output of the current i-th layer. Operation B5 will ultimately output the result F. i The input is fed into the last convolutional layer H of the deep self-attention module. conv In the middle, it can be expressed by the formula as follows:

[0157] F d =H conv (F i (10)

[0158] F d These are the deep image features that are ultimately extracted.

[0159] Operation C inputs both deep and shallow features extracted from the image into the feature reconstruction module for convolution, generating a haze-free image. First, the kernel size is 3x3 in the first convolutional layer, then in the second, and finally in the third convolutional layer. The final output is the haze-free result. The specific formula is:

[0160] I out =H r (F0+F d (11)

[0161] Where H r This is the feature restoration module; F0 represents the extracted shallow features. d It is the extracted deep features, I out The final output is the generated endoscope image without fog.

[0162] According to an embodiment of the present invention, in order to balance accuracy and computational efficiency, M can be 8.

[0163] As a second aspect of the present invention, a dehazing training method for endoscopic images based on a deep learning model is also provided, wherein the deep learning model includes a feature extraction module, a self-attention transformation module, and a feature restoration module.

[0164] Training methods include:

[0165] The original fog-free endoscope sample image is input into the 3D rendering engine, and a fogged endoscope sample image corresponding to the original fog-free endoscope sample image is obtained through a smoke generation algorithm.

[0166] Input the foggy endoscope sample image into the feature extraction module to obtain a shallow feature sample map;

[0167] The shallow feature sample map, which includes at least one shallow feature sub-sample map, is input into the self-attention transformation module to obtain the deep feature sample map.

[0168] The shallow feature sample map and the deep feature sample map are fused to obtain the fused feature sample map;

[0169] Input the fused feature sample image into the feature restoration module to obtain a fog-free endoscope sample image;

[0170] Based on the original fog-free endoscope sample images and the fog-free endoscope sample images, the network parameters of the deep learning model are adjusted to obtain the trained deep learning model.

[0171] Specifically, the training method includes: acquiring raw images using an endoscopic camera; preprocessing the raw images to remove unclear, blurry, or other unacceptable data; filtering the collected images to identify foggy images and constructing a set of realistic foggy test images. Since supervised training of the network is required, paired data of foggy and fog-free images within the same scene needs to be constructed. Therefore, fog-free images are input into the 3D rendering engine, and its built-in smoke generation algorithm generates corresponding foggy images. Image pairs are constructed using the fog-free images and the generated foggy images to generate the training dataset.

[0172] According to an embodiment of the present invention, the network parameters of a deep learning model are adjusted based on the original haze-free endoscope sample image and the haze-free endoscope sample image to obtain a trained deep learning model, including:

[0173] The loss function is calculated based on the original fog-free endoscope sample image and the fog-free endoscope sample image. The network weights of the deep learning model are updated by backpropagation, thereby adjusting the network parameters of the deep learning model. After multiple iterations, the trained deep learning model is finally obtained.

[0174] As a third aspect of the present invention, an electronic device is also provided, comprising:

[0175] One or more processors;

[0176] Storage device for storing one or more programs.

[0177] When the one or more programs are executed by the one or more processors, the one or more processors perform the above-described method.

[0178] Compared with the prior art, the dehazing method provided by the embodiments of the present invention has a clearer field of view recovery capability, can generate dehazed images end-to-end from foggy images, solves the problem of color distortion in dehazing, and thus obtains higher quality smoke-free images.

[0179] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for defogging an endoscope image based on a deep learning model, the deep learning model comprising a feature extraction module, a self-attention transformation module and a feature restoration module; the method comprising: inputting a foggy endoscope image into the feature extraction module to obtain a shallow feature map; inputting the shallow feature map comprising at least one shallow feature sub-map into the self-attention transformation module to obtain a deep feature map; fusing the shallow feature map and the deep feature map to obtain a fused feature map; and inputting the fused feature map into the feature restoration module to obtain a fog-free endoscope image; wherein the self-attention transformation module comprises a first convolutional layer and M transformation sub-modules, the mth transformation sub-module comprises a second convolutional layer and N transformation units, the nth transformation unit comprises P transformation sub-units, the pth transformation sub-unit comprises a self-attention layer, M, N and P are all integers greater than or equal to 1, m is an integer greater than or equal to 1 and less than or equal to M, n is an integer greater than or equal to 1 and less than or equal to N, and p is an integer greater than or equal to 1 and less than or equal to P; wherein, in the case of M>1 and N>1, inputting the shallow feature map comprising at least one shallow feature sub-map into the self-attention transformation module to obtain a deep feature map comprises: in response to 1 in the case of p=1, The m n-1,P Intermediate feature map input m n,p Transform subunit, get m n,p Intermediate feature map; in the case of 1 The m n,p-1 Intermediate feature map input the m n,p Transform subunit, get the m n,p Intermediate feature map; The m n-1,P Intermediate feature maps and the m n,P Intermediate feature maps are fused to obtain the m n,P Fused feature maps; The mth n,P The mth n,P The mth Mth N，P input the intermediate feature map into the first convolutional layer to obtain the deep feature map; The method comprises the following steps: n,p-1 The method comprises the following steps: n,p The method comprises the following steps: n,p The method comprises the following steps: dividing the mth intermediate feature map into at least one mth intermediate feature sub-map; n,p-1 dividing the mth intermediate feature map into at least one mth intermediate feature sub-map; n,p-1 dividing the mth intermediate feature map into at least one mth intermediate feature sub-map; The at least one m n,p-1 intermediate feature sub-graph input a first m n,p self-attention layer to obtain a first m n,p self-attention feature map; and According to the first m n,p From the self-attention feature map, the mth n,p Intermediate feature map.

2. The method of claim 1, wherein, According to the first m n,p From the self-attention feature map, the first m n,p The intermediate feature map comprises: According to the first m n,p Self-attention feature map and the m n,p-1 Intermediate feature map, get the first m n,p Fusion feature map; The first m n,p The fusion feature map is cyclically shifted to obtain a second m n,p fusion feature map; the second m n,p fusion feature map input second m n,p self-attention layer, obtaining second m n,p self-attention feature map; According to the first m n,p The self-attention feature map and the second m n,p The self-attention feature map, obtaining the first m n,p The intermediate feature map.

3. The method of claim 2, wherein, The first m n,p The fused feature map is cyclically shifted to obtain a second m n,p The fused feature map includes: The first m n,p The fused feature map is cyclically shifted along a first direction and a second direction in sequence according to a preset length, wherein the preset length is less than a division size of the intermediate feature subgraph, and the first direction and the second direction are perpendicular. n,p-1 The fused feature map is cyclically shifted along a first direction and a second direction in sequence according to a preset length, wherein the preset length is less than a division size of the intermediate feature subgraph, and the first direction and the second direction are perpendicular.

4. The method of claim 1, wherein, In the at least one first m n,p-1 The intermediate feature subgraph input the first m n,p Before the self-attention layer, the first m n,p-1 The intermediate feature map input the first m n,p The transformation subunit obtains the first m n,p The intermediate feature map further comprises: The at least one mth feature sub-graph input is input to a first mth normalization layer for normalization processing. n,p-1 The intermediate feature sub-graph input is input to a first mth normalization layer for normalization processing. n,p The intermediate feature sub-graph input is input to a first mth normalization layer for normalization processing.

5. The method of claim 2, wherein, The first m n,p The self-attention feature map and the m n,p-1 The intermediate feature map, obtaining the first m n,p The fusion feature map comprises: The first m n,p Self-attention feature map and the m n,p-1 Intermediate feature map are fused to obtain a third m n,p Fusion feature map; The third m n,p Fusion feature map is input into the second m n,p Layer normalization layer and the first m n,p Fully connected layer processes, and the processed feature map is fused with the third m n,p Fusion feature map. The first m n,p Fusion feature map.

6. The method of claim 5, wherein, In the first m n,p The second m n,p The first m n,p The self-attention feature map, the first m n,p The intermediate feature map further comprises: The first m n,p The fusion feature map is input into a third m n,p A normalization layer is used for normalization processing.

7. The method of claim 2, wherein, The first m n,p The self-attention feature map and the second m n,p The self-attention feature map, obtaining the first m n,p The intermediate feature map comprises: The first m n,p The self-attention feature map and the second m n,p The fourth m n,p The fusion feature map; The fourth m n,p The fusion feature map is sequentially input into the fourth m n,p The layer normalization layer and the second m n,p The full connection layer processes to obtain the first m n,p The intermediate feature map.

8. A training method for defogging an endoscope image based on a deep learning model, the deep learning model comprising a feature extraction module, a self-attention transformation module and a feature restoration module; the training method comprising: inputting an original fog-free endoscope sample image into a three-dimensional rendering engine to obtain a foggy endoscope sample image corresponding to the original fog-free endoscope sample image through a smoke generation algorithm; inputting the foggy endoscope sample image into the feature extraction module to obtain a shallow feature sample map; inputting the shallow feature sample map comprising at least one shallow feature sub-sample map into the self-attention transformation module to obtain a deep feature sample map; fusing the shallow feature sample map and the deep feature sample map to obtain a fused feature sample map; inputting the fused feature sample map into the feature restoration module to obtain a fog-free endoscope sample image; adjusting network parameters of the deep learning model according to the original fog-free endoscope sample image and the fog-free endoscope sample image to obtain a trained deep learning model; the trained deep learning model is used to implement the method of any one of claims 1-7.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to perform the method according to any one of claims 1-7.

Images