Endoscope image enhancement method based on pixel space domain offset correction
Through pixel spatial domain offset correction technology, the problems of low endoscopic image imaging quality and unstable light source are solved, and the image details and color are improved, exposure inhomogeneity is eliminated and the risk of misdiagnosis is reduced.
Patent Information
- Application Number
- CN202410952062.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-01
AI Technical Summary
The existing endoscopic image imaging quality is poor, the unstable light source leads to uneven exposure, making it difficult to effectively segment and learn feature space, which increases the possibility of misdiagnosis, and the existing image enhancement methods have failed to effectively solve the problem of insufficient light and excessive light. Complex effects.
Using an endoscopic image enhancement method based on pixel spatial domain offset correction, the PCA dimensionality reduction, VIT Transformer feature extraction, pseudo-normal pixel spatial feature generation, and pixel spatial domain offset estimator to calculate and correct underexposed and excessive areas. The offset is used for image enhancement using DCN and adaptive calculations.
Effectively eliminate blur and shadow of reflection in image boundaries, improve image details and colors, improve image quality, enhance exposure uniformity, and reduce the risk of misdiagnosis.
Smart Images

Figure CN118887147A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of endoscopic images, and in particular to an endoscopic image enhancement method based on pixel space domain offset correction. Background Art
[0002] At present, with the continuous improvement of science and technology, endoscopy has been widely used in the medical field. After imaging, it can be effectively segmented through the medical segmentation benchmark large model SAM, helping doctors to quickly and accurately observe the lesions, polyps, inflammation morphology and location size of tissues and organs. However, due to the poor quality of its imaging, it is often difficult to provide a large model for qualitative and quantitative learning. The quality and distribution of the light source of the endoscope are unstable and not ideal. Different areas often present different exposures, resulting in blurred adhesions and unclear textures, which makes it difficult for the model to learn effective feature space expressions, increasing the difficulty of SAM segmentation and increasing the possibility of misdiagnosis by doctors. Therefore, how to effectively deal with this problem, enhance the quality of endoscopic images, and strengthen the detailed features is of great research significance and value.
[0003] The existing image enhancement technologies are almost all used to deal with the effects of underexposure caused by insufficient light. However, they often fail to take into account the particularity of endoscopic imaging. When the quality of the light source is not high, it is very likely that the light source in a certain area will be concentrated during reflection, resulting in overexposure, and the light source in a certain area will be divergent, resulting in underexposure and weak imaging. Therefore, ordinary solutions cannot be used to consider a single problem. If we only address the phenomenon of underexposure caused by insufficient light, we can only balance the brightness information of the entire image and synthesize an image with uniform and clear illumination. Therefore, the key to effectively addressing the complex effects caused by insufficient and unstable light is to solve both the overexposure and underexposure problems in order to fully restore the image.
[0004] Therefore, it is necessary to design an endoscopic image enhancement method based on pixel spatial domain offset correction to enhance the quality of endoscopic imaging and eliminate problems such as reflection blur and shadow at the image boundary. Summary of the invention
[0005] 1. Technical issues to be resolved
[0006] In view of the shortcomings of the prior art, the present invention provides an endoscopic image enhancement method based on pixel spatial domain offset correction, which has the advantages of enhancing the quality of endoscopic imaging and eliminating problems such as reflective blur and shadows at the image boundaries, thereby solving the problems in the above-mentioned background technology.
[0007] (II) Technical solution
[0008] In order to achieve the above-mentioned purpose of enhancing the quality of endoscopic imaging and eliminating the problems of reflection blur and shadow at the image boundary, the present invention provides the following technical solutions: an endoscopic image enhancement method based on pixel space domain offset correction, including the need to select two types of a certain number of pictures, one type is a picture that contains both overexposed and underexposed pictures, and the other type is a picture that only contains overexposed or underexposed pictures. Randomly sample their image pixels, where the original image size is [H, W, 3], and the result of random sampling of its pixels is [N, 3], where N is equal to H×W, which is the number of vectors, and 3 is the vector dimension. The analysis process includes the following steps:
[0009] S1: Sample the target image pixels to construct a matrix N*C (N represents the amount of data, C represents the current dimension) data set X;
[0010] S2: Calculate the average value (decentralization), that is, subtract the average value of each feature from each feature. Get the standardized data set Z;
[0011] S3: Calculate the covariance matrix
[0012] S4: Calculate the K (here K is 2, the three-dimensional feature image needs to be reduced to a two-dimensional plane for visualization) eigenvalue λ 1 ,λ 2 and their corresponding eigenvectors c 1 , c 2 ;
[0013] S5: p=(C 1 C 2 ) construct an eigenvector matrix using the eigenvector as a column vector;
[0014] S6: The representation after dimensionality reduction to K-dimensional space: Y = ZP. The obtained Y is the new data set in the two-dimensional space. The new coordinates of each original data set point in the new two-dimensional principal component space are the corresponding rows of Y. The horizontal and vertical coordinates of each data point are (y i1 ,y i2 ).
[0015] The invention discloses an endoscopic image enhancement method based on pixel space domain offset correction, comprising a VIT Transformer feature extract module, a Feature Transform module, a pseudo-normal pixel space feature generator, a pixel space domain offset estimator and a pixel space domain corrector.
[0016] According to the above technical solution, the VIT Transformer feature extract feature extraction module further includes for a given input image X∈R 3*H*W, and then sent to VIT encoder for feature extraction before it needs to be inverted, X inversion =255-X (the distribution interval of each feature dimension is 0 to 255), and the final output is the enhanced image Y∈R 3 *H*W , Y is the final result after the correction of brightness, restoration of image details and color enhancement. Through the powerful feature extraction capability of VIT, we finally get two areas of underexposure and overexposure. The original image contains two inputs, X and X inversion , perform embedding operations on them, X∈R 3*H*W The token is input to the vit encoder through downsampling with a 16*16 convolution with a step size of 16. The formula is as follows:
[0017]
[0018] F b =F(token x ) (2)
[0019] Replace the input with X inversion , we can get F by the same logic d , in order to obtain the intermediate amount generated by the final features Feature B and Feature D, the formula is as follows:
[0020]
[0021] Among them, Feature B∈R 3*H*W is the feature map of the brightness overexposed area, Feature D∈R 3*H*W It is the feature map of the darkness underexposed area. After generating the two feature maps, the original input image X and Feature B and Feature D are sent as input to the pseudo-normal pixel feature space domain distributor to generate a pseudo-normal distribution feature map. The formula is as follows:
[0022]
[0023] in, The pseudo-normal pixel feature spatial domain distributor can be a common image enhancement method, such as histogram equalization, contrast-limited adaptive histogram equalization, LCDP, Feature N The pixel space domain offset O between it and the overexposed area feature map Feature B and the underexposed area feature map Feature D can be calculated by the pixel space domain offset estimator. B and O D .
[0024] According to the above technical solution, the pixel space domain offset estimator further includes firstly, it is necessary to perform the feature N Concat with Feature B / D to get the intermediate feature U N*2C*H*W , the formula is as follows:
[0025] U N*2C*H*W =concat(Feature N , FeatureB / D)
[0026] The DCN capture offset is performed in the pixel color space and spatial position. For Pixel Convolution (3*3 convolution kernel), the color correction offset captured in the pixel color space is recorded as Δc n , positionConvolution is the position captured by the spatial position, and the offset is recorded as Δp n , Δc n and Δp n With the original input Feature N After fusion in the channel dimension, the feature map of the offset is obtained by deformable convolution. The ScSE module is used to modulate the weight value between the channel dimension and the spatial dimension, and the original feature value is expanded and enhanced to form a new fusion weight Δw n In order to allocate different weights to different kernels in channels and spaces, so that they can focus on important pixels, the deformable convolution calculation formula in the pixel space domain direction is as follows:
[0027]
[0028] Among them, w is the weight of the convolution kernel, x is the feature of the input convolution operation, and p x +p y +Δp n is a two-dimensional variable of spatial position, Δw n is the fusion weight, which is the expression space weight used for output calculation. R = {(0,0), (0,1)…(1,1)} represents a 3×3 kernel network. n is the enumeration number of the element at the nth position in the R set. Here, it is a 3×3 network. The length of the R set is 9, and the range of n is (0-8).
[0029] According to the above technical solution, the pixel space domain corrector further includes: the input of this module is the original image, the darkening feature map offset, and the brightening feature map offset, and each input corresponds to three branches to process the images X and O respectively. B , O DThe three branches of information interaction between themselves and each other include the following parts: q , W k , W v (1×1 Note: All 1*1 convolutions described in this invention do not change the size of the input, but are just a way of expressing weight updates, which can change the number of channels) They are shared weight matrices (the three inputs are multiplied by these three weight matrices. Since the inputs are different, although the weights are the same, the inputs will be different). Their parameters are trainable parameter matrices obtained by back-propagation training of the loss function. After W q , W k , W v Matrix multiplication generates q i , k i 、v i ,i∈(D,X,B), here we can understand q i As the query matrix, k i is the bond matrix, v i is a value matrix, and adaptive calculations are performed on them. The formula is as follows:
[0030] A i =dot-product(q i ,k i ), i∈(D, X, B) (7)
[0031] Among them, dot-product is the dot multiplication operation of the matrix, q i , k i The corresponding position elements are multiplied and summed, and then the Norm is used to ensure the stability of the eigenvalues in the back propagation to obtain the adaptive matrix A i , A i It represents the associated similarity weight of two vectors, v i It represents the weight of the current attention map’s own information. Matrix multiplication is performed between them, and a linear operation of weighted summation of their similarities and weights is performed. The formula is as follows:
[0032]
[0033] Formula 8 corresponds to input branch O B , O D The first term is the adaptive calculation described above, the purpose is to obtain the important area of the offset. The second term is the cross-adaptive calculation, A i It not only interacts with its own information weight, but also interacts with the information weight of another input X. Finally, it combines the adaptive calculation of its own input X to obtain the final result Y. The formula is as follows:
[0034]
[0035] According to the above technical solution, S2 further includes two key part loss values, one is a pseudo-normal distribution feature map Feature used to supervise the intermediate process N And the final output enhanced image Y: Loss N ,Loss Y , the pseudo-normal distribution feature map Feature N Reconstruction loss with the true label value Loss N To measure the difference between the pseudo-normal result and the true result: the difference between the two should be as small as possible. N Infinitely close to the true value label, so the consistency loss is constructed as the formation of the intermediate supervision pseudo-normal distribution feature map, the formula is as follows:
[0036]
[0037] It is used to supervise the convolution matrix, feature matrix, eigenvalue, etc. of the whole process to generate the final enhanced image, including the following aspects: L m Reconstruction loss is used to measure the difference between the enhanced image and the original image at the pixel level. It can be achieved by calculating the MAE (absolute error) between the two. The formula is as follows:
[0038]
[0039] in, The square of the Euclidean distance (also called the square of the L2 norm) represented by
[0040] L cos Cosine similarity is used to construct the similarity between the enhanced image and the original image features and calculate the cosine similarity of the feature vectors. The formula is as follows:
[0041] L cos =-Cosine Similarity((φ(I y ),φ(I x ))
[0042] in, is the extracted feature;
[0043] L vit Perceptual loss measures the similarity of images through high-level features. It uses the pre-trained VIT to extract features and calculates the difference between the enhanced image and the original image features. It pays more attention to the perceptual quality of the image. The formula is as follows:
[0044]
[0045] Where n is the number of batches. is the feature value extracted by vit as the pre-trained network;
[0046] L s Structural similarity loss (SSIM) is used to measure the structural similarity between the enhanced image and the original image. It focuses on three aspects: brightness, structure, and contrast. The formula is as follows:
[0047] L s =1-SSIM(I y ,I x )
[0048] Among them, λ m , c , v , s , N , Y are the weight hyperparameters of the six loss functions, x represents the original input image, I y represents the enhanced image, Represents the true value label, and the final total Loss loss function is:
[0049] Loss Y =λ m L m +λ c L cos +λ v L vit +λ s L s (12)
[0050] Loss = λ N Loss N +λ Y Loss Y (13)
[0051] (III) Beneficial effects
[0052] Compared with the prior art, the present invention provides an endoscopic image enhancement method based on pixel space domain offset correction, which has the following beneficial effects:
[0053] The endoscopic image enhancement method based on pixel space domain offset correction performs dimensionality reduction operation on the image through PCA principal component analysis, and reduces it to a two-dimensional orthogonal plane for visualization. The x and y axes are mutually orthogonal features. Since the endoscopic image presents a special exposure form, the common brightness balance enhancement method cannot effectively enhance the image. After analyzing the pixel space domain, it is found that the overexposed and underexposed areas present an opposite distribution. They are offset relative to the pixels in the normal area, so the image enhancement can be completed by calculating these offsets and then correcting them. The darkening feature map (underexposed area) and the brightening feature map (exposure transition area) are constructed through Vit encoder and feature transform, and then the pseudo-normal distribution feature map is constructed using the existing equalization method. DCN (deformable convolution) is used to construct the offset of pixel color space and position space to model the pixel space domain direction offset. Adaptive and cross-adaptive calculations are used to correct this offset phenomenon, and self-attention is simulated to construct long-distance dependencies and fusion of different input information. The feature extraction and conversion capabilities of CNN are used to complete the correction. Finally, the loss function and optimizer are designed to update the parameters and features used in the above network. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is a diagram of the analysis process of the present invention;
[0055] Figure 2 The result graph of the PCA analysis of the samples of the present invention includes both overexposed and underexposed images;
[0056] Figure 3 This is a visualization result diagram of sampling PCA analysis of a normal non-exposed image of the present invention;
[0057] Figure 4 It is the overall process design diagram of the present invention;
[0058] Figure 5 The embedded operation diagram of the present invention;
[0059] Figure 6 Graph of pixel space domain offset estimator of the present invention;
[0060] Figure 7 This is a diagram of the pixel space domain corrector of the present invention. DETAILED DESCRIPTION
[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0062] See also Figure 1
[0063] The present invention provides a technical solution: an endoscopic image enhancement method based on pixel space domain offset correction, including the need to select two types of a certain number of pictures, one type is a picture that contains both overexposed and underexposed pictures, and the other type is a picture that only contains overexposed or underexposed pictures. Random sampling is performed on their image pixels, where the original image size is [H, W, 3], and the result of random sampling of its pixels is [N, 3], where N is equal to H×W, which is the number of vectors, and 3 is the vector dimension. The analysis process includes the following steps:
[0064] S1: Sample the target image pixels to construct a matrix N*C (N represents the amount of data, C represents the current dimension) data set X;
[0065] S2: Calculate the average value (decentralization), that is, subtract the average value of each feature from each feature. Get the standardized data set Z;
[0066] S3: Calculate the covariance matrix
[0067] S4: Calculate the K (here K is 2, the three-dimensional feature image needs to be reduced to a two-dimensional plane for visualization) eigenvalue λ 1 ,λ 2 and their corresponding eigenvectors c 1 , c 2 ;
[0068] S5: p=(C 1 C 2 ) construct an eigenvector matrix using the eigenvector as a column vector;
[0069] S6: The representation after dimensionality reduction to K-dimensional space: Y = ZP. The obtained Y is the new data set in the two-dimensional space. The new coordinates of each original data set point in the new two-dimensional principal component space are the corresponding rows of Y. The horizontal and vertical coordinates of each data point are (y i1 ,y i2 ).
[0070] The invention discloses an endoscopic image enhancement method based on pixel space domain offset correction, comprising a VIT Transformer feature extract module, a Feature Transform module, a pseudo-normal pixel space feature generator, a pixel space domain offset estimator and a pixel space domain corrector.
[0071] The VIT Transformer feature extraction module further includes for a given input image X∈R 3*H*W , and then sent to VIT encoder for feature extraction before it needs to be inverted, X inversion =255-X (the distribution interval of each feature dimension is 0 to 255), and the final output is the enhanced image Y∈R 3*H*W , Y is the final result after the correction of brightness, restoration of image details and color enhancement. Through the powerful feature extraction capability of VIT, we finally get two areas of underexposure and overexposure. The original image contains two inputs, X and X inversion , perform embedding operations on them, X∈R 3*H*W The token is input to the vit encoder through downsampling with a 16*16 convolution with a step size of 16. The formula is as follows:
[0072]
[0073] F b =F(token x ) (2)
[0074] Replace the input with X inversion , we can get F by the same logic d , in order to obtain the intermediate amount generated by the final features Feature B and Feature D, the formula is as follows:
[0075]
[0076] Among them, Feature B∈R 3*H*W is the feature map of the brightness overexposed area, Feature D∈R 3*H*W It is the feature map of the darkness underexposed area. After generating the two feature maps, the original input image X and Feature B and Feature D are sent as input to the pseudo-normal pixel feature space domain distributor to generate a pseudo-normal distribution feature map. The formula is as follows:
[0077]
[0078] in, The pseudo-normal pixel feature spatial domain distributor can be a common image enhancement method, such as histogram equalization, contrast-limited adaptive histogram equalization, LCDP, Feature N The pixel space domain offset O between it and the overexposed area feature map Feature B and the underexposed area feature map Feature D can be calculated by the pixel space domain offset estimator. B and O D .
[0079] The pixel space domain offset estimator further includes firstly, the feature obtained in the previous step needs to be N Concat with Feature B / D to get the intermediate feature U N*2C*H*W , the formula is as follows:
[0080] U N*2C*H*W =concat(Feature N , FeatureB / D)
[0081] The DCN capture offset is performed in the pixel color space and spatial position. For Pixel Convolution (3*3 convolution kernel), the color correction offset captured in the pixel color space is recorded as Δc n , positionConvolution is the position captured by the spatial position, and the offset is recorded as Δp n , Δc n and Δp n With the original input Feature N After fusion in the channel dimension, the feature map of the offset is obtained by deformable convolution. The ScSE module is used to modulate the weight value between the channel dimension and the spatial dimension, and the original feature value is expanded and enhanced to form a new fusion weight Δw n In order to allocate different weights to different kernels in channels and spaces, so that they can focus on important pixels, the deformable convolution calculation formula in the pixel space domain direction is as follows:
[0082]
[0083] Among them, w is the weight of the convolution kernel, x is the feature of the input convolution operation, and p x +p y +Δp n is a two-dimensional variable of spatial position, Δw nis the fusion weight, which is the expression space weight used for output calculation. R = {(0,0), (0,1)…(1,1)} represents a 3×3 kernel network. n is the enumeration number of the element at the nth position in the R set. Here, it is a 3×3 network. The length of the R set is 9, and the range of n is (0-8).
[0084] The pixel space domain corrector further includes the input of this module is the original image, the dark feature map offset, the bright feature map offset, each input corresponds to three branches to process the image X and O respectively B , O D The three branches of information interaction between themselves and each other include the following parts: q , W k , W v (1×1 Note: All 1*1 convolutions described in this invention do not change the size of the input, but are just a way of expressing weight updates, which can change the number of channels) They are shared weight matrices (the three inputs are multiplied by these three weight matrices. Since the inputs are different, although the weights are the same, the inputs will be different). Their parameters are trainable parameter matrices obtained by back-propagation training of the loss function. After W q , W k , W v Matrix multiplication generates q i , k i 、v i ,i∈(D,X,B), here we can understand q i As the query matrix, k i is the bond matrix, v i is a value matrix, and adaptive calculations are performed on them. The formula is as follows:
[0085] A i =dot-product(q i ,k i ), i∈(D, X, B) (7)
[0086] Among them, dot-product is the dot multiplication operation of the matrix, q i , k i The corresponding position elements are multiplied and summed, and then the Norm is used to ensure the stability of the eigenvalues in the back propagation to obtain the adaptive matrix A i , A i It represents the associated similarity weight of two vectors, v i It represents the weight of the current attention map’s own information. Matrix multiplication is performed between them, and a linear operation of weighted summation of their similarities and weights is performed. The formula is as follows:
[0087]
[0088] Formula 8 corresponds to input branch O B , O D The first term is the adaptive calculation described above, the purpose is to obtain the important area of the offset. The second term is the cross-adaptive calculation, A i It not only interacts with its own information weight, but also interacts with the information weight of another input X. Finally, it combines the adaptive calculation of its own input X to obtain the final result Y. The formula is as follows:
[0089]
[0090] S2 further includes two key loss values, one is the pseudo-normal distribution feature map Feature used to supervise the intermediate process N And the final output enhanced image Y: Loss N ,Loss Y , the pseudo-normal distribution feature map Feature N Reconstruction loss with the true label value Loss N To measure the difference between the pseudo-normal result and the true result: the difference between the two should be as small as possible. N Infinitely close to the true value label, so the consistency loss is constructed as the formation of the intermediate supervision pseudo-normal distribution feature map, the formula is as follows:
[0091]
[0092] It is used to supervise the convolution matrix, feature matrix, eigenvalue, etc. of the whole process to generate the final enhanced image, including the following aspects: L m Reconstruction loss is used to measure the difference between the enhanced image and the original image at the pixel level. It can be achieved by calculating the MAE (absolute error) between the two. The formula is as follows:
[0093]
[0094] in, The square of the Euclidean distance (also called the square of the L2 norm) represented by
[0095] L cos Cosine similarity is used to construct the similarity between the enhanced image and the original image features and calculate the cosine similarity of the feature vectors. The formula is as follows:
[0096] L cos =-Cosine Similarity((φ(I y ),φ(I x ))
[0097] in, is the extracted feature;
[0098] L vit Perceptual loss measures the similarity of images through high-level features. It uses the pre-trained VIT to extract features and calculates the difference between the enhanced image and the original image features. It pays more attention to the perceptual quality of the image. The formula is as follows:
[0099]
[0100] Where n is the number of batches. is the feature value extracted by vit as the pre-trained network;
[0101] L s Structural similarity loss (SSIM) is used to measure the structural similarity between the enhanced image and the original image. It focuses on three aspects: brightness, structure, and contrast. The formula is as follows:
[0102] L s =1-SSIM(I y ,I x )
[0103] Among them, λ m , c , c , s , N , Y are the weight hyperparameters of the six loss functions, x represents the original input image, I y represents the enhanced image, Represents the true value label, and the final total Loss loss function is:
[0104] Loss Y =λ m L m +λ c L cos +λ v L vit +λ s L s (12)
[0105] Loss = λ N Loss N +λ Y Loss Y (13)
[0106] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0107] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. An endoscopic image enhancement method based on pixel spatial domain offset correction, characterized in that , including the need to select a certain number of two types of pictures, one type contains both overexposed and underexposed pictures, and the other type contains only overexposed or underexposed pictures. Randomly sample their image pixels. Here, the original image size is [H, W, 3]. The result of random sampling of its pixels is [N, 3], where N is equal to H×W, which is the number of vectors, and 3 is the vector dimension. The analysis process includes the following steps: S1: Sample the target image pixels to construct a matrix N*C (N represents the amount of data, C represents the current dimension) data set X; S2: Calculate the average value (decentralization), that is, subtract the average value of each feature from each feature to obtain the standardized data set Z; S3: Calculate the covariance matrix S4: Calculate the first K (here K is 2, the three-dimensional feature image needs to be reduced to a two-dimensional plane for visualization) eigenvalues λ1, λ2 and their corresponding eigenvectors c1, c2; S5: p = (C1C2) constructs an eigenvector matrix using the eigenvector as a column vector; S6: The representation after dimensionality reduction to K-dimensional space: Y = ZP. The obtained Y is the new data set in the two-dimensional space. The new coordinates of each original data set point in the new two-dimensional principal component space are the corresponding rows of Y. The horizontal and vertical coordinates of each data point are (y i1 ,y i2 ).
2. The endoscopic image enhancement method based on pixel spatial domain offset correction according to claim 1, characterized in that: It includes VIT Transformer feature extract module, Feature Transform module, pseudo-normal pixel space feature generator, pixel space domain offset estimator and pixel space domain corrector.
3. The endoscopic image enhancement method based on pixel spatial domain offset correction according to claim 1, characterized in that: The VIT Transformer feature extract feature extraction module further includes: For a given input image X∈R 3*H*W , and then sent to VIT encoder for feature extraction before it needs to be inverted, X inversion =255-X (the distribution interval of each feature dimension is 0 to 255), and the final output is the enhanced image Y∈R 3*H*W , Y is the final result after the correction of brightness, restoration of image details and color enhancement. Through the powerful feature extraction capability of VIT, we finally get two areas of underexposure and overexposure. The original image contains two inputs, X and X inversion , perform embedding operations on them, X∈R 3*H*W The token is input to the vit encoder through downsampling with a 16*16 convolution with a step size of 16. The formula is as follows: F b =F(token x ) (2) Replace the input with X inversion , we can get F by the same logic d , in order to obtain the intermediate amount generated by the final features Feature B and Feature D, the formula is as follows: Among them, Feature B∈R 3*H*W is the feature map of the brightness overexposed area, Feature D∈R 3*H*W It is the feature map of the darkness underexposed area. After generating the two feature maps, the original input image X and Feature B and Feature D are sent as input to the pseudo-normal pixel feature space domain distributor to generate a pseudo-normal distribution feature map. The formula is as follows: in, The pseudo-normal pixel feature spatial domain distributor can be a common image enhancement method, such as histogram equalization, contrast-limited adaptive histogram equalization, LCDP, Feature N The pixel space domain offset O between it and the overexposed area feature map Feature B and the underexposed area feature map Feature D can be calculated by the pixel space domain offset estimator. B and O D .
4. The endoscopic image enhancement method based on pixel space domain offset correction according to claim 2, characterized in that: The pixel space domain offset estimator further comprises: First, we need to get the Feature from the previous step N Concat with Feature B / D to get the intermediate feature U N*2C*H*W , the formula is as follows: U N*2C*H*W =concat(Feature N ,Feature B / D) The DCN capture offset is performed in the pixel color space and spatial position. For Pixel Convolution (3*3 convolution kernel), the color correction offset captured in the pixel color space is recorded as Δc n , position Convolution is the position captured by the spatial position, and the offset is recorded as Δp n , Δc n and Δp n With the original input Feature N After fusion in the channel dimension, the feature map of the offset is obtained by deformable convolution. The ScSE module is used to modulate the weight value between the channel dimension and the spatial dimension, and the original feature value is expanded and enhanced to form a new fusion weight Δw n In order to allocate different weights to different kernels in channels and spaces, so that they can focus on important pixels, the deformable convolution calculation formula in the pixel space domain direction is as follows: Among them, w is the weight of the convolution kernel, x is the feature of the input convolution operation, and p x +p y +Δp n is a two-dimensional variable of spatial position, Δw n is the fusion weight, which is the expression space weight used for output calculation. R = {(0,0), (0,1)…(1,1)} represents a 3×3 kernel network. n is the enumeration number of the element at the nth position in the R set. Here, it is a 3×3 network. The length of the R set is 9, and the range of n is (0-8).
5. The endoscopic image enhancement method based on pixel space domain offset correction according to claim 1, characterized in that: The pixel space domain corrector further comprises: The input of this module is the original image, the darkening feature map offset, and the brightening feature map offset. Each input corresponds to three branches that process images X and O respectively. B , O D The three branches of information interaction between themselves and each other include the following parts: q , W k , W v (1×1 Note: All 1*1 convolutions described in this invention do not change the size of the input, but are just a way of expressing weight updates, which can change the number of channels) They are shared weight matrices (the three inputs are multiplied by these three weight matrices. Since the inputs are different, although the weights are the same, the inputs will be different). Their parameters are trainable parameter matrices obtained by back-propagation training of the loss function. After W q , W k , W v Matrix multiplication generates q i , k i 、v i ,i∈(D,X,B), here we can understand q i As the query matrix, k i is the bond matrix, v i is a value matrix, and adaptive calculations are performed on them. The formula is as follows: A i =dot-product(q i ,k i ),i∈(D、X、B) (7) Among them, dot-product is the dot multiplication operation of the matrix, q i , k i The corresponding position elements are multiplied and summed, and then the Norm is used to ensure the stability of the eigenvalues in the back propagation to obtain the adaptive matrix A i , A i It represents the associated similarity weight of two vectors, v i It represents the weight of the current attention map’s own information. Matrix multiplication is performed between them, and a linear operation of weighted summation of their similarities and weights is performed. The formula is as follows: Formula 8 corresponds to input branch O B , O D The first term is the adaptive calculation described above, the purpose is to obtain the important area of the offset. The second term is the cross-adaptive calculation, A i It not only interacts with its own information weight, but also interacts with the information weight of another input X. Finally, it combines the adaptive calculation of its own input X to obtain the final result Y. The formula is as follows:
6. The endoscopic image enhancement method based on pixel space domain offset correction according to claim 1, characterized in that: The S2 further includes two key loss values, one is the pseudo-normal distribution feature map Feature used to supervise the intermediate process N And the final output enhanced image Y: Loss N ,Loss Y , the pseudo-normal distribution feature map Feature N Reconstruction loss with the true label value Loss N To measure the difference between the pseudo-normal result and the true result: the difference between the two should be as small as possible. N Infinitely close to the true value label, so the consistency loss is constructed as the formation of the intermediate supervision pseudo-normal distribution feature map, the formula is as follows: It is used to supervise the convolution matrix, feature matrix, eigenvalue, etc. of the whole process to generate the final enhanced image, including the following aspects: L m Reconstruction loss is used to measure the difference between the enhanced image and the original image at the pixel level. It can be achieved by calculating the MAE (absolute error) between the two. The formula is as follows: in, The square of the Euclidean distance (also called the square of the L2 norm) represented by L cos Cosine similarity is used to construct the similarity between the enhanced image and the original image features and calculate the cosine similarity of the feature vectors. The formula is as follows: L cos =-Cosine Similarity((φ(I y ),φ(I x )) in, is the extracted feature; L vit Perceptual loss measures the similarity of images through high-level features. It uses the pre-trained VIT to extract features and calculates the difference between the enhanced image and the original image features. It pays more attention to the perceptual quality of the image. The formula is as follows: Where n is the number of batches. is the feature value extracted by vit as the pre-trained network; L s Structural similarity loss (SSIM) is used to measure the structural similarity between the enhanced image and the original image. It focuses on three aspects: brightness, structure, and contrast. The formula is as follows: L s =1-SSIM(I y ,I x ) Among them, λ m , c , v , s , N , Y are the weight hyperparameters of the six loss functions, x represents the original input image, I y represents the enhanced image, Represents the true value label, and the final total Loss loss function is: Loss Y =λ m L m +λ c L cos +λ v L vit +λ s L s (12) Loss=λ N Loss N +λ Y Loss Y (13)
Citation Information
Cited By
High-definition digital imaging method suitable for ear-nose-throat endoscope
CN119850495A
