Endoscope image reflection restoration method based on detection network and filtering fusion

Abstract

The invention relates to an endoscope image reflection restoration method based on detection network and filtering fusion, and belongs to the technical field of medical image computer processing. The method aims at improving detection and removal of specular reflection in an endoscope image, a variant of TransUnet is adopted in the detection stage, and U-Net and Transform architectures are combined; the method comprises Atrous spatial pyramid pooling (ASPP), information bottleneck (IB) and a convolutional block attention module (CBAM). The enhancement functions improve the identifiability of specular highlight and solve the problem of weak generalization in endoscope data concentration. In the removal stage, the invention designs an enhanced median filtering module for removing specular reflection. The module first applies Gaussian blur to the repaired region and then calculates color information from the non-reflective region. According to the method, mirror reflection is effectively eliminated from the endoscope image, clearer and more accurate visual information is provided for clinicians, and improvement of diagnosis precision is facilitated.

Description

Technical Field

[0001] The invention belongs to the technical field of medical image computer processing, and in particular relates to an endoscopic image reflection repair method based on detection network and filter fusion. Background Art

[0002] Endoscopic imaging is the cornerstone of minimally invasive surgery (MIS) and is gaining popularity due to its critical role in improving surgical outcomes, facilitating postoperative analysis, and ensuring diagnostic accuracy. However, the integrity of endoscopic images is often compromised by a variety of artifacts, chief among which are specular reflections, smoke, noise, and changing lighting conditions. Among these artifacts, specular reflections are particularly detrimental as they can severely impair the surgeon's field of view, potentially leading to misdiagnosis or surgical complications. These reflections obscure important details of color, texture, and structure, and can be misinterpreted as intrinsic elements of the scene. This uncertainty, combined with the inherent limitations of viewing angles and the presence of artifacts, poses a significant challenge to monitoring disease progression. Therefore, specular reflections in endoscopic images remain a key factor that undermines image quality and diagnostic reliability.

[0003] Addressing the image degradation caused by specular reflections in endoscopic imaging has been the focus of extensive research. Strategies explored include methods based on color space analysis, two-color reflectance modeling, and image inpainting techniques. The HSV color space was first applied to identify highlights, although the robustness was limited in complex scenes. Adaptive robust principal component analysis (AdaRPCA) was introduced, and accelerated adaptive non-convex robust principal component analysis (AANC-RPCA) was proposed. These principal component analysis-based methods typically perform sparsity and low-rank decomposition on endoscopic images to extract reflection masks. Despite their effectiveness, these methods still rely on fixed threshold settings, which limits their generalization ability and effectiveness in identifying various reflective regions.

[0004] Until now, many studies have focused on identifying and mitigating specular reflections in endoscopic images, with some success. However, these approaches still have several major limitations.

[0005] First, methods that rely on color differences for threshold segmentation are easily affected by specific colors in the image, reducing their applicability in different scenarios.

[0006] Second, methods that rely on color differences for threshold segmentation are easily affected by specific colors in the image, reducing their applicability in different scenarios.

[0007] Finally, although deep learning has been widely used in image segmentation and inpainting, few studies have utilized deep learning models to accurately detect highlight regions.

[0008] Specular reflection detection, as a key link in image segmentation, has received significant attention in recent years. It mainly focuses on accurately identifying and segmenting specular reflection areas from images. Specular reflection detection techniques can be roughly divided into two methods: traditional image processing methods and deep learning-based methods. Traditional specular reflection detection methods usually rely on threshold segmentation techniques. A two-step method has been proposed to detect the reflection area: first convert the image into a grayscale image, then detect and separate the specular highlights based on the grayscale threshold, and perform dilation on them. However, this method is limited by the fixed threshold and is only applicable to images that are usually dark. Adaptive robust principal component analysis (AdaRPCA) has been introduced, and accelerated adaptive non-convex robust principal component analysis (AANC-RPCA) has been proposed. These principal component analysis-based methods usually perform sparsity and low-rank decomposition on endoscopic images to extract reflection masks. Despite their effectiveness, these methods still rely on fixed threshold settings, which limits their generalization ability and effectiveness in identifying various reflection areas.

[0009] Deep learning methods have shown promise in specular reflection detection but also face generalization challenges due to the limited availability of datasets. A hybrid strategy combining transfer learning and weakly supervised learning was adopted to train a lightweight U-Net model despite inaccurate labels. However, this model still faces challenges in accurately detecting small reflective areas. ResNet50 and DeepLabv3+ were used to improve detection accuracy. However, the various forms, shapes, and sizes of specular reflections in endoscopic images pose unresolved challenges to current detection methods. These challenges lead to two main problems: over-segmentation (false positives) and under-segmentation (false negatives). These problems are particularly prominent when the appearance of the reflective area is similar to the surface tissue of the organ.

[0010] Given the outstanding performance of TransUnet in image segmentation, many TransUnet-based models have emerged in recent years. Since TransUnet was created by combining Transformers with U-Net, significantly improving the segmentation accuracy in complex scenes, the SwinUnet model was introduced, replacing the convolutional layers of U-Net with Swin Transformer. AA-TransUnet was proposed, which further optimized TransUnet through block attention model (CBAM) and depthwise separable convolution. In endoscopic image analysis, U-Net and TransUnet models significantly improved image segmentation performance by integrating attention mechanisms. However, the diversity and complexity of specular reflections in endoscopic images pose ongoing challenges.

[0011] In endoscopic image processing, removing specular reflection is an important task. Some people replace the value of the specular reflection area with the average intensity of the pixels in the non-specular reflection area, and some people use Gaussian blur to "smooth non-reflective color pixels. Some people use multi-resolution technology to process colposcopic images. Some people separate the scattered and specular reflection components based on the two-color reflection model, and some people restore the highlight area through multi-scale dynamic image expansion and fusion. Some people have developed a deep learning framework, but the inference time is long despite its superior performance. Some people use adaptive robust principal component analysis (RPCA) to fill in the specular area. However, these methods have limitations in preserving texture details. Summary of the invention

[0012] The technical problem to be solved by the present invention is to propose a new endoscopic image reflection restoration method SDMFFN based on detection network and filter fusion in view of the problems and shortcomings of the above-mentioned prior art. The method uses an enhanced TransUnet model S-TransUnet in the detection stage, integrating Atrous spatial pyramid pooling (ASPP), information bottleneck (IB) and CBAM to achieve accurate highlight detection, improve multi-scale feature extraction capabilities, and compress key features and filter out redundant features. The method smoothes the reflection area and integrates color information to achieve natural restoration. SDMFFN improves visual clarity and diagnostic accuracy, providing clearer images without mirror interference.

[0013] In order to solve the above problems, the present invention adopts the following technical solutions:

[0014] The present invention is an endoscopic image reflection repair method based on detection network and filter fusion, which is characterized in that the steps are as follows:

[0015] S1: Preliminary preparation of endoscope image data

[0016] Prepare an endoscopic image dataset, and use the image data of unsaturated specular reflection as training and testing data for the model;

[0017] The endoscopic image dataset described in the present invention can come from an existing public endoscopic image dataset, such as the CVC-ClinicSpec dataset from the Catalan Vision Center CVC, or from an endoscopic image dataset collected by various hospitals, or a mixed dataset of the two; there are no specific requirements for the endoscopic image dataset, but only the image data of unsaturated specular reflection is used as the training and test data of the model; the reflection intensity of the unsaturated specular reflection image is low, the details are clearly visible, and it appears as scattered granular reflection points, which can be broadly distinguished according to the commonly used methods in the field.

[0018] S2: Endoscopic image de-reflection stage 1:

[0019] S2.1 Initialize the image segmentation model S-TransUnet

[0020] The encoder parameters are initialized according to the nature of the task. The encoder parameters consist of a series of convolutional layers and pooling layers, which are responsible for extracting basic features from the input endoscopic image; these features are then projected into the embedding space through a learnable linear transformation to provide input for subsequent Transformer processing; the Transformer module is used to capture the global spatial relationship in the image and enhance the global representation ability of the features; next, the decoder parameters are initialized. The decoder parameters consist of multiple upsampling and convolutional layers, which gradually upsample the encoded features and fuse them with the high-level features from the encoder to finally generate a high-precision segmentation mask; this process is achieved through skip connections, which pass the high-level features of the encoder directly to the decoder to ensure that contextual information is preserved;

[0021] Initialize an optimizer, which is used to effectively update model parameters throughout the training process;

[0022] Initialize the loss function, which will be selected based on the characteristics of the segmentation task to evaluate model performance and guide model training;

[0023] At the same time, the information bottleneck IB module and the atrous convolutional pyramid ASPP module are initialized; the IB module optimizes the feature extraction process of the model by maximizing the mutual information between task-related features and the target; the ASPP module extracts multi-scale features by using convolution operations with different dilation rates, which is particularly suitable for reflective areas with variable sizes and shapes in endoscopic images;

[0024] S2.2: Setting the optimizer and loss function

[0025] Select the appropriate optimizer and loss function based on the complexity of the model and the requirements of the light spot segmentation task;

[0026] S2.3: Image segmentation model S-TransUnet training

[0027] 2.3.1 Data Preprocessing

[0028] The training data set and the validation data set are divided according to the set ratio; the input image data of the training set is preprocessed by random horizontal flipping, random vertical flipping and standardized data enhancement methods to improve the generalization ability of the model and prevent overfitting; for the image data of routine, early lesion and late lesion scenes, image smoothing and morphological transformation are introduced for data enhancement operations;

[0029] 2.3.2 Forward Propagation

[0030] The input image is first processed by the encoder part of U-Net to obtain the primary feature map; in the forward propagation process, the input image is first fed into the encoder part of U-Net to extract the primary feature map; the primary feature map is then fed into the ASPP module to capture multi-scale contextual information; the ASPP module extracts features of different scales through parallel convolutional layers with different expansion rates, and its output is expressed as:

[0031] ASPP(X)=Concat(C1(X,r1),C2(X,r2),C3(X,r3))

[0032] Among them, X is the input feature map, C i are different expansion rates r i The output of the hole convolution layer, Concat means splicing in the channel dimension;

[0033] Three atrous convolutional layers are used, with expansion rates of r1, r2, and r3 respectively; the concatenated feature maps are fed into the subsequent Transformer network, and the self-attention mechanism is used to further refine the features and capture the long-distance dependencies in the image; the feature maps processed by the Transformer are then passed through the information bottleneck IB module. The goal of the information bottleneck is to maximize the mutual information between the intermediate feature Z and the output Y, while minimizing the mutual information between the input X and the intermediate feature Z; the objective function of the information bottleneck is expressed as:

[0034]

[0035] Where T is the centralized bottleneck representation, and the optimization goal is to maximize the mutual information between T and the target Y, and minimize the mutual information between T and the input X;

[0036] 2.3.3 Loss Calculation and Back Propagation

[0037] Calculate the loss value based on the model's predicted output and the true label, including cross entropy loss and Dice loss; use the back propagation algorithm to calculate the gradient and update the model parameters based on the loss;

[0038] 2.3.4 Segmentation model parameter update

[0039] Update model parameters according to the optimizer strategy; introduce a learning rate scheduler to dynamically adjust the learning rate to ensure that the model can converge stably;

[0040] 2.3.5 Segmentation Model Validation

[0041] After each training round, the model is validated using the validation dataset, and the Dice coefficient, IoU, and HD95 evaluation indicators are calculated to comprehensively evaluate the generalization ability of the model.

[0042] S2.4: Segmentation model saving

[0043] The weights are retained every hundred rounds, and a best weight is defined. If the weight saved this time is the best, the best is saved as the current model parameter, and the relevant performance indicators are recorded to ensure that the final best model parameters can be retained.

[0044] S2.5: Output of the segmentation model S-TransUnet

[0045] After the model training and optimization process is completed, the model is saved; the saved model is used to evaluate the training dataset and the test dataset, and reliable mask information is provided for the restoration stage to accurately lock the reflective area; the model is applied to the images of the two datasets to generate predicted segmentation masks; these masks will be used for subsequent image restoration; after the predicted masks are generated, the model performance is measured by comparing them with the real masks in the test dataset, using quantitative evaluation indicators to determine whether the segmentation effect of the model meets the requirements of the restoration task; the predicted masks that meet the requirements of the restoration task will be directly used to guide the subsequent image restoration steps;

[0046] S3: Stage 2 of Endoscopic Image De-reflection: Image Inpainting Computational Model

[0047] The input of this stage is composed of the original image and the mask obtained by the previous step, and based on this, the reflective area of ​​the endoscope image is repaired;

[0048] The mask data is used to detect the reflective areas and a median filter is applied to correct the overexposed pixels and preserve the edges using its noise immunity. Subsequently, a Gaussian blur is applied to smooth these areas and reduce boundary artifacts. To enhance the texture of the restored area, a local binary pattern (LBP) analysis is used.

[0049] Calculate the LBP value distribution inside and outside the initially restored mask area M, and create a histogram by counting the number of LBP occurrences to quantify the texture features for similarity evaluation; the formula of the LBP histogram is as follows:

[0050]

[0051] where the Kronecker delta function () is used, k∈{0,1,…,255}, representing the possible LBP values; then, the similarity between the two histograms is measured by calculating the chi-square distance as follows,

[0052]

[0053] where ∈ is a small constant; subsequently, the chi-square distance is converted into a similarity measure, denoted by S, and normalized to the interval [0,1]:

[0054]

[0055] The determination of the centroid color of the masked area is enhanced by measuring the texture similarity between the initial inpainted masked area and the non-masked area of ​​the original image; the centroid color of each masked area M(x, y) is calculated as the weighted average of the pixels in the local window W(x, y), taking into account the spatial distribution and texture similarity weights; the expression of C(x, y) is as follows:

[0056]

[0057] The centroid color C(x,y) of each pixel (x,y) in the mask area M(x,y) is calculated by the formula; where W(x,y) refers to the window containing the pixel at the coordinate (x,y), and is the area within which the centroid color is determined by calculation; the centroid color is a weighted average calculated based on the pixels in the non-mask area within the local window W(x,y); the weight S takes into account the spatial distribution and texture similarity to ensure that the centroid color accurately represents the local visual features; I(u,v) is the color value of the pixel (u,v) in the original image, and M(u,v) is the mask value of the corresponding pixel in the mask area, where M(u,v)=1 indicates that the pixel is part of the mask, and M(u,v)=0 indicates that the pixel is not part of the mask;

[0058] With the centroid color determined, continue filling the masked area by applying this color to all pixels within the mask;

[0059]

[0060] The formula applies the calculated centroid color C(x,y) to all pixels within the mask area M(x,y); if the pixel (x,y) is part of the mask (M(x,y) = 1), its color value is set to the centroid color C(x,y); if the pixel is not part of the mask (M(x,y) = 0), its original color value I(x,y) is maintained; this process minimizes the effect of specular reflection while preserving the texture and color consistency of the image;

[0061] A falloff mask is applied to the filled image to smooth the transition between filled and adjacent unfilled areas:

[0062]

[0063] The formula calculates the attenuation mask D(x,y) used to smooth the transition between filled and adjacent unfilled areas; the attenuation mask is a value between 0 and 1, which depends on the mask area M(x,y) and the accumulation of the mask area within the local window W(x,y); the parameter k controls the strength of the attenuation;

[0064] Finally, the pixel value of the restored image is calculated by the weighted average of the attenuated mask and the filled image, and the weighted average of the corresponding pixel values ​​of the original image; the pixel value of the final restored image is calculated using the following formula:

[0065] I″(x,y)=D(x,y)·I′(x,y)+(1-D(x,y))·I(x,y) (10)

[0066] The formula calculates the final restored image I″(x,y); it is calculated by the weighted average of the attenuation mask D(x,y) and the filled image I′(x,y)I′(x,y), and the weighted average of the corresponding pixel values ​​of the original image I(x,y).

[0067] The above-described method of the present invention, its further preferred technical solution is:

[0068] 1. In step S1, the Retinex method is used to enhance the image to improve the visibility of the reflective area. The contrast and brightness of the image are improved by simulating the adaptability of the human eye to changes in illumination, thereby improving the image quality and providing more accurate data for subsequent image analysis and diagnosis.

[0069] 2. In step S2, the initialized optimizer is SGD or Adam.

[0070] 3. In step S2.2: Setting the optimizer and loss function The specific scheme includes:

[0071] 2.2.1 The stochastic gradient descent SGD optimization strategy is used, and the exponential decay learning rate scheduler is combined to optimize the model parameters. The learning rate adjustment formula is as follows:

[0072] 1. t =lr0·exp(-λ·t)

[0073] Among them, lr0 is the initial learning rate, λ is the decay rate hyperparameter, and t is the current round;

[0074] 2.2.2. According to the characteristics of the light spot segmentation task, the cross entropy loss CEL and the unified focus loss UFL are introduced for comparison; the cross entropy loss function is defined as follows:

[0075]

[0076] Among them, yi is the true label, p i is the predicted probability, and N is the total number of categories.

[0077] 4. In step S3, the local texture is quantified by comparing the intensity of the central pixel with the intensity of the pixels in its neighborhood using the local binary pattern (LBP) analysis:

[0078]

[0079] Where P represents the number of neighborhood pixels, the coordinates of the neighborhood pixels within the radius R of the central pixel (x, y) are considered to calculate the LBP feature pixel value; s(z) is a sign function, if and only if I(x k ,y k )≥I(x,y, and 0 otherwise. The feature map is used to evaluate the texture similarity between the unmasked area of ​​the original image and the initially restored masked area.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] 1. In order to improve the detection and removal of specular reflections in endoscopic images, a new two-stage method, specular detection median filter fusion network (SDMFFN), is proposed. This innovative strategy optimizes the detection and removal of specular reflections through two key stages: detection and removal. The detection stage adopts S-TransUnet, a variant of TransUnet, which combines the advantages of U-Net and Transformer architectures. It includes Atrous Spatial Pyramid Pooling (ASPP), Information Bottleneck (IB), and Convolutional Block Attention Module (CBAM). These enhancements improve the recognizability of specular highlights and solve the problem of weak generalization in endoscopic datasets. In the removal stage, an enhanced median filter module for removing specular reflections is designed. The module first applies Gaussian blur to the repaired area and then calculates color information from the non-reflective area. The proposed method effectively removes specular reflections from endoscopic images, providing clinicians with clearer and more accurate visual information, thereby improving diagnostic accuracy and surgical outcomes.

[0082] 2. This paper introduces S-TransUnet, an enhanced model for highlight detection in endoscopic images. It integrates ASPP and information bottleneck techniques into the TransUnet framework to improve segmentation accuracy. The model uses CBAM to focus on key features and improve the detection of reflective areas under different conditions. This approach strengthens the model's feature extraction capabilities at different scales and filters out irrelevant information, thereby significantly improving the model's segmentation accuracy.

[0083] 3. In the second stage, the present invention introduces an innovative method specifically designed to process specular reflection areas in endoscopic images. The core advantage of this method lies in its ability to extract peripheral features and fuse texture similarities, which not only removes highlight interference but also maintains the natural visual coherence and texture consistency of the image. This process not only captures the relative intensity changes between pixels, but also retains the local texture information of the image, providing key visual cues for image restoration.

[0084] 4. In the present invention, the S-TransUnet model solves these challenges by integrating the convolutional block attention module (CBAM) and the Atrous spatial pyramid pooling (ASPP) module, further improving the recognition ability of reflection points and achieving satisfactory segmentation results. The present invention adopts an improved median filter restoration algorithm with a color compensation mechanism, aiming to restore the endoscopic image more smoothly while preserving the texture details. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A process for removing specular reflections from a mirror image;

[0086] Figure 2 This is the network structure diagram of the S-TransUnet model;

[0087] Figure 3 for Figure 2 Schematic diagram of the module structure in the middle, with the encoding module on the left, the Transformer module in the middle, and the information bottleneck on the right;

[0088] Figure 4 This is a performance comparison diagram of the segmentation models in the embodiments;

[0089] Figure 5 This is a performance comparison diagram of the reflection removal model in the embodiment;

[0090] Figure 6 It is the result diagram on the G-endoscope dataset in the embodiment. DETAILED DESCRIPTION

[0091] The specific technical solutions of the present invention are further described below to help those skilled in the art further understand the present invention without limiting the rights thereof.

[0092] Example 1: Endoscopic image reflection restoration method based on detection network and filter fusion

[0093] 1. Methods

[0094] Endoscopic examination plays a vital role in medical diagnosis, however, the presence of specular reflection areas often interferes with clinicians' assessment of diseased tissues. Reflections are usually divided into two types: unsaturated specular reflections and saturated specular reflections. Unsaturated specular reflections have low intensity, and details are clearly visible, appearing as scattered granular reflective points, which are the repair objects of the present invention; while saturated specular reflections have high intensity, appearing as continuous blocks or regions, which may obscure important information in the image. In order to enable the model to work effectively under diverse clinical conditions, the endoscopic image dataset can also be divided into endoscopic images in different scenarios such as conventional classes, early lesions, and late lesions as needed. This can improve the generalization ability of the model and avoid repeated learning of a single feature.

[0095] The endoscopic image dataset used in the method of this embodiment consists of two parts: the first part is the CVC-ClinicSpec dataset, which comes from the Catalonia Visual Centre (CVC), and the second part is an endoscopic image dataset from a hospital. Through the identification of professionals, images containing mirror reflection areas are screened out from the dataset, and the mirror reflection areas are further divided into two types: saturated mirror reflection and unsaturated mirror reflection. Finally, endoscopic images with unsaturated mirror reflection are extracted, and a new dataset is constructed to support the research work of this embodiment.

[0096] In order to solve the above problems for non-saturated specular endoscope images, the inventors proposed a two-stage method: a specular reflection detection module and a specular reflection removal module. The specific process is as follows:

[0097] Endoscope reflection detection and repair hybrid network usage steps based on S-TransUnet

[0098]

[0099]

[0100] 1.1 Mirror reflection detection

[0101] The S-TransUnet model is used, which is designed for detecting reflective areas in endoscopic images. Figure 1 As shown in the first stage of the TransUnet model, the channel attention module (CAM) and the spatial attention module (SAM) are integrated to form the convolutional block attention module (CBAM). Figure 2As shown in the figure, in the encoder path, CBAM is embedded in the CNN sublayer of the CNN transform layer, while in the decoder path, CBAM is placed after consecutive convolution operations. This design enables the model to emphasize channel and spatial features that are critical for detecting reflective areas under fluctuating conditions in endoscopic imaging. In addition, the S-TransUnet model also integrates the Atrous Spatial Pyramid Pooling (ASPP) module, which significantly enhances the model's sensitivity to feature scale through multi-scale Atrous convolution.

[0102] 1.1.1. Encoder S-TransUnet utilizes a combination of convolutional neural networks (CNNs) and transformers in its encoder to capture image features. Once the CNN extracts feature maps, these features are subsequently projected into different embedding spaces through learnable linear transformations. This process allows spatial information to be retrieved from image patches.

[0103] 1.1.2. Skip Connections By merging skip features with decoder features, the network is able to retain key high-level information, which significantly enhances the performance of image segmentation. We utilize skip connections to directly incorporate these high-level features into the segmentation process, ensuring that the segmentation output benefits from the rich contextual information provided by the skip connections.

[0104] 1.1.3. Decoder In the decoder stage, a series of upsamplers are cascaded to perform multi-stage upsampling and feature decoding, and finally produce the segmentation result. Figure 3 As shown in (left), each decoder block consists of a 2x upsampling step, a feature concatenation stage, and a convolution operation. After each decoder block, the size of the feature map is doubled in length and width, while the number of channels is halved. After processing through three decoder blocks, the size of the feature map is reduced to half the size of the original image. The final segmentation map is then derived from the segmentation head, which includes an upsampling layer followed by a linear layer.

[0105] 1.1.4. Information Bottleneck In order to enhance the robustness and generalization ability of the S-TransUnet model, we adopt the information bottleneck (IB) principle. This strategy focuses on extracting task-related features from the input data while minimizing irrelevant information. The specific optimization objectives are as follows:

[0106]

[0107] Where T is a centralized bottleneck representation, the prediction robustness and effectiveness of the model in complex scenes are improved. The optimization goal is to maximize the mutual information between T and the target Y and minimize the mutual information between T and the input X, thereby enhancing the generalization ability of the model to new endoscopic image segmentation tasks.

[0108] 1.2 Specular Reflection Removal

[0109] In this phase, we introduce an innovative image restoration method to address the highlight interference problem in endoscopic images. The advantage of this method lies in its feature extraction and texture fusion capabilities, which reduces highlight interference without compromising the natural coherence and texture of the image.

[0110] like Figure 1 As shown in the second stage of , we use the mask data to detect reflective areas and apply a median filter to correct overexposed pixels, taking advantage of its noise immunity to preserve edges. Subsequently, a Gaussian blur is applied to smooth these areas and reduce boundary artifacts. To enhance the texture of the restored areas, we utilize Local Binary Pattern (LBP) analysis, which quantifies local texture by comparing the intensity of the center pixel with the intensity of pixels in its neighborhood, as detailed below:

[0111]

[0112] Where P represents the number of neighborhood pixels, and the coordinates of the neighborhood pixels within the radius R of the center pixel (x, y) are considered to calculate the LBP feature pixel value. In addition, s(z) is a sign function if and only if I(x k ,y k )≥I(x,y), and 0 otherwise. Using these feature maps, we evaluated the texture similarity between the unmasked area of ​​the original image and the initially restored masked area. We calculated the distribution of LBP values ​​inside and outside the initially restored masked area M, and created a histogram by counting the number of LBP occurrences to quantify the texture features for similarity evaluation. The formula for the LBP histogram is as follows:

[0113]

[0114] where the Kronecker delta function () is used, k∈{0,1,…,255}, representing the possible LBP values. Subsequently, the similarity between two histograms is measured by calculating the chi-square distance as follows,

[0115]

[0116] where ∈ is a small constant. Subsequently, the chi-square distance is converted into a similarity measure, denoted by S, and normalized to the interval [0,1]:

[0117]

[0118] We enhance the determination of the centroid color of the masked region by measuring the texture similarity between the initial inpainted masked region and the non-masked region of the original image. The centroid color of each masked region M(x,y) is calculated as the weighted average of the pixels within the local window W(x,y), taking into account both the spatial distribution and the texture similarity weights. This approach ensures that the centroid color accurately represents the local visual features. The expression of C(x,y) is as follows:

[0119]

[0120] This formula calculates the centroid color C(x,y) for each pixel (x,y) in the mask region M(x,y). Where W(x,y) refers to the window containing the pixel at coordinate (x,y) and is the region within which the centroid color is determined by calculation. The centroid color is a weighted average calculated based on the pixels in the non-masked area within the local window W(x,y). The weights S(u,v) take into account spatial distribution and texture similarity to ensure that the centroid color accurately represents the local visual features. Here, I(u,v) is the color value of the pixel (u,v) in the original image, and M(u,v) is the mask value of the corresponding pixel in the mask region, where M(u,v)=1 means that the pixel is part of the mask and M(u,v)=0 means that the pixel is not part of the mask.

[0121] With the centroid color determined, we proceed to fill the masked area by applying this color to all pixels within the mask. This approach minimizes the effect of specular reflections while preserving the texture and color consistency of the image.

[0122]

[0123] This formula applies the calculated centroid color C(x,y) to all pixels within the mask region M(x,y). If the pixel (x,y) is part of the mask (M(x,y) = 1), its color value is set to the centroid color C(x,y). If the pixel is not part of the mask (M(x,y) = 0), its original color value I(x,y) is retained. This process minimizes the effects of specular reflections while preserving the texture and color consistency of the image.

[0124] Furthermore, to enhance the restoration effect, an attenuation mask is applied to the filled image to smooth the transition between filled and adjacent unfilled regions, ensuring that the restored image looks more natural and harmonious.

[0125]

[0126] This formula calculates the attenuation mask D(x,y), which is used to smooth the transition between filled and adjacent unfilled areas. The attenuation mask is a value between 0 and 1 that depends on the mask area M(x,y) and the accumulation of the mask area within the local window W(x,y). The parameter k controls the strength of the attenuation. This attenuation mask ensures that the restored image looks more natural and harmonious.

[0127] Ultimately, the pixel value of the restored image is calculated by the weighted average of the attenuated mask and the filled image, as well as the weighted average of the corresponding pixel values ​​of the original image. This comprehensive calculation ensures seamless integration between the restored area and the original image, enriches the details, maintains consistent textures, and thus improves the overall quality of the image. The pixel value of the final restored image is calculated using the following formula:

[0128] I″(x,y)=D(x,y)·I′(x,y)+(1-D(x,y))·I(x,y). (10)

[0129] This formula calculates the final restored image I″(x,y). It is calculated by the weighted average of the attenuation mask D(x,y) and the filled image I′(x,y)I′(x,y), as well as the weighted average of the corresponding pixel values ​​of the original image I(x,y). This comprehensive calculation ensures seamless integration between the restored area and the original image, enriches the details, maintains consistent texture, and thus improves the overall image quality.

[0130] 2. Experiments and Results

[0131] Experimental Datasets To verify the effectiveness of the algorithm in detecting and segmenting specular reflections in endoscopic images, the inventors analyzed two datasets: the G-Endoscope dataset of a hospital and the CVC-ClinicSpec dataset. The inventors used indicators such as Dice coefficient, IoU, HD95, NIQE, and COV to evaluate the performance.

[0132] CVC-ClinicSpec dataset: This colonoscopy image dataset is annotated by professionals and includes 612 images with detailed segmentation of reflective areas.

[0133] G-Endoscopy Dataset: In collaboration with a hospital, the dataset contains endoscopic images in various surgical scenarios, covering organs with different reflective properties, such as colorectal cancer, colon polyps, and other conditions. It contains 34 carefully annotated images, and the Ground Truth labels of the reflective areas are accurately marked by expert colonoscopists, providing a valuable resource for the advancement of medical image analysis technology.

[0134] For evaluating specular highlight detection, three metrics are crucial: Dice coefficient, IoU, and HD95. The Dice coefficient evaluates the overlap of binary segmentation. IoU measures segmentation accuracy through the intersection-union ratio. HD95 focuses on the 95th percentile of the Hausdorff distance, providing a robust segmentation alignment metric that reduces the impact of outliers. Its calculation formula is presented here:

[0135]

[0136] Where TP (True Positive) refers to the number of positive cases that are correctly identified. FP (False Positive) refers to the number of positive cases that are incorrectly identified. FN (False Negative) refers to the number of positive cases that are not identified.

[0137] 2.2 Mirror reflection detection

[0138] In order to verify the superior performance of the detection algorithm, a comparative study was conducted using the CVC-ClinicSpec dataset to compare several methods including ASISH, JSHDR, UNet, TransUnet, IDRM and SHMGAN.

[0139] Figure 4 The detection performance of these methods on the CVC-ClinicSpec dataset is intuitively demonstrated. Figure 4 As shown, the first column shows the original endoscopic images and the second column provides the true labels for accuracy evaluation. In Table 1, we can observe that our method improves the Dice coefficient by 0.51% compared to the original TransUnet. Although it is only a slight improvement in value, it can better capture the fine light spots that TransUnet does not capture. In addition, HD95 is significantly reduced, down 1.47 pixels compared to the original TransUnet. The average HD95 value of our method is about 11.88% of the ASISH method and 51.05% of the TransUnet method. The analysis reveals that although the ASISH method has difficulties in losing light spots in the mirror area, the method is affected by boundary artifacts. TransUnet is effective in segmenting the mirror area, but it produces a granular texture. In contrast, the SDMFFN algorithm of the present invention performs well in accurately segmenting light spots, avoiding boundary artifacts, and maintaining a natural appearance, thereby ensuring the accuracy and reliability of segmentation.

[0140] ASISH: Automatic segmentation and inpainting of specular highlights for endoscopic imaging

[0141] Source:

[0142] Arnold M., Ghosh A., Ameling S., Lacey G. Automatic segmentation and inpainting of specular highlights for endoscopic imaging[J]. EURASIP J ImageVideo Process, 2010, vol. 2010: 1–12.

[0143] JSHDR: A multi-task network for joint specular highlight detection and removal (Multi-task network for joint specular highlight detection and removal) Source:

[0144] Fu G., Zhang Q., Zhu L., Li P., Xiao C. A multi-task network for joint specular highlight detection and removal [C]. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, 2021: 7752-7761. (Fu Gang, Zhang Qiang, Zhu Lin, Li Peng, Xiao Chen. A multi-task network for joint specular highlight detection and removal [C]. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, 2021: 7752-7761.)

[0145] TransUnet:Transunet:Transformers make strong encoders for medical image segmentation (Transunet: Transformers provide powerful encoders for medical image segmentation)

[0146] Source:

[0147] Chen J., Lu Y., Yu Q., Luo X., Adeli E., Wang Y., et al. TransUnet: Transformers make strong encoders for medical image segmentation[J]. arXiv preprint arXiv:2102.04306, 2021. (Chen Jie, Lu Yu, Yu Qi, Luo Xiang, Adri., Wang Yi, et al. TransUnet: Transformers make strong encoders for medical image segmentation[J]. arXiv preprint, arXiv:2102.04306, 2021.)

[0148] SHMGAN: Joint network for specular highlight detection and adversarial generation of specular-free images trained with polarimetric data (Joint network for specular highlight detection and adversarial generation of specular-free images, which is trained with polarimetric data)

[0149] Source:

[0150] Anwer A., ​​Ainouz S., Saad NM, Ali SSA, Merialdeau F. Joint network for specular highlight detection and adversarial generation of specular-free images trained with polarimetric data[J]. Neurocomputing, 2023, 559: 126769. (Anwar, Ainouz, Saad, Ali, Merialdeau. Joint network for specular highlight detection and adversarial generation of specular-free images trained with polarimetric data[J]. Neurocomputing, 2023, 559: 126769.)

[0151] IDRM: Highlight Removal Network Based on an Improved DichromaticReflection Model (Highlight Removal Network Based on an Improved DichromaticReflection Model)

[0152] Source:

[0153] Hu K., Huang Z., Wang X. Highlight Removal Network Based on an Improved Dichromatic Reflection Model[C]. In Proceedings of the ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2024: 2645-2649. (Hu Kai, Huang Zheng, Wang Xin. Highlight Removal Network Based on an Improved Dichromatic Reflection Model[C]. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2024: 2645-2649.)

[0154] Table 1 Comparison of segmentation model performance data.

[0155]

[0156]

[0157] 2.3 Mirror reflection repair

[0158] The inventors evaluated the performance of the proposed algorithm for removing endoscopic image reflections and compared it with several established methods including ASISH, CASRS, and JSHDR. In order to objectively measure the effect of removing specular reflections, two standard no-reference image quality assessment metrics were used: the natural image quality evaluator (NIQE) and the coefficient of variation (COV). These metrics allowed the inventors to quantitatively evaluate the effectiveness of the algorithm in reducing endoscopic image reflection artifacts.

[0159] In qualitative analysis, Figure 5 A comparative visual evaluation of the images processed by various methods is shown. The results show that in the second column, the method proposed by CASRS, while trying to remove specular reflections, leaves noticeable dark spots, resulting in image degradation and chromatic aberration. In the third column, the method by ASISH introduces a blurring effect, especially at the edges of the image, which is not conducive to maintaining image quality. The fourth column shows the technique by JSHDR, which, when applied, causes the color filling of the reflective areas to unnaturally match the surrounding environment, resulting in significant color differences and disruption of the overall aesthetic harmony. The fifth column shows the method by SHMGAN, which shows high accuracy in repairing highlights, but the color of the repaired area looks a bit dim and slightly out of harmony with the surrounding environment.

[0160] In contrast, the method proposed by the present invention, such as Figure 6As shown in the sixth column, not only the specular reflection is effectively removed, but also the texture and details of the image are preserved with greater refinement, thus providing a more natural visual result and significantly improving the quality of the endoscopic image. In order to rigorously verify the effectiveness and superiority of the proposed highlight removal algorithm, we conducted a comprehensive quantitative analysis using 311 endoscopic images as test samples. We evaluated and compared the performance of various algorithms on two key indicators: Natural Image Quality Evaluator (NIQE) and Coefficient of Variation (COV). These quantitative indicators can objectively reflect the visual quality of the restored image and the effect of highlight removal.

[0161] In Table 2, through comparative analysis, it is found that the algorithm proposed in the present invention has achieved significant optimization in the average NIQE value. As a non-reference image quality evaluation indicator based on the statistical characteristics of natural scenes, the lower the value of NIQE, the closer it is to natural images. Compared with the CASRS algorithm, the NIQE value of the method of the present invention is reduced by 1.53, which is 29.08% of the CASRS method, demonstrating its advantage in improving image quality. In addition, in the comparison of the average COV value, the method of the present invention performs well, and the average COV value is reduced by 1.22% compared with the latest IDRM method. The restoration technology of the present invention adopts calculation methods such as filtering, which not only improves the numerical accuracy, but also avoids the problems of long training time and large number of parameters of deep learning models. This series of numerical changes not only confirms the ability of our algorithm to remove highlights and restore image details, but also statistically proves its effectiveness in improving overall image quality.

[0162] Table 2 Average values ​​of reflection area evaluation indicators.

[0163]

[0164] CASRS: Content-aware specular reflection suppression based on adaptive image inpainting and neural network for endoscopic images (Application of content-aware specular reflection suppression based on adaptive image inpainting and neural network for endoscopic images)

[0165] Source:

[0166] Shen DF, Guo JJ, Lin GS, Lin JY Content-aware specular reflection suppression based on adaptive image inpainting and neural network for endoscopic images. Comput. Methods Programs Biomed. 2020, 192, 105414. (Shen Dafang, Guo Jiajun, Lin Guangshen, Lin Junying. Content-aware specular reflection suppression based on adaptive image inpainting and neural network for endoscopic images [J]. Computer Methods and Programs in Biomedicine, 2020, 192: 105414.)

[0167] In addition, the experiment was generalized and verified on the G-Endoscope dataset to further evaluate the robustness and applicability of the proposed method in different scenarios and conditions. Figure 6 As shown in the figure, the first row is the original endoscopic image with unsaturated specular highlights, the second row is the reflective area mask obtained by the S-TransUnet model, and the third row is the endoscopic image without unsaturated reflective areas after restoration by SDMFFN. The model of the method of the present invention performs well on this dataset, indicating its effectiveness and reliability in practical applications.

[0168] 2.4 Ablation study

[0169] In order to explore the role of the IB module and the ASPP module in enhancing the specular reflection segmentation of endoscopic images, the inventors conducted an ablation study on these two components to investigate their specific contributions to the model. Table 3 shows the quantitative results of the first stage of the ablation study. In the experiment, the inventors independently evaluated the performance of the IB module and the ASPP module, and then combined the two to observe the effect when used simultaneously. The results show that when the IB module is used alone, the model has shown a certain degree of performance improvement in the endoscopic image segmentation task. On this basis, after integrating the ASPP module, the performance of the model is further enhanced, indicating that the ASPP module plays an important role in capturing highlight areas of different scales.

[0170] Table 3 shows the ablation study of different modules in the proposed SDMFFN.

[0171]

[0172] Experiments show that:

[0173] During the process of model testing and application, the inventors evaluated the model to ensure that it can accurately identify and segment reflective areas in colorectal images of different scenes. In the first stage, the model receives colorectal images as input, then predicts the mask of the reflective area in the image, and calculates parameters such as Dice index, IoU and HD95 to measure the segmentation effect. These indicators help us understand the accuracy and reliability of the model in identifying reflective areas.

[0174] In the second stage, the model uses the original image and the mask generated in the first stage as input to perform detailed restoration of the reflective areas marked by the mask. In this process, the model analyzes the image texture features and fills the reflective areas based on this to achieve a smooth and natural transition and reduce or eliminate the visual interference caused by reflections. This restoration process not only improves the image quality, but also helps with subsequent diagnosis and analysis.

[0175] Throughout the testing process, the model’s performance will be continuously monitored and evaluated to ensure its effectiveness and accuracy in real-world applications. In this way, it can be ensured that the model can provide high-quality segmentation and restoration results when processing colorectal images to meet clinical and research needs.

[0176] SDMFFN is a two-stage hybrid model, in which the segmentation model S-TransUnet obtains multi-scale information features by fusing atrous convolutional pyramids (ASPP), reduces feature omissions in the encoding stage, and uses the U-Net network to capture local features, combining CNN with Transformer, and using information to ensure important information features while taking into account contextual information. After training, testing, and application, the system will accurately segment the reflective area based on the input image. In the second stage, the repair model calculates the features around the mask to perform detailed repairs on the reflective area. By analyzing the image texture features and filling the reflective area based on this, the model achieves a smooth and natural transition and reduces the visual interference caused by reflections. Such repair processing not only improves image quality, but also helps with subsequent diagnosis and analysis.

[0177] In summary, the two-stage image restoration method of the present invention uses the S-TransUnet model to solve the specular reflection problem in endoscopic images. The model combines the advantages of U-Net and Transformer architectures and is equipped with ASPP, IB and CBAM modules to enhance the detection of reflective areas. These modules enable the model to capture local and global features and meet the challenges of endoscopic imaging conditions. We also introduced an innovative restoration technique to pre-process the reflective area using median filtering and Gaussian blurring, and analyze the local texture using LBP feature extraction, thereby improving the segmentation accuracy, including DICE, HD95 and IoU scores. Our method ensures texture consistency and provides smoother restoration results, with better NIQE and CoV indicators than other methods. The method of the present invention is aimed at the smooth removal of specular highlights in endoscopic images and has medical application value. It is mainly effective for the restoration of non-saturated highlights.

Claims

1. A method for restoring endoscopic image reflection based on detection network and filter fusion, characterized in that: The steps are as follows: S1: Preliminary preparation of endoscope image data Prepare an endoscopic image dataset, and use the image data of unsaturated specular reflection as training and testing data for the model; S2: Endoscopic image de-reflection stage 1: S2.1 Initialize the image segmentation model S-TransUnet Initialize the encoder parameters according to the nature of the task. The encoder parameters consist of a series of convolutional and pooling layers responsible for extracting basic features from the input endoscopic image. These features are then projected into the embedding space through a learnable linear transformation, providing input for subsequent Transformer processing; The Transformer module is used to capture the global spatial relationship in the image and enhance the global representation ability of features; Next, the decoder parameters are initialized. The decoder parameters consist of multiple upsampling and convolutional layers, which gradually upsample the encoded features and fuse them with the high-level features from the encoder to finally generate a high-precision segmentation mask; This process is achieved through skip connections, which pass high-level features from the encoder directly to the decoder, ensuring that contextual information is preserved; Initialize an optimizer, which is used to effectively update model parameters throughout the training process; Initialize the loss function, which will be selected based on the characteristics of the segmentation task to evaluate model performance and guide model training; At the same time, the information bottleneck IB module and the atrous convolutional pyramid ASPP module are initialized; the IB module optimizes the feature extraction process of the model by maximizing the mutual information between task-related features and the target; the ASPP module extracts multi-scale features by using convolution operations with different dilation rates, which is particularly suitable for reflective areas with variable sizes and shapes in endoscopic images; S2.2: Setting the optimizer and loss function Select the appropriate optimizer and loss function based on the complexity of the model and the requirements of the light spot segmentation task; S2.3: Image segmentation model S-TransUnet training 2.3.1 Data Preprocessing The training data set and the validation data set are divided according to the set ratio; the input image data of the training set is preprocessed by random horizontal flipping, random vertical flipping and standardized data enhancement methods to improve the generalization ability of the model and prevent overfitting; for the image data of routine, early lesion and late lesion scenes, image smoothing and morphological transformation are introduced for data enhancement operations; 2.3.2 Forward Propagation The input image is first processed by the encoder part of U-Net to obtain the primary feature map; During the forward propagation process, the input image is first fed into the encoder part of U-Net to extract the primary feature map; The primary feature map is then fed into the ASPP module to capture multi-scale contextual information; The ASPP module extracts features of different scales through parallel convolutional layers with different expansion rates, and its output is expressed as: ASPP(X)=Concat(C1(X,r1),C2(X,r2),C3(X,r3)) Among them, X is the input feature map, C i are different expansion rates r i The output of the hole convolution layer, Concat means splicing in the channel dimension; Three atrous convolutional layers are used, with expansion rates of r1, r2, and r3 respectively; the concatenated feature maps are fed into the subsequent Transformer network, and the self-attention mechanism is used to further refine the features and capture the long-distance dependencies in the image; the feature maps processed by the Transformer are then passed through the information bottleneck IB module. The goal of the information bottleneck is to maximize the mutual information between the intermediate feature Z and the output Y, while minimizing the mutual information between the input X and the intermediate feature Z; the objective function of the information bottleneck is expressed as: Where T is the centralized bottleneck representation, and the optimization goal is to maximize the mutual information between T and the target Y, and minimize the mutual information between T and the input X; 2.3.3 Loss Calculation and Back Propagation Calculate the loss value based on the model's predicted output and the true label, including cross entropy loss and Dice loss; use the back propagation algorithm to calculate the gradient and update the model parameters based on the loss; 2.3.4 Segmentation model parameter update Update model parameters according to the optimizer strategy; introduce a learning rate scheduler to dynamically adjust the learning rate to ensure that the model can converge stably; 2.3.5 Segmentation Model Validation After each training round, the model is validated using the validation dataset, and the Dice coefficient, IoU, and HD95 evaluation indicators are calculated to comprehensively evaluate the generalization ability of the model. S2.4: Segmentation model saving The weights are retained every hundred rounds, and a best weight is defined. If the weight saved this time is the best, the best is saved as the current model parameter, and the relevant performance indicators are recorded to ensure that the final best model parameters can be retained. S2.5: Output of the segmentation model S-TransUnet After the model training and optimization process is completed, the model is saved; the saved model is used to evaluate the training dataset and the test dataset, and reliable mask information is provided for the restoration stage to accurately lock the reflective area; the model is applied to the images of the two datasets to generate predicted segmentation masks; these masks will be used for subsequent image restoration; after the predicted masks are generated, the model performance is measured by comparing them with the real masks in the test dataset, using quantitative evaluation indicators to determine whether the segmentation effect of the model meets the requirements of the restoration task; the predicted masks that meet the requirements of the restoration task will be directly used to guide the subsequent image restoration steps; S3: Stage 2 of Endoscopic Image De-reflection: Image Inpainting Computational Model The input of this stage is composed of the original image and the mask obtained by the previous step, and based on this, the reflective area of ​​the endoscope image is repaired; The mask data is used to detect the reflective areas and a median filter is applied to correct the overexposed pixels and preserve the edges using its noise immunity. Subsequently, a Gaussian blur is applied to smooth these areas and reduce boundary artifacts. To enhance the texture of the restored area, a local binary pattern (LBP) analysis is used. Calculate the LBP value distribution inside and outside the initially restored mask area M, and create a histogram by counting the number of LBP occurrences to quantify the texture features for similarity evaluation; the formula of the LBP histogram is as follows: where the Kronecker delta function () is used, k∈{0,1,…,255}, representing the possible LBP values; then, the similarity between the two histograms is measured by calculating the chi-square distance as follows, where ∈ is a small constant; subsequently, the chi-square distance is converted into a similarity measure, denoted by S, and normalized to the interval [0,1]: The determination of the centroid color of the masked area is enhanced by measuring the texture similarity between the initial inpainted masked area and the non-masked area of ​​the original image; the centroid color of each masked area M(x, y) is calculated as the weighted average of the pixels in the local window W(x, y), taking into account the spatial distribution and texture similarity weights; the expression of C(x, y) is as follows: The centroid color C(x,y) of each pixel (x,y) in the mask area M(x,y) is calculated by the formula; where W(x,y) refers to the window containing the pixel at the coordinate (x,y), and is the area within which the centroid color is determined by calculation; the centroid color is a weighted average calculated based on the pixels in the non-mask area within the local window W(x,y); the weight S takes into account the spatial distribution and texture similarity to ensure that the centroid color accurately represents the local visual features; I(u,v) is the color value of the pixel (u,v) in the original image, and M(u,v) is the mask value of the corresponding pixel in the mask area, where M(u,v)=1 indicates that the pixel is part of the mask, and M(u,v)=0 indicates that the pixel is not part of the mask; With the centroid color determined, continue filling the masked area by applying this color to all pixels within the mask; The formula applies the calculated centroid color C(x,y) to all pixels within the mask area M(x,y); if the pixel (x,y) is part of the mask (M(x,y) = 1), its color value is set to the centroid color C(x,y); if the pixel is not part of the mask (M(x,y) = 0), its original color value I(x,y) is maintained; this process minimizes the effect of specular reflection while preserving the texture and color consistency of the image; A falloff mask is applied to the filled image to smooth the transition between filled and adjacent unfilled areas: The formula calculates the attenuation mask D(x,y) used to smooth the transition between filled and adjacent unfilled areas; the attenuation mask is a value between 0 and 1, which depends on the mask area M(x,y) and the accumulation of the mask area within the local window W(x,y); the parameter k controls the strength of the attenuation; Finally, the pixel value of the restored image is calculated by the weighted average of the attenuated mask and the filled image, and the weighted average of the corresponding pixel values ​​of the original image; the pixel value of the final restored image is calculated using the following formula: I″(x,y)=D(x,y)·I′(x,y)+(1-D(x,y))·I(x,y) (10) The formula calculates the final restored image I″(x,y); it is calculated by the weighted average of the attenuation mask D(x,y) and the filled image I′(x,y)I′(x,y), and the weighted average of the corresponding pixel values ​​of the original image I(x,y).

2. The method according to claim 1, characterized in that In step S1, the Retinex method is used to enhance the image to improve the visibility of the reflective area, and the contrast and brightness of the image are improved by simulating the adaptability of the human eye to changes in illumination, thereby improving the image quality and providing more accurate data for subsequent image analysis and diagnosis.

3. The method according to claim 1, characterized in that In step S2, the initialized optimizer is SGD or Adam.

4. The method according to claim 1, characterized in that: In step S2.2: Setting the optimizer and loss function The specific scheme includes: 2.2.1 The stochastic gradient descent SGD optimization strategy is used, and the exponential decay learning rate scheduler is combined to optimize the model parameters. The learning rate adjustment formula is as follows: lr t =lr0·exp(-λ·t) Among them, lr0 is the initial learning rate, λ is the decay rate hyperparameter, and t is the current round; 2.2.

2. According to the characteristics of the light spot segmentation task, the cross entropy loss CEL and the unified focus loss UFL are introduced for comparison; the cross entropy loss function is defined as follows: Among them, y i is the true label, p i is the predicted probability, and N is the total number of categories.

5. The method according to claim 1, characterized in that In step S3, the local texture is quantified by comparing the intensity of the central pixel with the intensity of the pixels in its neighborhood using the local binary pattern (LBP) analysis: Where P represents the number of neighborhood pixels, and the coordinates of the neighborhood pixels within the radius R of the center pixel (x, y) are considered to calculate the LBP feature pixel value; s(z) is a sign function if and only if I(x k ,y k )≥I(x,y), and 0 otherwise. The feature map is used to evaluate the texture similarity between the unmasked area of ​​the original image and the initially restored masked area.

Images