Endoscope artifact segmentation method and system based on physical model

Abstract

The application discloses an endoscope artifact segmentation method and system based on a physical model, and relates to the technical field of medical image processing, wherein the method comprises the following steps: obtaining endoscope calibration parameters by using radiation calibration and geometric calibration; preprocessing the collected endoscope image, including noise suppression and gamma correction, and converting the image from an RGB space to a color space matched with the physical model to separate diffuse reflection and specular reflection components; establishing a physical reflection model based on the Fresnel law and the micro-surface reflection theory to deduce the local surface normal vector, the specular reflection coefficient and the diffuse reflectivity of each pixel; constructing a specular probability map, using an adaptive threshold segmentation to obtain an initial highlight mask, and performing morphological optimization to accurately position the highlight area, and outputting an artifact segmentation result through pixel segmentation. Thus, by accurately modeling the relationship between the light source and the reflection, the influence of the specular highlight artifact in the endoscope image on the image quality is effectively solved.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, specifically focusing on artifact detection and segmentation in endoscopic images, and particularly to a physical model-based method and system for endoscopic artifact segmentation. Background Technology

[0002] With the widespread application of endoscopic technology in surgery, gastrointestinal examinations, and minimally invasive treatments, doctors increasingly rely on images to determine the location and condition of lesions. However, endoscopic images suffer from a very common and difficult-to-avoid artifact—spectral highlights. Specular highlights arise from strong reflections between the mucosal surface and the illumination source, appearing as overexposed or bright spots in the image. These bright patches not only obscure the true texture of lesions but also cause automated analysis algorithms to malfunction. For example, deep learning models may misclassify highlighted areas as lesions or other structures, affecting diagnostic accuracy. Therefore, developing effective artifact detection and segmentation methods is crucial for improving endoscopic image quality and the effectiveness of downstream automated analysis.

[0003] Early research primarily relied on simple thresholding or morphological operations to detect highlight regions by setting a fixed brightness threshold, followed by restoration using interpolation or smoothing of surrounding pixels. While computationally simple, these methods are sensitive to lighting variations, particularly under different brightness conditions, making it difficult to balance detection accuracy and computational efficiency. Subsequent research shifted focus to image enhancement and color space transformation, identifying highlight regions by recognizing high-brightness, low-saturation pixels in color spaces such as HSV or Lab. However, these methods struggle to remain stable under complex structures or extreme lighting conditions.

[0004] In recent years, the introduction of deep learning has improved the accuracy of artifact removal. Researchers have made significant progress in image inpainting and visual quality improvement by automatically learning highlight features through convolutional neural networks. For example, the Partial Attention Network (PatNet) combines artifact segmentation with image inpainting, employing brightness thresholding segmentation and partial convolution to inpaint highlight regions. Furthermore, some works have explored graph generative adversarial networks or recurrent consistency networks for artifact elimination. However, deep models rely on large amounts of labeled data, resulting in high training and inference costs, and they lack an explanation of the physical processes involved in endoscopic optical imaging. Summary of the Invention

[0005] This application provides a physical model-based method, system, storage medium, computer program product, and electronic device for endoscopic artifact segmentation, which aims to at least solve the problems of inaccurate artifact detection and low computational efficiency in current related technologies.

[0006] In a first aspect, embodiments of this application provide a physical model-based method for endoscopic artifact segmentation. The method includes: acquiring endoscopic images and performing radiometric and geometric calibration on the endoscopic imaging system to obtain endoscopic calibration parameters, which include light source brightness, camera response curve, camera intrinsic parameters, and light source direction vector; preprocessing the acquired endoscopic images: suppressing imaging noise using noise suppression and gamma correction, and converting the image from RGB space to a color space matched by the physical model to separate diffuse reflectance from incident illumination; for the preprocessed endoscopic images, establishing a physical reflection model based on Fresnel's law and microsurface reflection theory, where the brightness of each pixel is the superposition of diffuse and specular reflection terms, and using a shape-to-brightness inference algorithm to estimate the local surface area of ​​each pixel. The surface normal vector is used to solve for the specular reflection coefficient and diffuse reflectance based on the endoscope calibration parameters and the local surface normal vector. The brightness deviation and saturation information of each pixel in the physical reflection model are analyzed, and a specular probability map is constructed by combining the specular reflection coefficient of each pixel. An adaptive threshold is used to segment the specular probability map to obtain an initial specular mask. The brightness deviation is the deviation between the pixel brightness and the global brightness of the image. A physical morphological structure kernel is constructed based on the light source direction vector and the local surface normal vector of each pixel. The initial specular mask is morphologically optimized to obtain a target specular mask. Based on the target specular mask, the target specular region is located in the preprocessed endoscope image, and the artifact segmentation result is output through pixel segmentation.

[0007] Secondly, embodiments of this application provide an endoscopic artifact segmentation system based on a physical model. The system includes: an image acquisition and calibration unit for acquiring endoscopic images and performing radiometric and geometric calibration on the endoscopic imaging system to obtain endoscopic calibration parameters, which include light source brightness, camera response curve, camera intrinsic parameters, and light source direction vector; an image preprocessing unit for preprocessing the acquired endoscopic images: suppressing imaging noise using noise suppression and gamma correction, and converting the image from RGB space to a color space matched by the physical model to separate diffuse reflectance from incident illumination; and a reflection model modeling unit for establishing a physical reflection model for the preprocessed endoscopic images, based on Fresnel's law and microsurface reflection theory, where the brightness of each pixel is a superposition of diffuse and specular reflection terms, and using a shape-to-brightness inference algorithm to estimate the local surface area of ​​each pixel. The system comprises: a surface normal vector, which calculates the specular reflection coefficient and diffuse reflectance based on the endoscope calibration parameters and the local surface normal vector; an initial mask segmentation unit, which analyzes the brightness deviation and saturation information of each pixel in the physical reflection model, and constructs a specular probability map by combining the specular reflection coefficient of each pixel, and segments the specular probability map using an adaptive threshold to obtain an initial specular mask; the brightness deviation is the deviation between the pixel brightness and the global brightness of the image; a target mask segmentation unit, which constructs a physical morphological structure kernel based on the light source direction vector and the local surface normal vector of each pixel, and obtains a target specular mask by morphological optimization of the initial specular mask; and a specular artifact segmentation unit, which locates the target specular region from the preprocessed endoscope image based on the target specular mask, and outputs artifact segmentation results by pixel segmentation.

[0008] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the physical model-based endoscopic artifact segmentation method of any embodiment of this application.

[0009] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the physical model-based endoscopic artifact segmentation method of any embodiment of this application.

[0010] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the physical model-based endoscopic artifact segmentation method of any embodiment of this application.

[0011] The endoscopic artifact segmentation method and system based on a physical model provided in this application can achieve at least the following technical effects:

[0012] (1) By establishing a physical reflection model based on Fresnel's law and micro-surface reflection theory, the brightness of each pixel can be decomposed into diffuse reflection and specular reflection. This can effectively distinguish between the highlight area and the normal tissue area. Furthermore, under different lighting and light source conditions, the parameters of the reflection model can be automatically adjusted based on the endoscope calibration parameters to accurately capture the changes in specular highlights.

[0013] (2) By combining endoscope calibration parameters and image brightness information, the local surface normal vector of each pixel is estimated, making the calculation of specular reflection coefficient and diffuse reflectance more accurate. This not only enhances the ability to predict specular highlights, but also automatically adapts to different imaging conditions under large changes in illumination, thus improving the robustness of artifact segmentation.

[0014] (3) By analyzing the brightness deviation and saturation information of each pixel and combining it with the specular reflection coefficient, a specular probability map was constructed. An adaptive threshold segmentation method was used to extract a preliminary specular mask from it. By dynamically adjusting the threshold, the specular region could be accurately identified and segmented in complex scenes. After the preliminary specular probability map segmentation, the specular mask was morphologically optimized by constructing a physical morphological structure kernel, thereby further refining the extraction of artifact regions and ensuring that the final artifact segmentation result was more accurate.

[0015] This technical solution, through the introduction of a physical reflection model combined with morphological optimization and adaptive thresholding, achieves precise segmentation of highlight regions. It not only accurately reflects the optical imaging process of endoscopic images, avoiding the black-box reasoning process in deep learning, but also provides high-quality input data for automated analysis algorithms, further improving endoscopic image quality and diagnostic accuracy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an example of a physical model-based endoscopic artifact segmentation method according to an embodiment of this application is shown.

[0018] Figure 2 A flowchart illustrating an example of endoscopic image artifact restoration processing according to an embodiment of this application is shown.

[0019] Figure 3 A flowchart illustrating an example of physical reflection modeling according to an embodiment of this application is shown.

[0020] Figure 4 A simulation illustration shows an example of the comparison between the average IoU values ​​of various baseline methods and the IoU values ​​of physical model methods under different brightness categories in an image segmentation task.

[0021] Figure 5 This is a schematic diagram illustrating an example of the simulation results, showing a comparison between the average ROC curves of various baseline methods and the ROC curves of the physical model methods in the test set.

[0022] Figure 6 A structural block diagram of an example of a physical model-based endoscopic artifact segmentation system according to an embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] It should be noted that, in current related technologies, some experts and scholars have proposed novel technical directions for medical image processing and analysis. Specifically, these mainly include the following directions: artifact detection based on brightness classification and adaptive thresholding, content-aware adaptive image inpainting algorithms, and artifact removal methods based on partial attention networks.

[0025] In artifact detection based on brightness classification and adaptive thresholding, specular reflections in endoscopic images severely interfere with physician observation and computer vision algorithms. Existing specular highlight detection methods are generally only applicable to images with medium brightness, while specular highlight removal methods are only suitable for images without large reflective areas. For high-resolution and textured medical images, these methods not only produce poor restoration results but also suffer from low computational efficiency. To address these issues, this paper proposes an adaptive thresholding detection scheme based on brightness classification: first, the image is classified into brightness categories (high, medium, and low), and then an adaptive threshold function that varies with brightness is used to detect absolute specular regions. In the restoration stage, the sample filling priority function is improved, and a local priority search strategy is introduced to enhance efficiency. Although this method achieves some success under different brightness levels, the algorithm relies on setting thresholds and empirical classifications, making it difficult to handle complex lighting changes and extreme reflection conditions.

[0026] In content-aware adaptive image inpainting algorithms, a content-aware specular reflection suppression method based on adaptive image inpainting and neural networks is proposed. This method includes three main steps: reflection detection, reflection region classification, and reflection occlusion. Specifically, a threshold plus morphological dilation is used to locate specular regions, and an adaptive image inpainting algorithm is applied to different regions. The inpainting parameters are adjusted by analyzing local image content. This method can automatically and accurately locate and occlude specular reflection regions in endoscopic images and outperforms some traditional methods in terms of the Structural Similarity Index Measure (SSIM). However, this algorithm still relies on empirical thresholds and local texture analysis, and the inpainting process lacks physical constraints, especially when dealing with complex surface reflections and irregularly shaped highlight areas, which can easily lead to artifacts or over-smoothing.

[0027] In this artifact removal method based on a partial attention network, deep learning is used for specular highlight removal. First, the image is divided into highlight and non-highlight regions using brightness thresholding. Then, a partially convolutional network with an integrated attention mechanism is used to repair the highlight regions, and a mask dataset containing random jump points is constructed for training. Experimental comparisons show that this method outperforms several traditional segmentation and repair methods in both subjective and objective metrics, and can achieve specular highlight removal on irregularly shaped surfaces. However, this method relies on a large amount of training data and a complex model, resulting in high training overhead. More importantly, this network primarily learns reflection patterns from the data, lacking explicit modeling and interpretation capabilities of the physical processes of endoscopic imaging, and is sensitive to changes in light source and instrument position.

[0028] In addition, the academic community has proposed various specular removal methods based on image decomposition, gradient domain inpainting, or generative adversarial networks. For example, research has employed Sobolev image inpainting combined with adaptive segmentation, introducing the Sobolev inner product within a partial differential equation framework to achieve texture smoothing; other methods use Cycle-GAN or Mask-guided Cycle-GAN for unsupervised specular removal. Although these methods improve the realism of the restorations, they still rely heavily on training data and network structure, and the generated results may exhibit instability.

[0029] Therefore, current research on related technologies mostly focuses on algorithmic frameworks, network structures, or image enhancement, while lacking in-depth consideration of endoscopic imaging physical models, light source distribution, and tissue surface reflectivity. Such algorithms, lacking physical constraints, are prone to failure when faced with complex scenarios (such as varying mucosal surface morphology, changing light source angles, or instrument interference).

[0030] It should be understood that the above description of the relevant technologies is intended only to help the public better understand the inventive spirit and motivation of this application, and is not intended to limit this application. Furthermore, the technical solutions described in the above-mentioned relevant technologies are not prior art, and may also be undisclosed technical solutions, such as those under research or in the laboratory stage.

[0031] The technical solutions in this application, including the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information, comply with relevant laws and regulations and do not violate public order and good morals.

[0032] Figure 1 A flowchart illustrating an example of a physical model-based endoscopic artifact segmentation method according to an embodiment of this application is shown.

[0033] Regarding the execution subject of the method in the embodiments of this application, it can be any controller or processor with computing or processing capabilities, such as an endoscope artifact segmentation processor. By implementing an artifact segmentation method based on a physical reflection model, combined with endoscope calibration parameters and local surface normal vector estimation, it can accurately and stably segment the specular highlight region. In some examples, it can be integrated into an electronic device or terminal through software, hardware, or a combination of both, and the type of terminal or electronic device can be diverse.

[0034] like Figure 1 As shown, in step S110, endoscopic images are acquired, and the endoscopic imaging system is calibrated radiometrically and geometrically to obtain endoscopic calibration parameters.

[0035] Specifically, endoscopic images need to be acquired through an endoscopic imaging system, typically using the camera module of an endoscope to obtain real-time images of the patient's body. To ensure the accuracy of subsequent image processing, the endoscopic imaging system undergoes radiometric and geometric calibration to obtain calibration parameters for the endoscope, such as light source brightness, camera response curve, camera intrinsic parameters, and light source direction vector. These calibrations eliminate geometric distortions and uneven illumination within the endoscopic imaging system.

[0036] In radiometric calibration, accurate light source brightness versus camera response curves are obtained by measuring the brightness of the light source (including light source intensity and illumination range) and the response characteristics of the camera. The camera response curve describes the nonlinear response of the camera to different brightness signals.

[0037] In geometric calibration, the camera's intrinsic and extrinsic parameters are calibrated to obtain the camera's intrinsic parameters (including focal length, principal point, distortion coefficient, etc.) and the light source direction vector (the direction of the light source relative to the endoscope). These parameters can provide the camera's geometric information and the geometric distortion correction of the image, so as to more accurately reproduce the real scene of the endoscope image.

[0038] In step S120, the acquired endoscopic images are preprocessed: noise suppression and gamma correction are used to suppress imaging noise, and the images are converted from RGB space to a color space that matches the physical model to separate diffuse reflectance from incident illumination.

[0039] In the image preprocessing stage, noise is removed and the effects of nonlinear illumination in imaging are corrected. Simultaneously, the image space is transformed to fit the subsequent physical model. Specifically, endoscopic images are often affected by imaging noise, especially in low-light environments where noise is particularly noticeable. Therefore, image denoising techniques (such as Gaussian filtering and nonlocal mean denoising) are applied to effectively remove noise while preserving as much detail as possible. Furthermore, due to the nonlinear imaging characteristics of endoscopic images, particularly the nonlinear response of the camera to illumination, gamma correction is used in preprocessing to adjust the image brightness values ​​to a linear space. This suppresses nonlinear effects in imaging, making the image brightness closer to the brightness distribution of the real scene.

[0040] Next, the image is converted from the RGB color space to a color space more suitable for physical modeling (such as HSV or Lab). This conversion helps to separate diffuse reflectance (the material properties of the reflective surface) from incident light (which is determined by the direction of the light source and the intensity of the light).

[0041] Therefore, image preprocessing effectively suppresses noise and brightness anomalies, ensuring a clear distinction between highlight areas and background. Color space conversion and brightness adjustment improve the accuracy of highlight area segmentation, which helps to effectively separate highlight areas from normal areas in the image.

[0042] In step S130, for the preprocessed endoscope image, a physical reflection model is established based on Fresnel's law and micro-surface reflection theory, in which the brightness of each pixel is the superposition of diffuse reflection and specular reflection terms. The local surface normal vector of each pixel is estimated using the shape-to-brightness inference algorithm. The specular reflection coefficient and diffuse reflectance are solved based on the endoscope calibration parameters and the local surface normal vector.

[0043] In physical reflection modeling, Fresnel's law and microsurface reflection theory are introduced to establish a physical reflection model that includes the brightness information of each pixel. Fresnel's law describes the reflection and refraction behavior of light at the interface of different media. In endoscopic imaging, due to the difference between specular and diffuse reflection, Fresnel's law can effectively distinguish the contribution of these reflection modes to image brightness. Furthermore, the microsurface model is used to describe the reflection effect of light on the microstructure of an object's surface. In endoscopic images, the reflective characteristics of tissues can often be approximated as a combination of diffuse and specular reflection. The brightness contributions of these two parts are superimposed to obtain the brightness value of each pixel, thus constructing the physical reflection model.

[0044] Furthermore, based on the pixel brightness distribution in the physical reflection model, a shape-from-illumination algorithm is used to estimate the local normal vector of each pixel. This normal vector reflects the surface orientation at that point, helping the system understand the interaction between light and the surface. Then, based on the estimated normal vector and the endoscope calibration parameters, the specular reflection coefficient and diffuse reflectance of each pixel can be calculated. The specular reflection coefficient represents the surface smoothness, while the diffuse reflectance reflects the light scattering characteristics of the surface material.

[0045] By introducing a physical reflection model, the light reflection behavior in the image is reconstructed, especially the specular reflection region, so that the brightness of each pixel can be accurately decomposed into the contributions of specular reflection and diffuse reflection. Thus, by simulating the actual imaging process, the limitations of simple thresholding or color space methods in traditional approaches are reduced, ensuring accurate identification of specular reflection regions under complex lighting conditions.

[0046] In step S140, the brightness deviation and saturation information of each pixel in the physical reflection model are analyzed, and a specular probability map is constructed by combining the specular reflection coefficient of each pixel. An adaptive threshold is used to segment the specular probability map to obtain the initial specular mask.

[0047] Here, a specular probability map is constructed based on the specular reflection coefficient and diffuse reflectance calculated by the physical reflection model, and then segmented to obtain a preliminary specular mask.

[0048] In the physical reflection model, there is a certain deviation between the brightness of each pixel and the global brightness of the image. This brightness deviation reflects whether the pixel is affected by strong specular reflection. By combining these deviations with the specular reflection coefficient of each pixel, a specular probability map can be obtained, which accurately represents the probability that each pixel is specularly reflective.

[0049] Furthermore, based on the values ​​of the specular probability map, an adaptive threshold segmentation algorithm dynamically sets a threshold to distinguish between highlight and non-highlight regions. Unlike fixed threshold methods, the adaptive threshold comparison method can flexibly adjust according to the specific brightness distribution of the image model, making the initial segmentation of highlight regions more accurate.

[0050] In step S150, a physical morphological structure kernel is constructed based on the light source direction vector and the local surface normal vector of each pixel, and the target specular mask is obtained by morphological optimization of the initial specular mask.

[0051] Here, a physical morphology structure kernel is used to optimize the initial specular mask, resulting in a more accurate target specular mask.

[0052] Specifically, by combining the light source direction vector and the local surface normal vector of each pixel, a morphological structure kernel adapted to endoscopic imaging is constructed. This structure kernel is used to process the morphological optimization of the highlight region, further eliminating missegmentation caused by noise or edge effects.

[0053] For example, morphological operations (such as dilation and erosion) are used to optimize the initial specular mask, repairing the broken areas and removing artifacts in non-target specular areas, thereby greatly improving the accuracy of artifact segmentation and ensuring that the final specular mask is more consistent with the real specular distribution.

[0054] In step S160, the target highlight region is located from the preprocessed endoscope image based on the target highlight mask, and the artifact segmentation result is output through pixel segmentation.

[0055] Here, based on the optimized target highlight mask, the target highlight region is located from the preprocessed endoscopic image, and the final artifact segmentation result is output through pixel-level segmentation. The preprocessed image undergoes noise removal and color correction to avoid interference from artifacts and background noise in the original image on highlight region segmentation. Specifically, the highlight region in the preprocessed endoscopic image is located according to the target highlight mask, and through pixel-level fine segmentation, an accurate mask of the highlight region is finally generated, which can effectively remove the interference of the highlight region on other lesion information.

[0056] Through the embodiments of this application, by combining physical reflection models, local normal vector estimation, adaptive segmentation and morphological optimization, the accuracy and stability of artifact segmentation are significantly improved, enabling adaptive and high-precision artifact processing under different exposure conditions.

[0057] Figure 2 A flowchart illustrating an example of endoscopic image artifact restoration processing according to an embodiment of this application is shown.

[0058] In step S210, the target highlight region is located from the preprocessed endoscope image based on the target highlight mask, and the artifact segmentation result is output through pixel segmentation.

[0059] For details regarding artifact segmentation in step S210, please refer to [link / reference]. Figure 1 The description of the operations in the code will not be repeated here.

[0060] In step S220, based on the diffuse reflectance obtained by the physical reflection model, it is determined whether the diffuse reflectance in the target highlight area satisfies the condition of uniform distribution.

[0061] Here, a uniform diffuse reflectance distribution detection mechanism is employed. Based on the diffuse reflectance calculated from the physical reflection model, the diffuse reflectance value of each pixel represents the surface material's ability to scatter light at that location. In highlight areas, uneven surface reflection can lead to noticeable bright patches, while uniform diffuse reflectance usually indicates a relatively stable light distribution, suggesting that the highlight portion of this area has good illumination uniformity, thus allowing for direct repair using simple methods.

[0062] In step S231, when the uniform distribution condition is met, the occluded texture is filled in according to the diffuse reflectance solved by the physical reflection model to output the repaired image.

[0063] When the diffuse reflectance of the target highlight area satisfies the condition of uniform distribution, it indicates that the highlight portion of that area can be repaired using a simple filling method. In this case, the diffuse reflectance, calculated using a physical reflection model, is used to fill in the occluded texture. Specifically, based on the diffuse reflectance values ​​of surrounding pixels, interpolation methods (such as bilinear interpolation or higher-order interpolation) are employed to fill in the texture within the highlight area. Thus, by utilizing the uniform distribution of diffuse reflectance, the repair process can maintain the naturalness and consistency of the image.

[0064] In step S233, when the uniform distribution condition is not met, the diffuse reflectance of the target highlight region is optimized using a physically constrained image inpainting model, and the occluded texture is filled in to output the inpainted image.

[0065] When the diffuse reflectance of the target highlight area does not meet the condition of uniform distribution, the restoration process requires more complex and refined processing. In this case, a physically constrained image restoration model is used to optimize the diffuse reflectance of the target highlight area. Specifically, by using physical constraints (such as illumination consistency and surface normal consistency), the diffuse reflectance of the target area is optimized, resulting in a better match between the restored texture and the surrounding area. For example, the image restoration model uses an iterative optimization method, based on the diffuse reflectance and illumination information of the surrounding area, to continuously adjust the texture of the target highlight area until it meets the physical constraints, ensuring the naturalness and accuracy of the restoration effect.

[0066] Regarding the implementation details of step S110, in some examples of embodiments of this application, by photographing standard grayscale plates with different reflectivities and recording the actual illuminance, the camera response function of the endoscope camera and the reference luminance of the light source are solved based on the correspondence between luminance and digital output; wherein, the camera response function reflects the camera's response characteristics to different intensities of light, and the reference luminance is the luminance of the light source under standard illumination conditions. Furthermore, the intrinsic parameters and distortion parameters of the endoscope camera are obtained using a checkerboard calibration plate, and the light source direction vector is estimated based on the known positions of the geometric calibration plate and the reflectivity standard grayscale plate.

[0067] Specifically, a standard grayscale plate with known reflectivity is selected, such as a black-and-white grayscale plate whose grayscale values ​​cover the range of illumination intensity from pure black to pure white. The reflectivity of each region on the grayscale plate is known, thus serving as a calibration reference. Images of the standard grayscale plate are captured under different illumination conditions to ensure coverage of various exposure conditions from low to high light, facilitating the consideration of light source intensity under different brightness conditions during calibration. The actual illuminance value in each acquired image is recorded using a standard illuminance meter, ensuring that the brightness value output by the camera matches the true illuminance value. This achieves accurate calibration of the camera response function and the reference brightness of the light source, ensuring that the light source intensity and the camera's response to illumination in the endoscopic images can be accurately represented by a mathematical model.

[0068] Then, a brightness-direction calibration model for the light source is established. By capturing reflection information from multiple light source angles, the light source direction vector corresponding to each light source angle can be calculated. By combining camera response and reference brightness, a model relating light source brightness to direction is established.

[0069] Specifically, by capturing endoscopic images from different angles, the position of the light source and its relative position to the optical center of the camera are recorded, thereby estimating the direction vector of the light source.

[0070]

[0071] In the formula, It is the direction vector of the light source. It is the position of the light source. It is the optical center of the endoscope. Normalized(·) represents the normalization operation, and the calculation result will provide an accurate geometric model for the light source direction vector.

[0072] Based on known light source intensity and standard illuminance, a light source brightness-direction calibration model is established. The relationship between the light source brightness and direction at different light source angles is calibrated using the following formula:

[0073]

[0074] In the formula, I raw For the intensity of the acquired uncalibrated image, C camera For the camera response function, I baseline I is the reference brightness of the light source. calibrated This represents the calibrated brightness of the light source.

[0075] By establishing the relationship between light source brightness and direction, the calibration process of the light source was optimized, thereby enabling precise correction of the light source direction and brightness in endoscopic images. This ensures that the endoscopic system can provide stable and high-quality image data under different lighting conditions.

[0076] Specifically, using a standard checkerboard calibration board and leveraging known geometry and corner extraction methods, the focal length, principal point coordinates, and distortion coefficients of the endoscopic camera are calculated. These parameters are then used to describe the camera's optical model.

[0077] [u,v]=[f x ,f y ,c x ,c y [X,Y,Z] T Equation (3)

[0078] In the formula, [u,v] represents the image coordinates, [X,Y,Z] represents the world coordinates, and f x ,f y c is the focal length. x ,c y Using the principal point coordinates, the correspondence between each pixel in the image and the actual scene is ensured.

[0079] Therefore, by combining the checkerboard calibration plate for geometric calibration, accurate camera intrinsic parameters and distortion parameters can be obtained, which further improves the accuracy of the light source direction vector estimation, thereby ensuring the geometric accuracy and illumination consistency of the endoscopic image.

[0080] Regarding the details of image preprocessing in step S120, imaging noise in the endoscopic image is reduced through noise suppression and gamma correction, and the image is converted to a color space that matches the physical model through color space conversion.

[0081] Specifically, Gaussian filtering is used for filtering:

[0082]

[0083] In the formula, G(i,j) is the Gaussian kernel, k is the filter size, and I(x+i,y+j) is the neighboring pixel value.

[0084] Furthermore, gamma correction is used to correct the uneven brightness of the endoscopic image, making the brightness distribution of the image closer to the natural brightness perceived by the human eye. In addition, the gamma value is set according to the brightness and contrast requirements of the imaging system, for example, between 0.4 and 2.2.

[0085] Furthermore, diffuse and specular reflection components in endoscopic images are often mixed in the RGB color space, making subsequent reflection separation difficult. To address this issue, color space conversion can transform the image from RGB space to a physically matched color space (such as CIE LAB or XYZ space), separating the diffuse reflection component from the incident light. For example, converting the image from RGB space to CIE LAB color space, where L represents the luminance component and A and B represent the chromaticity components, allows for clearer separation of diffuse and specular reflection components, reducing the impact of lighting variations on image analysis and enhancing the robustness of artifact restoration.

[0086] Figure 3 A flowchart illustrating an example of physical reflection modeling according to an embodiment of this application is shown.

[0087] like Figure 3 As shown, in step S310, the incident light direction is calculated based on the light source direction vector to initialize the defined surface normal vector, and a physical reflection model of pixel intensity, diffuse reflectivity and specular reflectivity is established.

[0088] The core of the physical reflection model is to decompose the brightness of an image into diffuse reflection and specular reflection. To build this model, the direction of the light source needs to be accurately calculated. Based on the relationship between the position of the light source and the direction of the incident light, the intensity of the light source is estimated and used to calculate the reflected brightness of each pixel. The diffuse reflection term is simulated using the Lambertian reflection model, and the specular reflection term uses the Fresnel reflectance factor.

[0089] I(x,y)=I d (x,y)+I s(x,y), formula (5)

[0090] I d (x,y)=I inc (x,y)·R d (x,y)·cos(θ i Equation (6)

[0091]

[0092] In the formula, (x,y) represents the position coordinates of any pixel in the preprocessed endoscopic image, and I(x,y) represents the pixel brightness value. d (x,y) represents the luminance value of the diffuse reflection term of a pixel, I s (x,y) represents the brightness value of the specular reflection term of the pixel; θ i It is the angle between the incident light and the surface normal vector, I inc (x,y) is the incident light intensity, I d (x,y) is the diffuse reflection term, R d (x,y) is diffuse reflectance, θ r R is the angle between the reflected light and the surface normal vector, n is the specular index, and R is the surface normal vector. s (x,y) is the specular reflection coefficient.

[0093] By calculating the direction of the light source, the brightness of each pixel in an image can be precisely decomposed into diffuse and specular reflections, thus enabling a better understanding and processing of the reflective components in endoscopic images. This allows for the effective simulation and prediction of reflected light in endoscopic images, improving the accuracy of subsequent image processing and artifact repair.

[0094] In step S320, the local surface normal vector of each pixel is calculated from the brightness information of the physical reflection model using a shape-to-brightness inference algorithm.

[0095] There is a direct relationship between the brightness gradient and the surface normal vector of an image. The Shape-to-Brightness (SFS) algorithm estimates the local surface normal vector of each pixel based on the image's brightness information. The reflection model is then optimized based on this normal vector. The brightness gradient of the image reflects the spatial variation of the surface normal vector.

[0096] Brightness gradient The difference in brightness in the horizontal and vertical directions of the image can be calculated:

[0097]

[0098] In the formula, and These represent unit vectors in the horizontal and vertical directions, respectively. This represents the brightness gradient of the image.

[0099] Based on the brightness gradient, the normal vector for each pixel can be obtained by minimizing the error in the normal vector estimation. Assume the relationship between incident light intensity and image brightness is as follows:

[0100]

[0101] In the formula, This represents the image brightness gradient at pixel (x, y) in the physical reflection model. It is the calculated surface normal vector at pixel (x,y), and I0 is a constant. This represents the direction vector of the light source.

[0102] In some implementations, the surface normal vector is optimized using the least squares method by calculating the brightness gradient in the image and matching it with the actual brightness. The optimization objective is to minimize the error between the estimated normal vector and the image brightness, and regularization is used to smooth the normal vector and reduce the influence of noise. Thus, by optimizing the normal vector, the geometric information of surface reflection can be accurately derived.

[0103] In step S330, the angle between the reflected light and the incident light and the surface normal vector is updated using the calculated local surface normal vector of each pixel to optimize the physical reflection model, and the specular reflection coefficient and diffuse reflectance are solved by minimizing the prediction error of the physical reflection model.

[0104] In some implementations, the specular reflection coefficient and diffuse reflectance are optimized to minimize the difference between the actual image brightness and the predicted error of the physical reflection model. The least squares method is used to optimize the brightness of each pixel, thereby solving for the optimal reflection coefficient. The goal is to optimize the reflection coefficient by minimizing the difference between the actual image brightness and the brightness of the reflection model. The optimization objective is:

[0105]

[0106] In the formula, I pixel (x,y) represents the actual pixel brightness at pixel (x,y) in the preprocessed endoscopic image. s (x,y) * I represents the luminance value of the specular reflection term at pixel (x,y) in the optimized physical reflection model. d (x,y) * Ω represents the luminance value of the diffuse reflection term at pixel (x,y) in the optimized physical reflection model, and Ω is the image domain, representing the set of all pixels in the preprocessed endoscopic image.

[0107] Here, the reflection error of each pixel is calculated using the image pixel set Ω, and the reflection coefficient is adjusted using the least squares method. Total variation regularization is used to ensure the smoothness of the normal vector, avoiding the loss of image details. At the same time, the reflection coefficient is optimized to make the distribution of specular reflection and diffuse reflection more reasonable, thereby improving the accuracy of the reflection model and enabling it to better match the brightness distribution in the actual image.

[0108] It should be noted that in the modeling and optimization of the physical reflection model, the model is initialized by assuming a normal vector. Initially, the light source direction vector is used as the assumed normal vector. By calculating the brightness of diffuse and specular reflection components, a preliminary reflection model can be quickly constructed. Based on this, the SFS algorithm is used to derive the true surface normal vector based on image brightness information, and the normal vector and reflection coefficient are optimized using the least squares method to make the reflection model more accurate. As the normal vector is gradually optimized, the model is continuously improved, ultimately obtaining a reflection coefficient that best matches the actual image brightness, thereby effectively improving the quality of endoscopic images and removing artifacts and noise. Furthermore, the optimization process of the reflection coefficient ensures the accurate calculation of diffuse and specular reflection, which helps to accurately separate specular and diffuse reflection components in endoscopic images.

[0109] Regarding the implementation details of step S140, in some examples of embodiments of this application, a specular probability map is constructed by analyzing the brightness deviation and saturation information in the physical reflection model and combining the reflection characteristics of each pixel.

[0110] P sp (x,y)=α·ΔI(x,y)+β·S(x,y), formula (12)

[0111] ΔI(x,y)=|I pixel (x,y)-I avg |, Equation (13)

[0112]

[0113] In the formula, P sp (x,y) represents the specular probability map value at pixel (x,y), indicating the probability that the pixel is a specular reflection area; ΔI(x,y) represents the pixel brightness deviation, α and β represent the brightness deviation contribution coefficient and saturation contribution coefficient, respectively, S(x,y) represents the pixel saturation, and I max I represents the maximum pixel brightness of all pixels in the physical reflection model. avg This represents the average pixel brightness of all pixels in the physical reflection model, where N represents the total number of pixels in the physical reflection model.

[0114] Here, brightness deviation is the difference between the actual brightness of each pixel and the global brightness of the image. A larger deviation indicates that the pixel is more likely to be a highlight area. Saturation is an important feature of reflected light in an image, representing the degree of "saturation" of pixel brightness. The closer the saturation value is to 1, the closer the pixel is to a reflective highlight area. Based on the brightness deviation and saturation information of each pixel, a specular probability map can be constructed. This image reflects the probability that each pixel will become a specular reflection area; specularly reflecting pixels will have larger brightness deviations and higher saturation. Furthermore, by adjusting α and β, the detection effect for highlight areas can be optimized.

[0115] An adaptive threshold segmentation method is used to calculate the adaptive threshold, and the specular probability map is segmented based on the adaptive threshold to obtain the initial specular mask.

[0116] After the specular probability map is constructed, an adaptive thresholding method is used to extract the initial specular mask. The adaptive threshold can be automatically adjusted according to the local features of the image to accurately segment the specular reflection area under different lighting conditions.

[0117] Specifically, the adaptive threshold can be dynamically adjusted based on the local brightness information of the image. The calculation method is as follows:

[0118] T(x,y)=mean(P sp (x,y))+λ·stddev(P sp (x,y)), formula (16)

[0119] For each pixel, if its specular probability value is greater than the adaptive threshold, then the pixel is considered to belong to the specular reflection region, and an initial specular mask is constructed:

[0120]

[0121] In the formula, H initial (x,y) represents the mask value at pixel position (x,y) in the initial specular mask, T(x,y) represents the pixel adaptive threshold, and mean(P) sp (x,y)) represents the local mean of the mirror probability map, λ represents the threshold sensitivity adjustment coefficient, and stddev(P) sp (x,y) represents the standard deviation of the mirror probability plot.

[0122] By combining brightness deviation and saturation information, specular and diffuse reflection regions can be effectively distinguished. The adaptive thresholding method ensures that the extraction of the highlight mask is unaffected by global exposure changes, and can automatically adjust according to local features, ensuring accurate extraction of highlight regions and achieving more precise quantization output of mask boundaries.

[0123] Regarding the implementation details of step S150, in some examples of embodiments of this application, a physical morphological structure kernel is constructed based on the light source direction vector and the local surface normal vector of each pixel.

[0124] It should be noted that in traditional morphological optimization, the structure kernel (or morphological filter) is designed based on local image features (such as brightness, texture, etc.). However, in this embodiment, a physical morphological structure kernel is constructed by combining the light source direction vector and the local surface normal vector. This allows the structure kernel to better reflect the reflection characteristics of the image. The information-driven structure kernel design based on the physical reflection model enables morphological optimization to be adjusted according to real lighting and reflection conditions, thereby improving the extraction accuracy of highlight areas.

[0125]

[0126] In the formula, K(x,y) represents the physical morphological structure kernel at pixel position (x,y), and θ s It is the angle between the light source direction vector and the pixel surface normal vector at (x,y). This is an indicator function used to determine whether the position of a pixel meets the reflection condition based on the surface normal vector and the light source direction vector of the pixel position (x,y).

[0127] Here, the value of the indicator function depends on the relationship between the light source direction vector and the surface normal vector. If the angle between the light source and the normal vector is less than the set reflection angle threshold, then the pixel is a reflective area and the indicator function value is 1; otherwise, the indicator function value is 0.

[0128] The angle θ between the light source direction and the normal vector at each pixel is calculated. s By combining the indicator function, a physical morphological structure kernel corresponding to each pixel is constructed, which enables the structure kernel to be adjusted according to the local reflection characteristics of the image, thereby playing a precise optimization role in image processing.

[0129] The initial specular mask is subjected to dilation and erosion operations using a physical morphology structure core to optimize the specular region, thereby obtaining the target specular mask.

[0130] The mask area is expanded by dilation. If the structure kernel overlaps with the current pixel, the mask value at that pixel location is 1 (indicating it belongs to the target area); otherwise, it is 0. Furthermore, the mask area is shrunk by erosion. The mask value is 1 only when the structure kernel completely covers the current pixel; otherwise, it is 0.

[0131]

[0132] Morphological dilation can expand the initial highlight area, especially for partially occluded highlight areas, helping to restore these areas and make the highlight area more complete. Erosion, on the other hand, is used to refine the boundaries of highlight areas, avoiding over-expansion caused by dilation. Erosion removes over-expanded portions of the image, preserving the precise highlight area.

[0133] H final (x,y) = K(x,y)! H dilated (x,y), formula (20)

[0134] In the formula, The symbol '!' represents the dilation operation in morphology, '!' represents the erosion operation in morphology, and 'H' represents the erosion operation. dilated (x,y) represents the specular mask after dilation, H final (x,y) represents the target specular mask after expansion and erosion operations.

[0135] The embodiments of this application effectively combine the information of the light source and the surface normal vector, so that morphological operations are not limited to local image features, but can also be adjusted based on the laws of physical reflection, which can dynamically adapt to the reflection characteristics of different scenes and improve the effect of image restoration and detail recovery.

[0136] Regarding the implementation details of endoscopic image artifact repair processing, in some examples of embodiments of this application, the uniform distribution condition is defined based on the standard deviation of diffuse reflectance within the target highlight region, which is calculated using diffuse reflectance obtained from a physical reflectance model.

[0137] Therefore, by calculating the standard deviation of diffuse reflectance within the highlight region of the target, it can be determined whether the diffuse reflectance estimation of that region is stable.

[0138]

[0139] In the formula, Ω s It is the highlight area of ​​the target. It represents the standard deviation of diffuse reflectance within the highlight region of the target.

[0140] when When the value of T indicates that the diffuse reflectance estimation result obtained through the physical reflectance model is stable, the diffuse reflectance obtained through the physical reflectance model can be directly used to repair the highlight area of ​​the target; threshold This indicates the set threshold for uniform distribution.

[0141] By using standard deviation to ensure that the repair process is only performed when necessary, the repair efficiency is improved, and over-repair of accurate areas is avoided. This can effectively reduce the possibility of incorrect repair and ensure that the texture of the highlight area is restored to a natural and smooth state.

[0142] when If the result is unstable, it indicates that the diffuse reflectance estimation result obtained by the physical reflection model is unstable. Therefore, the diffuse reflectance obtained by the physical reflection model is optimized by using the Poisson equation.

[0143]

[0144] The weighting function w(x,y) controls the intensity of the restoration process, ensuring that image edges and areas with strong textures receive more restoration effects.

[0145]

[0146] In the formula, Let represent the gradient operator, w(x,y) represent the local weight function, and η represent the adjustment parameter. This represents the maximum value of the global gradient in the preprocessed endoscopic image. This represents the local gradient of a pixel.

[0147] By optimizing the diffuse reflectance, artifacts can be eliminated while preserving detail in highlight areas. The Poisson equation ensures that the diffuse reflectance restoration results in highlight areas transition smoothly and naturally with the surrounding areas.

[0148] Pixel-level image restoration is performed based on the diffuse reflectance of each pixel in the highlight region of the target.

[0149] I repaired (x,y)=R d (x,y)·I inc (x,y)+I s (x,y) * Equation (24)

[0150] In the formula, I repaired (x,y) represents the repaired image pixel value at pixel position (x,y).

[0151] Here, the restored image combines optimized diffuse reflectance and incident light intensity information, enabling the final image to recover more realistic details and have a natural brightness transition.

[0152] This application's embodiments introduce a discrimination mechanism based on the standard deviation of diffuse reflectance, which can automatically identify areas where the diffuse reflectance estimation results are unstable during the restoration process. When the standard deviation of diffuse reflectance exceeds a set threshold, it indicates that the diffuse reflectance estimation for that area is unstable, triggering a Poisson equation optimization process based on the physical reflection model. This process smooths and optimizes the diffuse reflectance of the target area, ensuring a smooth spatial transition and consistency with the texture of the surrounding areas. The optimized diffuse reflectance is then combined with the incident light intensity and the brightness of the specular reflection portion to generate the final restored image. Thus, by combining the physical characteristics of the image with local image information, artifacts are effectively reduced, and the natural transition of highlight areas is restored, ensuring a smooth brightness transition for the overall image.

[0153] To verify the effectiveness of the proposed physical model artifact segmentation method, this paper designs a series of comparative experiments. The experiments use the publicly available EndoScene dataset and 200 gastrointestinal endoscopic images collected from a hospital. Highlight regions are labeled as ground truth, and the following methods are compared:

[0154] Brightness classification with adaptive thresholding (BC), i.e., the method of Nie et al.;

[0155] Content-aware adaptive inpainting (CAI), namely the method of Shen et al.;

[0156] PatNet, also known as the method of Zhang et al.;

[0157] The physical modeling method proposed in this paper is called Physical.

[0158] Evaluation indicators

[0159] The following metrics were used to evaluate the precision and quality of specular segmentation:

[0160] IoU (Intersection over Union): The ratio of the intersection of the segmented region and the true value;

[0161] Precision and Recall: Reflect the accuracy of highlight area detection;

[0162] F1 score: Harmonic mean of precision and recall;

[0163] SSIM (Structural Similarity): The structural similarity between the restored image and the original image without highlights;

[0164] Running time: Evaluate algorithm efficiency.

[0165] Segmentation performance comparison

[0166] Figure 4This is a simulation illustration showing an example of the comparison between the average IoU values ​​of baseline methods and the IoU values ​​of physical model methods under different brightness categories in an image segmentation task.

[0167] like Figure 4 As shown, the baseline method exhibits relatively low IoU scores under low and high brightness conditions, but performs better under medium brightness conditions. In contrast, the proposed physical model method achieves IoU scores exceeding 0.85 across all brightness categories, demonstrating stability and superiority in both low and high brightness scenes, significantly outperforming the baseline method. Therefore, the physical model method demonstrates stronger robustness and performance when handling image segmentation tasks under varying brightness conditions.

[0168] Figure 5 This diagram illustrates a simulation result comparing the average ROC (Receiver Operating Characteristic) curves of the baseline methods with those of the physical model method in the test set. The horizontal axis represents the False Positive Rate (FPR), and the vertical axis represents the True Positive Rate (TPR). By comparing the ROC curves of the two methods, it can be seen that the curve of the proposed method is significantly better than that of the baseline methods, exhibiting a higher TPR and a lower FPR. Especially when the false positive rate is low, the proposed method can maintain a high true positive rate, indicating that the physical model method has stronger detection capabilities and higher accuracy under high brightness or complex lighting conditions. This further verifies the superiority and robustness of the physical model method in artifact highlight region segmentation.

[0169] Table 1. Comparison of performance metrics of different methods on the test set.

[0170]

[0171] As shown in Table 1, our proposed method significantly outperforms other methods in IoU, F1 score, and SSIM. Although computation involves physical modeling, its average runtime is only slightly higher than the thresholding method and far lower than PatNet's deep network training due to the use of linear solutions and local optimization. It should be noted that the BC method has low recall in high-brightness scenes; the CAI method has a long runtime due to the complexity of region classification and local search; and PatNet's training time is enormous and difficult to apply in real-time.

[0172] This paper addresses the problem of specular highlight artifacts in endoscopic images severely impacting diagnosis and automated analysis, proposing a physical model-based artifact segmentation method. First, the endoscopic equipment is radiatively and geometrically calibrated to establish the physical relationship between incident light and pixel intensity. Then, the specular coefficient and diffuse reflectance are updated by inferring brightness from the simplified Fresnel reflection model and shape. A probabilistic fusion strategy is used to generate a specular probability map, which is then combined with adaptive thresholding and physical morphological operations to achieve accurate segmentation. Finally, texture restoration is performed using the weighted Poisson equation under physical constraints. Experimental results show that this method significantly outperforms existing techniques in both segmentation accuracy and restoration quality, and also exhibits good operational efficiency and generalization ability.

[0173] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0174] Figure 6 A structural block diagram of an example of a physical model-based endoscopic artifact segmentation system according to an embodiment of this application is shown.

[0175] like Figure 6 As shown, the endoscopic artifact segmentation system 600 based on a physical model includes an image acquisition and calibration unit 610, an image preprocessing unit 620, a reflection model modeling unit 630, an initial mask segmentation unit 640, a target mask segmentation unit 650, and a specular artifact segmentation unit 660.

[0176] The image acquisition and calibration unit 610 is used to acquire endoscopic images and perform radiometric and geometric calibration on the endoscopic imaging system to obtain endoscopic calibration parameters, which include light source brightness, camera response curve, camera intrinsic parameters and light source direction vector.

[0177] The image preprocessing unit 620 is used to preprocess the acquired endoscopic images: suppressing imaging noise using noise suppression and gamma correction, and converting the image from RGB space to a color space that matches the physical model to separate diffuse reflectance from incident illumination.

[0178] The reflection modeling unit 630 is used to establish a physical reflection model for the preprocessed endoscope image, based on Fresnel's law and micro-surface reflection theory, in which the brightness of each pixel is superimposed by the diffuse reflection term and the specular reflection term. It uses a shape-to-brightness inference algorithm to estimate the local surface normal vector of each pixel, and solves the specular reflection coefficient and diffuse reflectance according to the endoscope calibration parameters and the local surface normal vector.

[0179] The initial mask segmentation unit 640 is used to analyze the brightness deviation and saturation information of each pixel in the physical reflection model, and construct a specular probability map by combining the specular reflection coefficient of each pixel. The specular probability map is segmented by an adaptive threshold to obtain an initial specular mask; the brightness deviation is the deviation between the pixel brightness and the global brightness of the image.

[0180] The target mask segmentation unit 650 is used to construct a physical morphological structure kernel based on the light source direction vector and the local surface normal vector of each pixel, and obtain the target specular mask by performing morphological optimization on the initial specular mask.

[0181] The specular artifact segmentation unit 660 is used to locate the target specular region from the preprocessed endoscope image based on the target specular mask, and output the artifact segmentation result through pixel segmentation.

[0182] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, server, or network device) to perform the steps of any of the physical model-based endoscopic artifact segmentation methods described above.

[0183] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of any of the above-described physical model-based endoscopic artifact segmentation methods.

[0184] In some embodiments, this application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform steps of a physical model-based endoscopic artifact segmentation method.

[0185] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0186] The electronic devices in this application can exist in various forms, including but not limited to: mobile communication devices, ultra-mobile personal computer devices, portable entertainment devices, or other airborne electronic devices with data interaction functions.

[0187] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for segmenting endoscopic artifacts based on a physical model, characterized in that, The method includes: Acquire endoscopic images and perform radiometric and geometric calibration on the endoscopic imaging system to obtain endoscopic calibration parameters, which include light source brightness, camera response curve, camera intrinsic parameters, and light source direction vector; The acquired endoscopic images are preprocessed: noise suppression and gamma correction are used to suppress imaging noise, and the images are converted from RGB space to a color space that matches the physical model to separate diffuse reflectance from incident light. For the preprocessed endoscopic images, a physical reflection model is established based on Fresnel's law and microsurface reflection theory, in which the brightness of each pixel is the superposition of diffuse reflection and specular reflection terms. The local surface normal vector of each pixel is estimated using a shape-to-brightness inference algorithm. The specular reflection coefficient and diffuse reflectance are solved based on the endoscope calibration parameters and the local surface normal vector. The brightness deviation and saturation information of each pixel in the physical reflection model are analyzed, and a specular probability map is constructed based on the brightness deviation and saturation information. An adaptive threshold is used to segment the specular probability map to obtain an initial specular mask; the brightness deviation is the deviation between the pixel brightness and the global brightness of the image. A physical morphological structure kernel is constructed based on the light source direction vector and the local surface normal vector of each pixel. The target specular mask is obtained by morphological optimization of the initial specular mask. Based on the target highlight mask, the target highlight region is located in the preprocessed endoscope image, and the artifact segmentation result is output through pixel segmentation.

2. The method according to claim 1, characterized in that, After locating the target highlight region from the preprocessed endoscope image based on the target highlight mask and outputting artifact segmentation results through pixel segmentation, the method further includes: Based on the diffuse reflectance obtained through the physical reflection model, it is determined whether the diffuse reflectance in the target highlight area satisfies the condition of uniform distribution. When the uniform distribution condition is met, the occluded texture is filled in according to the diffuse reflectance solved by the physical reflection model to output the repaired image; When the uniform distribution condition is not met, the diffuse reflectance of the target highlight region is optimized using a physically constrained image inpainting model, and the occluded texture is filled in to output the inpainted image.

3. The method according to claim 1, characterized in that, The process of performing radiometric and geometric calibration on the endoscopic imaging system to obtain endoscopic calibration parameters includes: By photographing standard grayscale plates with different reflectivities and recording actual illuminance, the camera response function of the endoscopic camera and the reference illuminance of the light source are calculated based on the correspondence between illuminance and digital output. The camera response function reflects the camera's response characteristics to light of different intensities, and the reference illuminance is the illuminance of the light source under standard illumination conditions. ， In the formula, The intensity of the acquired uncalibrated image, For camera response function, The reference brightness of the light source The calibrated light source brightness; The intrinsic parameters and distortion parameters of the endoscope camera are obtained using a checkerboard calibration plate, and the light source direction vector is estimated based on the known positions of the geometric calibration plate and the reflectivity standard grayscale plate.

4. The method according to claim 1, characterized in that, For the preprocessed endoscopic image, a physical reflection model is established based on Fresnel's law and microsurface reflection theory, where the brightness of each pixel is the superposition of diffuse and specular reflection terms. A shape-to-brightness inference algorithm is used to estimate the local surface normal vector of each pixel. The specular reflection coefficient and diffuse reflectance are then solved based on the endoscopic calibration parameters and the local surface normal vector, including: The incident light direction is calculated based on the light source direction vector to initialize the defined surface normal vector, and a physical reflection model is established based on pixel intensity, diffuse reflectance, and specular reflectance coefficient. The diffuse reflectance term is simulated using the Lambertian reflection model, and the specular reflectance term uses the Fresnel reflectance factor. ， ， ， In the formula, This represents the position coordinates of any pixel in the preprocessed endoscopic image. Represents the brightness value of a pixel. The brightness value representing the diffuse reflection term of a pixel. The brightness value representing the specular reflection of a pixel; It is the angle between the incident light and the surface normal vector. It is the intensity of the incident light. It is the diffuse reflection term. It is diffuse reflectance. The angle between the reflected light and the surface normal vector. Highlight index, The specular reflection coefficient; The local surface normal vector for each pixel is calculated from the brightness information of the physical reflection model using a shape-to-brightness inference algorithm. ， In the formula, For the physical reflection model at the pixel point The image brightness gradient, It is the calculated value at the pixel point Surface normal vector, It is a constant. Represents the direction vector of the light source; The physical reflection model is optimized by updating the angle between the reflected light and the incident light and the surface normal vector using the calculated local surface normal vector of each pixel. The specular reflection coefficient and diffuse reflectance are then solved by minimizing the prediction error of the physical reflection model. ， ， In the formula, This indicates the pixel position in the preprocessed endoscopic image. The actual pixel brightness This indicates the optimized physical reflection model at the pixel level. The brightness value of the specular reflection term. This indicates the optimized physical reflection model at the pixel level. The luminance value of the diffuse reflection term. The image domain represents the set of all pixels in the preprocessed endoscopic image.

5. The method according to claim 1, characterized in that, The process involves analyzing the brightness deviation and saturation information of each pixel in the physical reflection model, constructing a specular probability map based on the brightness deviation and saturation information, and segmenting the specular probability map using an adaptive threshold to obtain an initial specular mask, including: By analyzing the brightness deviation and saturation information in the physical reflection model and combining it with the reflection characteristics of each pixel, a specular probability map is constructed: ， ， ， ， In the formula, This indicates the pixel position in the preprocessed endoscopic image. The actual pixel brightness Indicates at pixel point The specular probability map value at a given location represents the probability that the pixel is a specular reflection area; Indicates pixel brightness deviation. and These represent the contribution coefficients of brightness deviation and saturation, respectively. Indicates pixel saturation. This represents the maximum pixel brightness of all pixels in the physical reflection model. This represents the average pixel brightness of all pixels in the physical reflection model. N This represents the total number of pixels in the physical reflection model. Indicates the pixel index. Indicates the first The position coordinates of each pixel; An adaptive threshold is calculated using an adaptive threshold segmentation method, and the specular probability map is segmented based on the adaptive threshold to obtain an initial specular mask. ， ， In the formula, Indicates the pixel position in the initial specular mask The mask value, Indicates pixel adaptive threshold, This represents the local mean of the mirror probability plot. This represents the threshold sensitivity adjustment coefficient. This represents the standard deviation of the mirror probability plot.

6. The method according to claim 1, characterized in that, The step of constructing a physical morphological structure kernel based on the light source direction vector and the local surface normal vector of each pixel, and obtaining the target specular mask by morphological optimization of the initial specular mask, includes: Based on the light source direction vector and the local surface normal vector of each pixel, a physical morphological structure kernel is constructed: ， In the formula, Indicates the position of a pixel. The physical morphological structure of the nucleus. It is the direction vector of the light source and The angle between the pixel surface normals at that location. This is an indicator function used to determine the position of a pixel. The surface normal vector and the light source direction vector are used to determine whether the position of the corresponding pixel meets the reflection condition; Represents pixels The local surface normal vector at that location; Represents the direction vector of the light source; A physical morphology structure kernel is used to perform dilation and erosion operations on the initial specular mask to optimize the specular region, thereby obtaining the target specular mask: ， ， In the formula, This refers to the dilation operation in morphology. This refers to the erosion operation in morphology. Represents the number of pixels in the initial specular mask. Mask value at that location, This represents the specular mask after the dilation operation. This indicates the target specular mask after expansion and etching operations.

7. The method according to claim 2, characterized in that, The uniform distribution condition is defined based on the standard deviation of diffuse reflectance within the target's highlight region, which is calculated using the diffuse reflectance obtained from the physical reflectance model. ， In the formula, It is the highlight area of ​​the target. The standard deviation of diffuse reflectance within the highlight region of the target; Indicates diffuse reflectance; Indicates the highlight area of ​​the target Diffuse reflectance of each pixel Calculate the standard deviation; when When the result of the diffuse reflectance estimation obtained by the physical reflectance model is stable, the diffuse reflectance obtained by the physical reflectance model can be directly used to repair the highlight area of ​​the target. This indicates the set threshold for uniform distribution; when If the result indicates that the diffuse reflectance estimate obtained through the physical reflectance model is unstable, then the diffuse reflectance obtained through the physical reflectance model is optimized using the Poisson equation: ， ， In the formula, Represents the gradient operator. Represents the local weight function. Indicates the adjustment parameter. This represents the maximum value of the global gradient in the preprocessed endoscopic image. For pixel local gradients; Pixel-level image inpainting is performed based on the diffuse reflectance of each pixel in the highlight region of the target: ， In the formula, Indicates pixel position The restored image pixel values, Indicates the intensity of the incident light; This indicates the optimized physical reflection model at the pixel level. The brightness value of the specular reflection term at that location.

8. An endoscopic artifact segmentation system based on a physical model, characterized in that, include: The image acquisition and calibration unit is used to acquire endoscopic images and perform radiometric and geometric calibration on the endoscopic imaging system to obtain endoscopic calibration parameters, which include light source brightness, camera response curve, camera intrinsic parameters and light source direction vector. The image preprocessing unit is used to preprocess the acquired endoscopic images: suppressing imaging noise using noise suppression and gamma correction, and converting the image from RGB space to a color space that matches the physical model to separate diffuse reflectance from incident illumination. The reflection modeling unit is used to establish a physical reflection model for the preprocessed endoscope image, based on Fresnel's law and micro-surface reflection theory, in which the brightness of each pixel is the superposition of diffuse reflection and specular reflection terms. It uses a shape-to-brightness inference algorithm to estimate the local surface normal vector of each pixel, and solves the specular reflection coefficient and diffuse reflectance based on the endoscope calibration parameters and the local surface normal vector. An initial mask segmentation unit is used to parse the brightness deviation and saturation information of each pixel in the physical reflection model, and construct a specular probability map based on the brightness deviation and saturation information. An adaptive threshold is used to segment the specular probability map to obtain an initial specular mask. The brightness deviation is the deviation between pixel brightness and the overall brightness of the image; The target mask segmentation unit is used to construct a physical morphological structure kernel based on the light source direction vector and the local surface normal vector of each pixel, and obtain the target specular mask by performing morphological optimization on the initial specular mask; The specular artifact segmentation unit is used to locate the target specular region from the preprocessed endoscope image based on the target specular mask, and output the artifact segmentation result through pixel segmentation.

Images