An endoscope image self-supervised depth estimation method based on a physical light compensation model

Abstract

The application relates to the technical field of computer vision and medical image processing, and discloses an endoscope image self-supervision depth estimation method based on a physical light supplement model, which aims to solve the problem that local dark light in an endoscope scene image leads to inaccurate depth estimation, the method comprising the following steps: introducing at least one virtual light source to establish a physical light supplement model during an endoscope examination process of an object; acquiring video data sets of the endoscope examination process under conditions of not containing the physical light supplement model and containing the physical light supplement model respectively; constructing a depth estimation network, the depth estimation network comprising an encoder, a depth decoder, an albedo decoder, a virtual light source multi-layer perception and a depth refinement network, and training the depth estimation network based on self-supervision learning; and inputting an endoscope image to be processed into the trained depth estimation network to output a depth map of the endoscope image to be processed. The method can improve the accuracy of depth estimation of an endoscope image.

Description

Technical Field

[0001] This application relates to the fields of computer vision and medical image processing technology, and more specifically, to a self-supervised depth estimation method for endoscopic images based on a physical illumination model, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Currently, in minimally invasive surgery and endoscopic examinations, the physical size limitations of endoscopes restrict visual information input, often posing challenges to surgeons such as insufficient spatial perception and limited operational precision. Especially in confined and complex diagnostic or surgical environments, visual errors can lead to operational mistakes, jeopardizing patient safety.

[0003] In recent years, computer vision technology has been widely applied to endoscopic spatial perception. Among these applications, 3D spatial reconstruction and augmented reality (AR) navigation systems are core applications, providing doctors with reliable spatial location information and surgical reference coordinates. Depth information is crucial for 3D spatial reconstruction and AR enhancement. However, endoscopic images commonly suffer from insufficient lighting, complex tissue surface reflections, and frequent occlusion, limiting the effectiveness of traditional depth estimation methods based on image texture and structural information in practical applications. Furthermore, endoscopic scenes exhibit unique near-field illumination characteristics, meaning that brightness information in images often depends on the complex interaction between the light source position, the tissue surface normal direction, and reflection properties. The lack of modeling for these physical properties easily leads to biased depth estimation results due to lighting interference, especially in low-light areas.

[0004] Existing depth estimation methods still have the following drawbacks: the enhancement effect on the image is global, that is, the brightness is also increased in well-lit areas, which can easily cause overexposure and also affect the depth estimation of normal areas. Summary of the Invention

[0005] To address the problem of inaccurate depth estimation caused by localized dim lighting in endoscopic scene images in existing technologies, this invention provides a self-supervised depth estimation method for endoscopic images based on a physical illumination model, a computer device, a computer-readable storage medium, and a computer program product. This method can effectively improve the illumination distribution of endoscopic images, thereby enhancing the accuracy of depth estimation for endoscopic images.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a self-supervised depth estimation method for endoscopic images based on a physical illumination model is provided, the method comprising:

[0007] A physical lighting model is established by introducing at least one virtual light source during the endoscopic examination of the object.

[0008] Video datasets of endoscopic examination procedures were obtained under conditions without physical illumination models and with physical illumination models, respectively. The video datasets include images of multiple endoscopes and corresponding camera pose information. The video datasets of endoscopic examinations with physical illumination models also include virtual light source information.

[0009] A depth estimation network is constructed, which includes an encoder, a depth decoder, an albedo decoder, a virtual light source multilayer perceptron, and a depth refinement network. The encoder is used to extract color features from the image, the depth decoder is used to predict the initial depth map based on the color features, the albedo decoder is used to predict the albedo map based on the color features, the virtual light source multilayer perceptron is used to predict the physical illumination parameters based on the color features, and the depth refinement network is used to predict the depth refinement map based on the pixel-by-pixel rendering features of the image, the initial depth map, and the color features.

[0010] Using the depth refinement map, albedo map, physical illumination parameters, and corresponding camera pose information, multi-source pixel-by-pixel rendering is performed to generate reconstructed images; the reconstructed images and the imaging images are compared to realize the training of the depth estimation network based on self-supervised learning.

[0011] The endoscope image to be processed is input into the trained depth estimation network, which outputs the depth map of the endoscope image to be processed.

[0012] Furthermore, the endoscope image to be processed is input into a trained depth estimation network, which outputs a depth map of the endoscope image to be processed. This includes inputting the endoscope image to be processed into the trained depth estimation network and outputting a depth refinement map, an albedo map, and physical illumination parameters of the endoscope image to be processed. Using the depth refinement map, albedo map, and physical illumination parameters of the endoscope image to be processed, multi-light source pixel-by-pixel rendering is performed to generate a reconstructed image of the endoscope image to be processed. Based on the reconstructed image of the endoscope image to be processed, the depth map of the endoscope image to be processed is obtained.

[0013] Furthermore, using the depth refinement map, albedo map, physical illumination parameters, and corresponding camera pose information, multi-source pixel-by-pixel rendering is performed to generate a reconstructed image. This includes generating a surface normal map corresponding to each pixel in the image based on the depth refinement map; calculating the illumination contribution of each pixel in the image to both the camera's built-in light source and the virtual light source, based on the physical illumination parameters and corresponding camera pose information, where the illumination contribution includes the illumination direction and attenuation factor; obtaining a first pixel-by-pixel shadow field rendered by the camera's built-in light source based on the surface normal map corresponding to each pixel, the illumination direction corresponding to the camera's built-in light source, and the attenuation factor; obtaining a second pixel-by-pixel shadow field rendered by the virtual light source based on the surface normal map corresponding to each pixel, the illumination direction corresponding to the virtual light source, and the attenuation factor; and then superimposing the first and second pixel-by-pixel shadow fields and combining them with the albedo map to obtain the reconstructed image.

[0014] Furthermore, the video dataset also includes true depth maps, true albedo maps, and surface normal maps corresponding to the imaging images of multiple endoscopes.

[0015] Furthermore, the loss functions used in the training process of the depth estimation network include the scale translation invariance loss function and the depth regularization loss function, which are used to constrain the stability of the depth refinement map and the true depth map under different scales and displacement conditions, respectively; the virtual normal loss function and the pixel-by-pixel illumination loss function are used to constrain the surface normal map calculated from the depth refinement map and the illumination rendering result, respectively; the albedo loss function is used to constrain the accuracy of the albedo map; and the virtual light source loss function is used to align the physical lighting parameters with the virtual light source information to constrain the accuracy of multi-light source pixel-by-pixel rendering.

[0016] Furthermore, at least one virtual light source is configured to face a dark area, which is an area where the illuminance is less than a preset value.

[0017] Furthermore, the aforementioned self-supervised depth estimation method for endoscopic images based on the physical illumination model also includes calculating a pixel-by-pixel rendering image based on the initial depth map using pixel-by-pixel rendering; and extracting features from the pixel-by-pixel rendering image using an encoder to obtain the pixel-by-pixel rendering features of the imaging image.

[0018] According to a second aspect of the invention, a computer device is also provided, comprising a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of any of the methods described above.

[0019] According to a third aspect of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the above methods.

[0020] According to a fourth aspect of the invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0021] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0022] This invention provides a self-supervised depth estimation method for endoscopic images based on a physical illumination model. It introduces a physical illumination model into the depth estimation network, expanding the single light source to multiple light sources to simulate the multi-directional lighting environment in actual surgery. By adding virtual light sources and dynamically adjusting their positions and intensities, the illuminance distribution of the endoscopic image is effectively improved, which helps to enhance the accuracy of depth estimation in insufficiently lit areas (dark areas) and strengthens the adaptability of the depth estimation network to complex lighting scenarios. Furthermore, using the depth refinement map, albedo map, physical illumination parameters, and corresponding camera pose information, multi-light source pixel-by-pixel rendering based on the physical illumination model is performed to generate a reconstructed image. The depth estimation network is guided to learn physically consistent surface geometry through self-supervised learning, thereby improving the accuracy of depth estimation for endoscopic images. Moreover, it does not require expensive real depth annotation data and can be trained directly using image reconstruction loss. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a self-supervised depth estimation method for endoscopic images based on a physical illumination model, provided for an embodiment of this application;

[0025] Figure 2 A training architecture diagram of the depth estimation network provided in the embodiments of this application;

[0026] Figure 3 An architectural diagram of the DINOv2 encoder provided in the embodiments of this application;

[0027] Figure 4 An architecture diagram of the depth decoder and albedo decoder provided in the embodiments of this application;

[0028] Figure 5 An architecture diagram of a deep refinement network provided in an embodiment of this application;

[0029] Figure 6A schematic diagram of multi-light source pixel-by-pixel rendering provided in an embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0032] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0033] like Figure 1 As shown, a self-supervised depth estimation method for endoscopic images based on a physical illumination model is provided. This method can be executed by a terminal or by a server communicating with the terminal via a network. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers. The method is illustrated using a terminal as an example, including the following steps:

[0034] Step 101: Establish a physical lighting model by introducing at least one virtual light source during the endoscopic examination of the object.

[0035] The virtual light source can be a spotlight model. At least one virtual light source can be configured to face the dark area, which is a region with an illuminance lower than a preset value, to provide physical illumination-based supplemental lighting enhancement to the dark area, thereby increasing the brightness of the dark area and helping to improve the accuracy of depth estimation.

[0036] Step 102: Obtain video datasets of the endoscopic examination process under conditions that do not include a physical illumination model and conditions that include a physical illumination model.

[0037] The video dataset includes images from multiple endoscopes, along with corresponding true depth maps, true albedo maps, surface normal maps, and camera pose information. The camera pose information includes the position and orientation of the camera's built-in light source. The video dataset containing endoscopic examinations under physical illumination model conditions also includes virtual light source information, including the position and orientation of the virtual light source.

[0038] An albedo map is a type of texture map used to define the color and base brightness of an object's surface in the absence of lighting.

[0039] Step 103: Construct a depth estimation network, which includes an encoder, a depth decoder, an albedo decoder, a virtual light source multilayer perceptron, and a depth refinement network. The encoder is used to extract color features from the image, the depth decoder is used to predict the initial depth map based on the color features, the albedo decoder is used to predict the albedo map based on the color features, the virtual light source multilayer perceptron is used to predict the physical illumination parameters based on the color features, and the depth refinement network is used to predict the depth refinement map based on the pixel-by-pixel rendering features of the image, the initial depth map, and the color features.

[0040] Among them, the physical supplementary lighting parameters include the position and direction of the additional light source.

[0041] Step 104: Using the depth refinement map, albedo map, physical illumination parameters, and corresponding camera pose information, perform multi-source pixel-by-pixel rendering to generate a reconstructed image; compare the reconstructed image with the imaging image to achieve self-supervised learning-based training of the depth estimation network.

[0042] Step 105: Input the endoscope image to be processed into the trained depth estimation network and output the depth map of the endoscope image to be processed.

[0043] For example, the terminal inputs the endoscope image to be processed into a trained depth estimation network, and outputs a depth refinement map, an albedo map, and physical illumination parameters of the endoscope image to be processed; using the depth refinement map, albedo map, and physical illumination parameters of the endoscope image to be processed, multi-light source pixel-by-pixel rendering is performed to generate a reconstructed image of the endoscope image to be processed; based on the reconstructed image of the endoscope image to be processed, the depth map of the endoscope image to be processed is obtained.

[0044] The aforementioned self-supervised depth estimation method for endoscopic images based on a physical illumination model introduces a physical illumination model into the depth estimation network, expanding the single light source to multiple light sources to simulate the multi-directional lighting environment in actual surgery. By adding virtual light sources and dynamically adjusting their positions and intensities, the illuminance distribution of the endoscopic image is effectively improved, which helps to enhance the accuracy of depth estimation in insufficiently lit areas, i.e., dark areas, and strengthens the adaptability of the depth estimation network to complex lighting scenarios. Furthermore, using the depth refinement map, albedo map, physical illumination parameters, and corresponding camera pose information, multi-light source pixel-by-pixel rendering based on the physical illumination model is performed to generate a reconstructed image. Through self-supervised learning training, the depth estimation network is guided to learn physically consistent surface geometry, thereby improving the accuracy of depth estimation for endoscopic images.

[0045] In one embodiment, a self-supervised depth estimation method for endoscopic images based on a physical illumination model includes the following steps:

[0046] Step 1: Prepare two sets of datasets. One set is an existing video dataset of colonoscopy, including RGB images (i.e., imaging images), true depth maps, surface normal maps, and camera pose information. The other set is a video dataset of colonoscopy with virtual light sources, including virtual light source RGB images, true depth maps, surface normal maps, phase pose information, and virtual light source information.

[0047] Step 2, Establishment and training of the depth estimation network: Introduce a physical illumination model on the training strategy of existing methods to achieve physical illumination-based illumination enhancement.

[0048] Step 3, perform depth estimation on the image to be processed: input the image to be processed into the depth estimation network, and use the output of the depth estimation network to perform multi-light source pixel-by-pixel rendering.

[0049] Step 1 specifically includes:

[0050] Colonoscopy video data was collected as the training dataset for this embodiment, using the C3VD dataset and the dataset presented in this paper as examples. The C3VD dataset contains high-definition clinical colonoscopy and high-fidelity colon models, including 22 registered video sequences. The dataset presented in this paper consists of 14 video sequences, obtained by adding virtual light sources to VR-Caps 3D intestinal organ models and capturing endoscopic data.

[0051] The dataset collection process in this paper is as follows: A virtual environment is simulated using Unity to record the camera's movement path and light source positions, generating a multimodal dataset containing RGB images, true depth maps, true Albedo maps, surface normal maps, and camera poses. The core principle is to use scripts to control the movement trajectories of the camera and light sources, combined with Unity's Recorder tool to capture scene data. First, the camera's movement trajectory in the virtual scene is recorded via script and saved as a JSON file to ensure the path can be traced back and verified. Light source positions are recorded synchronously to optimize the lighting effect of the camera path and enhance data realism. Versions with and without light sources are collected separately to ensure data consistency.

[0052] In this embodiment, by adding virtual light sources, two sets of datasets are obtained: one including virtual light sources and one excluding virtual light sources. Without affecting the normal area illumination, the addition of virtual light sources enhances the brightness of local dark areas through rendering, thereby obtaining training datasets under different lighting conditions. Training the deep learning network based on the training datasets under different lighting conditions can indirectly improve the accuracy of the deep learning network in depth estimation of endoscopic images obtained under different lighting conditions, especially those containing complex dark areas.

[0053] Step 2 specifically includes:

[0054] like Figure 2 As shown, the depth estimation network consists of a DINOv2 encoder, a depth decoder (DPT-Depth Head), an albedo decoder (also known as an Albedo decoder) (DPT-Albedo Head), a virtual light source multilayer perceptron (MLP), and a depth refinement network (PPS-Refinement). This network takes an RGB image as input and outputs a refined depth map through progressive feature extraction and refinement. Albedo plot, along with the position and orientation of additional light sources, are then used to refine the plot using depth. The system performs multi-source pixel-by-pixel rendering of Albedo images and the positions and orientations of additional light sources to generate reconstructed images, thereby achieving self-supervised depth estimation of endoscopic images based on a physical illumination model.

[0055] Specifically, the training process of the depth estimation network is as follows:

[0056] First, the RGB image is input to the DINOv2 encoder, which extracts the RGB features from the image. These RGB features serve as input to the depth decoder and Albedo decoder, where the depth decoder processes the RGB features and outputs an initial depth map. The Albedo decoder processes RGB features and outputs a predicted Albedo map, which is used to define the albedo of the object's surface in subsequent rendering.

[0057] Secondly, based on the initial depth map The DINOv2 encoder calculates a pixel-per-pixel rendered image (PPS image) using pixel-per-pixel rendering (also known as PPS rendering). Then, it extracts features from this PPS image to obtain pixel-per-pixel rendered features (PPS features) of the RGB image, capturing lighting-related geometric information. Simultaneously, a virtual light source multilayer perceptron (MLP) uses the RGB features as input and predicts the position and orientation of additional light sources through a two-layer fully connected network, supporting multi-source lighting simulation.

[0058] Then, the depth refinement network PPS-Refinement receives the initial depth map. RGB and PPS features are used as inputs to optimize depth details, and the final output is a refined depth map. This improves the accuracy and physical consistency of depth estimation.

[0059] Finally, using the depth refinement plot The system uses Albedo images, the location and orientation of built-in camera lights, and the location and orientation of additional light sources to perform multi-light source pixel-by-pixel rendering to generate a reconstructed image. The reconstructed image is compared with the original RGB image to achieve joint optimization through self-supervised training, which can improve the accuracy of depth estimation network in the dark areas of endoscopes.

[0060] In one embodiment, such as Figure 3 As shown, the encoder uses the DINOv2 encoder. DINOv2, as the overall encoder architecture, is based on the Vision Transformer (ViT-S) model, including patch embedding, position embedding, and a 24-layer Transformer encoder. The input RGB image, after patch processing and position embedding, is converted into a token sequence. RGB features are extracted using a multi-head self-attention mechanism (MHSA), a feedforward neural network (FFN), and a normalization layer (LayerNorm).

[0061] In one embodiment, such as Figure 4 As shown, the depth decoder and Albedo decoder adopt the DPTHead architecture, which includes a reprojection layer, an upsampling convolutional layer, a multi-scale feature fusion layer (RefineNet), and an output layer. RGB features are projected into multi-scale CNN features, the upsampling convolutional layer amplifies and adjusts the channels, and the multi-scale feature fusion layer (RefineNet) integrates the features step by step. Finally, the depth decoder outputs the initial depth map. The Albedo decoder outputs a predicted Albedo graph.

[0062] In one embodiment, such as Figure 5 As shown, the deep refinement network PPS-Refinement includes a CrossAttention Module, a FeatureEncoder layer, a FeatureLinearModulation (FiLM) layer, a Downsample layer, a four-layer U-Net network, an Upsample layer, and a DepthAddition layer. (The initial depth map is used as an example.) Using PPS features as input, a multi-head cross-attention module calculates and synthesizes a feature set, which is then converted into convolutional features by a feature encoding layer. A feature linear modulation layer adjusts the features through scaling and offsetting. After downsampling to reduce the size, the feature set is processed through four U-Net layers, and an upsampling layer restores the size. Finally, the feature set is added to the initial depth map in a depth refinement layer to obtain the final depth refinement map. .

[0063] In one embodiment, the virtual light source multilayer perceptron (MLP) consists of two fully connected layers, the parameters of which are shown in Table 1, used to predict the position and orientation of an additional light source using RGB features as input.

[0064] Table 1

[0065]

[0066] As shown in Table 1, the input of the first fully connected layer is 384-dimensional RGB color features, and the output is 128-dimensional hidden features; the input of the second fully connected layer is the same 128-dimensional hidden features, and the output is 5-dimensional virtual light source parameters, which are used to predict the position and direction of the additional light source.

[0067] In one embodiment, to achieve high-quality image illumination enhancement and improve the accuracy of depth estimation, this embodiment designs multiple loss functions during training to guide the depth estimation network to learn physically consistent depth information. Specifically, targeted loss functions are designed for the depth refinement network, the virtual light source multilayer perceptron, and the Albedo decoder. First, a scale translation invariance loss function and a depth regularization loss function are introduced to constrain the stability of the predicted depth refinement map and the true depth map under different scales and displacement conditions. Second, a virtual normal loss function and a pixel-by-pixel illumination loss function are introduced to constrain the surface normal map calculated from the predicted depth refinement map and the illumination rendering result, enabling the depth estimation network to better capture the geometric and lighting relationships under the physical illumination model. Then, an albedo loss function is introduced to constrain the accuracy of the predicted albedo map. Finally, a virtual light source loss function is introduced to align the predicted position and direction of the additional light source with the true values ​​(i.e., the position and direction of the virtual light source) to constrain the accuracy of multi-light source pixel-by-pixel rendering.

[0068] In one embodiment, the self-supervised training process is guided by the following loss function:

[0069]

[0070] in, This represents the scale translation invariance loss function. This represents the depth regularization loss function. Represents the virtual normal loss function. This represents the pixel-by-pixel illumination loss function. Represents the albedo loss function. Represents the virtual light source loss function. represents the weight coefficients of the scale translation invariant loss function, depth regularization loss function, virtual normal loss function, pixel-by-pixel illumination loss function, albedo loss function, and virtual light source loss function, respectively. This represents the total loss function.

[0071] The following formula is used as the scale translation invariance loss function. :

[0072]

[0073] in, It is L1 loss. It is a depth refinement map predicted by a depth estimation network. It is a true depth map. This represents the total number of sample points used to calculate the scale translation invariance loss.

[0074] L1 loss is a commonly used loss function in regression tasks, defined as the absolute value of the difference between the predicted and the true values. .

[0075] Indicates the first i The predicted depth value of each pixel. Indicates the first i The true depth value of each pixel, where the symbol ^ indicates that the value is the result after scale normalization. Scale normalization refers to transforming both the predicted and true depth values ​​to a common, comparable scale. This is usually calculated by subtracting the mean and dividing by the standard deviation, thereby eliminating the effects of overall scaling and translation, and making the loss function focus more on shape consistency rather than absolute numerical matching.

[0076] The following formula is used as the depth regularization loss function. :

[0077]

[0078] in, It represents the gradient of an image in the horizontal direction (x-axis direction), and measures the rate of change of pixel values ​​in the horizontal direction. The gradient of an image represents the vertical direction (y-axis direction), measuring the rate of change of pixel values ​​in the vertical direction. For depth maps, the gradient can reveal the boundaries of depth changes. If the gradient of a depth map is large at a certain location, it indicates that the depth value is changing rapidly at that location in the image, which usually corresponds to the edge of an object or a place where the depth is discontinuous.

[0079] , , , Indicates the first i The residual (error) between the predicted depth value and the true depth value at each pixel.

[0080] This is a detailed depth map predicted by the depth estimation network after scaling and translation. It is a scaled and translated true depth map. and The formulas are as follows:

[0081] in, It is a scaling factor. It is a translation factor, based on and The least squares method is used to calculate the result, and the formula is as follows:

[0082] The following formula is used as the virtual normal loss function. :

[0083]

[0084] in, It is the number of surface normal sample points used to calculate the loss. It is a surface normal map calculated from the true depth map. It is a surface normal map calculated from the depth refinement map predicted by the depth estimation network.

[0085] The following formula is used as the pixel-by-pixel illumination loss function. :

[0086]

[0087] in, It is the height of the image (i.e., the image map), which is the number of pixels in the vertical direction of the image; It is the width of the image, that is, the number of pixels in the horizontal direction of the image; It is a location The binary mask at that location controls whether that pixel is included in the loss; It is a pixel-by-pixel rendering image of a single light source, calculated from the predicted depth refinement image and the real Albedo image; It is a pixel-by-pixel rendering image of a single light source, calculated from the real depth map and the real Albedo map.

[0088] The following formula is used as the albedo loss function. :

[0089]

[0090] in, It is a pixel-by-pixel rendering image of a single light source, calculated from the real depth map and the real Albedo map. It is a pixel-by-pixel rendering image of a single light source, calculated from the real depth map and the Albedo map predicted by the Albedo decoder.

[0091] The following formula is used as the virtual light source loss function. :

[0092]

[0093] in, , It is a multi-light source pixel-by-pixel rendering map calculated using the real depth map, the real Albedo map, and the real position and orientation of additional light sources (i.e., the position and orientation of virtual light sources). It is a multi-light source pixel-by-pixel rendering map calculated using a real depth map, a real Albedo map, and the position and orientation of additional light sources predicted by MLP.

[0094] In this embodiment, through the joint optimization of the above-mentioned multiple loss functions, this embodiment can achieve physically consistent self-supervised depth estimation without the need for real depth annotation, thereby improving the accuracy of depth prediction in low-light areas.

[0095] In one embodiment, such as Figure 6 As shown, Figure 6 This is a schematic diagram of multi-source pixel-by-pixel rendering provided in an embodiment of this application. Based on single-source pixel-by-pixel illumination rendering (PPS) and PPSNet by Paruchuri et al., this embodiment introduces a multi-source illumination mechanism by adding virtual light sources. The virtual light sources and the camera's built-in light sources work together to achieve pixel-by-pixel illumination rendering from multiple light sources, thereby enhancing the brightness of local dark areas through rendering without affecting the illumination of normal areas.

[0096] Based on the linear superposition property of multi-source lighting models, the following multi-source pixel-wise shadow field is proposed. The calculation formula is as follows:

[0097]

[0098] in, This represents the per-pixel shadow field rendered using different light sources.

[0099] refer to Figure 6 An endoscope image can be reconstructed by rendering the illumination effect of external light sources and its own Albedo color effect. The depth map D(X) analyzes the illumination and reflection of the light source in space and the geometric information of the surface of the illuminated organ. In other words, the pixel-by-pixel rendering reconstruction result of PPS depends on the accuracy of depth estimation. Therefore, supervising the reconstruction training with image data under different lighting conditions can indirectly improve the accuracy of depth estimation network under different lighting conditions.

[0100] For example, multi-source pixel-by-pixel rendering is performed using a depth refinement map, an albedo map, physical illumination parameters, and corresponding camera pose information to generate a reconstructed image. This includes generating a surface normal map corresponding to each pixel in the image based on the depth refinement map; calculating the illumination contribution of each pixel in the image to both the camera's built-in light source and a virtual light source, based on the physical illumination parameters and corresponding camera pose information, where the illumination contribution includes the illumination direction and attenuation factor; obtaining a first pixel-by-pixel shadow field rendered by the camera's built-in light source based on the surface normal map corresponding to each pixel, the illumination direction corresponding to the camera's built-in light source, and the attenuation factor; obtaining a second pixel-by-pixel shadow field rendered by the virtual light source based on the surface normal map corresponding to each pixel, the illumination direction corresponding to the virtual light source, and the attenuation factor; and then superimposing the first pixel-by-pixel shadow field and the second pixel-by-pixel shadow field and combining them with the albedo map to obtain the reconstructed image.

[0101] Specifically, based on the physical illumination parameters and the corresponding camera pose information, the illumination contribution of each pixel in the image is calculated to correspond to the camera's built-in light source and the virtual light source, respectively. The illumination contribution includes the illumination direction and the attenuation factor. Specifically, the first illumination direction of any point on the object's surface is calculated based on the position of the camera's built-in light source, and the second illumination direction of any point on the object's surface is calculated based on the position of the additional light source. The first attenuation factor of any point on the object's surface is calculated based on the first illumination direction, the camera direction, and the position of the camera's built-in light source. The second attenuation factor of any point on the object's surface is calculated based on the second illumination direction, the direction of the additional light source, and the position of the additional light source.

[0102] In one embodiment, the formula for calculating the multi-light source pixel-by-pixel rendered image (i.e., the reconstructed image) is as follows:

[0103]

[0104] in, Indicates the first lighting direction. Indicates the second lighting direction. Indicates the first attenuation factor. Indicates the second attenuation factor. Represents the surface normal diagram. Represents the true Albedo diagram. This represents a pixel-by-pixel rendering image with multiple light sources. Represents the first pixel-by-pixel shadow field. This represents the second pixel-by-pixel shadow field.

[0105] Step 3 specifically includes:

[0106] After completing the dataset construction and network training, this embodiment uses the trained depth estimation network based on a physical illumination model to estimate the depth of the endoscopic image to be processed, obtaining the corresponding depth refinement map, the predicted Albedo map, and physical illumination parameters such as the position and direction of the additional light source. Subsequently, based on the above output results, a physical consistency reconstruction of the endoscopic image to be processed is performed using a multi-light source pixel-by-pixel rendering mechanism. This rendering process comprehensively considers the camera's built-in light source, virtual additional light source, illumination attenuation factor, and surface normal information, providing directional illumination for dark areas while maintaining the brightness of normal areas unaffected, thereby generating an enhanced reconstructed image. The reconstructed image is then processed using standard depth estimation algorithms (such as PPSNet, Monodepth2, EndoSLAM) to obtain the depth map of the endoscopic image to be processed.

[0107] Through the above steps, high-precision depth estimation of endoscopic scene images can be achieved, and reconstructed images enhanced by supplemental lighting can be output, providing more reliable depth information and illumination balance for subsequent 3D reconstruction, augmented reality navigation, or clinical analysis.

[0108] This application also provides a computer device, the internal structure diagram of which can be shown as follows: Figure 7As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a self-supervised depth estimation method for endoscopic images based on a physical illumination model.

[0109] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0110] like Figure 7 As shown, this application also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the above-described method embodiments.

[0111] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0112] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0113] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0114] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0115] The above description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An endoscope image self-supervised depth estimation method based on a physical light compensation model, characterized in that, The method comprises the steps of: establishing a physical light compensation model by introducing at least one virtual light source in an endoscopy process of an execution object; respectively acquiring video data sets of the endoscopy process under conditions of not containing the physical light compensation model and containing the physical light compensation model; the video data sets comprise imaging images of multiple endoscopes and corresponding camera pose information, and the video data set of the endoscopy under the condition of containing the physical light compensation model further comprises virtual light source information; constructing a depth estimation network, the depth estimation network comprising an encoder, a depth decoder, an albedo decoder, a virtual light source multi-layer perception and a depth refinement network; wherein the encoder is used to extract color features of the imaging images, the depth decoder is used to predict an initial depth map according to the color features, the albedo decoder is used to predict an albedo map according to the color features, the virtual light source multi-layer perception is used to predict physical light compensation parameters according to the color features, and the depth refinement network is used to predict a depth refinement map according to pixel-by-pixel rendering features of the imaging images, the initial depth map and the color features; performing multi-light source pixel-by-pixel rendering by using the depth refinement map, the albedo map, the physical light compensation parameters and the corresponding camera pose information to generate a reconstructed image; comparing the reconstructed image and the imaging image to realize training of the depth estimation network based on self-supervised learning; inputting an endoscopy image to be processed into the trained depth estimation network to output a depth map of the endoscopy image to be processed.

2. The method of claim 1, wherein, The inputting of the endoscopy image to be processed into the trained depth estimation network to output the depth map of the endoscopy image to be processed comprises: inputting the endoscopy image to be processed into the trained depth estimation network to output a depth refinement map, an albedo map and physical light compensation parameters of the endoscopy image to be processed; performing multi-light source pixel-by-pixel rendering by using the depth refinement map, the albedo map and the physical light compensation parameters of the endoscopy image to be processed to generate a reconstructed image of the endoscopy image to be processed; obtaining a depth map of the endoscopy image to be processed based on the reconstructed image of the endoscopy image to be processed.

3. The method of claim 1, wherein, The performing of the multi-light source pixel-by-pixel rendering by using the depth refinement map, the albedo map, the physical light compensation parameters and the corresponding camera pose information to generate the reconstructed image comprises: generating a surface normal map corresponding to each pixel point in the imaging image according to the depth refinement map; calculating illumination contributions of each pixel point in the imaging image corresponding to a camera built-in light source and a virtual light source respectively according to the physical light compensation parameters and the corresponding camera pose information, the illumination contributions comprising illumination directions and attenuation factors; obtaining a first pixel-by-pixel shadow field rendered by the camera built-in light source according to the surface normal map corresponding to each pixel point and the illumination directions and attenuation factors corresponding to the camera built-in light source; obtaining a second pixel-by-pixel shadow field rendered by the virtual light source according to the surface normal map corresponding to each pixel point and the illumination directions and attenuation factors corresponding to the virtual light source; The first pixel-wise shadow field, the second pixel-wise shadow field are superimposed and then combined with the albedo map to obtain a reconstructed image.

4. The method of claim 1, wherein, The video dataset further includes real depth maps, real albedo maps and surface normal maps corresponding to imaging maps of multiple endoscopes.

5. The method of claim 4, wherein, The loss function of the depth estimation network in the training process includes: A scale translation invariance loss function and a depth regularization loss function are respectively used to constrain the stability of the depth refinement map and the real depth map under different scales and displacements. A virtual normal loss function and a pixel-wise lighting loss function are respectively used to constrain the surface normal map calculated from the depth refinement map and the lighting rendering result. An albedo loss function is used to constrain the accuracy of the albedo map. A virtual light source loss function is used to align the physical light compensation parameters with the virtual light source information to constrain the accuracy of the multi-light source pixel-wise rendering.

6. The method of claim 1, wherein, At least one virtual light source is configured to face an illumination dark area, which is an area with an illuminance less than a preset value.

7. The method of claim 1, wherein, The method further includes: Based on the initial depth map, a pixel-wise rendering map is calculated using pixel-wise rendering; Features of the pixel-wise rendering map are extracted by the encoder to obtain pixel-wise rendering features of the imaging map.

8. A computer device, comprising: A memory, a processor and a computer program stored in the memory are included, and the processor executes the computer program to implement the steps of the method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is executed by a processor to implement the steps of the method of any one of claims 1-7.

10. A computer program product, characterised in that, A computer program is included, and the computer program is executed by a processor to implement the steps of the method of any one of claims 1-7.

Images