Monocular depth estimation method based on active perception
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING JISHUITAN HOSPITAL
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-05
Smart Images
Figure CN122140369A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intraoperative navigation technology for surgical robots, and more specifically, to a monocular depth estimation method based on active perception. Background Technology
[0002] In robot-assisted minimally invasive surgery, especially endoscopic surgery, real-time and accurate 3D scene perception is fundamental to achieving autonomous or semi-autonomous operation, augmented reality navigation, and precise surgical planning. Currently, the most widely used endoscopic imaging systems in clinical practice are monocular cameras. This is because monocular endoscopes are simple in structure, low in cost, easy to integrate into narrow surgical channels, and have been widely accepted by surgeons in long-term clinical practice.
[0003] Monocular depth estimation technology is considered a highly promising solution due to its advantage of requiring only a single endoscopic camera to infer the 3D structure of a scene. However, directly applying existing monocular depth estimation models trained on natural images to surgical scenarios faces significant challenges. Surgical environments contain numerous textureless tissue surfaces, specular reflections (such as wet tissue surfaces and instrument reflections), and complex occlusions. These factors inherently introduce ambiguity and high uncertainty into depth information inferred from monocular images. Existing models show a significant performance degradation on such "out-of-distribution" data, and their predictions are often unreliable, posing safety risks when directly used for surgical navigation.
[0004] To compensate for the limitations of pure visual perception, an intuitive approach is to introduce sparse yet precise tactile depth measurements provided by robot proprioception as a truth constraint. However, in the dynamic and fragile surgical environment, each active robot contact measurement involves time costs, risks of tissue disturbance, and potential disruptions to the surgical procedure. Therefore, how to maximize the accuracy and reliability of depth estimation across the entire surgical field of view with the fewest necessary tactile measurements has become a critical but unresolved problem. Summary of the Invention
[0005] The purpose of this invention is to propose a monocular depth estimation method based on active perception, which enhances the accuracy and robustness of monocular depth estimation through sparse proprioceptive (tactile) depth measurement, while minimizing measurement costs (reducing the number of robot touches).
[0006] To achieve the above objectives, this invention proposes a monocular depth estimation method based on active perception for intraoperative navigation in robot-assisted surgery. The method includes: S1: Acquire a monocular RGB image of the surgical area using an endoscopic imaging system; S2: Obtain the initial set of sparse depth measurements; S3: Input the RGB image and the sparse depth measurement set at the current time into a depth estimation ensemble model; wherein the sparse depth measurement set is used as an additional input condition of the model in a manner aligned with the spatial configuration of the RGB image; the depth estimation ensemble model outputs an ensemble dense depth prediction map as a depth estimate of the current scene, and a prediction uncertainty map for evaluating the reliability of the estimation. S4: Based on the predicted uncertainty map, calculate its gradient with respect to the measurement values in the current sparse depth measurement set, and generate a gradient map indicating the measurement information value of each pixel. S5: Based on the gradient map, the Stein variational gradient descent algorithm is used to actively select the next set of image spatial locations that can minimize the overall uncertainty as the points to be explored. S6: Under the guidance of the intraoperative navigation system, the tactile sensing end of the surgical robot is controlled to move sequentially to the three-dimensional anatomical position in the body corresponding to the point to be explored to perform contact measurement and obtain new high-precision depth measurement values. S7: Update the newly acquired depth measurement value to the sparse depth measurement set, and return to step S3 to update the depth estimation ensemble model input; S8: Repeat steps S3 to S7 to iteratively optimize depth estimation and uncertainty assessment until the iteration stopping condition is met. Then, output the integrated dense depth prediction map finally generated by the depth estimation integrated model as an accurate three-dimensional scene model for surgical navigation.
[0007] Optionally, in step S1, the monocular RGB image is acquired in real time during the operation by a camera fixed to the front end of the endoscope.
[0008] Optionally, in step S2, the sparse depth measurement set is an empty set or contains several prior measurement points. Each measurement point includes a pixel coordinate with a definite position in the monocular RGB image and a depth measurement value corresponding to the pixel coordinate.
[0009] Optionally, in step S3, the depth estimation ensemble model includes multiple monocular depth estimation sub-models with the same structure and trained independently, each monocular depth estimation sub-model being based on the U-Net architecture; each monocular depth estimation sub-model receives the RGB image and the sparse depth measurement set as input, and outputs a dense depth prediction map corresponding to the input; The integrated dense depth prediction map is the pixel-level average of the dense depth prediction maps output by all monocular depth estimation sub-models; the prediction uncertainty map is the variance map of the dense depth prediction maps output by all monocular depth estimation sub-models at the corresponding pixel positions.
[0010] Optionally, step S4 specifically includes: The average uncertainty value of the prediction uncertainty map is calculated as the total uncertainty scalar, which is the average of the uncertainty values of all pixels in the prediction uncertainty map; The gradient of the total uncertainty scalar with respect to the sparse depth measurement set is calculated using the backpropagation algorithm to obtain a gradient map, which indicates the expected contribution of adding depth measurements at various locations in the image to reducing the overall uncertainty.
[0011] Optionally, step S5 specifically includes: The gradient map is negative and normalized to construct a target probability distribution as the optimization objective, such that high probability regions correspond to pixel positions with large absolute gradient values. Initialize multiple particles representing candidate measurement positions, with each particle corresponding to an image pixel coordinate; Using the Stein variational gradient descent algorithm, with the target probability distribution as the objective, the particle positions are iteratively updated so that the distribution of the particle set approximates the target probability distribution; The position of the particle that finally converges is selected as the point to be explored.
[0012] Optionally, step S6 specifically includes: The coordinates of the selected point to be explored in the image space are combined with the pre-calibrated camera intrinsic parameter matrix and back-projected onto the camera coordinate system to form a three-dimensional ray. The intraoperative navigation system, based on optical or electromagnetic positioning data, converts the three-dimensional rays to a unified patient coordinate system and generates the robot end effector motion path; Control the surgical robot to move along the end effector path until its tactile sensing end detects a contact event, and record the robot joint angle at this time; Based on the robot's forward kinematics model, the three-dimensional coordinates of the contact point in the patient's coordinate system are calculated and further converted into precise depth measurements relative to the camera.
[0013] Optionally, in step S6, the tactile sensing end is a capacitive tactile sensor, torque sensor, capacitive sensor, or bioimpedance sensor integrated into the end of the surgical robot.
[0014] Optionally, in step S7, updating the sparse depth measurement set with the newly acquired depth measurement value includes: The newly acquired depth measurements are added to the existing sparse depth measurement set in the form of pixel coordinate-depth value pairs, forming a more informative conditional input for the next depth estimation.
[0015] Optionally, in step S8, the iteration stopping condition is one of the following three: The total uncertainty scalar of the predicted uncertainty graph is lower than a preset threshold; The total number of tactile measurements performed has reached the preset limit. The cumulative measurement time exceeds the preset time limit.
[0016] The beneficial effects of this invention are as follows: This invention introduces sparse yet absolutely accurate robotic tactile measurements as "geometric anchors," providing crucial scale benchmarks and local truth constraints for monocular vision models. Combined with a depth estimation ensemble model, it achieves efficient fusion of visual appearance information and tactile geometric information, generating a dense depth map that is highly accurate at the measurement points and subject to global constraints. This transforms unreliable visual guesses into reliable spatial estimates. By explicitly quantifying prediction uncertainties and calculating their gradients, the system can accurately assess the information value of each potential measurement point. Furthermore, it intelligently plans the optimal sequence of measurement points using the Stein variational gradient descent (SVGD) algorithm, maximizing the reduction of global uncertainty while ensuring reasonable spatial dispersion. This ensures that every valuable tactile measurement is used effectively, achieving maximum accuracy improvement with minimal necessary contact. The entire process constitutes a complete "perception-evaluation-decision-execution-update" closed loop. Each new measurement immediately updates its internal state (depth map and uncertainty) and replans the next step. By setting preset iteration stopping conditions, the system can automatically terminate when a predetermined accuracy target or resource limit is reached, ultimately outputting a high-precision, high-confidence 3D scene model, providing a ready-to-use and reliable perception foundation for surgical navigation. In summary, this invention successfully transforms a costly and unreliable passive perception problem into a cost-controllable and progressively converging active optimization problem by introducing uncertainty-driven decision-making and a closed-loop active learning mechanism into the field of surgical robot perception. This provides robust and practical real-time 3D scene understanding capabilities for robot-assisted surgery.
[0017] The present invention has other features and advantages, which will be apparent from or will be set forth in detail in the accompanying drawings and the following detailed description, which together serve to explain the particular principles of the invention. Attached Figure Description
[0018] The above and other objects, features and advantages of the present invention will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.
[0019] Figure 1The diagram illustrates the steps of a monocular depth estimation method based on active sensing, according to an embodiment of the present invention. Detailed Implementation
[0020] The invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
[0021] This embodiment provides a monocular depth estimation method based on active perception for intraoperative navigation in robot-assisted surgery, such as... Figure 1 As shown, the method includes the following steps: S1: Acquire a monocular RGB image of the surgical area using an endoscopic imaging system; In this step, the monocular RGB image is acquired in real time during the operation using a camera fixed to the front end of the endoscope.
[0022] In one example, the endoscopic imaging system acquires a monocular RGB image of the surgical area. H and W represent the height and width of the image, respectively, and the resolution of this monocular RGB image can be selected as 512×512 pixels.
[0023] S2: Obtain the initial set of sparse depth measurements; In this step, the sparse depth measurement set includes several measurement points. Each measurement point includes a pixel coordinate with a definite position in the monocular RGB image and a depth measurement value corresponding to the pixel coordinate. The depth measurement value in the measurement point is triggered by a tactile sensor configured at the end of the robot's tactile perception when it contacts the tissue, and the three-dimensional coordinates of the contact point in the camera coordinate system are calculated based on the forward kinematics model of the surgical robot to convert it into a depth value.
[0024] In one example, an initial set of sparse depth measurements is obtained, defined as follows: ,in N S The current number of measurement points, ( x i , y i () represents the image pixel coordinates. d i This represents the corresponding depth measurement value. During the first iteration, this set may be empty. When the set is not empty, each element is ( x i , y i ,d i ),in( x i , y i () is an image I pixel coordinates on d i This is the corresponding depth value. This depth value d i This is achieved by controlling the end effector of a surgical robot (e.g., a concentric tube robot) to move it to a position corresponding to the pixel ( x i , y i The intersection of the 3D ray obtained by back-projection from the camera model and the tissue surface. When the end effector-integrated tactile sensor detects a contact event, the depth of the contact point in the camera coordinate system is calculated using the robot's forward kinematics model; this is the depth value. d i .
[0025] S3: Input the RGB image and the sparse depth measurement set at the current time into a depth estimation ensemble model; wherein the sparse depth measurement set is used as an additional input condition of the model in a manner aligned with the spatial configuration of the RGB image; the model outputs an ensemble dense depth prediction map as a depth estimate of the current scene, and a prediction uncertainty map for evaluating the reliability of the estimation. In this step, the depth estimation ensemble model includes multiple monocular depth estimation sub-models with identical structures and independently trained, each based on the U-Net architecture. Each monocular depth estimation sub-model receives the RGB image and the sparse depth measurement set as input and outputs a dense depth prediction map corresponding to the input. The ensemble dense depth prediction map is the pixel-level average of the dense depth prediction maps output by all monocular depth estimation sub-models. The prediction uncertainty map is the variance map of the dense depth prediction maps output by all monocular depth estimation sub-models at the corresponding pixel positions.
[0026] In one example, this step will use an image I The sparse depth set S (represented as a depth map of the same size, with unmeasured areas padded with -1) is input into the depth estimation ensemble model.
[0027] The depth estimation ensemble model consists of K (e.g., K=5) structurally identical but independently trained monocular depth estimation sub-models, denoted as... ,in i Represents model parameters; each sub-model Based on the U-Net architecture, each sub-model The input is the RGB image. I It is concatenated with the channels of a sparse depth measurement set S represented in mask form, and a dense depth prediction map is output independently. The integrated dense depth prediction map is the mean map of the predictions from all K sub-models. For any pixel position in the image ( x , y ), its depth value Calculate using the following formula:
[0028] in, Indicates the first k Each sub-model takes an image as input. I When dealing with a sparse depth set S, at pixel position ( x , y The predicted depth value; Simultaneously, by calculating the prediction variance, a prediction uncertainty diagram is obtained. U The prediction uncertainty graph U For the variance plot of all K sub-model predictions, for pixel location ( x , y Its uncertainty value U x,y Calculate using the following formula: .
[0029] In one example, the training process for each sub-model in the deep estimation ensemble model is as follows: (1) Training data preparation A 3D anatomical model was reconstructed using preoperative CT data. Numerous paired RGB images and dense, realistic depth maps were then rendered in a virtual environment. For example, in simulation environments like Unity, a virtual camera was placed inside the 3D trachea model for exploratory movement, rendering an RGB image corresponding to each viewpoint and a precisely registered dense, realistic depth map. To reduce the gap between the simulation and real surgical images, a style transfer network was used to post-process the simulated RGB images, making their visual style closer to real endoscopic images.
[0030] (2) Training input simulation: sparse depth condition generation
[0031] The key to training lies in simulating the sparse set of depth measurements S encountered during inference. In each training iteration, for a given complete ground truth depth map, the following steps are performed to construct the training input: From the complete depth map, a certain number of pixels (e.g., 10, 20, 50) are randomly sampled, and their depth values are taken as known true values. The depth values of the remaining pixels are set to a special mask value (-1).
[0032] Sampling strategies are diverse, including random uniform sampling, cluster-based sampling, and fixed probability sampling, to cover various spatial distribution patterns that may emerge in future tactile measurements.
[0033] In this way, for the same RGB image and its corresponding complete ground truth depth, the model will be exposed to a large number of depth condition inputs with different sparse patterns and densities during training, thus being forced to learn how to utilize any given sparse depth cues.
[0034] (3) Training objectives
[0035] Each sub-model employs a U-Net encoder-decoder architecture. Its input consists of a channel-stitched RGB image (3 channels) and the simulated sparse depth map described above (1 channel, with unmeasured points at -1).
[0036] Each sub-model in the depth estimation ensemble model is trained by minimizing a scale-invariant logarithmic loss function between the predicted and true depths. For a batch of data, the loss L is calculated as follows:
[0037] in, N For the number of effective pixels, d i This is the actual depth value. This represents the depth value predicted by the model. This loss function is insensitive to global scale changes, making it particularly suitable for monocular depth estimation tasks.
[0038] Train K (e.g., 5) of the above U-Net sub-models to form an ensemble model. Each sub-model uses the same network architecture and training dataset, but is initialized with different random weights and uses different data augmentation or batch order during training. This strategy ensures that each sub-model converges to a different local optimum in the loss function space, thus producing beneficial diversity in predictions.
[0039] Through the above process, each sub-model learned two core capabilities: (1) Conditional depth prediction: When sparse depth measurements are provided, the model can accurately follow these true values at the corresponding locations and perform reasonable smoothing and diffusion in the surrounding areas.
[0040] (2) Visual feature inference: In areas where depth measurement is lacking, the model must rely on visual cues such as texture, perspective, and occlusion in RGB images to infer depth.
[0041] The training objective of the entire ensemble model is not to pursue the highest accuracy of a single model, but to obtain a set of models with good and diverse predictive performance. These models make consistent predictions in simple regions (small variance), while their predictions may diverge in difficult regions (such as areas with missing textures or reflections) (large variance). This prediction variance is the core basis for quantifying uncertainty and guiding active perception in subsequent steps.
[0042] S4: Based on the predicted uncertainty map, calculate its gradient with respect to the measurement values in the current sparse depth measurement set, and generate a gradient map indicating the measurement information value of each pixel. This step specifically includes: The average uncertainty value of the prediction uncertainty map is calculated as the total uncertainty scalar, which is the average of the uncertainty values of all pixels in the prediction uncertainty map; The gradient of the total uncertainty scalar with respect to the sparse depth measurement set is calculated using the backpropagation algorithm to obtain a gradient map, which indicates the expected contribution of adding depth measurements at various locations in the image to reducing the overall uncertainty.
[0043] In one example, firstly, based on the predicted uncertainty graph... U Calculate the total uncertainty scalar for the entire scenario U total It is defined as the average value of all pixel values in the uncertainty map:
[0044] in, N = H × W The total number of pixels in the image. H and W These are the height and width of the image, respectively; Then, the total uncertainty scalar is calculated using automatic differentiation techniques. U total The gradient of the current sparse depth measurement set S This yields a gradient map; the gradient map is located at the pixel position. z gradient value ( U total right S z The partial derivative of represents the expected contribution of adding a depth measurement at that location to reducing the total uncertainty, where S z Indicates pixel position z The corresponding sparse depth measurement value at the location, if the position zIf it has been measured by touch (i.e., belongs to set S), then S z The true depth value at this location d i If the position z If it has not yet been measured by touch, then S z A predefined unmeasured identifier (e.g., 0 or -1) indicates that the current depth value is missing at this location.
[0045] S5: Based on the gradient map, the Stein variational gradient descent algorithm is used to actively select the next set of image spatial locations that can minimize the overall uncertainty as the points to be explored. This step specifically includes: S51: Negate and normalize the gradient map to construct a target probability distribution as the optimization objective, so that the high probability region corresponds to the pixel position with the large absolute value of the gradient. In one example, based on the gradient graph Construct a target probability distribution to guide particle sampling. p ( z First, the gradient is negative and normalized so that the high-probability region corresponds to the pixel position with the large absolute value of the gradient in the gradient map. S52: Initialize multiple particles representing candidate measurement positions, each particle corresponding to an image pixel coordinate; In one example, initialize a set containing M A collection of particles Each particle z i Represents the pixel coordinates of a candidate image; S53: Using the Stein Variational Gradient Descent (SVGD) algorithm, with the target probability distribution as the objective, iteratively update the particle positions so that the distribution of the particle set approximates the target probability distribution; In one example, the Stein variational gradient descent algorithm is used with respect to the target probability distribution. p ( z With the objective of iteratively updating particle positions to approximate the distribution of the particle set, the particle positions are determined. p ( z This process drives the particles toward p ( z High-probability region movement (maximizing information gain) while maintaining inter-particle distance through kernel function gradient terms (ensuring spatial diversity).
[0046] Each particle in each iteration z iUpdate according to the following rules:
[0047] in or Step size, ( z i ) is the update function, which represents the update at position z i The SVGD update vector at that location indicates the particle. z i The direction and magnitude of the movement to be performed in this iteration is a two-dimensional vector (corresponding to image pixel coordinates), the calculation of which depends on the target probability distribution. p ( z gradient of ) The calculation is as follows:
[0048] in, The radial basis function kernel is a measure of the interaction between two particles. z j and z i A function of similarity between them , s For bandwidth parameters; p ( z j ) for particles z j The target probability density value at the location is a distribution constructed from the normalized negative uncertainty gradient map; For the target probability distribution p ( z In particles z j The gradient (a two-dimensional vector) at the location, its direction p ( z The fastest growing sector; this is the driving force. For kernel function For the first parameter z j The gradient (a two-dimensional vector) points to... z j and z i The more dissimilar (i.e., farther away) direction, the more repulsive the force, preventing all particles from gathering at the same point.
[0049] S54: Select the particle position that finally converges as the point to be explored.
[0050] After a preset number of iterations, the final converged particle positions are used as the selected set of points to be explored.
[0051] S6: Under the guidance of the intraoperative navigation system, the tactile sensing end of the surgical robot is controlled to move sequentially to the three-dimensional anatomical position in the body corresponding to the point to be explored to perform contact measurement and obtain new high-precision depth measurement values. This step specifically includes: The coordinates of the selected point to be explored in the image space are combined with the pre-calibrated camera intrinsic parameter matrix and back-projected onto the camera coordinate system to form a three-dimensional ray. The intraoperative navigation system, based on optical or electromagnetic positioning data, converts the three-dimensional rays to a unified patient coordinate system and generates the robot end effector motion path; Control the surgical robot to move along the end effector path until its tactile sensing end detects a contact event, and record the robot joint angle at this time; Based on the robot's forward kinematics model, the three-dimensional coordinates of the contact point in the patient's coordinate system are calculated and further converted into precise depth measurements relative to the camera.
[0052] The tactile sensing end is a capacitive tactile sensor, torque sensor, capacitive sensor, or bioimpedance sensor integrated into the end of the surgical robot. Taking a concentric tube robot as an example, its tactile sensing end can consist of a guidewire or probe equipped with a capacitive contact sensor, which enters the body through the working channel of the endoscope.
[0053] Specifically, for each selected point to be explored, the intraoperative navigation system (e.g., based on optical positioning) combines it with the current endoscopic pose and calculates the target's three-dimensional position in the patient coordinate system through camera back-projection. The surgical robot's end effector moves along the planned path to the vicinity of this position until a tactile sensor detects contact, records the joint angles, and obtains the precise contact point depth through kinematic calculations. d new .
[0054] S7: Update the newly acquired depth measurement value to the sparse depth measurement set, and return to step S3 to update the depth estimation ensemble model input; In this step, updating the sparse depth measurement set with the newly acquired depth measurement values includes: adding the newly acquired depth measurement values, in the form of pixel coordinate-depth value pairs, to the existing sparse depth measurement set, thus forming a more informative conditional input for the next depth estimation. For example, adding the newly acquired measurement points ( x i , y i , d new Add it to the sparse depth measurement set S.
[0055] S8: Repeat steps S3 to S7 to iteratively optimize depth estimation and uncertainty assessment until the iteration stopping condition is met. Then, output the integrated dense depth prediction map finally generated by the depth estimation integrated model as an accurate three-dimensional scene model for surgical navigation.
[0056] In this step, the iteration stopping condition is one of the following three: the total uncertainty scalar of the prediction uncertainty graph is lower than a preset threshold; the total number of tactile measurements performed has reached a preset upper limit; or the cumulative measurement time exceeds a preset time limit.
[0057] Specifically, determine whether the iteration stopping condition is met. The stopping condition can be: total uncertainty. U total Below the preset threshold Or the total number of measurement points has reached the upper limit. N max If the cumulative measurement time exceeds the limit, or if the condition is not met, return to step S3 to re-evaluate and plan the depth using the updated set S, forming a closed-loop optimization. If the condition is met, output the final iteratively obtained integrated dense depth prediction map. This depth map It can be further converted into a 3D point cloud or mesh model, and after being registered with preoperative images, it provides surgeons with an augmented reality view in the navigation interface.
[0058] The technical principle of this invention lies in constructing a perception framework that is "uncertainty-driven and closed-loop active learning," the core logic of which is as follows: First, based on the inherent perceptual uncertainty of monocular vision in surgical scenarios, it is explicitly quantized into a variance map through a deep ensemble model. U Secondly, the objective of "reducing global uncertainty" is transformed into a differentiable optimization problem, which is solved by calculating the gradient. To evaluate the information value of each spatial location. Then, using the SVGD particle optimization algorithm, the information value (defined by the gradient map as the target distribution) is evaluated. p ( z The goal is to find the optimal balance between tactile feedback and the diversity of spatial exploration, thereby planning the most efficient sequence of tactile measurement points. Each tactile measurement provides the system with sparse but absolutely accurate geometric anchor points for updating and correcting the visual model. Through multiple iterative cycles of perception (vision + tactile) → quantization (uncertainty) → decision-making (gradient + SVGD) → execution (robot contact) → update (set), the area of perceptual uncertainty can be adaptively reduced until the accuracy or cost requirements are met.
[0059] Based on the above principles, the present invention achieves the following significant technical effects: 1) Fundamentally improved perception reliability: Through the deep fusion of tactile truth anchors and visual information, the scale ambiguity and local errors of monocular depth estimation are effectively corrected, generating a globally consistent and locally accurate dense 3D model.
[0060] 2) Achieves the optimal trade-off between cost and accuracy: Active planning based on uncertainty gradients ensures that every valuable tactile measurement is used at the position that best improves global accuracy, while the diversity guarantee of SVGD avoids measurement redundancy. Thus, with the fewest physical interactions, it achieves the level of accuracy that traditional methods require more measurements to obtain.
[0061] 3) It achieves adaptive, closed-loop perception capabilities: the algorithm is no longer a static estimator, but an intelligent agent capable of autonomous decision-making and dynamic optimization based on real-time perception status. The final output is not only a 3D model, but also a perception result with known confidence that can be directly used for safe navigation. This provides an innovative systematic solution to resolving the contradiction between accurate perception and low invasiveness in robot-assisted surgery.
[0062] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.
Claims
1. A monocular depth estimation method based on active perception for intraoperative navigation in robot-assisted surgery, characterized in that, The method includes: S1: Acquire a monocular RGB image of the surgical area using an endoscopic imaging system; S2: Obtain the initial set of sparse depth measurements; S3: Input the RGB image and the sparse depth measurement set at the current time into a depth estimation ensemble model; wherein the sparse depth measurement set is used as an additional input condition of the model in a manner aligned with the spatial configuration of the RGB image; the depth estimation ensemble model outputs an ensemble dense depth prediction map as a depth estimate of the current scene, and a prediction uncertainty map for evaluating the reliability of the estimation. S4: Based on the predicted uncertainty map, calculate its gradient with respect to the measurement values in the current sparse depth measurement set, and generate a gradient map indicating the measurement information value of each pixel. S5: Based on the gradient map, the Stein variational gradient descent algorithm is used to actively select the next set of image spatial locations that can minimize the overall uncertainty as the points to be explored. S6: Under the guidance of the intraoperative navigation system, the tactile sensing end of the surgical robot is controlled to move sequentially to the three-dimensional anatomical position in the body corresponding to the point to be explored to perform contact measurement and obtain new high-precision depth measurement values. S7: Update the newly acquired depth measurement value to the sparse depth measurement set, and return to step S3 to update the depth estimation ensemble model input; S8: Repeat steps S3 to S7 to iteratively optimize depth estimation and uncertainty assessment until the iteration stopping condition is met. Then, output the integrated dense depth prediction map finally generated by the depth estimation integrated model as an accurate three-dimensional scene model for surgical navigation.
2. The method according to claim 1, characterized in that, In step S1, the monocular RGB image is acquired in real time during the operation by a camera fixed to the front end of the endoscope.
3. The method according to claim 1, characterized in that, In step S2, the sparse depth measurement set is either an empty set or contains several prior measurement points. Each measurement point includes a pixel coordinate with a definite location in the monocular RGB image and a depth measurement value corresponding to that pixel coordinate.
4. The method according to claim 1, characterized in that, In step S3, the depth estimation ensemble model includes multiple monocular depth estimation sub-models with the same structure and trained independently. Each monocular depth estimation sub-model is based on the U-Net architecture. Each monocular depth estimation sub-model receives the RGB image and the sparse depth measurement set as input and outputs a dense depth prediction map corresponding to the input. The integrated dense depth prediction map is the pixel-level average of the dense depth prediction maps output by all monocular depth estimation sub-models; the prediction uncertainty map is the variance map of the dense depth prediction maps output by all monocular depth estimation sub-models at the corresponding pixel positions.
5. The method according to claim 3, characterized in that, Step S4 specifically includes: The average uncertainty value of the prediction uncertainty map is calculated as the total uncertainty scalar, which is the average of the uncertainty values of all pixels in the prediction uncertainty map; The gradient of the total uncertainty scalar with respect to the sparse depth measurement set is calculated using the backpropagation algorithm to obtain a gradient map, which indicates the expected contribution of adding depth measurements at various locations in the image to reducing the overall uncertainty.
6. The method according to claim 1, characterized in that, Step S5 specifically includes: The gradient map is negative and normalized to construct a target probability distribution as the optimization objective, such that high probability regions correspond to pixel positions with large absolute gradient values. Initialize multiple particles representing candidate measurement positions, with each particle corresponding to an image pixel coordinate; Using the Stein variational gradient descent algorithm, with the target probability distribution as the objective, the particle positions are iteratively updated so that the distribution of the particle set approximates the target probability distribution; The position of the particle that finally converges is selected as the point to be explored.
7. The method according to claim 1, characterized in that, Step S6 specifically includes: The coordinates of the selected point to be explored in the image space are combined with the pre-calibrated camera intrinsic parameter matrix and back-projected onto the camera coordinate system to form a three-dimensional ray. The intraoperative navigation system, based on optical or electromagnetic positioning data, converts the three-dimensional rays to a unified patient coordinate system and generates the robot end effector motion path; Control the surgical robot to move along the end effector path until its tactile sensing end detects a contact event, and record the robot joint angle at this time; Based on the robot's forward kinematics model, the three-dimensional coordinates of the contact point in the patient's coordinate system are calculated and further converted into precise depth measurements relative to the camera.
8. The method according to claim 1 or 7, characterized in that, In step S6, the tactile sensing end is a capacitive tactile sensor, torque sensor, capacitive sensor, or bioimpedance sensor integrated into the end of the surgical robot.
9. The method according to claim 1, characterized in that, In step S7, updating the newly acquired depth measurement value to the sparse depth measurement set includes: The newly acquired depth measurements are added to the existing sparse depth measurement set in the form of pixel coordinate-depth value pairs, forming a more informative conditional input for the next depth estimation.
10. The method according to claim 1, characterized in that, In step S8, the iteration stopping condition is one of the following three: The total uncertainty scalar of the predicted uncertainty graph is lower than a preset threshold; The total number of tactile measurements performed has reached the preset limit. The cumulative measurement time exceeds the preset time limit.