Three-Dimensional Reconstruction Method for the Intraoperative Dynamic Scene of a Surgical Robot

Abstract

The present invention relates to the field of surgical robots, and discloses a method for three-dimensional reconstruction of intraoperative dynamic scenes of a surgical robot, including: collecting an image sequence of a surgical scene, generating a depth map and extracting a tissue mask; constructing a three-dimensional reconstruction model based on 3DGS, including: an initialization module extracting three-dimensional point clouds in each frame according to the depth map and the tissue mask, and generating a dense initial Gaussian point cloud by fusing multi-frame point clouds through a binary motion mask; a flexible deformation model assigning dynamic attributes to each Gaussian point in the initial Gaussian point cloud, and learning the motion curve of each Gaussian point through a Gaussian-type combined basis function; training the three-dimensional reconstruction model, and optimizing the explicit parameters of the Gaussian points and the weights and parameters of the basis function of the flexible deformation model by minimizing the total loss function; loading the trained three-dimensional reconstruction model into a surgical navigation system to generate a three-dimensional view of the surgical scene in real time. The present invention can achieve efficient and accurate reconstruction of intraoperative dynamic three-dimensional scenes of a surgical robot.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical robots, and more particularly, to a method for three-dimensional reconstruction of intraoperative dynamic scenes of a surgical robot. Background Art

[0002] In the field of surgical robots, accurate intraoperative three-dimensional scene reconstruction is crucial for surgical navigation and operation. Existing three-dimensional reconstruction technologies have some problems when dealing with dynamic surgical scenes. For example, traditional stereo matching and SLAM (Simultaneous Localization and Mapping) methods assume that the scene is static and cannot effectively handle the dynamic changes of tissues and tools during the surgical process. While the method based on NeRF (Neural Radiance Field) performs well in static scene reconstruction, it has a long training time, slow rendering speed, and lacks a clear geometric representation in dynamic scenes. In recent years, the 3D Gaussian Splatting (3DGS) technology has received extensive attention due to its high efficiency and high-quality rendering effect. 3DGS represents the scene as a three-dimensional Gaussian point cloud and adopts an efficient differential rendering algorithm to quickly generate high-quality images and depth maps. However, although the recent 3D Gaussian rendering methods have improved the rendering speed, they still have deficiencies in initialization and depth change capture, especially when dealing with complex dynamic surgical scenes. Summary of the Invention

[0003] The object of the present invention is to propose a method for three-dimensional reconstruction of intraoperative dynamic scenes of a surgical robot, which realizes the efficient and accurate reconstruction of intraoperative dynamic three-dimensional scenes of a surgical robot through an improved three-dimensional Gaussian rendering model.

[0004] To achieve the above object, the present invention proposes a method for three-dimensional reconstruction of intraoperative dynamic scenes of a surgical robot, including:

[0005] S1: Collect an image sequence of the surgical scene, where the image sequence includes various surgical scenes and image pairs of left and right views with dynamic changes; preprocess the image sequence, generate a depth map for each frame based on the preprocessed image pairs, and extract a tissue mask for each frame;

[0006] S2: Construct a three-dimensional reconstruction model based on 3DGS, and the three-dimensional reconstruction model includes:

[0007] Initialization module: used to extract a three-dimensional point cloud for each frame according to the depth map and the tissue mask, and generate a dense initial Gaussian point cloud by fusing multi-frame point clouds through a binary motion mask;

[0008] Flexible deformation model: used to assign dynamic attributes to each Gaussian point in the initial Gaussian point cloud, and learn the motion curve of each Gaussian point through Gaussian-type combined basis functions;

[0009] Rendering module: used to convert the Gaussian point cloud into a 2D image and a depth map through a differential rendering algorithm;

[0010] S3: Use the preprocessed image, depth map, and tissue mask as training data to train the 3D reconstruction model, and optimize the explicit parameters of the Gaussian points and the weights and parameters of the basis functions of the flexible deformation model by minimizing the total loss function;

[0011] S4: Load the trained 3D reconstruction model in the surgical navigation system, generate a 3D view of the surgical scene in real time, and output for surgical navigation and robot control.

[0012] Optionally, in step S1, the depth map is generated by a stereo matching algorithm, and the tissue mask is extracted by an automatic segmentation algorithm or a pre-trained segmentation model.

[0013] Optionally, in step S2, the calculation formula for extracting the 3D point cloud in each frame based on the depth map and the tissue mask is:

[0014]

[0015] where P i is the 3D point cloud in the i-th frame, I i is the i-th frame image, D i is the depth map of the current i-th frame, M i is the tissue mask of the i-th frame, K 1, K 2 are the camera intrinsic matrix and the extrinsic matrix respectively, and ⊙ represents element-wise multiplication.

[0016] Optionally, in step S2, the method for generating the binary motion mask includes:

[0017] Calculate the depth map difference between each frame and the reference frame to generate a binary motion mask, and the reference frame is the first frame;

[0018] The generation formula of the binary motion mask is:

[0019] B i = I (∣ D 0 - D i ∣ > τ ) ∪ (1 - M0) ∩ M i

[0020] wherein, B i is the binary motion mask of the i-th frame, I (⋅) is the indicator function, M 0 is the tissue mask of the reference frame, M i is the tissue mask of the current frame, D 0 is the depth map of the reference frame, τ is the threshold of depth change, used to judge whether the depth change of the pixel is significant.

[0021] Optionally, in step S2, a dense initial Gaussian point cloud is generated by fusing multi-frame point clouds through the binary motion mask, and the fusion formula is:

[0022] P ={ P 0, P 1 ⊙ B 1, …, P T ⊙ B T}

[0023] wherein, P represents the fused initial Gaussian point cloud, P 0 represents the three-dimensional point cloud of the first frame, P i represents the i -th frame of the three-dimensional point cloud, B i is the binary motion mask of the i -th frame, T represents the number of frames, i ∈ [1, T].

[0024] Optionally, in step S2, the flexible deformation model represents the complex motion of tissues through a Gaussian-type basis function combination, and the position, rotation, and scaling changes of each Gaussian point are controlled by learnable weights, centers, and variances, where:

[0025] (1)

[0026] wherein, t represents time, θ and σ are the learnable center positions and variances;

[0027] The change in the position of the Gaussian point along the X-axis is represented by a basis function combination as:

[0028] (2)

[0029] Among them, ψ μ,x ( t ; Θ μ,x ) represents the position change of the Gaussian point along the x-axis at time t ; Θ μ,x represents the parameter set of the position change along the x-axis, , represents the contribution of the j -th basis function in the motion curve of the position change along the x-axis, represents the central position of the j -th basis function in time, representing the j -th basis function in time width; represents the j -th Gaussian basis function, which is calculated using formula (1); j is an index variable used to traverse the set of basis functions, and the number of basis functions is determined by B , represented as j = 1, 2, …, B , and each basis function j has its own parameters, including the weight ω , the center θ , and the variance σ ;

[0030] The position of the Gaussian point along the x-axis at time t is expressed as:

[0031] μ x ( t ) = μ x (0) + ψ μ,x ( t )

[0032] Among them, μ x ( t ) represents the position of the Gaussian point along the x-axis at time t , μ x (0) represents the position of the Gaussian point along the x-axis at the initial time t = 0, ψ μ,x ( t ) represents the position change of the Gaussian point along the x-axis at time t , which is calculated from formula (2).

[0033] Optionally, in step S3, the process of training the three-dimensional reconstruction model includes:

[0034] The model updates the dynamic attributes of the Gaussian point cloud according to the input depth map and tissue mask;

[0035] The flexible deformation model adjusts the attributes of the Gaussian points according to the time series;

[0036] The rendering module renders the updated Gaussian point cloud into a two-dimensional image and a depth map;

[0037] Calculate the total loss through the total loss function. The total loss includes photometric loss, normalized depth loss, and total variation loss. The photometric loss is used to optimize the color difference between the rendered image and the actual image, and the normalized depth loss and the total variation loss are used to optimize the difference between the predicted depth map and the actual depth map;

[0038] According to the calculated total loss, update the model parameters through the backpropagation algorithm to minimize the total loss function.

[0039] Optionally, the calculation formula for the normalized depth regularization loss is:

[0040]

[0041] Where, is the normalized depth regularization loss, M is the mask, and D norm are the normalized predicted depth map and the actual depth map respectively. ⊙ represents the element-wise multiplication operation, and the mask M is applied to the difference calculation of the depth map.

[0042] Optionally, the calculation formula for the total variation loss is:

[0043]

[0044] Where, L smooth represents the depth smoothness loss, which is used to remove noise in the depth map and retain depth details, represents the number of valid pixels in the predicted depth map, is the result of Canny edge detection, indicating whether the pixel ( i , j ) is on the edge. If the pixel is on the edge, the value is 0, otherwise it is 1; represents the pixel ( i , j ) and its right pixel ( i +1,j the absolute value of the depth difference of indicating the pixel ( i , j ) and its lower pixel ( i , j + 1) of the absolute value of the depth difference.

[0045] Optionally, the total loss function is:

[0046]

[0047] where L colour is the original photometric loss of 3DGS, is the normalized depth loss, L smooth is the total variation loss, λ smooth is the weight coefficient of the smooth loss.

[0048] The beneficial effects of the present invention are as follows:

[0049] Through the improved three-dimensional Gaussian rendering framework and the adoption of 3D Gaussian rendering technology, the present invention significantly improves the rendering speed and meets the requirements of intraoperative real-time navigation. Through the flexible deformation model and depth optimization steps, it accurately captures the dynamic changes in the surgical scene and improves the geometric accuracy of three-dimensional reconstruction. By introducing normalized depth regularization and unsupervised depth smoothing, it enhances the model's ability to handle depth map noise and errors, improves the stability and reliability of the reconstruction results, and thus realizes the efficient and accurate reconstruction of the intraoperative dynamic surgical scene of the surgical robot.

[0050] The system of the present invention has other characteristics and advantages, which will be obvious from the accompanying drawings incorporated herein and the subsequent detailed description, or will be described in detail in the accompanying drawings incorporated herein and the subsequent detailed description, and these accompanying drawings and detailed description are used together to explain the specific principles of the present invention. Brief Description of the Drawings

[0051] By describing the exemplary embodiments of the present invention in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present invention will become more obvious. In the exemplary embodiments of the present invention, the same reference numerals generally represent the same components.

[0052] Figure 1 Shows a flowchart of the steps of a method for three-dimensional reconstruction of an intraoperative dynamic scene of a surgical robot according to the present invention. Detailed Description of the Embodiments

[0053] The present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present invention will be more thorough and complete, and will fully convey the scope of the present invention to those skilled in the art.

[0054] As Figure 1 shown, a three-dimensional reconstruction method for intraoperative dynamic scenes of a surgical robot according to the present invention includes:

[0055] S1: Collect an image sequence of the surgical scene, where the image sequence includes image pairs of various surgical scenes and dynamically changing left and right views; preprocess the image sequence, generate a depth map for each frame based on the preprocessed image pairs, and extract the tissue mask in each frame;

[0056] Specifically, use the stereoscopic endoscope (such as laparoscope, thoracoscope) of the surgical robot to collect the image sequence of the surgical video, including the image pairs of the left and right views, ensuring that the data covers various surgical scenes and dynamic changes. These data include images under different patients, different surgical stages, and different operation methods, ensuring that the model is exposed to rich variations. Preprocess the collected images, including operations such as correction and denoising, to improve the accuracy of subsequent processing.

[0057] Then, generate a depth map for each frame image through a stereo matching algorithm, and extract the depth information of the tissue region in combination with the tissue mask. The depth map D i can be expressed as:

[0058] D i = StereoMatching( I i,left , I i,right ), where I i,left and I i,right are the left and right view images of the i th frame respectively, and StereoMatching represents stereo matching.

[0059] Extract the tissue mask in each frame through an automatic segmentation algorithm or a pre-trained segmentation model M i :

[0060] M i = Segmentation( I i )

[0061] Among them, I i is the i input image of the i-th frame, including the left and right images in relative positions, and Segmentation represents segmentation.

[0062] S2: Construct a 3D reconstruction model based on 3DGS. The 3D reconstruction model includes: Initialization module: used to extract the 3D point cloud in each frame according to the depth map and tissue mask, and generate a dense initial Gaussian point cloud by fusing the point clouds of multiple frames through a binary motion mask;

[0063] Flexible deformation model: used to assign dynamic attributes to each Gaussian point in the initial Gaussian point cloud, and learn the motion curve of each Gaussian point through Gaussian-type combined basis functions;

[0064] Rendering module: used to convert the Gaussian point cloud into a 2D image and a depth map through a differential rendering algorithm;

[0065] In this step, a 3D reconstruction model based on 3DGS is constructed, which mainly includes an initialization module, a flexible deformation model, and a rendering module.

[0066] The initialization module is used to: extract the 3D point cloud in each frame based on the depth map and tissue mask, design a binary motion mask for each frame, identify regions with significant depth changes, and fuse the point clouds of all frames to form a dense initial Gaussian point cloud. Specifically, the point cloud generation is based on the camera parameters K 1 and K 2 to convert the depth map into a 3D point cloud P i :

[0067]

[0068]

[0069] Among them, P i is the 3D point cloud in the i-th frame, I i is the i-th frame image, D i is the depth map of the current i-th frame, M i is the tissue mask of the i-th frame, K 1, K 2 are the camera intrinsic matrix and extrinsic matrix respectively, and ⊙ represents element-wise multiplication.

[0070] Then, calculate the depth map difference between each frame and the reference frame (usually the first frame) to generate a binary motion mask B i :

[0071] B i = I (∣ D 0 - D i ∣> τ ) ∪ (1 - M 0) ∩ M i

[0072] wherein, B i is the binary motion mask of the i-th frame, I (⋅) is the indicator function, M 0 is the tissue mask of the first frame, M i is the tissue mask of the current frame, D 0 is the depth map of the first frame, τ is the threshold of depth change, used to judge whether the depth change of pixels is significant.

[0073] After that, multiple frames of point clouds are fused through the binary motion mask to form a dense initial Gaussian point cloud P：

[0074] P ={ P 0, P 1 ⊙ B 1, …, P T ⊙ B T}

[0075] wherein, P represents the initial Gaussian point cloud after fusion, P 0 represents the 3D point cloud of the first frame, P i represents the i -th frame of the 3D point cloud, B i is the binary motion mask of the i -th frame, T represents the number of frames, i ∈ [1, T].

[0076] The flexible deformation model is used to: represent the dynamic scene by using a deformation field, assign dynamic attributes including position, rotation, and scaling to each Gaussian point in the initial Gaussian point cloud, and learn the motion curve of each Gaussian point through Gaussian-type combined basis functions; specifically, the flexible deformation model represents the dynamic scene by using a deformation field, associates a set of learnable weights ω and parameters Θ, and represents the motion curve of the Gaussian point by using Gaussian-type basis functions.

[0077] Specifically, the flexible deformation model learns the motion curves of each Gaussian point through Gaussian basis functions and represents the complex motion of the tissue through the combination of Gaussian basis functions . The position, rotation, and scaling changes of each Gaussian point are controlled by learnable weights, centers, and variances, where:

[0078] (1)

[0079] where t represents time, θ and σ are the learnable center position and variance;

[0080] Each Gaussian point is associated with a set of learnable weights ω and parameters Θ μ , Θ r , Θ s . By linearly combining these weights and parameters with basis functions, the deformations of position, rotation, and scaling are represented respectively, where: Θ μ represents the set of parameters for position change, including the position change parameters along the x, y, and z axes Θ μ,x , Θ μ,y and Θ μ,z , Θ μ,x , Θ μ,y and Θ μ,z respectively include the corresponding weight, center, and variance parameters; Θ r represents the set of parameters for rotation change, including the weight, center, and variance of rotation; Θ s represents the set of parameters for scaling change, including the weight, center, and variance of scaling;

[0081] For example, the change in the position of the Gaussian point along the X axis is represented by the combination of basis functions as:

[0082] (2)

[0083] where ψ μ,x ( t ; Θ μ,x ) represents the change in the position of the Gaussian point along the x axis at time t , which is represented by the linear combination of basis functions; Θ μ,x represents the set of parameters for position change along the x axis, , including the weight in the x direction, center and variance , represents the contribution of the j -th basis function to the motion curve of position change along the x axis, represents the central position of the j -th basis function in time, and represents the j width of the -th basis function in time; j represents the j -th Gaussian basis function, and its calculation formula is formula (1); B is an index variable used to traverse the set of basis functions. The number of basis functions is determined by j and is expressed as B = 1, 2, …, j Each basis function ω j has its own parameters, including the learnable weight θ j (controlling the importance of the basis function), the learnable center σ j and variance

[0084] The position of the Gaussian point along the x - axis at time t is expressed as:

[0085] μ x ( t ) = μ x (0)+ ψ μ,x ( t ) (3)

[0086] where μ x ( t ) represents the position of the Gaussian point along the x - axis at time t , μ x (0) represents the position of the Gaussian point along the x - axis at the initial time t = 0, ψ μ,x ( t ) represents the change in the position of the Gaussian point along the x - axis at time t , which is calculated by formula (2).

[0087] Similarly, the changes in the position of the Gaussian point along the Y - axis and Z - axis are represented by the combination of basis functions as:

[0088]

[0089] where ψ μ,y ( t ; Θ μ,y) represents the position change of the Gaussian point along the y-axis at time t , where , are the weight, center, and variance parameters of the position change along the y-direction; ψ μ,z ( t ; Θ μ,z ) represents the position change of the Gaussian point along the z-axis at time t , where , are the weight, center, and variance parameters of the position change along the z-direction; The basis functions are all in the form of formula (1). The position update refers to the form of formula (3).

[0090] For the quaternion components of rotation, the dynamic change formula for each component of the quaternion r = r 1, r 2, r 3, r 4] T is as follows:

[0091]

[0092] where . The updated quaternion needs to be normalized.

[0093] The scaling factor s = s x , s y , s z T has the following dynamic change formula:

[0094]

[0095] where .

[0096] Finally, the position, rotation, and scaling updates of each point cloud at time t are jointly achieved by the above parameters (the initial value plus the change).

[0097] The rendering module is used to: convert the Gaussian point cloud into a 2D image and a depth map through a differential rendering algorithm; specifically, the rendering module performs rendering based on the existing 3DGS differential algorithm, and the process is as follows:

[0098] The influence of each Gaussian point is defined by a Gaussian distribution:

[0099]

[0100] where G ( x ) represents the value of the Gaussian distribution, which is used to measure the point​x At the Gaussian point μ the response intensity x represents a point in space, usually a three - dimensional vector x , y , z , μ represents the central position of the Gaussian point, usually a three - dimensional vector μ x , μ y , μ z , Σ represents the covariance matrix of the Gaussian point, which determines the shape and orientation of the Gaussian distribution, Σ −1 represents the inverse of the covariance matrix, which is used to calculate the exponential part of the Gaussian distribution.

[0101] Project the 3D Gaussian point onto the 2D image plane through the camera parameters, and calculate the projected covariance matrix Σ′:

[0102] Σ′ = JW Σ W T J T

[0103] where W is the perspective transformation matrix, J is the Jacobian matrix of the projection, Σ is the 3D covariance matrix of the Gaussian point, T represents the transpose.

[0104] Rasterize the projected 2D Gaussian, and calculate the pixel color and depth through alpha blending:

[0105]

[0106] where represents the color value of the rendered pixel x of represents the depth value of the rendered pixel x of N represents the set of Gaussian points that affect the pixel x of c i represents the i - th Gaussian point's color, d i represents the i - th Gaussian point's depth; α i represents the i - th Gaussian point's opacity at the pixel x of , where oi Represents the opacity parameter of the i th Gaussian point, Represents the i th Gaussian point at the pixel x with a two-dimensional Gaussian distribution value.

[0107] S3: Use the preprocessed image, depth map, and tissue mask as training data to train the 3D reconstruction model. Optimize the explicit parameters of the Gaussian points and the weights and parameters of the basis functions of the flexible deformation model by minimizing the total loss function to improve the reconstruction accuracy of the model;

[0108] In this step, the process of training the 3D reconstruction model includes:

[0109] The model updates the dynamic attributes of the initial Gaussian point cloud according to the input depth map and tissue mask, including position, rotation, and scaling, etc.;

[0110] The flexible deformation model adjusts the attributes of the Gaussian points according to the time series;

[0111] The rendering module renders the updated Gaussian point cloud into a 2D image and a depth map;

[0112] Calculate the total loss through the total loss function. The total loss includes photometric loss, normalized depth loss, and total variation loss. The photometric loss is used to optimize the color difference between the rendered image and the actual image, and the normalized depth loss and the total variation loss are used to optimize the difference between the predicted depth map and the actual depth map;

[0113] According to the calculated total loss, update the model parameters through the backpropagation algorithm to minimize the total loss function. Preferably, use an optimization algorithm (such as the Adam optimizer) to adjust the model parameters to minimize the loss function.

[0114] Specifically, by introducing the normalized depth loss, ensure that the predicted depth map is accurately aligned with the actual depth map; specifically, the calculation formula of the normalized depth regularization loss is:

[0115]

[0116] Where is the normalized depth regularization loss, M is the mask, and D norm are the normalized predicted depth map and the actual depth map respectively. ⊙ represents the element-wise multiplication operation, and the mask M is applied to the difference calculation of the depth map.

[0117] Furthermore, the total variation loss is used to remove the noise in the depth map, and Canny edge detection is applied as a mask to prevent over-smoothing of the edges; the calculation formula of the total variation loss is:

[0118]

[0119] where L smooth represents the depth smoothing loss, which is used to remove the noise in the depth map and retain the depth details, represents the number of valid pixels in the predicted depth map, is the result of Canny edge detection, indicating whether the pixel ( i , j ) is on the edge. If the pixel is on the edge, the value is 0; otherwise, it is 1; represents the absolute value of the depth difference between the pixel ( i , j ) and its right pixel ( i +1, j ), represents the absolute value of the depth difference between the pixel ( i , j ) and its lower pixel ( i , j +1).

[0120] The total loss function includes the photometric loss (color loss), depth loss, and depth smoothing loss:

[0121]

[0122] where L colour is the original photometric loss of 3DGS (calculated by the existing method), is the normalized depth loss, L smooth is the total variation loss, λ smooth is the weight coefficient of the smoothing loss.

[0123] By minimizing the total loss function, the explicit parameters of Gaussian points and the weights and parameters of the basis functions of the flexible deformation model are optimized to improve the reconstruction accuracy of the model. The weights and parameters of the basis functions are optimized by minimizing the loss function to capture the changes over time in attributes such as the position, rotation, and scaling of Gaussian points. Through explicit parameter optimization and the flexible deformation model, the dynamic change characteristics in the surgical scene are captured to provide an accurate three-dimensional view to assist surgical operations. It is preferably to use the Adam optimizer to jointly optimize the Gaussian point attributes (position, rotation, scaling, transparency, color) and the basis function parameters to achieve efficient parameter optimization, and use GPU acceleration to render and dynamically cull low-transparency Gaussian points (such as o i <0.01), parallelize the rasterization process, and accelerate the real-time rendering speed.

[0124] Purposes of training: (1) Through dynamic tissue deformation modeling (utilizing the flexible deformation model): Learn the dynamic deformation patterns of tissues in the surgical scene due to tool operations (such as pushing, pulling, cutting, suturing, etc.) and accurately simulate complex deformation processes. (2) High-precision three-dimensional reconstruction: Accurately reconstruct the geometric structure and surface details of tissues under the challenges of tool occlusion, lighting changes, and low-texture areas. (3) Real-time rendering ability: Through efficient parameter optimization, ensure that the model can generate three-dimensional scenes in real time during surgery (≥30 FPS) to meet the timeliness requirements of surgical navigation. (4) Enhancement of generalization ability: Enable the model to adapt to different surgical scenes (such as laparoscopy, thoracoscopy) and tissue types (such as soft tissues, vascular structures) to enhance clinical applicability.

[0125] Among them, the role of basis function parameter learning is: Through training, adjust the weights ( ω j ), centers ( θ j ), and variances ( σ j ) of Gaussian-type basis functions so that they can be combined to represent complex deformations. For example, when the number of basis functions B = 17 and the variance σ j of the j-th basis function = 1.0:

[0126]

[0127] These parameters are automatically optimized through training, enabling the model to capture local deformation characteristics at different time points (such as the instantaneous displacement caused by tool pushing and pulling). The localization characteristics of the basis functions allow the model to independently learn deformations in different time periods and avoid distortion caused by global coupling.

[0128] The role of multi-degree-of-freedom modeling is: Independently parameterize the position (x, y, z), rotation (quaternion), and scaling (each axis), covering all degrees of freedom in three-dimensional space to ensure the accurate expression of complex deformations.

[0129] The function of depth constraint joint optimization is: by normalizing the depth regularization loss Avoid unstable optimization caused by depth range differences, significantly improve geometric accuracy, force the rendered depth to be consistent with the true depth, and at the same time suppress noise. Through the total variation loss (L smooth ), suppress noise, combine Canny edge detection to protect tissue boundaries, and balance the needs of detail preservation and smoothing. At the same time, utilize photometric loss (color consistency) and depth loss (geometric accuracy) to extract complementary information from stereo endoscopic videos and enhance the model's scene understanding ability.

[0130] The function of occluded area completion is: use multi-frame point cloud fusion initialization to supplement the areas occluded by tools in a single frame and improve the reconstruction integrity.

[0131] The function of dynamic mask generation is: screen dynamic areas through motion masks ( B i ), focus on optimizing the significantly deformed parts, and fuse multi-frame point clouds to solve the single-frame occlusion problem. For example, the tissue occluded by the tool in the first frame may be visible in subsequent frames, and the model generates a more complete initial point cloud by fusing this information.

[0132] The main function of the flexible deformation model is to capture the dynamic change characteristics of Gaussian points during the surgical process and learn the change characteristics in the surgical scene. Specifically:

[0133] During the surgical process, tissues and tools move and deform, resulting in changes in the attributes of Gaussian points such as position, rotation, and scaling. During the training process, the flexible deformation model learns the patterns and rules of these changes by optimizing the weights and parameters of the basis functions. Surgery is a dynamic process with temporal continuity and dependence. The flexible deformation model can capture the changes of Gaussian points in the time series, thus accurately representing the dynamic characteristics of the surgical scene. In addition, during the surgical process, tissues are stretched, compressed, moved, etc., resulting in changes in their shape and structure. The flexible deformation model can learn these deformation characteristics, thus more accurately reconstructing and rendering the dynamic morphology of tissues. Further, the movement trajectory and mode of surgical tools also have an important impact on the dynamic changes of the surgical scene. The flexible deformation model can learn the movement patterns of tools, including operations such as entry, exit, rotation, and shearing, as well as the interaction between tools and tissues. As the surgery progresses, the occlusion relationship between tissues and tools will constantly change. The flexible deformation model can learn the change characteristics of these occlusion relationships during the training process to help the model better handle and recover the occluded areas.

[0134] In summary, through dynamic deformation modeling, depth constraint optimization, and efficient parameter learning, model training achieves high-precision real-time reconstruction of intraoperative dynamic scenes. By adapting the 3DGS technology to the intraoperative scene of the surgical robot during the training process, the deficiencies of traditional methods in dynamic modeling, occlusion handling, and real-time performance are addressed, providing reliable three-dimensional perception support for surgical navigation and robot operation.

[0135] Preferably, after the training is completed, the trained model parameters are saved, and the performance of the model is evaluated using the validation dataset to ensure the accuracy and robustness of the model in different surgical scenarios.

[0136] S4: Load the trained three-dimensional reconstruction model in the surgical navigation system to generate a three-dimensional view of the surgical scene in real time, and output it for surgical navigation and robot control.

[0137] Specifically, load the trained three-dimensional reconstruction model and related modules for image preprocessing, depth map, and tissue mask extraction in the surgical navigation system. During the surgical process, images are collected in real time, depth maps and tissue masks are generated and input into the model. The Gaussian point cloud is updated through the flexible deformation model to capture the dynamic changes of the surgical scene. The rendering module generates real-time three-dimensional views and depth maps for doctors to perform surgical navigation and operations. In the training and application processes of the method of the present invention, through technical means such as multi-frame data fusion, depth regularization, and unsupervised depth smoothing, the efficiency and accuracy of intraoperative dynamic three-dimensional reconstruction of the robot are ensured. Moreover, in the face of complex surgical scenarios and various interference factors, such as noise and occlusion, the model can maintain stable performance and provide reliable reconstruction results.

[0138] The embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments.

Claims

1. A three-dimensional reconstruction method for intraoperative dynamic scenes of a surgical robot, characterized in that, Including: S1: Collect an image sequence of the surgical scene, where the image sequence includes various surgical scenes and image pairs of left and right views with dynamic changes; Preprocess the image sequence, generate a depth map for each frame based on the preprocessed image pairs, and extract the tissue mask for each frame; S2: Construct a three-dimensional reconstruction model based on 3DGS, where the three-dimensional reconstruction model includes: Initialization module: Used to extract the three-dimensional point cloud for each frame according to the depth map and tissue mask, and generate a dense initial Gaussian point cloud by fusing multi-frame point clouds through a binary motion mask; Flexible deformation model: Used to assign dynamic attributes to each Gaussian point in the initial Gaussian point cloud, and learn the motion curve of each Gaussian point through Gaussian-type combined basis functions; Rendering module: Used to convert the Gaussian point cloud into a two-dimensional image and a depth map through a differential rendering algorithm; S3: Use the preprocessed images, depth maps, and tissue masks as training data to train the three-dimensional reconstruction model, and optimize the explicit parameters of the Gaussian points and the weights and parameters of the basis functions of the flexible deformation model by minimizing the total loss function; S4: Load the trained three-dimensional reconstruction model in the surgical navigation system, generate a three-dimensional view of the surgical scene in real time, and output for surgical navigation and robot control.

2. The three-dimensional reconstruction method for intraoperative dynamic scenes of a surgical robot according to claim 1, wherein In step S1, the depth map is generated through a stereo matching algorithm, and the tissue mask is extracted through an automatic segmentation algorithm or a pre-trained segmentation model.

3. The three-dimensional reconstruction method of the intraoperative dynamic scene of the surgical robot according to claim 1, wherein In step S2, the calculation formula for extracting the three-dimensional point cloud for each frame based on the depth map and tissue mask is: ， Among them, P i is the 3D point cloud in the i-th frame, I i is the i-th frame image, D i is the depth map of the current i-th frame, M i is the tissue mask of the i-th frame, K 1, K 2 are the camera intrinsic matrix and extrinsic matrix respectively, and ⊙ represents element-wise multiplication.

4. The intraoperative dynamic scene three-dimensional reconstruction method of the surgical robot according to claim 3, wherein In step S2, the generation method of the binary motion mask includes: Calculate the depth map difference between each frame and the reference frame, and generate a binary motion mask, where the reference frame is the first frame; The generation formula of the binary motion mask is: B i = I (∣ D 0- D i ∣> τ )∪(1- M 0)∩ M i wherein, B i is the binary motion mask of the i-th frame, I (·) is the indicator function, M 0 is the tissue mask of the reference frame, M i is the tissue mask of the current frame, D 0 is the depth map of the reference frame, τ is the threshold of depth change, used to determine whether the depth change of a pixel is significant.

5. The intraoperative dynamic scene three-dimensional reconstruction method of the surgical robot according to claim 4, wherein, In step S2, a dense initial Gaussian point cloud is generated by fusing multi-frame point clouds through a binary motion mask, and the fusion formula is: P ={ P 0, P 1⊙ B 1,…, P T ⊙ B T} Among them, P represents the initial Gaussian point cloud after fusion, P 0 represents the 3D point cloud of the 0th frame, P i represents the i 3D point cloud of the B i is the binary motion mask of the i frame, T represents the maximum index value of the frame sequence, i ∈ [1, T].

6. The three-dimensional reconstruction method of the intraoperative dynamic scene of the surgical robot according to claim 1, wherein In step S2, the flexible deformation model represents the complex motion of the tissue through a Gaussian basis function combination, and the position, rotation, and scaling changes of each Gaussian point are controlled by learnable weights, centers, and variances, where: （1） Among them, t represents time, θ and σ are learnable central positions and variances; The change of the position of the Gaussian point along the X-axis is represented by a combination of basis functions as: （2） Among them, ψ μ,x ( t ; Θ μ,x ) represents the position change of the Gauss point along the x-axis at time t ; Θ μ,x represents the parameter set of the position change along the x-axis, , represents the contribution of the j -th basis function to the motion curve of the position change along the x-axis, represents the central position of the j -th basis function in time, representing the j -th basis function's width in time; represents the j -th Gaussian basis function, which is calculated using formula (1); j is an index variable used to iterate through the set of basis functions. The number of basis functions is determined by B and is expressed as j = 1, 2, …, B . Each basis function j has its own parameters, including the weight ω , the center θ and the variance σ ; The position of the Gauss point along the x-axis at time t is expressed as: μ x ( t )= μ x (0)+ ψ μ,x ( t ) Among them, μ x ( t ) represents the position of the Gauss point along the x-axis at time t . μ x (0) represents the position of the Gauss point along the x-axis at the initial time t = 0. ψ μ,x ( t ) represents the change in the position of the Gauss point along the x-axis at time t , which is calculated by formula (2).

7. The three-dimensional reconstruction method for intraoperative dynamic scenes of a surgical robot according to claim 1, characterized in that In step S3, the process of training the three-dimensional reconstruction model includes: The model updates the dynamic attributes of the Gaussian point cloud according to the input depth map and tissue mask; The flexible deformation model adjusts the attributes of the Gaussian points according to the time series; The rendering module renders the updated Gaussian point cloud into a two-dimensional image and a depth map; Calculate the total loss through the total loss function, where the total loss includes photometric loss, normalized depth loss, and total variation loss. The photometric loss is used to optimize the color difference between the rendered image and the actual image, and the normalized depth loss and the total variation loss are used to optimize the difference between the predicted depth map and the actual depth map; According to the calculated total loss, update the model parameters through the backpropagation algorithm to minimize the total loss function.

8. The three-dimensional reconstruction method for intraoperative dynamic scenes of a surgical robot according to claim 7, characterized in that The calculation formula of the normalized depth loss is: ， wherein, is the normalized depth loss, M is the mask, and D norm are the normalized predicted depth map and the actual depth map respectively, and ⊙ represents the element-wise multiplication operation, applying the mask M to the depth map difference calculation.

9. The three-dimensional reconstruction method of the intraoperative dynamic scene of the surgical robot according to claim 8, wherein The calculation formula of the total variation loss is: ， Among them, L smooth represents the depth smoothness loss, which is used to remove noise in the depth map and retain depth details, represents the number of valid pixels in the predicted depth map, is the result of Canny edge detection, indicating whether the pixel ( i , j ) is on the edge. If the pixel is on the edge, the value is 0, otherwise it is 1; represents the absolute value of the depth difference between the pixel ( i , j ) and its right pixel ( i +1, j ); represents the absolute value of the depth difference between the pixel ( i , j ) and its bottom pixel ( i , j +1).

10. The intraoperative dynamic scene three-dimensional reconstruction method for a surgical robot according to claim 1, wherein The total loss function is: ， Among them, L colour is the original photometric loss of 3DGS, is the normalized depth loss, L smooth is the total variation loss, λ smooth is the weight coefficient of the smooth loss.

Images