Intraoperative scene overall dual-constraint dynamic registration navigation system

Abstract

The present application provides an intraoperative scene overall dual-constraint dynamic registration and navigation system, which relates to the field of model reconstruction. The system includes an image acquisition module for acquiring multiple frames of images and determining a frame of target image from the multiple frames of images; a posture optimization module for extracting a binary mask and jointly adjusting the projection constraint and the depth constraint; a parameter initialization module for initializing multiple target parameters of the model; a first parameter update module for dynamically updating a part of the target parameters of the model; and a second parameter update module for optimizing the color consistency of multiple frames of images. The present application combines dual-constraint camera posture estimation with the coefficient solution of the Gaussian model for intraoperative navigation, thereby improving the modularity of the algorithm. The intraoperative scene overall dual-constraint dynamic registration and navigation system can dynamically adapt to the rapidly changing tissue morphology during surgery by continuously collecting images, performing image data processing on the images, and updating parameters to ensure the quality of model reconstruction.

Description

Technical Field

[0001] The present application relates to the field of model reconstruction technology, and in particular to an intraoperative scene overall dual-constraint dynamic registration navigation system. Background Art

[0002] In minimally invasive surgery, endoscopes provide surgeons with critical visual information. Traditional registration and reconstruction techniques typically rely on single-view image processing, resulting in insufficient registration accuracy and poor robustness. Dynamic adaptation often fails when dealing with rapidly changing surgical scenarios, easily leading to misidentification of tissue deformation. Specifically, when performing computed tomography (CT) or magnetic resonance imaging (MRI) scans on patients to obtain three-dimensional medical imaging data, optical tracking technology is often used to align intraoperative images with three-dimensional models. This process relies on manual feature matching and is often affected by factors such as lighting changes, bubble interference, and tissue occlusion, resulting in reduced system accuracy and a lack of effective adaptive adjustment mechanisms when dealing with various dynamic changes.

[0003] Related technologies also exist for rapidly reconstructing 3D models of deformable tissues in surgical scenarios. For example, Deform3DGS leverages 3D Gaussian scattering technology to efficiently handle dynamic intraoperative changes through flexible deformation modeling, ensuring accurate and effective navigation within the surgical environment. By constructing a flexible Gaussian point cloud model, it enables tissue reconstruction from multiple perspectives.

[0004] However, current three-dimensional models lack robustness in intraoperative registration during rapid reconstruction and are easily limited by single-view information, leading to misidentification of tissue deformation. They perform poorly in terms of global consistency and cannot effectively handle rapidly changing tissue morphology in dynamic scenes. In complex scenes, they have difficulty adapting to factors such as blur and occlusion, resulting in suboptimal model reconstruction quality. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present application provides an intraoperative scene overall dual-constraint dynamic registration navigation system, which solves the problem that the current three-dimensional model rapid reconstruction technology cannot accurately handle the rapidly changing tissue morphology in dynamic scenes and the model reconstruction quality is not ideal.

[0006] To achieve the above objectives, this application is implemented through the following technical solutions:

[0007] An embodiment of the present application provides an intraoperative scene overall dual-constraint dynamic registration navigation system, which includes an image acquisition module, a posture optimization module, a parameter initialization module, a first parameter updating module, and a second parameter updating module.

[0008] Specifically, the image acquisition module is used to acquire multiple frames of images from the endoscope during minimally invasive surgery, and to determine a frame of target image from the multiple frames; the pose optimization module is used to process the target image using a pre-trained deep learning segmentation network to extract a binary mask, and to construct a constraint loss function to jointly adjust the projection constraint and the depth constraint to optimize the camera pose; the parameter initialization module is used to initialize multiple target parameters of the model based on the topological structure of the acquired preoperative three-dimensional Gaussian model and the characteristics of the target image; the first parameter update module is used to determine the reduction of the pixel-level photometric loss between the rendered image of the model and the target image as the optimization goal during the model adjustment and optimization process, and to dynamically update a part of the target parameters of the model so that the model can dynamically adapt to changes in tissue during the operation; the second parameter update module is used to optimize the color consistency of multiple frames of images, update another part of the target parameters of the model, and obtain the reconstructed target three-dimensional Gaussian model.

[0009] According to an embodiment of the present application, some target parameters include Gaussian center position, transparency, and covariance matrix, and another part of the target parameters includes color coefficients; the target three-dimensional Gaussian model is an overall three-dimensional model that is dynamically aligned to the intraoperative image and has color.

[0010] According to an embodiment of the present application, the aforementioned posture optimization module is specifically used to: use a pre-trained deep learning segmentation network to process the target image to distinguish the target tissue from the background, and extract the binary mask of the target tissue in the intraoperative environment; wherein the background of the target image includes illumination change information, bubbles and other instruments; after obtaining the binary mask, determine the projection constraint and depth constraint based on the geometric information of the acquired preoperative three-dimensional Gaussian model; construct a constraint loss function to adjust the projection constraint and depth constraint to optimize the camera pose; wherein the constraint loss function satisfies the expression:

[0011]

[0012] Where, is the total constraint loss, represents the rotation matrix, T represents the translation vector, and They are all weight parameters and can be adaptively adjusted during the optimization process to ensure the accuracy and stability of camera pose estimation; It is the projection loss and is used to characterize the Hausdorff distance between the projection of the preoperative 3D Gaussian model and the contour of the binary mask Mask. It is the depth loss and is related to the set depth estimation type; the depth estimation type is one of binocular depth estimation and monocular depth estimation.

[0013] According to an embodiment of the present application, when the depth estimation type is set to binocular depth estimation, the depth loss is calculated using Euclidean distance and the calculation process satisfies the expression:

[0014]

[0015] Where, is the depth predicted by the preoperative 3D Gaussian model in the binocular depth estimation scenario, is the depth actually observed in the binocular depth estimation scene; N is the number of samples used to calculate the Euclidean distance loss in binocular depth estimation.

[0016] According to an embodiment of the present application, when the depth estimation type is set to monocular depth estimation, a relative depth loss function is determined to compare the depth loss based on the relative depth, and the calculation process of the depth loss satisfies the expression:

[0017]

[0018] Where, , is the depth predicted by the preoperative 3D Gaussian model in the monocular depth estimation scenario, is the actual observed depth in the monocular depth estimation scene, It is a reference depth value and is used to convert the depth information into the form of relative depth; M is the number of samples used to calculate the relative depth loss in monocular depth estimation.

[0019] According to an embodiment of the present application, the Gaussian center position satisfies the expression: , where is the Gaussian center position, is the center position of the preoperative 3D model and represents the original position of Gaussian point i in the undeformed state; It is the adjustment amount based on the intraoperative image changes to characterize the deformation of the target tissue during actual surgery;

[0020] Transparency satisfies the expression: Where, For transparency, It is the input feature associated with Gaussian point i and is used to determine the transparency of Gaussian point i in the scene; Is an activation function and is used to transform the input features Mapped to a transparency value to describe the visibility of Gaussian point i;

[0021] The covariance matrix is ​​used to characterize the shape and extension of Gaussian point i to describe the distribution of Gaussian point i in space and satisfies the expression: , where is the covariance matrix, , , are the standard deviations of the Gaussian points in the X, Y, and Z directions, respectively, and represent the extent of the Gaussian points in the X, Y, and Z directions, respectively.

[0022] According to an embodiment of the present application, the aforementioned first parameter update module is specifically used to: construct a loss function corresponding to the pixel-level photometric loss to improve the similarity between the rendering result and the actual image by minimizing the loss; based on a preset image rendering function, generate a rendered image through the current model in the optimization process, and determine to reduce the pixel-level photometric loss between the rendered image of the model and the target image as the optimization goal; evaluate the model under the current parameters based on the loss function corresponding to the pixel-level photometric loss, and calculate the corresponding loss value; iteratively update a part of the target parameters of the model through the gradient descent method to continuously reduce the loss.

[0023] According to an embodiment of the present application, the loss function corresponding to the pixel-level luminosity loss satisfies the expression:

[0024]

[0025] in, is the loss function corresponding to the pixel-level photometric loss, H is the total number of pixels in the image, is a rendered image generated by the current model during the optimization process at position The pixel value of The actual intraoperative image is at position The pixel value of .

[0026] According to an embodiment of the present application, based on a preset image rendering function, the process of generating a rendered image by the current model in the optimization process satisfies the expression:

[0027]

[0028] Where, A rendered image generated for the current model, For transparency, is the Gaussian center position, is the covariance matrix, represents the position in the image, i represents the Gaussian point, Indicates the number of Gaussian points involved in rendering; is an image rendering function that satisfies the expression:

[0029]

[0030] in, is the determinant of the covariance matrix, represents the transpose of the matrix, represents the inverse matrix of the covariance matrix, Used to determine the rendering value generated at position x;

[0031] The aforementioned process of iteratively updating part of the target parameters of the model through the gradient descent method satisfies the expression: Where, Represents a part of the updated target parameters, Represents a part of the target parameters before updating, is the learning rate and is used to control the size of the adjustment step; is the loss function The gradient of is used to characterize the rate of change of the loss with respect to a portion of the target parameters.

[0032] According to an embodiment of the present application, the aforementioned second parameter update module is specifically used to: construct a color optimization loss function when multiple frames of images are aligned with the actual intraoperative scene, and the optimization goal of the color optimization loss function is to ensure the visual consistency of multiple frames of images under different viewing angles; iteratively update the color coefficient of the model by the gradient descent method; in each iteration, calculate the gradient according to the color optimization loss function, and update the color coefficient of the model to obtain the reconstructed target three-dimensional Gaussian model; wherein the color coefficient is used to characterize the spherical harmonic coefficient of the RGB channel and satisfies the expression: , where is the color coefficient, r, g, b are the color components of the red, green, and blue channels respectively.

[0033] According to an embodiment of the present application, the color optimization loss function satisfies the expression:

[0034]

[0035] Where, The first one is obtained by rendering the current model Color information of the image; The actual collection The color information of each image, n represents the total number of images; is the regularization weight, which is used to balance the weight between color consistency and smoothness; is the total variation regularization term and is used to smooth the color distribution between adjacent Gaussian nodes; Satisfies the expression:

[0036]

[0037] Where, Indicates the color in The value of the Gaussian nodes, Indicates the color in The value of the Gaussian nodes, Indicates the color in The value of the Gaussian nodes.

[0038] This application provides a dual-constraint dynamic registration navigation system for an intraoperative scene. Compared with existing technologies, it has the following advantages:

[0039] This application collects multiple frames of images during minimally invasive surgery, selects a frame of target image from the multiple frames for processing, and gradually reduces the pixel-level luminosity loss between the model's rendered image and the target image to update part of the target parameters of the preoperative three-dimensional Gaussian model; at the same time, color consistency optimization is performed based on the joint multi-frame images to avoid interference from factors such as light and perspective in multi-frame image acquisition, thereby improving the visual quality of the model; specifically, after extracting the binary mask, this application constructs a constraint loss function to jointly adjust the projection constraint and depth constraint. Compared with the traditional single constraint method, the positioning error is greatly reduced, and the estimation accuracy of the camera pose is improved; the system can be used throughout the entire operation, by continuously collecting images, updating parameters, and dynamically adapting to the rapidly changing tissue morphology during surgery to ensure the quality of model reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a structural diagram of an intraoperative scene overall dual-constraint dynamic registration navigation system provided by an embodiment of the present application;

[0042] Figure 2 This is a flow chart of the overall dual-constraint dynamic registration navigation of the intraoperative scene provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0044] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0045] The embodiment of the present application solves the problem that the current three-dimensional model rapid reconstruction technology cannot accurately handle the rapidly changing tissue morphology in dynamic scenes and the model reconstruction quality is not ideal by providing an overall dual-constraint dynamic registration navigation system for intraoperative scenes.

[0046] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0047] In minimally invasive surgery, endoscopes provide surgeons with critical visual information. However, traditional registration and reconstruction techniques face many challenges. These techniques typically rely on single-view image information for processing, resulting in insufficient registration accuracy and poor robustness. When dealing with rapidly changing surgical scenarios, dynamic adaptation is often not achieved, which can easily lead to misidentification of tissue deformation.

[0048] Traditional intraoperative registration systems, when acquiring 3D medical imaging data from CT or MRI scans, often use optical tracking technology to align intraoperative images with 3D models in real time. This process relies on manual feature matching, which is often affected by factors such as lighting variations, air bubble interference, and tissue occlusion. This results in reduced system accuracy and lacks effective adaptive adjustment mechanisms to cope with various dynamic changes.

[0049] Several other technical solutions are dedicated to rapidly reconstructing 3D models of deformable tissues in surgical scenarios. For example, Deform3DGS, based on 3D Gaussian scattering technology, uses flexible deformable modeling to efficiently handle dynamic intraoperative changes, ensuring accurate and effective navigation within the surgical environment. This solution constructs a flexible Gaussian point cloud model to achieve real-time reconstruction of tissues from multiple perspectives.

[0050] However, the related technologies have the following problems: (1) The robustness of intraoperative registration is insufficient and it is easily limited by single-view information, resulting in misidentification of tissue deformation. (2) It performs poorly in terms of global consistency and cannot effectively handle rapidly changing tissue morphology. (3) In complex scenes, it fails to automatically adjust weights to adapt to factors such as blur and occlusion, resulting in a decrease in reconstruction quality. (4) Existing algorithms have difficulty in quickly responding to changes in dynamic scenes and fail to achieve effective adaptive adjustment. (5) There is a lack of effective automatic adjustment mechanism for changes in intraoperative image quality, which affects the adaptability of the overall algorithm. Based on this, it is urgent to propose an intraoperative scene overall dual-constraint dynamic registration navigation system to solve the defects in the aforementioned related technologies.

[0051] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0052] The following first introduces an intraoperative scene overall dual-constraint dynamic registration navigation system provided by an embodiment of the present application.

[0053] The present invention provides a flow chart of an intraoperative scene overall dual-constraint dynamic registration navigation system 100, please refer to Figure 1 and Figure 2 The intraoperative scene overall dual-constraint dynamic registration navigation system 100 may include the following modules:

[0054] The image acquisition module 110 is used to acquire multiple frames of images from an endoscope during minimally invasive surgery and determine a frame of target image from the multiple frames of images.

[0055] The pose optimization module 120 is used to process the target image using a pre-trained deep learning segmentation network to extract a binary mask, and construct a constraint loss function to jointly adjust the projection constraint and the depth constraint to optimize the camera pose.

[0056] The parameter initialization module 130 is used to initialize multiple target parameters of the model based on the acquired topological structure of the pre-operative three-dimensional Gaussian model and in combination with the characteristics of the target image.

[0057] The first parameter updating module 140 is used to determine, during the process of model adjustment and optimization, reducing the pixel-level photometric loss between the model's rendered image and the target image as an optimization goal, and dynamically update a portion of the model's target parameters to enable the model to dynamically adapt to intraoperative tissue changes.

[0058] The second parameter updating module 150 is used to optimize the color consistency of multiple frames of images, update another part of the target parameters of the model, and obtain a reconstructed target three-dimensional Gaussian model.

[0059] The above is a specific implementation method of an overall dual-constraint dynamic registration navigation system for an intraoperative scene provided by an embodiment of the present application. It can be understood that the present application collects multiple frames of images in minimally invasive surgery, selects a frame of target image from the multiple frames for processing, and gradually reduces the pixel-level photometric loss between the rendered image of the model and the target image to update part of the target parameters of the preoperative three-dimensional Gaussian model; at the same time, the present application performs color consistency optimization based on the joint multi-frame images to avoid interference from factors such as light and viewing angle in multi-frame image acquisition, thereby improving the visual quality of the model.

[0060] Specifically, after extracting the binary mask, this application constructs a constrained loss function to jointly adjust the projection constraint and depth constraint. Compared with the traditional single constraint method, the positioning error is greatly reduced, and the estimation accuracy of the camera pose is improved; the system can be used throughout the entire operation, by continuously collecting images, updating parameters, and dynamically adapting to the rapidly changing tissue morphology during the operation to ensure the quality of model reconstruction.

[0061] According to an embodiment of the present application, any multiple modules among the image acquisition module 110, the pose optimization module 120, the parameter initialization module 130, the first parameter updating module 140, and the second parameter updating module 150 can be combined into a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module.

[0062] In some embodiments, the aforementioned part of the target parameters includes Gaussian center position, transparency, and covariance matrix, and the aforementioned other part of the target parameters includes color coefficients; the aforementioned target three-dimensional Gaussian model is an overall three-dimensional model that is dynamically aligned to the intraoperative image and has color.

[0063] The aforementioned posture optimization module 120 can be specifically used for:

[0064] S210. Use a pre-trained deep learning segmentation network to process the target image to distinguish the target tissue from the background, and extract the binary mask of the target tissue in the intraoperative environment; wherein the background of the target image includes illumination change information, bubbles and other instruments.

[0065] S220 : After obtaining the binary mask Mask, determine the projection constraint and the depth constraint based on the geometric information of the acquired pre-operative three-dimensional Gaussian model.

[0066] S230. Construct a constraint loss function to adjust the projection constraint and the depth constraint to optimize the camera pose.

[0067] In an embodiment of the present application, a binary mask is an image of the same size as the target image. The selected area can be marked as 1, and the remaining areas can be marked as 0. It can be understood that when an image is processed, masking before processing can limit the processing operation to occur only in a specific area; the present application can effectively exclude the background of illumination change information, bubbles and other instruments of the target image in the frame by extracting the binary mask of the target tissue in the intraoperative environment, so that subsequent processing can focus on the target tissue.

[0068] Furthermore, it is understandable that deep learning can realize image segmentation in the field of computer vision. After selecting the image segmentation network infrastructure, the accuracy and stability of image segmentation can be guaranteed through training; illustratively, the image segmentation network infrastructure can select FCN network, U-Net network, DeepLab series network, CRF-RNN network and Attention U-Net network, etc.

[0069] Based on this, the deep learning segmentation network can acquire the ability to effectively distinguish target tissue from the background through pre-training; in practical applications, the aforementioned target images can be collected in a complex intraoperative environment, and the background of the target image includes factors such as lighting changes, bubbles and other instruments, all of which will affect the robustness of traditional feature detection methods.

[0070] In contrast, deep learning methods can better adapt to different scenarios and changes, thereby providing clearer target detection results during surgery. The aforementioned deep learning segmentation network can extract the binary mask of the target tissue from the target image, thereby removing background noise and other interference factors. This application uses the joint optimization of projection constraints and depth constraints to improve the estimation accuracy of the camera pose; this application quickly solves the Gaussian model parameters based on a single-frame image to achieve dynamic adaptation to tissue deformation during surgery, so that the algorithm can update the model in time according to changes in the video stream during surgery to ensure the continuity and accuracy of the operation.

[0071] It should be noted that the projection error function based on the camera's extrinsic parameters can determine the projection loss, and the image captured by the depth camera contains depth information. After obtaining the binary mask, the next step of this application is to construct a constraint loss function to optimize the projection constraint and depth constraint. The projection constraint and depth constraint are based on the geometric information of the preoperative three-dimensional Gaussian model, which can improve the accuracy of the camera pose estimation. This application obtains the camera pose during surgery to ensure accurate alignment of the endoscopic image with the three-dimensional model. This application can capture the pose of surgical instruments or tissues to ensure effective visual guidance in complex surgical environments.

[0072] In some embodiments, the constraint loss function satisfies the expression:

[0073]

[0074] Where, is the total constraint loss, represents the rotation matrix, T represents the translation vector, and They are all weight parameters and can be adaptively adjusted during the optimization process to ensure the accuracy and stability of camera pose estimation; It is the projection loss and is used to characterize the Hausdorff distance between the projection of the preoperative 3D Gaussian model and the contour of the binary mask Mask. It is the depth loss and is related to the set depth estimation type; the depth estimation type is one of binocular depth estimation and monocular depth estimation.

[0075] In one example, when the depth estimation type is set to binocular depth estimation, the depth loss is calculated using Euclidean distance and the calculation process satisfies the expression:

[0076]

[0077] Where, is the depth predicted by the preoperative 3D Gaussian model in the binocular depth estimation scenario, is the depth actually observed in the binocular depth estimation scene; N is the number of samples used to calculate the Euclidean distance loss in binocular depth estimation.

[0078] In another example, when the depth estimation type is set to monocular depth estimation, a relative depth loss function is determined to compare the depth loss based on the relative depth, and the calculation process of the depth loss satisfies the expression:

[0079]

[0080] Where, , is the depth predicted by the preoperative 3D Gaussian model in the monocular depth estimation scenario, is the actual observed depth in the monocular depth estimation scene, It is a reference depth value and is used to convert the depth information into the form of relative depth; M is the number of samples used to calculate the relative depth loss in monocular depth estimation.

[0081] In some embodiments, the Gaussian center position satisfies the expression: , where is the Gaussian center position, is the center position of the preoperative 3D model and represents the original position of Gaussian point i in the undeformed state; It is an adjustment based on the intraoperative image changes to characterize the deformation of the target tissue during actual surgery.

[0082] Transparency satisfies the expression: Where, For transparency, It is the input feature associated with Gaussian point i and is used to determine the transparency of Gaussian point i in the scene; Is an activation function and is used to transform the input features Mapped to a transparency value to describe the visibility of Gaussian point i. It can be understood that by optimizing the parameters of the model, the model can be dynamically adapted to the changes of tissue during surgery; the input feature It is the input feature associated with Gaussian point i, which is used to determine the transparency of the point in the scene.

[0083] The covariance matrix is ​​used to characterize the shape and extension of Gaussian point i to describe the distribution of Gaussian point i in space and satisfies the expression: , where is the covariance matrix, , , are the standard deviations of the Gaussian points in the X, Y, and Z directions, respectively, and represent the extent of the Gaussian points in the X, Y, and Z directions, respectively.

[0084] In some embodiments, the first parameter updating module 140 may be specifically configured to:

[0085] S310, constructing a loss function corresponding to the pixel-level photometric loss to improve the similarity between the rendering result and the actual image by minimizing the loss;

[0086] S320, based on a preset image rendering function, generating a rendered image using the current model in the optimization process, and determining as an optimization goal to reduce the pixel-level luminosity loss between the rendered image of the model and the target image;

[0087] S330, evaluating the model under the current parameters based on the loss function corresponding to the pixel-level photometric loss, and calculating the corresponding loss value;

[0088] S340. Iteratively update a portion of the target parameters of the model through the gradient descent method to continuously reduce the loss.

[0089] In the embodiment of the present application, it can be understood that by reducing the pixel-level photometric loss between the rendered image and the intraoperative image, the accurate reproduction of the model in the surgical scene is ensured; by minimizing the loss This improves the similarity between the rendered image and the actual image. By updating model parameters, the model can quickly adapt to actual tissue changes during surgery, ensuring that the surgeon receives visual feedback during the procedure. This dynamic model update mechanism is crucial for efficient minimally invasive surgical navigation and processing.

[0090] It should be noted that this application divides the process of updating model parameters into two stages. This application divides the model parameters into two parts. One part of the target parameters is updated based on the features of a frame of target image, and the other part of the target parameters is updated based on multiple frames of endoscope images collected during minimally invasive surgery. In the first stage, this application iteratively updates part of the target parameters of the model through the gradient descent method, updating the Gaussian center position, transparency and covariance matrix of the model.

[0091] In one example, the loss function corresponding to the pixel-level photometric loss satisfies the expression:

[0092]

[0093] in, is the loss function corresponding to the pixel-level photometric loss, H is the total number of pixels in the image, is a rendered image generated by the current model during the optimization process at position The pixel value of The actual intraoperative image is at position The pixel value of .

[0094] In another example, the aforementioned process of generating a rendered image through the current model in the optimization process based on the preset image rendering function satisfies the expression:

[0095]

[0096] Where, A rendered image generated for the current model, For transparency, is the Gaussian center position, is the covariance matrix, represents the position in the image, i represents the Gaussian point, Indicates the number of Gaussian points involved in rendering; is an image rendering function that satisfies the expression:

[0097]

[0098] in, is the determinant of the covariance matrix, represents the transpose of the matrix, represents the inverse matrix of the covariance matrix, Used to determine the render value to generate at position x.

[0099] The process of iteratively updating a part of the target parameters of the model through the gradient descent method satisfies the expression: Where, Represents a part of the updated target parameters, Represents a part of the target parameters before updating, is the learning rate and is used to control the size of the adjustment step; is the loss function The gradient of is used to characterize the rate of change of the loss with respect to a portion of the target parameters.

[0100] In some embodiments, the second parameter updating module 150 may be specifically used to:

[0101] S410, constructing a color optimization loss function when the multiple frames of images are aligned with the actual intraoperative scene, wherein the optimization goal of the color optimization loss function is to ensure visual consistency of the multiple frames of images under different viewing angles;

[0102] S420, iteratively updating the color coefficients of the model by gradient descent method;

[0103] S430, in each iteration, calculating the gradient according to the color optimization loss function, and updating the color coefficient of the model to obtain a reconstructed target three-dimensional Gaussian model;

[0104] Among them, the color coefficient is used to represent the spherical harmonic coefficient of the RGB channel and satisfies the expression: , where is the color coefficient, r, g, b are the color components of the red, green, and blue channels respectively.

[0105] In the embodiment of the present application, it can be understood that after completing the single-frame camera pose estimation and solving part of the target parameters, the core of this step is to optimize the color consistency through multi-frame images, aiming to solve the problem of image color inconsistency under different perspectives; the intraoperative scene overall dual-constraint dynamic registration navigation system of the present application effectively solves the color inconsistency problem of images under different perspectives through a multi-frame color optimization strategy, provides a more coherent and realistic three-dimensional reconstruction effect, and provides doctors with a more reliable visual reference.

[0106] It's important to note that color coefficients are used to describe variations in tissue color to better match the appearance of tissue in surgical images. By optimizing color consistency across multiple frames, the algorithm significantly improves 3D reconstruction and enhances model visualization, providing physicians with more reliable and effective visual aids. This process not only improves color consistency but also enhances the overall effect and perception of depth, allowing surgeons to more accurately judge tissue structure during surgery.

[0107] It should also be noted that this application can capture the position of surgical instruments or tissues, ensuring effective visual guidance in complex surgical environments. For the update of multiple target parameters, this application adopts a dynamic weight adaptive adjustment mechanism, enabling the algorithm to intelligently respond to complex intraoperative environments. Regardless of lighting changes, image blur, or partial occlusion, the algorithm can maintain efficient and stable performance, greatly improving the practicality of the technology.

[0108] Through the above steps, the model parameters can quickly adapt to actual tissue changes during surgery. This dynamic model update mechanism is crucial for efficient minimally invasive surgical navigation and processing, providing a strong guarantee for the ultimate success of the operation. Doctors can focus more on the operation itself without having to pay too much attention to technical details, thereby improving overall surgical efficiency.

[0109] In one example, the color optimization loss function satisfies the expression:

[0110]

[0111] Where, The first one is obtained by rendering the current model Color information of the image; The actual collection The color information of each image, n represents the total number of images; is the regularization weight, which is used to balance the weight between color consistency and smoothness; is the total variation regularization term and is used to smooth the color distribution between adjacent Gaussian nodes; The project aims to suppress color mutations, smooth the colors of adjacent nodes, and achieve a more coherent and natural visual effect. It can effectively reduce color inconsistencies caused by lighting changes, imaging noise or other visual interference.

[0112] in, Satisfies the expression:

[0113]

[0114] Where, Indicates the color in The value of the Gaussian nodes, Indicates the color in The value of the Gaussian nodes, Indicates the color in The value of the Gaussian nodes.

[0115] It can be understood that in terms of multi-frame color consistency optimization, this application collects multiple frames of registered intraoperative images, constructs a color optimization loss function to ensure visual consistency under different perspectives, and uses the total variation regularization term to smooth the color distribution, thereby further improving the visual quality of the model.

[0116] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, detailed descriptions of known systems and methods are omitted here. In the above embodiments, specific steps of several modules are described and illustrated as examples. However, the system of the present application is not limited to the specific modules and steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0117] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0118] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0119] The above reference is made to the system according to the embodiment of the present disclosure and describes various aspects of the present disclosure. It should be understood that each box in the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed via the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more boxes of the flow chart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flow chart and the combination of the boxes in the block diagram and / or flow chart can also be implemented by the dedicated hardware that performs the specified function or action, or can be implemented by the combination of dedicated hardware and computer instructions.

[0120] In summary, compared with the prior art, this application has the following beneficial effects:

[0121] 1. This application collects multiple frames of images during minimally invasive surgery, selects a target image from the multiple frames for processing, and gradually reduces the pixel-level photometric loss between the model's rendered image and the target image to update the Gaussian center position, transparency, and covariance matrix of the preoperative three-dimensional Gaussian model; through the provided dynamic weight adaptive adjustment mechanism, the weight parameters in the loss function can be adjusted according to changes in intraoperative image quality, thereby improving the adaptability and robustness of the algorithm in complex scenarios.

[0122] 2. This application utilizes a combined optimization of projection and depth constraints to improve the accuracy of camera pose estimation. By rapidly solving Gaussian model parameters based on a single-frame image, dynamic adaptation to tissue deformation during surgery is achieved. This allows the algorithm to update the model in real time based on changes in the video stream during surgery, ensuring continuity and accuracy. The system can be used throughout the entire surgical procedure, continuously capturing images and updating parameters to dynamically adapt to rapidly changing tissue morphology during surgery, ensuring the quality of model reconstruction.

[0123] 3. This application performs color consistency optimization based on multiple frames of images to avoid interference from factors such as light and viewing angle during multi-frame image acquisition, thereby ensuring the accuracy of the model's color coefficient updates and improving the model's visual quality. In terms of multi-frame color consistency optimization, this application collects multiple registered images and constructs a color optimization loss function to ensure visual consistency under different viewing angles. It also uses a total variation regularization term to smooth the color distribution, further improving the model's visual quality.

[0124] 4. This application proposes a phased dynamic processing flow that combines dual-constraint camera pose estimation with Gaussian model coefficient solution for intraoperative navigation. This improves the modularity of the algorithm and makes the optimization of each step more efficient and accurate. Through the dual-constraint optimization strategy, this application has achieved a breakthrough in the accuracy of camera pose estimation. Compared with the traditional single-constraint method, the positioning error is greatly reduced, laying a technical foundation for accurate surgical navigation. This application greatly reduces the complexity of the technology through deep learning and adaptive algorithms.

[0125] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A dual-constraint dynamic registration navigation system for the entire intraoperative scene, characterized by: include: An image acquisition module is used to acquire multiple frames of images from an endoscope during minimally invasive surgery and determine a target image from the multiple frames of images; A pose optimization module is used to process the target image using a pre-trained deep learning segmentation network to extract a binary mask, and to construct a constraint loss function to jointly adjust the projection constraint and the depth constraint to optimize the camera pose; A parameter initialization module, configured to initialize multiple target parameters of the model based on the acquired topological structure of the preoperative three-dimensional Gaussian model and in combination with the characteristics of the target image; A first parameter updating module is configured to determine, during the process of model adjustment and optimization, reducing the pixel-level photometric loss between the rendered image of the model and the target image as an optimization goal, and dynamically updating a portion of the target parameters of the model so that the model dynamically adapts to changes in tissue during surgery; The second parameter updating module is used to optimize the color consistency of the multiple frames of images, update the target parameters of another part of the model, and obtain a reconstructed target three-dimensional Gaussian model; the target parameters of one part include the Gaussian center position, transparency, and covariance matrix, and the target parameters of the other part include color coefficients.

2. The intraoperative scene overall dual-constraint dynamic registration navigation system according to claim 1, characterized in that: The target three-dimensional Gaussian model is a colored overall three-dimensional model that is dynamically registered to the intraoperative image; The posture optimization module is specifically used for: The target image is processed using a pre-trained deep learning segmentation network to distinguish the target tissue from the background and extract a binary mask of the target tissue in the intraoperative environment; wherein the background of the target image includes illumination change information, bubbles, and other instruments; After obtaining the binary mask, determining the projection constraint and the depth constraint based on the geometric information of the acquired preoperative three-dimensional Gaussian model; Construct a constraint loss function to adjust the projection constraint and the depth constraint to optimize the camera pose; wherein the constraint loss function satisfies the expression: Where, is the total constraint loss, represents the rotation matrix, T represents the translation vector, and They are all weight parameters and can be adaptively adjusted during the optimization process to ensure the accuracy and stability of camera pose estimation; It is the projection loss and is used to characterize the Hausdorff distance between the projection of the preoperative 3D Gaussian model and the contour of the binary mask Mask. It is depth loss and is related to the set depth estimation type; the depth estimation type is one of binocular depth estimation and monocular depth estimation.

3. The intraoperative scene overall dual-constraint dynamic registration navigation system according to claim 2, characterized in that: When the depth estimation type is set to the binocular depth estimation, the depth loss is calculated using the Euclidean distance and the calculation process satisfies the expression: Where, is the depth predicted by the preoperative 3D Gaussian model in the binocular depth estimation scenario, is the depth actually observed in the binocular depth estimation scene; N is the number of samples used to calculate the Euclidean distance loss in binocular depth estimation.

4. The intraoperative scene overall dual-constraint dynamic registration navigation system according to claim 2, characterized in that: In the case where the depth estimation type is set to the monocular depth estimation, a relative depth loss function is determined to compare the depth loss based on the relative depth, and the calculation process of the depth loss satisfies the expression: Where, , is the depth predicted by the preoperative 3D Gaussian model in the monocular depth estimation scenario, is the actual observed depth in the monocular depth estimation scene, It is a reference depth value and is used to convert the depth information into the form of relative depth; M is the number of samples used to calculate the relative depth loss in monocular depth estimation.

5. The intraoperative scene overall dual-constraint dynamic registration navigation system according to claim 2, characterized in that: The Gaussian center position satisfies the expression: , where is the Gaussian center position, is the center position of the preoperative 3D model and represents the original position of Gaussian point i in the undeformed state; It is the adjustment amount based on the intraoperative image changes to characterize the deformation of the target tissue during actual surgery; The transparency satisfies the expression: Where, For transparency, It is the input feature associated with Gaussian point i and is used to determine the transparency of Gaussian point i in the scene; Is an activation function and is used to transform the input features Mapped to a transparency value to describe the visibility of Gaussian point i; The covariance matrix is ​​used to characterize the shape and extension of the Gaussian point i to describe the distribution of the Gaussian point i in space and satisfies the expression: , where is the covariance matrix, , , are the standard deviations of the Gaussian points in the X, Y, and Z directions, respectively, and represent the extent of the Gaussian points in the X, Y, and Z directions, respectively.

6. The intraoperative scene overall dual-constraint dynamic registration navigation system according to claim 4, characterized in that: The first parameter updating module is specifically configured to: Construct a loss function corresponding to the pixel-level photometric loss to improve the similarity between the rendering result and the actual image by minimizing the loss; Based on a preset image rendering function, a rendered image is generated by the current model in the optimization process, and reducing the pixel-level photometric loss between the rendered image of the model and the target image is determined as an optimization goal; Evaluate the model under the current parameters based on the loss function corresponding to the pixel-level photometric loss, and calculate the corresponding loss value; The target parameters of the part of the model are iteratively updated by the gradient descent method to continuously reduce the loss.

7. The intraoperative scene overall dual-constraint dynamic registration navigation system according to claim 6, characterized in that: The loss function corresponding to the pixel-level photometric loss satisfies the expression: in, is the loss function corresponding to the pixel-level photometric loss, H is the total number of pixels in the image, is a rendered image generated by the current model during the optimization process at position The pixel value of The actual intraoperative image is at position The pixel value of .

8. The intraoperative scene overall dual-constraint dynamic registration navigation system according to claim 6, characterized in that: The process of generating a rendered image based on the preset image rendering function by the current model in the optimization process satisfies the expression: Where, A rendered image generated for the current model, For transparency, is the Gaussian center position, is the covariance matrix, represents the position in the image, i represents the Gaussian point, Indicates the number of Gaussian points involved in rendering; is the image rendering function and satisfies the expression: in, is the determinant of the covariance matrix, represents the transpose of the matrix, represents the inverse matrix of the covariance matrix, Used to determine the rendering value generated at position x; The process of iteratively updating the target parameters of the model by the gradient descent method satisfies the expression: Where, represents the updated portion of the target parameters, represents the part of the target parameters before updating, is the learning rate and is used to control the size of the adjustment step; is the loss function The gradient is used to characterize the rate of change of the loss with respect to the target parameter of the part.

9. The intraoperative scene overall dual-constraint dynamic registration navigation system according to claim 2, characterized in that: The second parameter updating module is specifically configured to: When the multi-frame images are aligned with the actual intraoperative scene, a color optimization loss function is constructed, wherein the optimization goal of the color optimization loss function is to ensure visual consistency of the multi-frame images under different viewing angles; Iteratively updating the color coefficients of the model by gradient descent; In each iteration, the gradient is calculated according to the color optimization loss function, and the color coefficient of the model is updated to obtain a reconstructed target three-dimensional Gaussian model; The color coefficients are used to characterize the spherical harmonic coefficients of the RGB channels and satisfy the expression: , where is the color coefficient, r, g, b are the color components of the red, green, and blue channels respectively.

10. The intraoperative scene overall dual-constraint dynamic registration navigation system according to claim 9, characterized in that: The color optimization loss function satisfies the expression: Where, The first one is obtained by rendering the current model Color information of the image; The actual collection The color information of each image, n represents the total number of images; is the regularization weight, which is used to balance the weight between color consistency and smoothness; is the total variation regularization term and is used to smooth the color distribution between adjacent Gaussian nodes; Satisfies the expression: Where, Indicates the color in The value of the Gaussian nodes, Indicates the color in The value of the Gaussian nodes, Indicates the color in The value of the Gaussian nodes.

Images