Method and system for estimating 6D pose of instrument based on 3D Gaussian sputtering

Abstract

The present invention provides a method and system for estimating the 6D pose of an instrument based on 3D Gaussian sputtering, and relates to the field of computer vision. The present invention proposes a method for estimating the distance between an instrument and tissue based on binocular depth estimation, 6D pose estimation of an instrument, and depth completion. In view of the difficulties in estimating and tracking the 6D pose of an instrument caused by the characteristics of surgical instruments such as fast movement, low texture, and severe reflection, as well as the problem of decreased accuracy of tissue depth estimation caused by the presence of instruments, it is proposed to estimate the three-dimensional position of the instrument and tissue separately, and combine depth completion to remove the influence of instrument occlusion on tissue reconstruction. It can realize complete three-dimensional reconstruction of the surgical scene, three-dimensional position estimation and tracking of surgical instruments, and instrument-tissue relative distance estimation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method and system for estimating 6D posture of an instrument based on 3D Gaussian sputtering. Background Art

[0002] 6D pose estimation is an important task in the field of computer vision, which aims to estimate the position and posture of an object in three-dimensional space from images or point cloud data.

[0003] In the related art, the paper (3-D pose estimation of articulated instruments in robotic minimally invasive surgery [J]. IEEE transactions on medical imaging, 2018, 37(5): 1204-1213.) specifically discloses a 6D pose estimation and tracking method for articulated surgical instruments in minimally invasive surgery scenarios: the contour and optical flow features of the instrument in the image are used to construct the pose optimization target of the articulated instrument; the existing rigid CAD model is projected onto the image plane using the 6D pose to be optimized to obtain the projection mask of the instrument, and then the mask of the instrument on the image is obtained using the random forest algorithm, and the similarity metric function between the instrument projection mask and the segmentation mask is constructed, and the similarity metric function is maximized as the main optimization target; combined with the optical flow prediction results of the LK optical flow for adjacent frame images, the similarity metric function between the 2D coordinates after the pose change from the previous frame to the current frame and the 2D coordinates of the current frame predicted by the optical flow is constructed. Finally, the 6D pose of the instrument is iteratively solved by the gradient descent algorithm.

[0004] However, in the above-mentioned device pose estimation method based on region alignment, the rendering of the device model and the establishment of the sdf field of the 2D projection take a long time in each iteration, and the number of iterations of device pose optimization is large, both of which lead to the problem of serious time consumption in the above-mentioned method. In addition, compared with the effect in the z direction, the device segmentation mask constrains the pose estimation in the xy direction more, which makes the estimation accuracy in the z direction difficult to meet the requirements of practical applications. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a method and system for 6D pose estimation of an instrument based on 3D Gaussian sputtering, which solves the problem of time-consuming single-frame optimization caused by model rendering and 2D projection acquisition in traditional instrument pose estimation methods based on region alignment, and the problem that the accuracy of instrument pose estimation does not meet the requirements of actual applications due to the constraint of a single segmentation mask.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A 6D pose estimation method for an instrument based on 3D Gaussian sputtering, comprising:

[0010] Reading the current frame image, obtaining the device mask through semantic segmentation; and using a binocular depth estimation algorithm to perform depth estimation on the current frame image, and respectively obtaining a tissue depth map and a device depth map in combination with the device mask;

[0011] Initialize the device pose, use the pre-reconstructed device Gaussian model to perform 3D Gaussian sputtering rendering, and obtain the view, projection mask, and depth map under the corresponding pose;

[0012] A total loss function is constructed based on the rendering result, the current frame image, its device mask, and the device depth map; and the device pose is updated and learned through a back propagation algorithm to minimize the total loss function and obtain the optimal 6D pose of the device in the current frame image.

[0013] Preferably, after obtaining the optimal 6D pose estimation result of the device of the current frame image, the device 6D pose estimation method further includes:

[0014] Using a thin plate spline interpolation method to perform depth completion on the tissue depth map to obtain a three-dimensional tissue surface;

[0015] According to the three-dimensional tissue surface, obtaining the three-dimensional coordinates of the tissue in the current frame camera coordinate system;

[0016] According to the optimal 6D pose estimation result of the instrument, combined with the reconstructed Gaussian model of the instrument, the three-dimensional coordinates of the instrument in the camera coordinate system of the current frame are obtained;

[0017] The relative distance between the instrument and the tissue is obtained according to the three-dimensional coordinates of the tissue and the instrument in the camera coordinate system of the current frame.

[0018] Preferably, the initializing the device posture comprises:

[0019] For the first frame of image, a rendering view of the instrument Gaussian model on the image plane is obtained in combination with the camera model, the laparoscope is moved so that the image coincides with the area where the rendering view is located, and the instrument posture at this time is used as the initial instrument posture of the first frame of image;

[0020] For images other than the first frame, the device pose of the current frame is initialized based on the optimal device pose of the previous frame and the optical flow information of two consecutive frames.

[0021] Preferably, the reconstruction process of the Gaussian model of the device includes:

[0022] Acquire several images with known camera poses and containing all views of the device to reconstruct a sparse point cloud of the device;

[0023] Initialize each point of the sparse point cloud as a Gaussian to obtain an initialized Gaussian model of the instrument;

[0024] In combination with the aforementioned image sequence consisting of images with known camera poses and containing all viewing angles of the instrument and their corresponding camera poses, the initialized instrument Gaussian model is iteratively optimized to obtain the reconstructed instrument Gaussian model.

[0025] Preferably, the image sequence composed of the images of the aforementioned known camera poses and containing all viewing angles of the instrument and the corresponding camera poses is combined to iteratively optimize the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; comprising:

[0026] The color consistency loss function between the 3D Gaussian rendering view and the original image that only retains the instrument part is constructed as follows:

[0027]

[0028] in, u, q, s Respectively represent the center position, rotation, and size of any Gaussian; , L is the number of images included in the image sequence; L 1 indicates a norm loss; The camera pose is R l The process of performing 3D Gaussian differentiable rendering of an image; G i Indicates i Gaussian kernel, , I is the number of Gaussians; It indicates the original image from the image sequence where only the instrument part is retained;

[0029] A multi-region alignment loss function is introduced to constrain the projection shape of each articulated part on the reconstructed device Gaussian model on the image plane and the device mask on the original image. M lc Align them separately; the multi-region alignment loss function is as follows:

[0030]

[0031] in, , C Indicates the number of articulated parts of the device; The camera pose is Rl The process of performing 3D Gaussian differentiable rendering projection;

[0032] The total loss function of the instrument Gaussian model reconstruction process is:

[0033]

[0034] in, α represents the corresponding weight coefficient;

[0035] By minimizing the total loss function L recons , obtain the reconstructed instrument Gaussian model , as shown below:

[0036] (4).

[0037] Preferably, the method constructs a total loss function based on the rendering result, the current frame image, the device mask, and the device depth map; and updates and learns the device pose through a back propagation algorithm to minimize the total loss function and obtain the optimal 6D pose of the device in the current frame image; including:

[0038] Construct the color consistency loss function between the 3D Gaussian rendering view and the current frame image as follows:

[0039]

[0040] in, t Indicates the current moment; L 1 represents a norm loss, T t Indicates the current frame equipment posture; Represents a rendered view; ; M t The instrument mask representing the current image segmentation;

[0041] Construct the regional alignment loss function between the 3D Gaussian differentiable rendering projection image and the device mask of the current frame image as follows:

[0042]

[0043] in, , They represent the first c The projection shape of each articulated part on the image plane and the instrument mask of the current frame image; ; ;

[0044] Construct the depth consistency loss between the rendered depth map and the device depth map of the previous frame as follows:

[0045]

[0046] in, 、 Respectively represent the rendered depth map and the device depth map of the previous frame image; It is the process of rendering the depth map of the Gaussian model of the device;

[0047] The total loss function is constructed as:

[0048]

[0049] in, α 1 、α 2 、α 3 represent the corresponding weight coefficients respectively;

[0050] Then the device posture is adjusted by back propagation algorithm. Iterate to minimize L pose , get the optimal 6D pose estimation result of the device in the current frame image, as shown below:

[0051] .

[0052] A 6D pose estimation system for an instrument based on 3D Gaussian sputtering, comprising:

[0053] An image processing module is used to read the current frame image, obtain the device mask through semantic segmentation; and use a binocular depth estimation algorithm to perform depth estimation on the current frame image, and respectively obtain a tissue depth map and a device depth map in combination with the device mask;

[0054] The scene rendering module is used to initialize the instrument posture, perform 3D Gaussian sputtering rendering using the pre-reconstructed instrument Gaussian model, and obtain the view, projection mask and depth map under the corresponding posture;

[0055] The pose estimation module is used to construct a total loss function based on the rendering result, the current frame image, its device mask, and the device depth map; and to update and learn the device pose through the back propagation algorithm, minimize the total loss function, and obtain the optimal 6D pose of the device in the current frame image.

[0056] Preferably, after the posture estimation module obtains the optimal 6D posture estimation result of the device of the current frame image, the device 6D posture estimation system further includes:

[0057] A depth completion module, used to perform depth completion on the tissue depth map using a thin plate spline interpolation method to obtain a three-dimensional tissue surface;

[0058] A distance calculation module is used to obtain the three-dimensional coordinates of the tissue in the current frame camera coordinate system based on the three-dimensional tissue surface; obtain the three-dimensional coordinates of the instrument in the current frame camera coordinate system based on the optimal 6D pose estimation result of the instrument in combination with the reconstructed instrument Gaussian model; and obtain the instrument-tissue relative distance based on the three-dimensional coordinates of the tissue and the instrument in the current frame camera coordinate system.

[0059] A storage medium stores a computer program for 6D pose estimation of an instrument based on 3D Gaussian sputtering, wherein the computer program enables a computer to execute the 6D pose estimation method of the instrument as described above.

[0060] An electronic device, comprising:

[0061] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the instrument 6D pose estimation method as described above.

[0062] (III) Beneficial effects

[0063] The present invention provides a 6D pose estimation method and system for an instrument based on 3D Gaussian sputtering. Compared with the prior art, it has the following beneficial effects:

[0064] In the present invention, firstly, the current frame image is read, and the instrument mask is obtained through semantic segmentation; and the binocular depth estimation algorithm is used to perform depth estimation on the current frame image, and the tissue depth map and the instrument depth map are obtained respectively in combination with the instrument mask; secondly, the instrument posture is initialized, and the pre-reconstructed instrument Gaussian model is used for 3D Gaussian sputtering rendering to obtain the view, projection mask and depth map under the corresponding posture; then, according to the rendering result and the current frame image and its instrument mask and instrument depth map, a total loss function is constructed, and the instrument posture is updated and learned through the back propagation algorithm, the total loss function is minimized, and the optimal 6D posture of the instrument of the current frame image is obtained. The present invention estimates the instrument posture by aligning the instrument Gaussian model and image features, which can realize accurate and robust restoration of the 6D posture of the instrument in complex surgical scenes; and the instrument posture estimation is realized based on the framework of 3D Gaussian sputtering, which can realize rapid restoration of the instrument posture in a small number of iterations and shorter iteration time. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A block diagram of a device 6D pose estimation method based on 3D Gaussian sputtering provided in an embodiment of the present invention;

[0067] Figure 2 A block diagram of another device 6D pose estimation method based on 3D Gaussian sputtering provided in an embodiment of the present invention;

[0068] Figure 3 An overall flow chart of a method for estimating 6D posture of an instrument based on 3D Gaussian sputtering provided in an embodiment of the present invention;

[0069] Figure 4 A flowchart of device posture estimation provided by an embodiment of the present invention;

[0070] Figure 5 A schematic diagram of an articulated motion coordinate system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] The embodiments of the present application provide a method and system for device 6D pose estimation based on 3D Gaussian sputtering, which solves the problem of time-consuming single-frame optimization caused by model rendering and 2D projection acquisition in traditional device pose estimation methods, and the problem that the accuracy of device pose estimation does not meet the requirements of actual applications due to the constraint of a single segmentation mask.

[0073] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0074] The relative position and interaction status of the instrument and tissue can provide high-precision positioning and navigation information in real time, which helps to improve the accuracy and safety of minimally invasive surgery. The embodiment of the present invention proposes a method for estimating the distance between instruments and tissues based on binocular depth estimation, instrument 6D pose estimation and depth completion. In view of the difficulties in estimating and tracking the 6D pose of instruments caused by the characteristics of surgical instruments such as fast movement, low texture, and severe reflection, as well as the problem of decreased accuracy of tissue depth estimation caused by the presence of instruments, it is proposed to estimate the three-dimensional position of instruments and tissues separately, and combine depth completion to remove the impact of instrument occlusion on tissue reconstruction. It can achieve complete three-dimensional reconstruction of the surgical scene, three-dimensional position estimation and tracking of surgical instruments, and relative distance estimation between instruments and tissues.

[0075] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0076] Embodiment 1:

[0077] like Figure 1 The embodiment of the present invention provides a method for estimating 6D posture of an instrument based on 3D Gaussian sputtering, comprising:

[0078] S1, reading the current frame image, obtaining the device mask through semantic segmentation; and using a binocular depth estimation algorithm to perform depth estimation on the current frame image, and respectively obtaining a tissue depth map and a device depth map in combination with the device mask;

[0079] S2, initialize the device posture, use the pre-reconstructed device Gaussian model to perform 3D Gaussian sputtering rendering, and obtain the view, projection mask and depth map under the corresponding posture;

[0080] S3. Construct a total loss function based on the rendering result, the current frame image, its device mask, and the device depth map; and update and learn the device pose through the back propagation algorithm to minimize the total loss function and obtain the optimal 6D pose of the device in the current frame image.

[0081] The embodiments of the present invention estimate the instrument pose by aligning the instrument Gaussian model and image features, which can accurately and robustly restore the 6D pose of the instrument in complex surgical scenarios; and implement instrument pose estimation based on a 3D Gaussian sputtering framework, which can quickly restore the instrument pose in a shorter iteration time and a small number of iterations.

[0082] In an optional embodiment, if Figure 2 As shown, after obtaining the optimal 6D pose estimation result of the device of the current frame image, the device 6D pose estimation method further includes:

[0083] S4, using a thin plate spline interpolation method to perform depth completion on the tissue depth map to obtain a three-dimensional tissue surface;

[0084] S5. According to the three-dimensional tissue surface, the three-dimensional coordinates of the tissue in the current frame camera coordinate system are obtained; according to the optimal 6D pose estimation result of the instrument, combined with the reconstructed instrument Gaussian model, the three-dimensional coordinates of the instrument in the current frame camera coordinate system are obtained; according to the three-dimensional coordinates of the tissue and the instrument in the current frame camera coordinate system, the relative distance between the instrument and the tissue is obtained.

[0085] The embodiment of the present invention provides a depth-completed instrument-tissue interaction distance estimation method. The acquired relative position and interaction status of the instrument-tissue can provide high-precision positioning and navigation information in real time, which helps to improve the accuracy and safety of minimally invasive surgery.

[0086] Figure 3 The overall flow chart of the method is given. The embodiment of the present invention designs an instrument-tissue interaction distance estimation method based on instrument 6D pose estimation, depth estimation and depth completion, including but not limited to 6D pose estimation of surgical instruments and instrument-tissue relative distance estimation in endoscopic surgery or robot-assisted surgery. It mainly includes: depth estimation and instrument segmentation based on binocular endoscopic images, 3D Gaussian model reconstruction of surgical instruments, 6D pose estimation of surgical instruments based on 3D Gaussian optimization, tissue depth completion and instrument-tissue relative distance estimation, etc.

[0087] Next, we will combine Figure 3 The following are the steps of the above solution:

[0088] In step S1, the current frame image is read, and the instrument mask is obtained through semantic segmentation; and the depth of the current frame image is estimated using a binocular depth estimation algorithm, and a tissue depth map and an instrument depth map are respectively obtained in combination with the instrument mask.

[0089] In this step, the semantic segmentation algorithm (Random Forest, SAM, PerSAM, etc.) is used to perform multi-value segmentation on the image to obtain the device mask on the image. ,in K Indicates the number of articulations of the instrument. And the depth of the endoscope image is estimated using a binocular depth estimation method (MIOL algorithm, CREStereo algorithm, etc.) to obtain a depth map D t , and combined with the device mask M t , and obtain the tissue depth map of the non-device occluded area D t (1- M t ) and the device depth map of the visible part of the device Dt M t .in, t Indicates the current moment.

[0090] In step S2, the instrument posture is initialized, and 3D Gaussian sputtering rendering is performed using the pre-reconstructed instrument Gaussian model to obtain the view, projection mask and depth map under the corresponding posture.

[0091] Note that 3D Gaussian splatter rendering is mainly used for new view synthesis. The 3D position of the object represented by the 3D Gaussian is determined by the center position of each Gaussian. , Rotate ,size ,color ,density σ Together, it is not suitable for expressing the 6D pose transformation of dynamic objects.

[0092] Accordingly, the embodiment of the present invention pre-establishes a 3D Gaussian model of the surgical instrument (ie, a reconstructed Gaussian model of the instrument), thereby converting the 3D Gaussian optimization into Gaussian posture optimization, ie, posture optimization of the surgical instrument.

[0093] Specifically, the reconstruction process of the Gaussian model of the device includes:

[0094] First, obtain several images with known camera poses and all viewing angles of the instrument to reconstruct the sparse point cloud of the instrument; specifically,

[0095] a) Fix the position of the surgical instrument, aim the camera at the instrument and shoot around it to obtain several instrument images covering all viewing angles. I l , , L is the number of images. However, due to the textureless and specular reflection characteristics of surgical instruments, it is not easy to solve the relative camera pose through the image features between adjacent frames. Feature points can be provided by attaching marker points on the instrument or fixing a checkerboard in the shooting field of view.

[0096] b) The sparse point cloud of the captured scene is reconstructed through SFM, as well as the 2D-3D corresponding points between the image and the SFM point cloud.

[0097] c) Scale the sfm point cloud according to the real scale of the device, restore the absolute scale of the sparse point cloud, and use the 2D-3D corresponding points to obtain the absolute camera pose of each image R l .

[0098] d) Use the segmentation algorithm to segment the instrument and obtain the segmentation mask of the instrument to remove the background outside the instrument in the image and obtain an image with only the instrument .

[0099] e) According to the camera pose R l Known images , use the SFM algorithm to reconstruct the sparse point cloud of the instrument M sfm , used as initialization for 3D Gaussian model reconstruction.

[0100] Then, each point of the sparse point cloud is initialized as a Gaussian G i , , the properties of each Gaussian are: the center position of the Gaussian , Rotate ,size ,color ,density σ , the probability of belonging to the foreground f .

[0101] It should be noted that the parameters that need to be optimized in the reconstruction process of the surgical instrument Gaussian model in the embodiment of the present invention are only: the center position of each Gaussian , Rotate ,size .

[0102] In addition, the instrument Gaussian model at this stage { G i} is initialized as follows: Equipment point cloud M sfm Each point is initialized as a Gaussian G i , , I is the number of Gaussians, and the position and color of the point cloud are initialized to the center position of the Gaussians u and color c , Gaussian rotation q Initialized to {1,0,0,0}, the distance from each point to the nearest neighbor is initialized to the size of the Gaussian, and the density of the Gaussian σ Initialized to 1, the probability of belonging to the foreground f Initialized to 1.

[0103] On this basis, the image sequence composed of the images of the aforementioned known camera poses and containing all viewing angles of the instrument and its corresponding camera poses is combined to iteratively optimize the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; including:

[0104] The color consistency loss function between the 3D Gaussian rendering view and the original image that only retains the instrument part is constructed as follows:

[0105]

[0106] in, L 1 indicates a one-norm loss. It indicates the original image from the image sequence where only the instrument part is retained; The camera pose is R l The process of performing 3D Gaussian differentiable rendering of an image.

[0107] In addition, to ensure that the reconstructed instrument Gaussian model is an articulated structure, the embodiment of the present invention introduces a multi-region alignment loss function to constrain the projection shape of each articulated part on the reconstructed instrument Gaussian model on the image plane and the instrument mask on the original image. M lc Align them separately; the multi-region alignment loss function is as follows:

[0108]

[0109] in, , C Indicates the number of articulated parts of the device; The camera pose is R l The process of performing 3D Gaussian differentiable rendering projection;

[0110] The total loss function of the instrument Gaussian model reconstruction process is:

[0111]

[0112] in, α represents the corresponding weight coefficient;

[0113] By minimizing the total loss function L recons , obtain the reconstructed instrument Gaussian model , as shown below:

[0114]

[0115] The optimized instrument Gaussian model { G i}, recorded as , i Represents the index of the Gaussian kernel.

[0116] In getting the Gaussian model of the device { G i,0}, the rigid motion between the articulated parts of the surgical instrument TIn the embodiment of the present invention, the DH parameter method is used to define the articulated motion of the instrument, and the rod and the left clamp (the right clamp is symmetrical to the left clamp) of the instrument are regarded as a two-link structure, as shown in the schematic diagram. Figure 5 As shown in the figure, local coordinate systems are defined for the instrument rod and the left clamp (the right clamp is symmetrical to the left clamp) respectively. F 1. F 2. Establish the camera coordinate system F cam The device pose estimation solves the transformation matrix from the device's own coordinate system to the camera coordinate system. At this time, the local coordinate system of the device F 1. F 2 points below X , the formula for transforming to the camera coordinate system is as follows:

[0117]

[0118] in, , Represents the local coordinate system F 1. F 2 points below, Figure 5 The points of the left (right) clamp are in the local coordinate system F 2. The points on the instrument rod are in the local coordinate system F 1, is the instrument rod coordinate system F 1 to the camera coordinate system F cam The transformation matrix, It is the coordinate system of the left clamp of the instrument F 2 to the instrument rod coordinate system F 1's transformation matrix, the left clamp coordinate system of the instrument F 2 Must be around the rod coordinate system F 1 z-axis rotation, so one degree of freedom in Euler angles θ The local coordinate system F 1. F 2-room rotation.

[0119] In summary, the rigidity transformation of the articulated device T It can be expressed as ,in , R represents a 3D rotation matrix, Represents a 3D translation vector.

[0120] After clarifying the reconstruction process of the Gaussian model of the instrument and the rigid motion parameters corresponding to the Gaussian model, Figure 4 and Figure 5As shown, based on the defined Gaussian model, the pose parameters of the instrument are optimized to obtain the 6D pose of the articulated instrument. Given the current frame binocular image, the current frame depth map is obtained by the binocular depth estimation algorithm, and the segmentation mask of the instrument is obtained by the instrument segmentation algorithm. Among them, the input image, depth map and instrument mask are used as the supervision information for subsequent pose learning. Based on the initialization position of the instrument, the view, depth map and projection mask under the corresponding pose are rendered by 3D Gaussian sputtering ( Figure 4 It is expressed as rendered image, segmentation, and depth information, and is compared with the original image, semantic segmentation mask, and depth estimation result ( Figure 4 The loss functions are constructed separately for the image, segmentation, and depth information (expressed in the figure), so that the subsequent steps can construct the total loss function based on this, and then the device posture is updated and learned through the back propagation algorithm to obtain the optimal 6D posture of the device in the current frame image.

[0121] In the stage of device posture optimization, the update of the device Gaussian model follows the following formula:

[0122]

[0123] in, i represents the index of the Gaussian kernel, t Indicates the current frame. is the property of the reconstructed instrument Gaussian model, The update analogy formula (a) is as follows:

[0124] when When it is a point belonging to the rod of the instrument, it is updated as follows:

[0125]

[0126] Indicates that Convert to local coordinate system F 1 is the transformation matrix; It is determined after the instrument Gaussian model is reconstructed and the corresponding posture parameters are defined.

[0127] when When it belongs to the left (right) clamp of the instrument, update as follows:

[0128]

[0129] Indicates that Convert to local coordinate system F 2 is the transformation matrix; It is determined after the instrument Gaussian model is reconstructed and the corresponding posture parameters are defined.

[0130] The initialization of the instrument posture is mainly divided into two cases.

[0131] First, for the first frame of image, the rendering view of the instrument Gaussian model on the image plane is obtained in combination with the camera model, the laparoscope is moved so that the image and the area where the rendering view is located coincide with each other, and the instrument posture at this time is used as the initial instrument posture of the first frame of image;

[0132] Second, for images other than the first frame, the device pose of the current frame is initialized based on the optimal device pose of the previous frame and the optical flow information of two consecutive frames.

[0133] Current frame image I t Initial instrument posture From the previous frame of equipment pose T t-1 and a rough estimate of the 3D translation get.

[0134]

[0135] in, The calculation of is as follows:

[0136]

[0137] in, Represents the mean value of the predicted optical flow from the previous frame to the current frame in the instrument area, Represents the average depth value of the instrument area in the previous frame, f x , f y Respectively indicated in the camera x , y The focal length in the direction.

[0138] In step S3, a total loss function is constructed based on the rendering result and the current frame image and its instrument mask and instrument depth map; and the instrument posture is updated and learned through the back propagation algorithm to minimize the total loss function and obtain the optimal 6D posture of the instrument in the current frame image.

[0139] Furthermore, the optimization of the device posture is mainly performed by rendering the view and the original image I t Loss of color consistency between L im , Rendered instrument projection mask and instrument segmentation mask M t The region alignment loss between L region , Rendered device depth map And the device depth estimation results D t The depth consistency loss between L depth To construct the optimization target.

[0140] In the iteratively updated device pose T t Under the action of , the color consistency loss function between the 3D Gaussian rendering view and the current frame image is constructed as follows:

[0141]

[0142] in, t Indicates the current moment; L 1 represents a norm loss, T t Indicates the current frame equipment posture; Represents a rendered view; ; M t The instrument mask representing the current image segmentation;

[0143] Construct the regional alignment loss function between the 3D Gaussian differentiable rendering projection image and the device mask of the current frame image as follows:

[0144]

[0145] in, , They represent the first c The projection shape of each articulated part on the image plane and the instrument mask of the current frame image; ; ;

[0146] Construct the depth consistency loss between the rendered depth map and the device depth map of the previous frame as follows:

[0147]

[0148] in, 、 Respectively represent the rendered depth map and the device depth map of the previous frame image; It is the process of rendering the depth map of the Gaussian model of the device;

[0149] The total loss function is constructed as:

[0150]

[0151] in, α 1 、α 2、α 3 represent the corresponding weight coefficients respectively;

[0152] Then the device posture is adjusted by back propagation algorithm. Iterate to minimize L pose , get the optimal 6D pose estimation result of the device in the current frame image, as shown below:

[0153]

[0154] In step S4, a thin plate spline interpolation method is used to perform depth completion on the tissue depth map to obtain a three-dimensional tissue surface.

[0155] In order to reduce the accuracy of tissue depth estimation and the loss of tissue depth in the occluded area caused by device occlusion, the present invention introduces a depth completion method based on thin plate spline interpolation. This method uses the depth estimation result to take the pixel points of the 10-pixel wide area of ​​the device mask neighborhood as control points. , Belong to the mask ( 1-M ) and the corresponding depth values ​​in the area where it is located, R Represents the number of pixels on the laparoscope image, and fits a surface that passes through all control points and can ensure the minimum bending energy, so as to correct and complete the three-dimensional information of the tissue in the vicinity of the instrument and its occluded area.

[0156] Now, we use the binocular depth estimation algorithm to obtain the tissue depth without device occlusion and unaffected by it, take the pixel points of the 10-pixel-wide area of ​​the device mask neighborhood as control points, and use these control points to fit a surface. The surface fitting is done by the norm of the predicted value and the true value. L a and minimum bending loss L b To construct the objective function L TPS , the loss function used can be described as follows:

[0157]

[0158] in L a is the fitting term, L b The constrained fitting surface has the minimum bending energy while passing through all control points. λ is the weight coefficient. L a The calculation of is as follows:

[0159]

[0160] inp r Represents a pixel, , express p r Corresponding to the depth value on the fitted surface. L b The calculation of is as follows:

[0161]

[0162] minimize L TPS , solve , fitting the complete tissue depth information into a surface , , Represents a depth value as follows:

[0163]

[0164] It can be understood that the embodiments of the present invention address the difficulty of three-dimensional reconstruction of the in vivo environment caused by occlusion of tissue by surgical instruments in endoscopic scenarios. It selects areas with high confidence in the depth estimation results as control points, combines fitting energy and minimum bending energy, and achieves the goal of removing the influence of instrument occlusion on the three-dimensional reconstruction of tissue.

[0165] In step S5, the three-dimensional coordinates of the tissue in the current frame camera coordinate system are obtained according to the three-dimensional tissue surface; the three-dimensional coordinates of the instrument in the current frame camera coordinate system are obtained according to the optimal 6D pose estimation result of the instrument in combination with the reconstructed instrument Gaussian model; the relative distance between the instrument and the tissue is obtained according to the three-dimensional coordinates of the tissue and the instrument in the current frame camera coordinate system.

[0166] The three-dimensional tissue surface is obtained by tissue depth completion , the initial tissue depth map D t ( 1-M t ) is completed as , combined with the camera model, the three-dimensional coordinates of the tissue in the camera coordinate system can be calculated .

[0167]

[0168] Among them, D2P(*) is based on pixel coordinates, depth map and camera intrinsic parameters K The process of obtaining a three-dimensional point cloud.

[0169] After the above steps, the optimal 6D posture of the instrument is obtained , assuming that the initialization Gaussian kernel center position of the current frame instrument Gaussian model is ,I Represents the number of Gaussian kernels. The center of the Gaussian kernel is simplified to the point cloud of the instrument to obtain the three-dimensional coordinates of the surgical instrument point cloud model in the current frame camera coordinate system. .

[0170] The three-dimensional coordinates of the points on the rod of the instrument model in the current frame camera coordinate system are calculated as follows:

[0171]

[0172] At the point on the left (right) side of the instrument model, the three-dimensional coordinates in the camera coordinate system of the current frame are calculated as follows:

[0173]

[0174] in, , Respectively indicate that u i,0 Transform to the local coordinate system as shown in the figure F 1 and F 2 down.

[0175] Successfully obtaining the device and organization Based on the three-dimensional coordinates, the relative distance between the instrument and the tissue is calculated as follows:

[0176]

[0177] in, , I and R The number of point clouds referring to instruments and tissues, respectively.

[0178] So far, given the binocular laparoscope image, the embodiment of the present invention successfully gives the three-dimensional position of the articulated instrument and can provide the relative position information of the instrument and tissue to achieve the purpose of auxiliary decision-making and surgical path planning.

[0179] Embodiment 2:

[0180] The embodiment of the present invention provides a 6D pose estimation system for an instrument based on 3D Gaussian sputtering, comprising:

[0181] An image processing module is used to read the current frame image, obtain the device mask through semantic segmentation; and use a binocular depth estimation algorithm to perform depth estimation on the current frame image, and respectively obtain a tissue depth map and a device depth map in combination with the device mask;

[0182] The scene rendering module is used to initialize the instrument posture, perform 3D Gaussian sputtering rendering using the pre-reconstructed instrument Gaussian model, and obtain the view, projection mask and depth map under the corresponding posture;

[0183] The pose estimation module is used to construct a total loss function based on the rendering result, the current frame image, its device mask, and the device depth map; and to update and learn the device pose through the back propagation algorithm, minimize the total loss function, and obtain the optimal 6D pose of the device in the current frame image.

[0184] In an optional embodiment, after the posture estimation module obtains the optimal 6D posture estimation result of the instrument in the current frame image, the instrument 6D posture estimation system further includes:

[0185] A depth completion module, used to perform depth completion on the tissue depth map using a thin plate spline interpolation method to obtain a three-dimensional tissue surface;

[0186] A distance calculation module is used to obtain the three-dimensional coordinates of the tissue in the current frame camera coordinate system based on the three-dimensional tissue surface; obtain the three-dimensional coordinates of the instrument in the current frame camera coordinate system based on the optimal 6D pose estimation result of the instrument in combination with the reconstructed instrument Gaussian model; and obtain the instrument-tissue relative distance based on the three-dimensional coordinates of the tissue and the instrument in the current frame camera coordinate system.

[0187] Embodiment 3:

[0188] An embodiment of the present invention provides a storage medium storing a computer program for 6D pose estimation of an instrument based on 3D Gaussian sputtering, wherein the computer program enables a computer to execute the 6D pose estimation method of the instrument as described in Example 1.

[0189] Embodiment 4:

[0190] An embodiment of the present invention provides an electronic device, including:

[0191] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the instrument 6D pose estimation method as described in Example 1.

[0192] It can be understood that the instrument 6D pose estimation system, storage medium and electronic device based on 3D Gaussian sputtering provided in the embodiments of the present invention correspond to the instrument 6D pose estimation method based on 3D Gaussian sputtering provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the instrument 6D pose estimation method, and will not be repeated here.

[0193] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0194] 1. The embodiment of the present invention does not require additional equipment or data, and only uses binocular endoscopic images to achieve complete three-dimensional reconstruction of the surgical scene, and assists in surgical planning and navigation through real-time estimated three-dimensional information of instruments and tissues.

[0195] 2. The embodiment of the present invention is based on the research idea of ​​"reconstruction-optimization". By aligning the reconstructed instrument model and image features to estimate the instrument posture, it can accurately and robustly restore the 6D posture of the instrument in complex surgical scenarios and estimate the relative distance between the instrument and the tissue.

[0196] 3. The embodiment of the present invention realizes device posture estimation based on the framework of 3D Gaussian sputtering, and can quickly restore the device posture in a small number of iterations.

[0197] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0198] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 6D pose estimation method for an instrument based on 3D Gaussian sputtering, characterized in that: include: Read the current frame image and obtain the device mask through the semantic segmentation algorithm; and performing depth estimation on the current frame image using a binocular depth estimation algorithm, and respectively acquiring a tissue depth map and an instrument depth map in combination with the instrument mask; Initialize the device pose, use the pre-reconstructed device Gaussian model to perform 3D Gaussian sputtering, and obtain the view, projection mask and depth map under the corresponding pose; According to the rendering result, the current frame image, the device mask, and the device depth map, a total loss function is constructed; and the device posture is updated and learned through the back propagation algorithm to minimize the total loss function and obtain the optimal 6D posture of the device in the current frame image; The reconstruction process of the Gaussian model of the device includes: Acquire several images with known camera poses and containing all views of the device to reconstruct a sparse point cloud of the device; Initialize each point of the sparse point cloud as a Gaussian to obtain an initialized Gaussian model of the instrument; Combining the aforementioned image sequence consisting of the images with known camera poses and containing all viewing angles of the instrument and their corresponding camera poses, iteratively optimizing the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; The method constructs a total loss function based on the rendering result, the current frame image, its device mask, and the device depth map; and updates and learns the device posture through a back propagation algorithm to minimize the total loss function and obtain the optimal 6D posture of the device in the current frame image; including: Construct the color consistency loss function between the 3D Gaussian rendering view and the current frame image as follows: in, t Indicates the current moment; L 1 represents a norm loss, T t Indicates the current frame equipment posture; I t Indicates the current frame image; Represents a rendered view; , Represents the process of performing 3D Gaussian differentiable rendering of an image. G i Indicates i Gaussian kernel, , I is the number of Gaussians; M t The instrument mask representing the current image segmentation; Construct the regional alignment loss function between the 3D Gaussian differentiable rendering projection image and the device mask of the current frame image as follows: in, , C Indicates the number of articulated parts of the device; , They represent the first c The projection shape of each articulated part on the image plane and the instrument mask of the current frame image; , Represents the process of performing 3D Gaussian differentiable rendering projection; ; Construct the depth consistency loss between the rendered depth map and the device depth map of the previous frame as follows: in, 、 Respectively represent the rendered depth map and the device depth map of the previous frame image; It is the process of rendering the depth map of the Gaussian model of the device; The total loss function is constructed as: in, α 1 、α 2 、α 3 represent the corresponding weight coefficients respectively; Then the device posture is adjusted by back propagation algorithm. Iterate to minimize L pose , get the optimal 6D pose estimation result of the device in the current frame image, as shown below:

2. The device 6D pose estimation method according to claim 1, characterized in that: After obtaining the optimal 6D pose estimation result of the device of the current frame image, the device 6D pose estimation method further includes: Using a thin plate spline interpolation method to perform depth completion on the tissue depth map to obtain a three-dimensional tissue surface; According to the three-dimensional tissue surface, obtaining the three-dimensional coordinates of the tissue in the current frame camera coordinate system; According to the optimal 6D pose estimation result of the instrument, combined with the reconstructed Gaussian model of the instrument, the three-dimensional coordinates of the instrument in the camera coordinate system of the current frame are obtained; The relative distance between the instrument and the tissue is obtained according to the three-dimensional coordinates of the tissue and the instrument in the camera coordinate system of the current frame.

3. The device 6D pose estimation method according to claim 1 or 2, characterized in that: The initialization device posture comprises: For the first frame of image, a rendering view of the instrument Gaussian model on the image plane is obtained in combination with the camera model, the laparoscope is moved so that the image coincides with the area where the rendering view is located, and the instrument posture at this time is used as the initial instrument posture of the first frame of image; For images other than the first frame, the device pose of the current frame is initialized based on the optimal device pose of the previous frame and the optical flow information of two consecutive frames.

4. The device 6D pose estimation method according to claim 1, characterized in that: The image sequence composed of the images of the aforementioned known camera poses and containing all viewing angles of the instrument and the corresponding camera poses is combined to iteratively optimize the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; comprising: The color consistency loss function between the 3D Gaussian rendering view and the original image that only retains the instrument part is constructed as follows: in, u, q, s Respectively represent the center position, rotation, and size of any Gaussian; , L is the number of images included in the image sequence; L 1 indicates a norm loss; The camera pose is R l The process of performing 3D Gaussian differentiable rendering of an image; It indicates the original image from the image sequence where only the instrument part is retained; A multi-region alignment loss function is introduced to constrain the projection shape of each articulated part on the reconstructed device Gaussian model on the image plane and the device mask on the original image. M lc Align them separately; the multi-region alignment loss function is as follows: in, The camera pose is R l The process of performing 3D Gaussian differentiable rendering projection; The total loss function of the instrument Gaussian model reconstruction process is: in, α represents the corresponding weight coefficient; By minimizing the total loss function L recons , obtain the reconstructed instrument Gaussian model , as shown below:

5. A 6D pose estimation system for an instrument based on 3D Gaussian sputtering, characterized in that: The method for estimating the 6D pose of the device according to claim 1 comprises: An image processing module is used to read the current frame image, obtain the device mask through semantic segmentation; and use a binocular depth estimation algorithm to perform depth estimation on the current frame image, and respectively obtain a tissue depth map and a device depth map in combination with the device mask; The scene rendering module is used to initialize the instrument posture, perform 3D Gaussian sputtering rendering using the pre-reconstructed instrument Gaussian model, and obtain the view, projection mask and depth map under the corresponding posture; The pose estimation module is used to construct a total loss function based on the rendering result, the current frame image, its device mask, and the device depth map; and to update and learn the device pose through the back propagation algorithm, minimize the total loss function, and obtain the optimal 6D pose of the device in the current frame image.

6. The device 6D pose estimation system according to claim 5, characterized in that: After the posture estimation module obtains the optimal 6D posture estimation result of the device of the current frame image, the device 6D posture estimation system further includes: A depth completion module, used to perform depth completion on the tissue depth map using a thin plate spline interpolation method to obtain a three-dimensional tissue surface; A distance calculation module is used to obtain the three-dimensional coordinates of the tissue in the current frame camera coordinate system based on the three-dimensional tissue surface; obtain the three-dimensional coordinates of the instrument in the current frame camera coordinate system based on the optimal 6D pose estimation result of the instrument in combination with the reconstructed instrument Gaussian model; and obtain the instrument-tissue relative distance based on the three-dimensional coordinates of the tissue and the instrument in the current frame camera coordinate system.

7. A storage medium, characterized in that: It stores a computer program for device 6D pose estimation based on 3D Gaussian sputtering, wherein the computer program enables a computer to execute the device 6D pose estimation method as described in any one of claims 1 to 4.

8. An electronic device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the instrument 6D pose estimation method as described in any one of claims 1 to 4.

Images