Real-time liver volume evaluation method based on liver surface reconstruction in digital twin surgery

Abstract

The embodiment of the invention provides a real-time liver volume evaluation method based on liver surface reconstruction in digital twin surgery, and relates to the field of medicine and artificial intelligence. The method comprises the following steps: acquiring a first three-dimensional model of the liver, wherein the first three-dimensional model is generated based on image data which is acquired before an operation and is related to the liver; the method comprises the following steps: acquiring image data of the liver in the liver resection; extracting feature information related to the liver based on the image data, and performing feature matching based on the extracted feature information; based on a feature matching result, reconstructing a second three-dimensional model of the liver; performing point cloud registration on the first three-dimensional model and the second three-dimensional model; and performing liver volume evaluation based on a point cloud registration result. According to the method, the liver model can be reconstructed in real time in the operation, and the accuracy of liver volume evaluation is improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of medicine and artificial intelligence technology, and in particular, to a real-time liver volume assessment method based on liver surface reconstruction during digital twin surgery. Background Art

[0002] Liver surgery is one of the most complex surgical procedures, used to treat a variety of liver diseases, such as cirrhosis, liver cancer, and liver abscesses. It carries high risks and potential complications. Post-hepatectomy liver failure (PHLF) is a major source of morbidity and mortality after major liver surgery, and its risk is closely related to the volume of the residual liver. A key to reducing PHLF is to reserve sufficient functional residual liver volume during the planning and execution of liver surgery.

[0003] In existing technologies, the residual liver volume is assessed using three-dimensional models generated by preoperative CT or MRI scans. These models can provide individualized liver structural information before surgery and predict the residual liver volume by setting the pre-cutting line. However, because the liver morphology changes dynamically during surgery with respiration, pneumoperitoneum pressure, and the traction intervention of surgical instruments, and because the actual surgical resection path differs significantly from the pre-planned cutting line, assessments based on preoperative models often fail to accurately reflect the actual liver resection situation. Therefore, it is impossible to accurately assess the actual residual liver volume after liver resection during surgery. Summary of the Invention

[0004] The embodiments of the present disclosure provide a real-time liver volume assessment method based on liver surface reconstruction during digital twin surgery, which is used to reconstruct a liver model in real time during surgery and improve the accuracy of liver volume assessment.

[0005] According to one aspect of an embodiment of the present disclosure, a real-time liver volume assessment method based on liver surface reconstruction during digital twin surgery is provided, comprising: Acquire a first three-dimensional model of the liver, where the first three-dimensional model is generated based on image data related to the liver acquired before surgery; During hepatectomy, imaging data of the liver is obtained; Extracting liver-related feature information based on the image data, and performing feature matching based on the extracted feature information; Based on the result of feature matching, a second three-dimensional model of the liver is reconstructed; Performing point cloud registration on the first three-dimensional model and the second three-dimensional model; Liver volume estimation was performed based on the results of point cloud registration.

[0006] In a feasible embodiment, during the hepatectomy, acquiring liver imaging data includes at least one of the following: Using a three-dimensional image acquisition device, video streams of the liver during surgery are collected from multiple angles; extracting key frame data from the video stream; The spatial coverage overlap area between adjacent images in the video stream and / or the key frame data is within a preset percentage range.

[0007] In a feasible embodiment, extracting liver-related feature information based on the image data, and performing feature matching based on the extracted feature information, includes: Performing feature detection on the image data to extract a first key point related to the liver surface texture and a second key point related to the liver geometric edge; Calculating similarities of feature descriptors between key frames of the image data, and establishing cross-visual feature point correspondences based on the similarities, wherein the feature descriptors indicate image information of a local area around the first key point and / or the second key point; Based on the feature point correspondence and the RGB features and depth map information of the image data, stereo constraint matching is performed on the geometric discontinuity features of the off-section area.

[0008] In a feasible embodiment, the performing feature detection on the image data to extract first key points related to the liver surface texture and second key points related to the liver geometric edge includes: Performing deblurring processing on the image data; The feature point density of the first key point and the second key point in each frame is greater than or equal to a preset value.

[0009] In a feasible embodiment, reconstructing the second three-dimensional model of the liver based on the result of feature matching includes: Based on the results of feature matching, a sparse 3D model of the liver during surgery is constructed through incremental structure from motion (SfM). Based on the sparse three-dimensional model, a dense three-dimensional model of the liver during surgery is constructed through multi-view stereo matching (MVS) and determined as the second three-dimensional model.

[0010] In a feasible embodiment, constructing a sparse 3D model of the liver during surgery through SfM based on the result of feature matching includes: Determining an initial pose based on a result of the feature matching; From the result of the feature matching, two frames are obtained as initial key frames based on the degree of overlap and the number of matching points; Based on the feature point matching relationship of the initial key frame and the initial pose, generating an initial three-dimensional point cloud through triangulation processing; For each subsequent frame, the pose of the frame is determined based on the matching relationship between the initial three-dimensional point cloud and the feature points of the frame, and the three-dimensional point cloud is expanded by triangulation to obtain a sparse three-dimensional model.

[0011] In a feasible embodiment, determining the initial pose based on the result of the feature matching includes: Determining internal parameters of a three-dimensional image acquisition device based on pre-calibrated parameters of the three-dimensional image acquisition device and / or real-time calibration data during surgery, wherein the three-dimensional image acquisition device is used to acquire the image data; Calculating a fundamental matrix based on the intrinsic parameters of the three-dimensional image acquisition device and the result of the feature matching; The basic matrix is ​​decomposed to obtain an initial pose.

[0012] In a feasible embodiment, the construction of the sparse three-dimensional model further includes: Performing global processing on the poses and three-dimensional point clouds of the sparse three-dimensional model through bundle adjustment to update all poses and three-dimensional point clouds; And / or, if the liver is deformed during the operation, the position and 3D point cloud of the sparse 3D model are updated by using a sliding window and / or abnormal frame elimination method.

[0013] In a feasible embodiment, performing point cloud registration on the first three-dimensional model and the second three-dimensional model includes: Calculating fast point feature histogram (FPFH) features for a first point cloud of the first three-dimensional model and a second point cloud of the second three-dimensional model respectively; Determining initial corresponding point pairs of the first point cloud and the second point cloud based on the FPFH features; Mapping the first point cloud to the second point cloud based on the NDT algorithm using the initial corresponding point pairs to obtain a first transformation matrix; Determining nearest neighbor pairs of the first point cloud and the second point cloud based on the first transformation matrix; Determining a weight coefficient for each point pair based on the distance or geometric consistency information of the nearest neighbor point pairs; Based on the weight coefficients, the first transformation matrix is ​​iteratively updated until a preset condition is met to obtain a second transformation matrix.

[0014] In a feasible embodiment, the liver volume assessment based on the result of point cloud registration includes: Generate a closed 3D mesh based on the results of point cloud registration; Volume calculation is performed based on the three-dimensional grid to determine information related to the liver volume, where the information includes a ratio of the residual liver to the whole liver.

[0015] In a feasible embodiment, the method further includes at least one of the following: Displaying a point cloud registration result and a volume evaluation result on a display device, wherein the point cloud registration result is displayed through a three-dimensional model, and the volume evaluation result includes at least one of first information related to the resection volume, second information related to the residual volume, or third information related to the point cloud registration accuracy; In response to an operation on the three-dimensional model, adjusting the angle, size and / or viewing angle of the three-dimensional model; and / or, in response to an operation on the pre-cut line of the liver, adjusting the pre-cut line displayed on the three-dimensional model; When the volume assessment result indicates that the residual liver volume is less than a preset percentage, an alarm is triggered.

[0016] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the real-time liver volume assessment method based on liver surface reconstruction during digital twin surgery as described in the above embodiment.

[0017] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the real-time liver volume assessment method based on liver surface reconstruction during digital twin surgery described in the above embodiment is implemented.

[0018] According to one aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the real-time liver volume assessment method based on liver surface reconstruction during digital twin surgery as described in the above embodiment.

[0019] The technical solutions provided by the embodiments of the present disclosure have the following beneficial effects: The disclosed embodiment provides a real-time liver volume assessment method based on digital twin intraoperative liver surface reconstruction. Specifically, during liver resection, image data of the liver can be obtained, and based on the image data, feature information related to the liver can be extracted and feature matching can be performed based on the extracted feature information. Then, based on the result of the feature matching, a second three-dimensional model of the liver can be reconstructed to achieve dynamic reconstruction of the three-dimensional model of the liver during surgery, more accurately reflecting intraoperative changes. On this basis, in order to reduce the assessment error, a first three-dimensional model of the liver can be obtained. The first three-dimensional model is generated based on image data related to the liver collected before surgery. The first three-dimensional model and the second three-dimensional model are point cloud registered, and finally the liver volume is assessed based on the result of the point cloud registration. The disclosed embodiment can reconstruct the liver model in real time during surgery, improve the accuracy of liver volume assessment, and help improve the safety and accuracy of liver surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments of the present disclosure.

[0021] Figure 1 A schematic flow chart of a method for real-time liver volume assessment based on liver surface reconstruction during digital twin surgery provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of a processing flow provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of a point cloud registration process provided by an embodiment of the present disclosure; Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following describes embodiments of the present disclosure in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions of the embodiments of the present disclosure.

[0023] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include the plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present disclosure mean that the corresponding features can be implemented as the features, information, data, steps, operations, elements, and / or components presented, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" or "A, B" indicates implementation as "A," or implementation as "B," or implementation as "A and B."

[0024] The term "based on" used in various embodiments of the present disclosure can be interpreted as meaning that the premise, condition, or information on which the basis is based is not exclusive, but at least one or a portion of it. This means that there is at least one clear basis, and other possible bases are not excluded.

[0025] The relevant technologies have the following deficiencies in terms of real-time performance, accuracy, and matching with intraoperative images: (1) Insufficient real-time performance: The preoperative three-dimensional model cannot be updated in time during the operation and cannot reflect the changes in the liver during the operation. (2) Poor accuracy: Due to factors such as deformation and bleeding during the operation, the registration accuracy between the preoperative model and the actual liver in the relevant navigation technology is low and relies on manual interaction. (3) Non-interventional: Although the remnant liver volume can be assessed by three-dimensional reconstruction based on the first postoperative CT scan, the operation has already ended and it is impossible to effectively intervene in the surgical operation, making it difficult to change the outcome of postoperative liver failure.

[0026] In response to at least one of the above technical problems, the embodiments of the present disclosure provide a real-time liver volume assessment method based on liver surface reconstruction during digital twin surgery. Specifically, a real-time three-dimensional model of the liver is reconstructed through intraoperative dynamic imaging to construct a high-precision digital twin of the liver. The three-dimensional model generated before the operation is adaptively aligned with the three-dimensional model reconstructed in real time during the operation to reduce the assessment error caused by intraoperative deformation. When the liver volume is assessed based on the point cloud alignment results, the accuracy of the liver volume assessment can be effectively improved.

[0027] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present disclosure and the technical effects produced by the technical solutions of the present disclosure. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0028] The following combination Figures 1 to 3 The method provided in the embodiment of the present disclosure is described in detail.

[0029] In one feasible embodiment, the real-time liver volume assessment method based on intraoperative liver surface reconstruction using digital twin surgery provided in the embodiments of the present disclosure can be executed by a terminal, which can run a client or service platform. The terminal (also referred to as a device) can be, but is not limited to, a smartphone, tablet, laptop, desktop computer, wearable electronic device (such as a smartwatch), smart medical device, AR / VR device, etc.

[0030] A digital twin is a digital representation of a physical entity or system created in a virtual space through digital means, enabling it to reflect the physical entity's state, behavior, and performance in real time. In the disclosed embodiments, the physical entity can be a human organ, such as the liver. Based on the data of the physical entity, a digital replica, such as a 3D model of the liver, is constructed to simulate its structure, function, and behavior.

[0031] Specifically, if Figure 1As shown, the method provided in the embodiment of the present disclosure includes S101 to S106: S101 . Acquire a first three-dimensional model of the liver, where the first three-dimensional model is generated based on image data related to the liver acquired before surgery.

[0032] S102. During liver resection, obtain imaging data of the liver.

[0033] S103 . Extracting liver-related feature information based on the image data, and performing feature matching based on the extracted feature information.

[0034] S104: Reconstruct a second three-dimensional model of the liver based on the result of feature matching.

[0035] S105: Perform point cloud registration on the first three-dimensional model and the second three-dimensional model.

[0036] S106. Evaluate the liver volume based on the result of point cloud registration.

[0037] Alternatively, the liver-related image data may be obtained by high-resolution CT (computed tomography) or MRI (magnetic resonance imaging) scans of the liver. For example, CT scans the liver from multiple angles, reconstructing cross-sectional (axial) two-dimensional images through computer reconstruction. MRI uses magnetic fields and radiofrequency pulses to excite hydrogen nuclei in the human body and reconstructs images by detecting the signals released.

[0038] Optionally, once liver-related image data is obtained, an image processing algorithm can be used to separate the liver-related portion from the image to remove interference from other tissues. The multi-layer 2D images are then spatially aligned to ensure the accuracy of the generated 3D model, and the 2D images are converted into a 3D model. Exemplarily, a CT image or MRI image is imported into an abdominal medical 3D reconstruction system for image segmentation and 3D reconstruction, generating a first preoperative 3D model of the liver.

[0039] Optionally, the image data can be processed into a suitable format, such as DICOM (a standard format for medical images that can include metadata such as patient information and scanning parameters) into PLY (Polygon File Format), STL (Stereolithography), or OBJ (Wavefront Object) formats. Due to the large size and complex format of DICOM, converting it into a common 3D model format such as PLY, STL, or OBJ can simplify the data structure to meet the input requirements of the 3D reconstruction system and improve processing efficiency.

[0040] Optionally, during liver resection surgery, real-time intraoperative data acquisition can be performed to obtain continuous dynamic imaging of the liver. For example, a 3D laparoscopic RGB camera can capture continuous dynamic images of the liver from multiple angles during surgery. This multi-angle capture eliminates blind spots associated with a single viewpoint, fully covering the liver's surface anatomy and providing redundant perspective information for 3D reconstruction.

[0041] Optionally, feature detection can be performed based on the image data to obtain liver-related feature information, such as geometric features and texture features. On this basis, feature matching can be performed to achieve cross-visual feature alignment and stereo constraint matching, thereby improving the accuracy of cross-sectional reconstruction.

[0042] In the disclosed embodiment, a first 3D model is generated before surgery to display information such as preoperative liver morphology and vascular distribution, providing auxiliary information for the surgeon during surgery. It can also serve as a reference for intraoperative 3D model registration and quantify liver deformation during surgery. The acquisition of intraoperative images can provide real-time updates of the liver's status, compensating for limitations in preoperative data. For example, preoperatively, the local structure of the liver may be unclear due to issues such as difficulty breathing. Intraoperative images can supplement this key information, improving the safety and accuracy of the surgery.

[0043] Optionally, a second 3D model of the liver can be reconstructed in real time during surgery to capture real-time changes in the liver. Point cloud registration of the preoperative first 3D model with the intraoperative second 3D model aligns the preoperative and intraoperative models, eliminating modal differences and correcting deformation errors caused by various reasons. The registered model can then be directly used for volume assessment, avoiding errors caused by model misalignment and improving the accuracy of liver volume assessment.

[0044] In a feasible embodiment, in S102, obtaining liver image data during liver resection includes at least one of the following steps A1 to A2: Step A1: Using a three-dimensional image acquisition device, video streams of the liver during surgery are acquired from multiple angles.

[0045] Alternatively, the 3D image acquisition device can be a 3D laparoscopic RGB camera. This device can capture continuous dynamic images of the liver from multiple angles during surgery, generating a video stream of the liver. Multi-angle acquisition ensures that all areas of the liver are fully covered, avoiding information loss due to blind spots. The video stream can record real-time changes in the liver during surgery.

[0046] In one example, a 3D laparoscope (such as the DPM 4K 3D fluorescence laparoscope), a multi-camera array, or an intraoperative ultrasound probe can be used to scan the liver surface with a fixed or dynamic trajectory to collect liver data.

[0047] Step A2: extract key frame data from the video stream.

[0048] Optionally, due to the large amount of video data stream, when acquiring a video stream of the liver, image processing technology can be used to perform sub-second time segmentation, extracting continuous keyframe data at a rate of 30 frames per second (as an example only). Based on the dual discrimination mechanism of instrument occlusion detection and image stability assessment, image sequences with rich information, no instrument occlusion, and minimal local deformation are automatically screened out. Several keyframe images are extracted to form a feature view set, significantly improving the subsequent point cloud reconstruction quality and modeling robustness, reducing the amount of computation and complexity, removing redundant information, and improving processing efficiency. For example, keyframes can be selected from frames with a large amount of information, such as frames covering major liver structures (such as blood vessels and tumor boundaries). Keyframes can be screened through image similarity analysis, deep learning models, and the like.

[0049] Optionally, the spatial overlap between adjacent images in the video stream and / or keyframe data is within a preset percentage. In one example, the spatial overlap between adjacent images in the video stream and / or keyframe data can be controlled to be 70-80%. Proper setting of the overlap can avoid data redundancy and information loss that could affect the integrity of the 3D reconstruction.

[0050] In a feasible embodiment, in S103, feature information related to the liver is extracted based on the image data, and feature matching is performed based on the extracted feature information, including steps B1 to B3: Step B1: Perform feature detection on the image data to extract first key points related to the liver surface texture and second key points related to the liver geometric edge.

[0051] Optionally, feature processing can capture intraoperative liver imaging data in real time, including RGB information, depth map information, and cross-sectional geometry information. For the input image data, a feature detection algorithm can be used to identify the liver area in the image and extract the corresponding key points. Among them, the feature detection algorithm can be SIFT (Scale Invariant Feature Transform, a scale-space-based image local feature description algorithm that remains invariant to image scaling, rotation, and affine transformation), SURF (Speeded-Up Robust Features, an algorithm that can improve computing speed based on SIFT), ORB (Oriented FAST and Rotated BRIEF, a fast local feature extraction algorithm) and other algorithms, or it can be a feature detection method based on deep learning.

[0052] Optionally, the first key point related to the liver surface texture can reflect changes in the liver surface texture, such as vascular bifurcations, ligament attachment points and other features; the second key point related to the liver geometric edge can reflect the boundary position and shape characteristics of the liver in the image. For example, the second key point can be used to characterize the overall contour and shape of the liver, providing key information for the geometric modeling and positioning of the liver.

[0053] Optionally, in step B1, feature detection is performed on the image data to extract first key points related to the liver surface texture and second key points related to the liver geometric edge, including steps B11 to B12: Step B11: Deblurring the image data.

[0054] Optionally, considering that the image data may be affected by various factors (such as equipment jitter, noise interference) during the acquisition and transmission process, resulting in blurred images, in order to reduce or eliminate the blurred components in the image and improve the clarity and quality of the image, the image data can be pre-processed after the intraoperative data is acquired, such as Figure 2 As shown, as deblurred.

[0055] Step B12: Using a GPU, construct a Gaussian difference pyramid in parallel to extract the first key point and the second key point.

[0056] Optionally, a Gaussian difference pyramid is a multi-scale image representation method that performs Gaussian blurring on the original image at different scales and calculates the difference between Gaussian blurred images at adjacent scales to obtain several differential images at different scales. The combination of related images can form a Gaussian difference pyramid.

[0057] Alternatively, a GPU (graphics processing unit) has powerful parallel computing capabilities. By distributing the computational tasks involved in constructing the Gaussian difference pyramid across multiple GPU computing units, processing can be accelerated. For example, Gaussian blur and difference calculations can be performed in parallel for different image regions or at different scales.

[0058] Optionally, a feature point density of the first key point and the second key point in each frame is greater than or equal to a preset value.

[0059] Optionally, considering that a sufficient number of key points can provide richer image feature information, while too few key points may lead to matching failures or very low reconstruction accuracy, after extracting key points, the number of key points in each frame can be counted and compared with a preset feature point density threshold (such as density ≥ 200 points / frame). If the number of key points in a frame is less than the preset value, the parameters of the key point extraction algorithm (such as the scale range of Gaussian blur, contrast threshold, etc.) can be adjusted and key point extraction can be repeated until the key point density in each frame or most frames meets the set requirements. Maintaining a stable key point density across different frames helps improve the robustness and reliability of the algorithm.

[0060] Step B2: Calculate the similarity of feature descriptors between key frames of the image data, and establish a cross-visual feature point correspondence based on the similarity, wherein the feature descriptors indicate image information of a local area around the first key point and / or the second key point.

[0061] Optionally, corresponding feature descriptors can be calculated for the extracted first and second key points. A feature descriptor is a mathematical expression of the image information of the local area around the key point, and may include image information such as the color, texture, and gradient of the area. By calculating the similarity of feature descriptors between key frames (such as Euclidean distance and Hamming distance), feature descriptors with high similarity (such as higher than a set similarity threshold) can be found, and then cross-visual feature point correspondences (i.e., pairs of feature points that match each other in different key frames) can be established, that is, the mutual correspondences between key points in different frames can be determined.

[0062] In the disclosed embodiments, establishing cross-visual correspondences between feature points facilitates accurate matching and localization of the liver in image data from different viewing angles and time points. These correspondences can characterize changes in the liver's position and morphology across frames. Furthermore, the localized image information carried by the feature descriptors facilitates accurate identification and matching of key points in complex imaging environments, improving matching accuracy and robustness.

[0063] Step B3: performing stereo constraint matching on the geometric discontinuity features of the off-section area based on the feature point correspondence and the RGB features and depth map information of the image data.

[0064] Optionally, the RGB features of the image data may be extracted during the feature detection process.

[0065] Optionally, the depth map information can provide the distance information from each point in the image data to the camera. Feature matching combined with the depth map information is conducive to more accurate description of the three-dimensional spatial position and shape of the liver.

[0066] Optionally, when performing stereo constraint matching on the geometric discontinuity features of the out-of-section areas that may exist in the image data (such as partial occlusion of the liver due to image acquisition, or partial missing or separated areas of the liver due to surgical operations), the positional relationship of the key points in the three-dimensional space can be analyzed, and the depth map information can be used to match and correct the feature points of the discontinuous areas, so that the images of the liver in different frames can be more accurately matched and fused.

[0067] In the disclosed embodiment, stereo constraint matching is performed in combination with RGB features and depth map information, which can improve the registration accuracy of image data, reduce matching errors caused by geometric discontinuities, improve the accuracy of end face reconstruction, and provide effective reference data for the subsequent reconstruction of the three-dimensional liver model.

[0068] In a feasible embodiment, in S104, based on the result of feature matching, a second three-dimensional model of the liver is reconstructed, including steps C1 to C2: Step C1: Based on the results of feature matching, a sparse 3D model of the liver during surgery is constructed through incremental structure from motion (SfM).

[0069] Alternatively, incremental structure-from-motion (SfM) can be used to estimate the camera pose (such as the device's position and orientation during data acquisition) based on feature matching results and reconstruct the scene's 3D structure. The construction of a sparse 3D model can provide preliminary 3D spatial information for liver model reconstruction, such as the positions of key liver points in 3D space, providing a preliminary understanding of the liver's morphology and spatial distribution from a 3D perspective.

[0070] Optionally, based on the result of feature matching in step C1, a sparse 3D model of the liver during surgery is constructed by SfM, including steps C11 to C14: Step C11: Determine the initial pose based on the result of the feature matching.

[0071] Optionally, the initial posture refers to the posture of the three-dimensional image acquisition device, which may include position and orientation information.

[0072] Optionally, in step C11, determining an initial pose based on the result of the feature matching includes steps C111 to C113: Step C111 : Determine the internal parameters of the three-dimensional image acquisition device based on pre-calibrated parameters of the three-dimensional image acquisition device and / or real-time calibration data during surgery. The three-dimensional image acquisition device is used to acquire the image data.

[0073] Optionally, before surgery, the three-dimensional image acquisition device (such as a laparoscopic camera) used to collect image data can be pre-calibrated, such as by using a specific calibration plate and calibration algorithm to obtain the internal parameters of the three-dimensional image acquisition device, such as focal length (such as horizontal focal length, vertical focal length), principal point (such as the intersection of the optical axis and the image plane in the image coordinate system), and distortion coefficient (such as radial distortion coefficient, tangential distortion coefficient).

[0074] Optionally, during the operation, slight movements of the device or environmental changes may affect the accuracy of the internal parameters of the three-dimensional image acquisition device. On this basis, real-time internal parameter data can be obtained through real-time calibration methods (such as calibration methods using known feature points during the operation).

[0075] Step C112: Calculate a basic matrix based on the internal parameters of the three-dimensional image acquisition device and the result of the feature matching.

[0076] Optionally, the fundamental matrix can describe the geometric relationship between corresponding points in the image at different perspectives. For example, the fundamental matrix can be calculated based on the internal parameters of the three-dimensional image acquisition device and the results of feature matching using the eight-point method (an algorithm based on the principle of least squares, which solves the fundamental matrix using at least 8 pairs of corresponding points) or the RANSAC algorithm (which can eliminate erroneous matching point pairs through random sampling and iteration to obtain a more accurate fundamental matrix).

[0077] Optionally, the basic matrix establishes a geometric constraint relationship between adjacent frames and may include relative motion information (such as rotation and translation) of a three-dimensional image acquisition device.

[0078] Step C113: Decompose the basic matrix to obtain an initial pose.

[0079] Optionally, the fundamental matrix can be converted into an intrinsic matrix of the 3D image acquisition device based on the intrinsic parameters of the 3D image acquisition device. Singular value decomposition (SVD) is then performed on the intrinsic matrix to obtain an initial pose of the 3D image acquisition device. The initial pose may include an initial rotation matrix and a translation vector.

[0080] Step C12: From the result of the feature matching, two frames are obtained as initial key frames based on the degree of overlap and the number of matching points.

[0081] Optionally, for the feature matching results of all adjacent frames, the overlap between the two frames and the number of matching points can be calculated. The overlap can be determined by calculating the ratio of the area of ​​the common coverage of the two frames to the area of ​​a single frame, and the number of matching points can be determined by the number of successfully matched feature point pairs in the two frames.

[0082] Optionally, when determining the initial keyframe based on overlap and the number of matching points, two frames with the highest overlap and the largest number of matching points can be selected as the initial keyframes. A high degree of overlap indicates that the two frames may contain a large amount of common information, and a large number of matching points indicates a high reliability of the correspondence between the two frames. The initial keyframe is the starting point for 3D reconstruction. Selecting an appropriate initial keyframe can ensure the quality and accuracy of the initial 3D point cloud, while setting constraints based on overlap and the number of matching points can help improve the accuracy of the initial 3D point cloud.

[0083] Step C13: generating an initial three-dimensional point cloud through triangulation based on the feature point matching relationship of the initial key frame and the initial pose.

[0084] Optionally, triangulation refers to determining the coordinates of these feature points in three-dimensional space through geometric calculations based on the matching relationship and initial pose of corresponding feature points in the two frames. For example, for a pair of corresponding feature points in two frames, the coordinates of the feature points in their respective image coordinate systems and the initial pose are known. Spatial geometric relationships (such as the similarity principle of triangles) can be used to establish a system of equations to solve the coordinates of the feature points in three-dimensional space. By triangulating all matching feature point pairs in the initial key frame, the coordinates of the corresponding feature points in three-dimensional space can be obtained, and the initial three-dimensional point cloud can be constructed accordingly.

[0085] Step C14: for each subsequent frame, based on the matching relationship between the initial three-dimensional point cloud and the feature points of the frame, determine the pose of the frame, and expand the three-dimensional point cloud by triangulation to obtain a sparse three-dimensional model.

[0086] Optionally, for each subsequent frame, feature point matching can be performed between it and the initial three-dimensional point cloud to determine the correspondence between the points in the initial three-dimensional point cloud and the feature points in the frame, and the PnP algorithm (Perspective-n-Point, an algorithm that solves the pose by knowing n points in three-dimensional space and their projection points in the image) can be used to determine the pose of the frame relative to the initial three-dimensional point cloud based on the correspondence and the internal parameters of the three-dimensional image acquisition device.

[0087] Optionally, based on the pose of the new frame and the matching relationship between the feature points, the feature points in the new frame can be triangulated and added to the initial three-dimensional point cloud, thereby expanding the three-dimensional point cloud.

[0088] The embodiment of the present disclosure can obtain a sparse three-dimensional model by adding new perspectives frame by frame and expanding the point cloud. The sparse three-dimensional model includes three-dimensional information of the liver at different perspectives.

[0089] Optionally, the construction of the sparse three-dimensional model further includes step C01 and / or step C02: Step C01: Perform global processing on the poses and three-dimensional point clouds of the sparse three-dimensional model through bundle adjustment to update all poses and three-dimensional point clouds.

[0090] Alternatively, bundle adjustment (BA) is a global optimization method based on the least squares principle. In the construction of a sparse 3D model, which may contain certain errors, bundle adjustment uses all poses and 3D point clouds as optimization variables and the reprojection error as the objective function. Reprojection error refers to the difference between the actual observed image points and the reprojection error of the 3D point cloud onto the image plane based on the current estimated pose. By minimizing the sum of the reprojection errors of all feature points, the poses and 3D point clouds are adjusted, which helps improve the accuracy of the 3D reconstruction results and ensure the consistency of the entire sparse 3D model.

[0091] Step C02: If the liver is deformed during the operation, the position and three-dimensional point cloud of the sparse three-dimensional model are updated by using a sliding window and / or abnormal frame elimination method.

[0092] Alternatively, a sliding window is a local optimization method that estimates poses and updates the 3D point cloud for a set of consecutive frames within the current window. If the liver deforms during surgery, as its morphology and structure change over time, using a sliding window to limit the number of frames involved in the optimization allows for processing of the most recent set of frames, reducing computational effort and enabling faster response to liver deformation.

[0093] During intraoperative data collection, various factors may cause abnormal frames to be generated. Inaccurate feature matching results in these abnormal frames may lead to errors in pose estimation and 3D point cloud generation. By inspecting each frame and removing abnormal frames that do not meet requirements (such as a small number of matching points or large reprojection errors), we can ensure that pose estimation and point cloud updates are performed on normal frames, improving the accuracy of the sparse 3D model.

[0094] Step C2: Based on the sparse three-dimensional model, a dense three-dimensional model of the liver during surgery is constructed through multi-view stereo matching (MVS), and is determined as a second three-dimensional model.

[0095] Alternatively, multi-view stereo (MVS) uses the camera pose and feature point information provided by the sparse 3D model to perform more detailed image matching and fusion, constructing a dense 3D model. Compared to a sparse 3D model, a dense 3D model can include richer details about the liver surface.

[0096] In one example, after obtaining a sparse point cloud from a sparse 3D model, the dense reconstruction function of COLMAP (an open-source SfM and MVS software) can be used to generate high-quality point cloud data through multi-view stereo matching. COLMAP's dense reconstruction function provides multiple parameters for users to adjust, such as the resolution of the depth map estimation, the sampling step, and the disparity range. Adjusting these parameters will affect the calculation time and point cloud quality. During implementation, the parameters can be adjusted according to actual needs to balance the calculation time and point cloud quality. For example, to achieve faster calculation speed, the resolution of the depth map estimation can be set to be lowered. To obtain higher-quality point cloud data, the sampling step can be increased or the disparity range can be adjusted. In addition, motion artifact points can be dynamically eliminated based on RANSAC (Random Sampling Consensus, an outlier detection algorithm) outlier detection. In point cloud data, some motion artifact points may be generated due to the existence of various factors (such as the movement of the acquisition device, the movement of the object, noise, etc.). These points do not conform to the actual geometric structure of the liver surface. Therefore, the point cloud data can also be processed based on RANSAC. A geometric model is fitted by randomly sampling a certain number of points, and the distance from other points to the model is counted. Points with a distance exceeding a certain threshold are determined as outliers and are eliminated. Optionally, the implementation of outlier elimination can be dynamic, and multiple iterations can be performed according to different data and needs to improve the accuracy of eliminating motion artifact points and improve the robustness of data processing.

[0097] In this disclosed embodiment, by adjusting the image viewport screening strategy, only image pairs with high parallax and low occlusion are retained for triangulation, eliminating weak matches and redundant views, effectively compressing the point cloud size and improving the geometric accuracy and computational efficiency of 3D reconstruction. Furthermore, an occlusion region masking mechanism is introduced into the original SIFT feature matching to avoid the generation of false point clouds caused by occlusion.

[0098] Optionally, 3D reconstruction of COLMAP can also be performed using a NeRF-based deep learning model (NeRF, Neural Radiance Fields).

[0099] Optionally, after obtaining the dense 3D model, a point cloud can be exported, and the generated point cloud can be exported to a format suitable for subsequent processing steps, such as PLY, OBJ, etc.

[0100] Optionally, the generated sparse point cloud data is used to perform high-precision modeling of the liver surface using 2D Gaussian Splatting technology. Specifically, each point in the point cloud is represented as a Gaussian kernel, which contains four key parameters: the position vector μ (representing the projection center of the point in the image plane), the two-dimensional covariance matrix Σ (used to describe the blur and directionality of the point in the image), the transparency coefficient α (controlling the visibility weight of the point in the final image), and the color vector c (extracted from the RGB pixel values ​​of the original laparoscopic image). By modeling these sparse points as a continuous Gaussian distribution, the system avoids the topological assumptions imposed by traditional triangulation and mesh construction, thereby better adapting to the complex non-rigid deformations of the liver surface and changes in local anatomical structures during surgery.

[0101] To achieve high-quality image reconstruction, the system incorporates a maximum likelihood estimation mechanism, constructs an optimization objective function based on pixel differences, and jointly trains all Gaussian kernel parameters to minimize the reprojection error between the reconstructed image and the real image. Specifically, the system compares the pixel-level L2 difference between each frame of the real laparoscopic image and the current Gaussian model rendering. It then uses gradient descent (such as the Adam optimizer) to iteratively update the spatial position, shape, transparency, and color of each Gaussian kernel, ensuring consistency in both geometric structure and visual perception. The entire training process is accelerated in parallel on a GPU, completing model optimization for key frames within 1–2 seconds, meeting intraoperative real-time requirements.

[0102] The rendering process uses the differentiable rendering framework Soft Rasterization, which supports transparent overlay and depth sorting of multiple Gaussian kernels on the image plane, thereby achieving accurate modeling of occlusion relationships. This rendering module can generate high-fidelity images in real time from different perspectives, supports user interaction (such as rotation, scaling, switching liver segment annotation views, etc.), and can also be fused with the original video image for display, enhancing the surgeon's spatial perception and anatomical positioning of the target area. In addition, the system supports rapid updating of local Gaussian kernel parameters during surgery. When a new video frame arrives or the liver surface undergoes local deformation due to traction, resection, etc., the system can only update the parameters of the affected area without retraining the entire point cloud model, thereby greatly improving modeling efficiency and system response speed.

[0103] In a possible embodiment, if Figure 3 As shown, in S105, the first three-dimensional model and the second three-dimensional model are point cloud registered, including steps D1 to D6: Step D1: Calculate fast point feature histogram (FPFH) features for the first point cloud of the first three-dimensional model and the second point cloud of the second three-dimensional model respectively.

[0104] Optionally, the FPFH feature can be a local feature descriptor that can capture the geometric feature information of points in the point cloud. In one example, for the first point cloud of the first three-dimensional model and the second point cloud of the second three-dimensional model, the normal vector of each point cloud can be calculated separately, such as using the principal component analysis (PCA) method, by calculating the covariance matrix of the neighborhood points around the point, and its eigenvector corresponds to the direction of the normal vector. Based on the normal vector information, the simple point feature histogram (SPFH) of each point can be calculated. The SPFH can describe the relative geometric relationship between points in the neighborhood around the point, such as the angle and distance between the point and the neighborhood points. Then, for each point, the FPFH feature of the point can be determined based on the SPFH of the point and its neighborhood points and a specific weight calculation.

[0105] Step D2: Determine initial corresponding point pairs of the first point cloud and the second point cloud based on the FPFH feature.

[0106] Optionally, for each point in the first point cloud, the point with the most similar FPFH can be searched in the second point cloud as a candidate corresponding point. For example, a feature similarity measurement method such as Euclidean distance, cosine similarity, etc. is used to calculate the similarity of the FPFH features of each point in the two point clouds, and the point pair with the highest similarity is used as the initial corresponding point pair.

[0107] Step D3: Map the first point cloud to the second point cloud based on the NDT algorithm using the initial corresponding point pairs to obtain a first transformation matrix.

[0108] Optionally, the Normal Distributions Transform (NDT) algorithm can be used to divide the point cloud space into several small grids (voxels). Within each grid, a probability distribution model (possibly a Gaussian distribution) is fitted based on the distribution of the point cloud within the grid. This allows the point cloud of the entire 3D model to be represented by a two-dimensional Gaussian kernel parameter set centered around position, covariance matrix, transparency, and color. Based on this, the initial corresponding point pairs are used to measure the degree of match between the transformed first point cloud and the second point cloud using an objective function (e.g., by calculating the sum of the probabilities that the transformed points fall within the corresponding grids of the second point cloud). The transformation matrix that maximizes the objective function is determined, which is also known as the first transformation matrix.

[0109] In the disclosed embodiments, the above-described coarse registration can quickly find a rough transformation relationship, aligning the two point clouds to similar positions and poses, thereby reducing the computational complexity of subsequent fine registration and the likelihood of falling into a local optimum. Optionally, the above-described registration algorithm can employ a Gaussian Mixture Model (GMM) or Fully Convolutional Geometric Features (FCGF).

[0110] Step D4: Determine the nearest neighbor point pairs of the first point cloud and the second point cloud based on the first transformation matrix.

[0111] Optionally, the first point cloud is transformed using the first transformation matrix to obtain a transformed point cloud. For each point in the transformed first point cloud, the closest point is found in the second point cloud to form a nearest neighbor pair. In one example, a KD Tree can be used to accelerate the nearest neighbor search process, wherein the KD Tree is a data structure that recursively divides the point cloud space into multiple subspaces, and can quickly exclude a large number of irrelevant points when searching for nearest neighbor points, thereby improving search efficiency. Among them, the nearest neighbor pair is the basis of the iteration of the ICP (Iterative Closest Point) algorithm, which finds the correspondence between the two point clouds by finding the nearest neighbor points.

[0112] Step D5: Determine the weight coefficient of each point pair based on the distance or geometric consistency information of the nearest neighbor point pairs.

[0113] Optionally, when calculating based on distance, a distance threshold can be set, such that the smaller the distance between the point pairs, the larger the weight coefficient. When calculating based on the consistency of the local geometric structure around the point pairs, the more similar the local geometric features (such as curvature and normal angle) of the two points in their respective point clouds are, the larger the weight coefficient.

[0114] During the registration process, the registration reliability of different point pairs is different. The weight coefficient can be used to make more reliable point pairs contribute more to the transformation matrix, thereby improving the registration accuracy.

[0115] Step D6: Iteratively update the first transformation matrix based on the weight coefficients until a preset condition is met to obtain a second transformation matrix.

[0116] Optionally, in each iteration, the nearest neighbor point pair can be found based on the first transformation matrix, the new transformation matrix update amount can be calculated based on the weight coefficient, the transformation matrix can be updated and the current registration error, such as RMSE (Root Mean Square Error), can be calculated. The above process is repeated until the RMSE is less than a set threshold, such as 0.5 mm, to obtain the second transformation matrix.

[0117] In the embodiment of the present disclosure, Figure 3 As shown in the figure, an adaptive point cloud registration method is used to effectively achieve accurate registration of the preoperative model and the intraoperative model through the combination of coarse registration and fine registration.

[0118] In a feasible embodiment, liver volume assessment is performed based on the result of point cloud registration in S106, including steps E1 to E2: Step E1: Generate a closed three-dimensional mesh based on the result of point cloud registration.

[0119] Optionally, the point cloud data after point cloud registration includes three-dimensional coordinate information of the liver surface. These points are distributed in space but do not form a continuous surface. When the point cloud data obtained from the point cloud registration is used as input, Poisson surface reconstruction (implemented by the CGAL library) can be used to calculate an indicator function based on the point cloud data. This function takes a value of 1 inside the object and a value of 0 outside the object. By calculating information such as the normal vector of each point in the point cloud, a vector field is constructed, and the Poisson equation is solved to obtain the indicator function. Then, using related functions in the CGAL library, isosurfaces are extracted based on the calculated indicator function, thereby generating a closed three-dimensional mesh. The closed three-dimensional mesh can accurately represent the surface shape of the liver and provide a geometric model for accurately calculating the liver volume.

[0120] Step E2: performing volume calculation based on the three-dimensional grid to determine information related to the liver volume, the information including the ratio of the residual liver to the whole liver.

[0121] Optionally, the generated 3D mesh can be divided into multiple tetrahedrons to simplify the volume calculation process. For each tetrahedron, the volume of the individual tetrahedron is calculated based on the 3D coordinates of its four vertices. By summing the volumes of all tetrahedrons, the volume of the object represented by the entire 3D mesh can be obtained, such as the volume of the liver.

[0122] Optionally, during surgery, the remnant liver and whole liver volumes can be calculated separately based on different procedures performed at different stages of the procedure. For example, at the beginning of surgery, point cloud acquisition and reconstruction of the entire liver can be performed to obtain the whole liver volume. During the procedure, as portions of the liver are removed, point cloud acquisition and reconstruction of the remaining liver are performed to calculate the remnant liver volume. Dividing the remnant liver volume by the whole liver volume can yield the remnant / whole liver ratio.

[0123] In a feasible embodiment, the method provided by the embodiment of the present disclosure further includes at least one of the following steps F1 to F3: Step F1: Display the point cloud registration result and the volume assessment result through a display device, wherein the point cloud registration result is displayed through a three-dimensional model, and the volume assessment result includes at least one of first information related to the resection volume, second information related to the residual volume, or third information related to the point cloud registration accuracy.

[0124] Step F2: In response to the operation on the three-dimensional model, adjust the angle, size and / or viewing angle of the three-dimensional model; and / or, in response to the operation on the liver pre-cutting line, adjust the pre-cutting line displayed on the three-dimensional model.

[0125] Step F3: triggering an alarm when the volume assessment result indicates that the residual liver volume is less than a preset percentage.

[0126] The system optionally integrates a user interface, allowing surgeons to monitor the 3D model, volumetric assessment results, and surgical progress in real time. This interface can include a real-time 3D view (such as a 3D model generated based on point cloud registration results) with rotation, zooming, and perspective switching, allowing surgeons to easily observe structural changes in the liver. The interface can also include a data display panel showing key information such as resection volume, residual volume, and registration accuracy. Furthermore, the interface can be configured with simple and intuitive operation buttons and gesture-based adjustments to facilitate surgeon review and analysis. To meet clinical needs, the system also provides a real-time volume calculation interface that estimates liver tissue volume changes based on the Gaussian kernel integral of the currently visible area, assisting surgeons in determining whether minimum remnant liver volume requirements are met. Furthermore, when the video input shows significant scene changes, strong camera motion, or sudden changes in tissue anatomical morphology, the system automatically triggers a "local reconstruction" mechanism, updating only the Gaussian kernel parameter set for the specified perspective, eliminating the need for global reconstruction. This effectively reduces computational overhead and ensures modeling stability during continuous surgical scenarios.

[0127] Optionally, to improve surgical safety, a threshold warning is set. For example, an alarm is triggered when the residual liver volume is <40%.

[0128] Optionally, for at least one of steps F1 to F3 above, an AR visualization method is also provided, using a head-mounted display device (such as a Hololens 2 headset) to overlay the registered liver model and real-time intraoperative images. This allows the surgeon to simultaneously observe the patient's actual location and the virtual liver model during surgery. The Hololens 2 headset may also have a built-in gesture recognition sensor that can capture the user's hand movements in real time. During surgery, the surgeon can use gestures to adjust the liver model's display angle or the pre-cutting line, such as by sliding a finger or clenching a fist to move, rotate, or scale the virtual resection path. Optionally, the Hololens 2 headset may also be configured with a voice interaction function, allowing the surgeon to perform specific operations during surgery, such as switching the liver model's display angle, using specific voice commands. Triggered alarms can also be used to alert the surgeon via the voice interaction function.

[0129] In an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the method provided in any optional embodiment of the present disclosure. Compared with the prior art, the present disclosure embodiment can achieve: a real-time liver volume assessment method based on liver surface reconstruction during digital twin surgery. Specifically, during liver resection, liver image data can be obtained, and based on the image data, feature information related to the liver is extracted and feature matching is performed based on the extracted feature information. Then, based on the result of the feature matching, a second three-dimensional model of the liver can be reconstructed to achieve dynamic reconstruction of the three-dimensional model of the liver during surgery, more accurately reflecting the changes during surgery. On this basis, in order to reduce the assessment error, a first three-dimensional model of the liver can be obtained. The first three-dimensional model is generated based on image data related to the liver collected before surgery, and the first three-dimensional model is point cloud registered with the second three-dimensional model. Finally, the liver volume is assessed based on the result of the point cloud registration. The present disclosure embodiment can reconstruct the liver model in real time during surgery, improve the accuracy of liver volume assessment, and help improve the safety and accuracy of liver surgery.

[0130] In an alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present disclosure.

[0131] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0132] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0133] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0134] The memory 4003 is used to store the computer program for executing the embodiments of the present disclosure, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiments.

[0135] Among them, electronic devices include but are not limited to: head-mounted display devices and terminal devices.

[0136] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0137] The embodiments of the present disclosure further provide a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiments when executed by a processor.

[0138] It should be understood that, although the flowcharts of the embodiments of the present disclosure indicate the various operation steps by arrows, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiments of the present disclosure, the implementation steps in each flowchart can be performed in other orders as required. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times. In scenarios where the execution times are different, the order of execution of these sub-steps or stages can be flexibly configured as required, and the embodiments of the present disclosure do not limit this.

[0139] The above description is only an optional implementation method for some implementation scenarios of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present disclosure, other similar implementation methods based on the technical ideas of the present disclosure also fall within the protection scope of the embodiments of the present disclosure.

Claims

1. A real-time liver volume assessment method based on liver surface reconstruction during digital twin surgery, characterized in that: include: Acquire a first three-dimensional model of the liver, where the first three-dimensional model is generated based on image data related to the liver acquired before surgery; During hepatectomy, imaging data of the liver is obtained; Extracting liver-related feature information based on the image data, and performing feature matching based on the extracted feature information; Based on the result of feature matching, a second three-dimensional model of the liver is reconstructed; Performing point cloud registration on the first three-dimensional model and the second three-dimensional model; Liver volume estimation was performed based on the results of point cloud registration.

2. The method according to claim 1, characterized in that During the hepatectomy, obtaining imaging data of the liver includes at least one of the following: Using a three-dimensional image acquisition device, video streams of the liver during surgery are collected from multiple angles; extracting key frame data from the video stream; The spatial coverage overlap area between adjacent images in the video stream and / or the key frame data is within a preset percentage range.

3. The method according to claim 1, characterized in that Extracting liver-related feature information based on the image data, and performing feature matching based on the extracted feature information, includes: Performing feature detection on the image data to extract a first key point related to the liver surface texture and a second key point related to the liver geometric edge; Calculating similarities of feature descriptors between key frames of the image data, and establishing cross-visual feature point correspondences based on the similarities, wherein the feature descriptors indicate image information of a local area around the first key point and / or the second key point; Based on the feature point correspondence and the RGB features and depth map information of the image data, stereo constraint matching is performed on the geometric discontinuity features of the off-section area.

4. The method according to claim 3, characterized in that The performing feature detection on the image data to extract first key points related to the liver surface texture and second key points related to the liver geometric edge includes: performing deblurring processing on the image data; Using a GPU, constructing a Gaussian difference pyramid in parallel to extract the first key point and the second key point; The feature point density of the first key point and the second key point in each frame is greater than or equal to a preset value.

5. The method according to claim 1, wherein The reconstructing a second three-dimensional model of the liver based on the result of the feature matching includes: Based on the results of feature matching, a sparse 3D model of the liver during surgery is constructed through incremental structure from motion (SfM). Based on the sparse three-dimensional model, a dense three-dimensional model of the liver during surgery is constructed through multi-view stereo matching (MVS) and determined as the second three-dimensional model.

6. The method according to claim 5, characterized in that The method of constructing a sparse 3D liver model during surgery based on the result of feature matching through SfM includes: Determining an initial pose based on a result of the feature matching; From the result of the feature matching, two frames are obtained as initial key frames based on the degree of overlap and the number of matching points; Based on the feature point matching relationship of the initial key frame and the initial pose, generating an initial three-dimensional point cloud through triangulation processing; For each subsequent frame, the pose of the frame is determined based on the matching relationship between the initial three-dimensional point cloud and the feature points of the frame, and the three-dimensional point cloud is expanded by triangulation to obtain a sparse three-dimensional model.

7. The method according to claim 6, characterized in that The determining of the initial pose based on the result of the feature matching includes: Determining internal parameters of a three-dimensional image acquisition device based on pre-calibrated parameters of the three-dimensional image acquisition device and / or real-time calibration data during surgery, wherein the three-dimensional image acquisition device is used to acquire the image data; Calculating a fundamental matrix based on the intrinsic parameters of the three-dimensional image acquisition device and the result of the feature matching; The basic matrix is ​​decomposed to obtain an initial pose.

8. The method according to claim 6, characterized in that The construction of the sparse three-dimensional model further includes: Performing global processing on the poses and three-dimensional point clouds of the sparse three-dimensional model through bundle adjustment to update all poses and three-dimensional point clouds; And / or, if the liver is deformed during the operation, the position and 3D point cloud of the sparse 3D model are updated by using a sliding window and / or abnormal frame elimination method.

9. The method according to claim 1, characterized in that The performing point cloud registration on the first three-dimensional model and the second three-dimensional model includes: Calculating fast point feature histogram (FPFH) features for a first point cloud of the first three-dimensional model and a second point cloud of the second three-dimensional model respectively; Determining initial corresponding point pairs of the first point cloud and the second point cloud based on the FPFH features; Mapping the first point cloud to the second point cloud based on the NDT algorithm using the initial corresponding point pairs to obtain a first transformation matrix; Determining nearest neighbor pairs of the first point cloud and the second point cloud based on the first transformation matrix; Determining a weight coefficient for each point pair based on the distance or geometric consistency information of the nearest neighbor point pairs; Based on the weight coefficients, the first transformation matrix is ​​iteratively updated until a preset condition is met to obtain a second transformation matrix.

10. The method according to claim 1, characterized in that The liver volume assessment based on the result of point cloud registration includes: Generate a closed 3D mesh based on the results of point cloud registration; Volume calculation is performed based on the three-dimensional grid to determine information related to the liver volume, where the information includes a ratio of the residual liver to the whole liver.

11. The method according to claim 1, wherein The method further comprises at least one of the following: Displaying a point cloud registration result and a volume evaluation result on a display device, wherein the point cloud registration result is displayed through a three-dimensional model, and the volume evaluation result includes at least one of first information related to the resection volume, second information related to the residual volume, or third information related to the point cloud registration accuracy; In response to an operation on the three-dimensional model, adjusting the angle, size and / or viewing angle of the three-dimensional model; and / or, in response to an operation on the pre-cut line of the liver, adjusting the pre-cut line displayed on the three-dimensional model; When the volume assessment result indicates that the residual liver volume is less than a preset percentage, an alarm is triggered.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the method according to any one of claims 1 to 11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images