Intraoperative real-time navigation system based on digital twinning

Abstract

The invention provides an intraoperative real-time navigation system based on digital twinning, and relates to the technical field of digital twinning, and the system comprises a data collection module which is used for obtaining the three-dimensional form data of a liver and a biliary tract through an intraoperative ultrasonic imaging device, and collecting the elastic modulus distribution parameters of liver parenchyma and a bile duct wall; the construction registration module is used for constructing a real-time digital twin in the liver and gall area based on the three-dimensional form data and the elastic modulus distribution parameters, and carrying out multi-mode non-rigid registration on the digital twin and the liver surface topography and the vascular structure which are subjected to optical coherence tomography in the operation; therefore, a dynamic space mapping relation between the organ structure and the image data is established. According to the method, real-time navigation in the operation is realized through digital twinning, the obstacle avoidance path is updated, and the safety and efficiency in the operation are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital twinning, in particular to an intraoperative real-time navigation system based on digital twinning. BACKGROUND

[0002] In hepatobiliary surgery, traditional intraoperative navigation technology has some limitations. Most traditional navigation systems may rely on preoperative CT or MRI static image data. Although two-dimensional monitoring can be performed with the help of ultrasound during the operation, it is difficult to real-time perceive the three-dimensional morphological changes of the liver parenchyma due to respiratory motion and the biomechanical properties of soft tissues such as bile duct walls. When the surgical instrument contacts or pulls the tissue around the intrahepatic bile duct, due to the lack of real-time acquisition and analysis capabilities of parameters such as tissue elastic modulus in traditional systems, it is difficult to accurately determine the stress deformation of soft tissues, thereby affecting the accurate control of the safe range of instrument operation.

[0003] In addition, the existing navigation technology has some deficiencies in the spatio-temporal synchronization fusion of multi-source data. There is some synchronization deviation between the intraoperative fluorescence imaging video stream and the instrument pose information collected by the electromagnetic positioning device, which leads to a visual delay between the instrument position displayed by the navigation system and the real anatomical structure, making it difficult for the operator to accurately determine the relative position of the instrument tip and the intrahepatic blood vessels and bile duct structure in real time, thereby increasing the risk of surgical operation. SUMMARY

[0004] The technical problem to be solved by the present application is to provide an intraoperative real-time navigation system based on digital twinning, which constructs a dynamic digital twin body through multi-modal data fusion and realizes real-time registration and feedback correction, thereby improving the navigation accuracy and surgical safety and providing more reliable operation guidance for the operator.

[0005] To solve the above technical problems, the technical solutions of the present application are as follows: In a first aspect, an intraoperative real-time navigation system based on digital twinning comprises: A data acquisition module is configured to acquire three-dimensional morphological data of the liver and biliary tract through an intraoperative ultrasound imaging device, and to acquire elastic modulus distribution parameters of the liver parenchyma and bile duct wall; A construction and registration module is configured to construct a real-time digital twin body of the hepatobiliary region based on the three-dimensional morphological data and the elastic modulus distribution parameters, and to perform multi-modal non-rigid registration between the digital twin body and the liver surface topography and vessel structure obtained by optical coherence tomography during the operation, so as to establish a dynamic spatial mapping relationship between the organ structure and the image data; A calculation and fusion module is configured to track the spatial pose of the surgical instrument in real time through an electromagnetic sensor device according to the dynamic spatial mapping relationship, and to synchronously fuse the intraoperative fluorescence imaging video stream, so as to calculate the displacement field of the liver lobe due to respiratory motion and the soft tissue deformation tensor; An update planning module is configured to dynamically update the structure and morphology of the digital twin according to the displacement field and the deformation tensor, and to calculate in real time an obstacle avoidance path and a safe needle insertion trajectory of the surgical instrument in the liver; A feedback correction module is configured to superimpose the obstacle avoidance path and the safe needle insertion trajectory to the visual field of the surgeon in real time, and to calculate a consistency measure of the registration error based on the reference marker points embedded in the liver, and if the measure is greater than a permissible threshold, to trigger a real-time correction mechanism of the mapping relationship and the path.

[0006] Further, a real-time digital twin of the liver and gallbladder region is constructed based on the three-dimensional morphological data and the elastic modulus distribution parameters, and the digital twin is multi-modal non-rigidly registered with the liver surface topography and the vascular structure obtained by the optical coherence tomography during the operation, to establish a dynamic spatial mapping relationship between the organ structure and the image data, including: Based on the three-dimensional morphological data, a geometric structure and a spatial topology for describing the liver parenchyma volume, the bile duct tree topology, and the intraliver vascular network are constructed; Based on the geometric structure and the spatial topology, and in combination with the elastic modulus distribution parameters, a three-dimensional elastic mechanical representation of the liver tissue endowed with biomechanical properties is generated; The geometric structure and the three-dimensional elastic mechanical representation are combined to form a real-time digital twin of the liver and gallbladder region with both anatomical structure and mechanical characteristics; The digital twin is subjected to feature extraction to obtain spatial distribution features of the liver surface morphology and the intraliver pipeline, and the distribution features are registered and matched with corresponding features of the real organ surface obtained by the optical coherence tomography during the operation, to obtain a registration and matching result; According to the registration and matching result, dynamic transformation parameters of the digital twin space to the real organ space are calculated, to establish a dynamic spatial mapping relationship between the image data and the physical organ structure.

[0007] Further, the digital twin is subjected to feature extraction to obtain spatial distribution features of the liver surface morphology and the intraliver pipeline, and the distribution features are registered and matched with corresponding features of the real organ surface obtained by the optical coherence tomography during the operation, to obtain a registration and matching result, including: Three-dimensional spatial coordinates and topological connection relationships of the intraliver bile duct tree branch points, the liver surface contour feature points, and the intraliver vascular intersection points are extracted from the digital twin as a first registration feature set; The liver and gallbladder region surface topography and the subcutaneous vascular structure image collected by the optical coherence tomography during the operation are subjected to enhancement and segmentation processing, and spatial distribution information of the corresponding bile duct wall contour, liver surface topography, and vascular branch points is extracted as a second registration feature set; The first registration feature set and the second registration feature set are subjected to non-rigid feature matching to establish a corresponding relationship between the feature points and to evaluate spatial consistency, to obtain an evaluation result; Based on the evaluation results, the non-linear transformation parameters are calculated to realize the spatial alignment of the two feature sets, so as to obtain the registration matching results.

[0008] Further, according to the dynamic spatial mapping relationship, the spatial pose of the surgical instrument is tracked in real time by the electromagnetic sensing device, and the intraoperative fluorescent imaging video stream is synchronously fused, and the displacement field and soft tissue deformation tensor generated by the respiratory motion of the liver lobe are calculated, including: By using the dynamic spatial mapping relationship, the original spatial signal of the surgical instrument collected by the electromagnetic sensing device in real time is converted to the spatial coordinate system corresponding to the real-time digital twin to obtain the position data of the surgical instrument in the digital twin space; Based on the position data, the real-time position and attitude of the surgical instrument in the digital twin space, i.e. six-degree-of-freedom pose information, are calculated; The six-degree-of-freedom pose information is time-synchronized and spatially fused with the real-time video stream of the liver and gallbladder region obtained by the intraoperative fluorescent imaging device to generate enhanced fluorescent images with spatial position markers; Based on the enhanced fluorescent images and the six-degree-of-freedom pose information, the periodic displacement field generated by the respiratory motion of the liver parenchyma is analyzed, and the deformation gradient tensor of the liver soft tissue region caused by the contact of the surgical instrument is calculated.

[0009] Further, based on the enhanced fluorescent images and the six-degree-of-freedom pose information, the periodic displacement field generated by the respiratory motion of the liver parenchyma is analyzed, and the deformation gradient tensor of the liver soft tissue region caused by the contact of the surgical instrument is calculated, including: The six-degree-of-freedom pose data is time-synchronized and spatially aligned with the intraoperative fluorescent imaging video stream to generate a fusion data stream; By using the dynamic spatial mapping relationship, the six-degree-of-freedom pose of the surgical instrument in the fusion data stream is mapped to the coordinate system of the real-time digital twin to obtain the motion trajectory of the surgical instrument in the digital twin space; Based on the motion trajectory of the surgical instrument, the real-time contact position between the tip of the instrument and the liver surface and the intraliver vascular structure represented by the digital twin is calculated to establish a dynamic spatial correspondence between the tip of the instrument and the surface points of the liver anatomical structure; Based on the dynamic spatial correspondence, the contact point position information associated with the main blood vessels and biliary structures in the liver is extracted, and the displacement of the corresponding pipeline structure in the digital twin is analyzed to calculate the rigid displacement vector; Based on the contact point position associated with the liver parenchyma soft tissue and the displacement information of the adjacent liver region in the dynamic spatial correspondence, the deformation mode of the liver soft tissue surface in the digital twin is analyzed, and the dynamic deformation gradient tensor is calculated.

[0010] Further, according to the displacement field and the deformation tensor, the structure and morphology of the digital twin are dynamically updated, and the obstacle avoidance path and safe needle insertion trajectory of the surgical instrument operating in the liver are calculated in real time, including: The rigid displacement vector and the dynamic deformation gradient tensor are combined to construct a boundary condition set for describing the dynamic deformation of the liver and gallbladder region; The boundary condition set is input into the liver and gallbladder tissue deformation simulation process to obtain the spatial coordinate data and tissue physical property data of each anatomical structure in the liver represented by the updated real-time digital twin; Based on the updated spatial coordinate data of the liver anatomical structure, a real-time spatial relationship mapping between the tip of the surgical instrument and important structures such as the intrahepatic bile duct, blood vessels, and liver surface is established; According to the real-time spatial relationship mapping, the real-time minimum spatial distance from the tip of the surgical instrument to the nearest key pipeline structure in the liver is calculated; Based on the real-time minimum spatial distance and the real-time feedback of the surgical instrument movement speed and direction from the electromagnetic sensing device, the real-time obstacle avoidance path and safe needle insertion trajectory of the surgical instrument in the liver parenchyma are dynamically generated.

[0011] Further, based on the real-time minimum spatial distance and the real-time feedback of the surgical instrument movement speed and direction from the electromagnetic sensing device, the real-time obstacle avoidance path and safe needle insertion trajectory of the surgical instrument in the liver parenchyma are dynamically generated, including: Based on the spatial distribution of the intrahepatic pipeline structure and blood vessels represented by the updated digital twin, the dynamic spatial obstacle distribution of the important structures in the liver around the surgical instrument is calculated; According to the dynamic spatial obstacle distribution, combined with the real-time acquisition of the surgical instrument position and movement direction data from the electromagnetic navigation device, the potential interference state between the instrument movement trajectory and the intrahepatic pipeline structure is dynamically evaluated and predicted to obtain a state evaluation result; Based on the state evaluation result, a safe and smooth real-time movement trajectory of the surgical instrument that avoids important vascular and biliary structures in the liver is calculated and generated as an obstacle avoidance path and needle insertion guide.

[0012] Further, the obstacle avoidance path and safe needle insertion trajectory are superimposed in real time on the surgeon's image field of view, and a consistency measure of the registration error is calculated based on the reference marker points embedded in the liver. If the measure is greater than the allowable threshold, a real-time correction mechanism for the mapping relationship and path is triggered, including: The real-time obstacle avoidance path and safe needle insertion trajectory information is superimposed in real time on the intraoperative image device field of view seen by the surgeon; Based on the reference marker points embedded in the anatomical position of the liver, the three-dimensional spatial coordinates of the corresponding marker points in the real-time digital twin and the real surgical field image are extracted respectively; The spatial deviation between the two sets of marker point coordinates is calculated to generate a consistency measure representing the spatial registration accuracy of the digital twin and the real liver and gallbladder structure. The consistency measure is compared with a preset allowable threshold in real time, when the consistency measure is greater than the allowable threshold, a real-time correction mechanism of the spatial mapping relationship and the obstacle avoidance path is triggered, including re-computing and adjusting the dynamic spatial mapping relationship between the real-time digital twin and the real hepatobiliary structure, and updating the real-time obstacle avoidance path and the safe needle insertion trajectory of the surgical instrument based on the corrected mapping relationship.

[0013] In a second aspect, a computing device includes: one or more processors; a memory device storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the system.

[0014] In a third aspect, a computer-readable storage medium stores a program, when the program is executed by a processor, the system is implemented.

[0015] The above scheme of the present application at least includes the following beneficial effects: By constructing the real-time digital twin of the hepatobiliary region and performing multi-modal non-rigid registration, a high-precision dynamic spatial mapping between virtual and real anatomical structures is established, combined with real-time tracking of the pose of the surgical instrument by electromagnetic navigation, the registration deviation caused by respiratory motion or tissue deformation is reduced, ensuring that the surgical instrument can always accurately point to the target lesion or operation point in the liver, real-time calculation of the displacement field of the liver lobe due to respiratory motion and the deformation tensor of the soft tissue caused by the contact of the surgical instrument, and dynamic updating of the digital twin and the obstacle avoidance path according to the displacement field and the deformation tensor, the generated real-time safe needle insertion trajectory can effectively guide the instrument to avoid important blood vessels, bile ducts and other key structures in the liver, reducing the risk of complications such as intraoperative bleeding and bile leakage, and ensuring the safety of the operation; by synchronously fusing real-time video streams such as intraoperative fluorescence imaging, the dynamic changes of the liver can be continuously captured, and the digital twin is updated synchronously, so that the virtual always keeps consistent with the shape and position of the real organ, ensuring the continuous reliability of the navigation guidance throughout the operation process, intuitive visualization and seamless superposition of the real-time obstacle avoidance path, reducing the time for the operator to repeatedly confirm the image and hesitate operation, automatic error monitoring and real-time correction mechanism can timely adjust the path and the mapping relationship, avoiding the interruption of the operation caused by positioning deviation, and improving the overall efficiency of the operation process. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a schematic diagram of an intraoperative real-time navigation system based on digital twin provided by an embodiment of the present application.

[0017] Figure 2The embodiment of the present application provides a schematic diagram of a process for dynamically updating the structure and morphology of a digital twin according to a displacement field and a deformation tensor, and calculating an obstacle avoidance path and a safe needle insertion trajectory of a surgical instrument in operation in a liver in real time. DETAILED DESCRIPTION

[0018] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0019] As shown in Figure 1 the embodiment of the present application proposes an intraoperative real-time navigation system based on digital twin, which comprises: a data acquisition module configured to acquire three-dimensional morphology data of a liver and biliary tract by an intraoperative ultrasonic imaging device, and collect elastic modulus distribution parameters of liver parenchyma and biliary duct wall; a construction registration module configured to construct a real-time digital twin of a liver and biliary region based on the three-dimensional morphology data and the elastic modulus distribution parameters, and perform multi-modal non-rigid registration between the digital twin and a liver surface topography and vessel structure of an optical coherence tomography in operation, so as to establish a dynamic spatial mapping relationship between organ structure and image data; a calculation and fusion module configured to track a spatial pose of a surgical instrument in real time through an electromagnetic sensing device according to the dynamic spatial mapping relationship, and synchronously fuse an intraoperative fluorescent imaging video stream, so as to calculate a displacement field and a soft tissue deformation tensor generated by a liver lobe due to respiratory motion; an update planning module configured to dynamically update the structure and morphology of the digital twin according to the displacement field and the deformation tensor, and calculate an obstacle avoidance path and a safe needle insertion trajectory of the surgical instrument in operation in the liver in real time; a feedback correction module configured to superimpose the obstacle avoidance path and the safe needle insertion trajectory to an image field of view of an operator in real time, and calculate a consistency measure of a registration error based on a reference marker point embedded in the liver, and if the measure is greater than an allowable threshold, trigger a real-time correction mechanism of the mapping relationship and the path.

[0020] In the embodiment of the present application, by acquiring the three-dimensional morphological data of the liver and biliary tract and the elastic modulus distribution parameters of the liver parenchyma and bile duct wall, the anatomical structure details and tissue mechanical properties of the liver and biliary region can be clearly presented, not only providing accurate and comprehensive original basis for surgical path planning, but also helping the operator to master the hardness difference of different tissues in advance, assisting in distinguishing normal tissue and lesion area, reducing the planning deviation caused by data loss or inaccuracy, and constructing a real-time digital twin of the liver and biliary region based on multi-dimensional data, combined with multi-modal non-rigid registration technology, the digital twin can be accurately corresponded with the liver surface topography and vascular structure, the operator can directly see the dynamic spatial mapping relationship between the internal structure of the organ and the external image through the digital twin, and clearly master the position and direction of the liver and biliary and vessels, avoiding misjudgment or injury caused by blurred anatomical structure cognition.

[0021] By real-time tracking of the spatial pose of the surgical instrument, fusion of intraoperative fluorescence imaging video stream, and simultaneous calculation of liver lobe displacement field and soft tissue deformation tensor caused by respiratory motion, the influence of respiratory motion and surgical operation on organ position and shape is reduced, which makes the instrument positioning no longer disturbed by the dynamic change of the organ, not only can the operator accurately master the relative position of the instrument and the lesion and important structure, but also can more clearly identify the lesion boundary through fluorescence imaging, improve the accuracy of operation, update the digital twin according to the real-time displacement and deformation of the organ, and simultaneously calculate the obstacle avoidance path and safe needle insertion trajectory, ensure that the surgical path always adapts to the dynamic state of the organ, even if the liver lobe is displaced due to respiration or operation, and the soft tissue deforms, the newly planned path can avoid blood vessels, bile ducts and other key structures, avoid bleeding, cholangi fistula and other complications caused by instrument misoperation, reduce the risk of surgery, and superimpose the planned path on the operator's image field of view in real time, so that the operator can intuitively compare the planned path with the actual operation scene, and calculate the registration error through the reference marker point and trigger the correction mechanism, if the error exceeds the allowable range, the spatial mapping relationship and the surgical path can be adjusted in real time to avoid operation errors caused by accumulated deviation.

[0022] In a preferred embodiment of the present application, a real-time digital twin of the liver and biliary region is constructed based on three-dimensional morphological data and elastic modulus distribution parameters, and the digital twin is multi-modal non-rigidly registered with the liver surface topography and vascular structure of the optical coherence tomography during the operation, to establish the dynamic spatial mapping relationship between the organ structure and the image data, which can include: Based on three-dimensional morphological data, the geometric structure and spatial topology of the liver parenchyma volume, bile duct tree topology and intraliver vascular network are constructed, which specifically includes: the obtained three-dimensional morphological original data of liver and gallbladder region is a large amount of discrete three-dimensional coordinate points generated by medical imaging equipment such as CT and MRI scanning, each coordinate point contains the numerical value of x, y and z three spatial dimensions. First, the noise points are removed. For each coordinate point, a spherical neighborhood with a radius of 0.5 mm is defined with the coordinate point as the center. The neighborhood radius can be adjusted according to the resolution of the original data. The higher the resolution, the smaller the neighborhood radius. The number of other coordinate points in the neighborhood is counted. If the number of points in the neighborhood of a certain coordinate point is less than a predetermined threshold, such as 5, the threshold is set according to the density characteristics of normal liver tissue. The adjacent coordinate points in the normal liver tissue region are densely distributed, and the number of points around the noise points is small. It is determined that the point is a noise point and is deleted. If the number of points reaches or exceeds the threshold, the point is retained, and the preprocessing is completed. According to the accuracy requirement of the data after preprocessing, such as hoping that the volume calculation error is not more than 0.1 cm 3 , the voxel edge length is set to 0.2 mm. The smaller the voxel edge length, the higher the calculation accuracy, but the larger the calculation amount. The balance between accuracy and efficiency needs to be balanced. The three-dimensional space of the coordinate points after preprocessing is divided along the x, y and z axis directions at an interval of 0.2 mm to form countless cubic voxels with an edge length of 0.2 mm. The spatial position of each voxel can be uniquely determined by its sequence number in the x, y and z axis directions, such as the voxel in the i-th row, the j-th column and the k-th layer.

[0023] For each voxel, it is judged whether it is a valid voxel. First, the number of liver parenchyma coordinate points contained in the voxel is counted. If the number of coordinate points in the voxel accounts for the total number of points that can be accommodated in the voxel, the total number of points that can be accommodated in the voxel is determined according to the sampling density of the original data, such as a sampling density of 100 points per cubic millimeter. The total number of points that can be accommodated in the voxel is about 0.2*0.2*0.2*100=0.8. In actual calculation, the proportion of integer 1 is taken. If the proportion is greater than or equal to 50%, that is, there is at least one coordinate point in the voxel, the voxel is a valid voxel. If the proportion is less than 50%, it is an invalid voxel. The number N of all valid voxels is counted. The volume of a single voxel is 0.2 mm*0.2 mm*0.2 mm=0.008 mm 3 , the total volume of liver parenchyma=effective voxel number N*0.008 mm 3 , finally the volume unit is converted to cm 3 (1 cm 3 =1000 mm 3 ).

[0024] When extracting the coordinate points related to the bile duct from the three-dimensional morphological data, first, the gray value corresponding to all coordinate points in the data is obtained. The gray values of different tissues in medical images are different. The gray value of the bile duct tissue is lower than that of the liver parenchyma because it contains bile. The gray value threshold range of the bile duct is set, such as 10-30 HU, HU is the unit of CT value. The range is determined according to a large number of clinical bile duct image data statistics to ensure that more than 95% of the bile duct tissue points fall within the range. All coordinate points are traversed, and points with a gray value between 10-30 HU are marked as bile duct points. The bile duct point set is screened out, and the region growing algorithm is used to divide the branch region. First, a bile duct point in the bile duct point set that has not been marked is randomly selected as an initial seed point. The connected distance threshold is set to 0.3 mm. The threshold is set according to the thickness of the bile duct wall. The thickness of the bile duct wall is about 0.2-0.4 mm to ensure that adjacent bile duct points can be classified into the same branch. All unmarked points within a 0.3 mm range around the seed point are checked. If a point is a bile duct point, it is marked as the same branch as the seed point, and the point is used as a new seed point. The above process is repeated until no new bile duct point can be added to the branch. At this time, a complete bile duct branch region is formed. Then, a seed point is selected again from the remaining unmarked bile duct points, and the region growing is repeated until all bile duct points are divided into corresponding branch regions.

[0025] The x coordinates of all bile duct points in the region are added and averaged to obtain the x coordinate of the center point. Similarly, the y and z coordinates are calculated to form the branch center point. The adjacent center points in the same branch are connected, which means that the points become seed points in the growing process. The direction of the branch is reflected, and the axis of the bile duct branch is obtained. The length of the axis is the sum of the distances between all adjacent center points of the branch. The connection relationship between branches is determined. If the end point of the axis of one branch and the middle point of the axis of another branch, except for the end point, the distance between other center points is ≤0.2 mm, which is the normal gap at the connection site of the bile duct branch, then it is determined that the two branches are connected. The former is the child branch of the latter. The connection relationship of all branches is sorted in this way to determine the main bile duct, the branch with the longest axis and the most connected branches, the first-level branch, the branch directly connected to the main bile duct, the second-level branch, the branch directly connected to the first-level branch, and other levels. Finally, a complete bile duct tree topology is constructed.

[0026] The intrahepatic blood vessels include arteries, veins, such as the portal vein and the hepatic vein. The gray value of different blood vessels is different. The gray value of the artery is higher than that of the vein because the artery contains arterial blood. The gray value threshold of the artery and the vein is set respectively. The artery threshold is 50-80HU, and the vein threshold is 20-40HU. Both are determined according to the clinical blood vessel image data statistics. All coordinate points of the three-dimensional morphological data are traversed. The points with the gray value between 50-80HU are marked as arterial points, and the points with the gray value between 20-40HU are marked as venous points. The arterial point set and the venous point set are obtained respectively. The region growing algorithm is used to divide the branch region for the arterial point set and the venous point set respectively. The connection distance threshold of the arterial branch is 0.4mm. The arterial wall is slightly thicker than the bile duct, about 0.3-0.5mm. The connection distance threshold of the venous branch is 0.5mm. The venous wall is thin, but the lumen is thick. The threshold is slightly larger to ensure complete division. The arterial branch region and the venous branch region are obtained respectively according to the same region growing steps.

[0027] The center point and the axis of each arterial branch and venous branch are calculated. The method is the same as that of the bile duct branch. The connection relationship between the arterial branches is judged, such as the connection between the hepatic artery trunk and the intrinsic arterial branch of the liver. The connection relationship between the venous branches is judged, such as the connection between the portal vein trunk and the left branch of the portal vein. The connection judgment threshold is 0.3mm. The spatial position relationship between the arterial branch and the venous branch is judged simultaneously. The shortest distance between the arterial branch axis and the venous branch axis is calculated. If the shortest distance is ≤1mm, the normal interval between the intrahepatic artery and vein, the two are recorded as adjacent relationship. The arterial network topology, the venous network topology and the adjacent relationship between the artery and the vein are integrated to form the intrahepatic blood vessel network topology.

[0028] Based on the geometric structure and spatial topology, and combined with the elastic modulus distribution parameters, a three-dimensional elastic mechanical characterization of the liver tissue endowed with biomechanical properties is generated, which specifically includes: through preoperative ultrasonic elastography or magnetic resonance elastography, the elastic modulus values of different regions of the liver tissue are obtained, the obtained parameters are presented in the form of regional coordinate range and elastic modulus value, such as x: 50-60 mm, y: 30-40 mm, z: 20-30 mm, elastic modulus: 5 kPa, the parameters are preprocessed, and it is checked whether there is region overlap, if the region coordinate ranges of two different parameters have intersection, if there is overlap, the elastic modulus value with the highest frequency in the overlapping region is taken as the final value of the region; if there is no overlap, the original parameters are directly retained; the previously constructed liver tissue geometric structure, including the spatial range of liver parenchyma, bile duct tree and blood vessel network, is divided into mechanical calculation units according to the region coordinate range of the elastic modulus parameter, if the region range of a certain elastic modulus parameter is large, such as the x direction span exceeds 5 mm, the region is subdivided along the x, y and z axis directions at an interval of 2 mm, the size of the subdivided unit is matched with the subsequent mechanical calculation accuracy, forming smaller mechanical calculation units, for each mechanical calculation unit, the elastic modulus value of the region where the unit is located is assigned to the unit, ensuring that each unit corresponds to only one elastic modulus value, and the integration of parameters and geometric structure is completed.

[0029] The liver tissue is treated as an isotropic material, i.e. the mechanical properties in different directions are the same, and the Poisson's ratio is set to 0.45, which is a typical Poisson's ratio of the liver tissue in a physiological state, reflecting the ratio of transverse deformation to longitudinal deformation of the tissue when it is squeezed, for each mechanical calculation unit, the stiffness coefficient is calculated according to the elastic modulus value, taking the tensile stiffness as an example, the calculation logic is that the stiffness coefficient is proportional to the elastic modulus and related to the geometric size of the unit, the larger the unit volume, the greater the deformation under the same external force, and the smaller the stiffness coefficient, in specific calculation, the shape of the unit is first determined, such as a tetrahedral unit, the characteristic size of the unit is measured, such as the longest edge length of the tetrahedron, then the elastic modulus and Poisson's ratio are combined to obtain the stiffness coefficient of the unit in the tensile direction through mechanical property derivation, i.e. the inverse of the tensile deformation amount that the unit can produce when subjected to a tensile force of 1 N, the smaller the deformation amount, the larger the stiffness coefficient, similarly, the shear stiffness coefficient is calculated, reflecting the ability of the unit to resist shear force, the compression stiffness coefficient reflects the ability to resist pressure, etc.

[0030] The elastic deformation capability is embodied in the external force-deformation amount correspondence. For each mechanical calculation unit, a series of external forces of different sizes, such as 0.1 N, 0.2 N, 0.5 N, 1 N, etc., are set to cover the range of external forces that can be applied in surgery, and the deformation amount under the action of each external force is calculated. Taking compression deformation as an example, if the unit compression stiffness coefficient is k (unit: N / mm), when the applied pressure is F (unit: N), the deformation amount is F / k (mm). Different external forces and corresponding deformation amounts are arranged into a table to form the elastic deformation capability data of the unit. The stiffness coefficients and elastic deformation capability data of all mechanical calculation units are integrated, and combined with the spatial positions of each unit in the geometric structure, such as whether the unit is on the surface of the liver or inside the liver, to form a three-dimensional elastic mechanical characterization of the liver tissue, that is, a complete mechanical description that can clearly describe how much deformation the liver tissue at any position will produce under different external forces.

[0031] The geometric structure and the three-dimensional elastic mechanical characterization are combined to form a liver and gallbladder region real-time digital twin body with both anatomical structure and mechanical characteristics, specifically including: each spatial position in the geometric structure is associated with a coordinate point and a mechanical calculation unit. For any coordinate point in the geometric structure, determine which mechanical calculation unit it belongs to by comparing the coordinate point with the spatial range of each unit. The elastic modulus, stiffness coefficient, and elastic deformation capability data of the unit are bound to the coordinate point to achieve one-to-one correspondence between the coordinate point and the mechanical properties. All coordinate points with bound mechanical properties are arranged according to the spatial distribution of the geometric structure to form fusion data. The fusion data is verified, and typical positions in the liver tissue, such as the hepatic portal and the edge of the right lobe of the liver, are selected to check whether the coordinate points at these positions have correctly bound corresponding mechanical properties and the mechanical property values meet the actual characteristics of the liver tissue in this region. For example, the elastic modulus of the hepatic portal should be higher than that of the edge of the liver due to the presence of more fibrous tissue. If there is an attribute error, such as the elastic modulus of the edge of the liver being too high, the error is traced back to the mechanical calculation unit division or parameter assignment step, and corrected before reintegration.

[0032] The fusion data is extremely large and can include millions of coordinate points and corresponding mechanical properties, and needs to be processed by dimension reduction. The principal component analysis method is adopted to statistically analyze the variation of the mechanical properties in the fusion data. For example, the elastic modulus values of all coordinate points in a certain area of the liver tissue are relatively small, such as being between 4-6 kPa, and the average elastic modulus value of the area is used to replace the specific value of each coordinate point, while the spatial range and key coordinate points of the area, such as the boundary points, are retained. In this way, the originally dispersed coordinate point data is integrated into simplified data of area-average mechanical property-key coordinate point. The data update time interval is set to 0.3 seconds. The interval is determined according to the response requirement of the surgical operation. An interval that is too short will increase the computational burden, and an interval that is too long will not be able to reflect the organ changes in real time. Within each update interval, the latest organ state data transmitted by the intraoperative device, such as OCT, is first received, such as the position change of the liver surface. Then, the difference between the latest data and the current digital twin data is compared, the coordinate deviation of the key coordinate points, such as the characteristic points of the liver surface, between the two is calculated, and if the deviation is ≤0.1 mm, no update is needed. If the deviation is >0.1 mm, the coordinate point position of the corresponding area in the digital twin is adjusted according to the deviation, and the mechanical properties of the area are updated synchronously. For example, the position change can cause the unit stress state to change, and the stiffness coefficient needs to be recalculated. The data is updated to ensure that the digital twin and the real organ state are synchronized in real time.

[0033] The feature extraction is performed on the digital twin to obtain the spatial distribution characteristics of the liver surface morphology and the intrahepatic ducts, and the distribution characteristics are matched with the corresponding characteristics of the real organ surface obtained by the intraoperative optical coherence tomography to obtain a matching result, including: extracting the three-dimensional spatial coordinates and topological connection relationship of the intrahepatic bile duct tree branch points, liver surface contour feature points and intrahepatic blood vessel junction points from the digital twin as a first matching feature set; performing enhancement and segmentation processing on the liver and gallbladder region surface morphology and subcutaneous vascular structure image collected by the intraoperative optical coherence tomography to extract the spatial distribution information of the corresponding bile duct wall contour, liver surface morphology and vascular branch point as a second matching feature set; performing non-rigid feature matching on the first matching feature set and the second matching feature set to establish the corresponding relationship between the feature points and evaluate the spatial consistency to obtain an evaluation result; based on the evaluation result, the non-linear transformation parameters for spatial alignment of the two feature sets are calculated to obtain the matching result, specifically including: extracting all liver surface coordinate points from the geometric structure of the digital twin, judging whether the coordinate points are located on the boundary of the liver parenchyma volume, if there are liver parenchyma points and non-liver parenchyma points in the neighborhood of a point, it is a surface point, the number is usually hundreds of thousands, using the Bezier surface fitting method, the surface points are divided into several 4x4 point arrays according to the x and y coordinates, the basic fitting unit of the Bezier surface, for each point array, adjust the surface control parameters to ensure that the surface can pass through most of the points in the point array, the deviation is ≤0.05mm, a small Bezier surface piece is generated, all surface pieces are spliced to ensure that the connection part of adjacent surface pieces is smooth and the tangent direction of adjacent surface pieces at the connection boundary is consistent, and finally a complete liver surface curve is formed; for the fitted liver surface curve, the curvature parameters are calculated, a sampling point with a density of 1 per square millimeter is selected on the curve to ensure that the entire curve is covered, for each sampling point, the equation of the surface piece where the sampling point is located is determined, the main curvature and , reflects the maximum bending degree along the x direction, reflects the minimum bending degree along the y direction, and the Gaussian curvature is x , reflecting the overall bending type of the surface, such as the positive Gaussian curvature of a convex surface and the negative Gaussian curvature of a concave surface, the normal vector is calculated, the direction vector perpendicular to the tangent plane is determined according to the tangent direction of the surface piece at the sampling point along the x and y tangents, which is the normal vector of the point, the convex and concave regions are counted, the Gaussian curvature threshold is set to 0.01mm -2 , the region with Gaussian curvature greater than 0.01mm -2 is defined as the convex region, and the region with Gaussian curvature less than -0.01mm -2The concave region is determined as a recessed region, and for each convex or recessed region, the area thereof is calculated, the number of sampling points in the region is counted, and the area and volume represented by each sampling point are multiplied.

[0034] For each branch of the bile duct tree and the vascular network, measurement points are selected at intervals of 1 mm on the axis thereof, ensuring that there are at least 5 measurement points for each branch. At each measurement point, a plane is established perpendicular to the axis of the branch, which intersects the boundary of the branch to form a circular cross-section. The cross-section of the pipeline is selected, and 8 directions are uniformly distributed on the cross-section, such as 0°, 45°, 90°,..., 315°. The distance between two points on the boundary of the cross-section in each direction is measured, i.e., the diameter in that direction. The average of the diameters in the 8 directions is taken as the diameter of the branch at the measurement point. The average diameter of the branch is obtained by averaging the diameters of all measurement points of the same branch. For each pipeline branch, the two end points of the axis, i.e., the start point and the end point of the branch, are taken, and the axis vector is calculated. The x component of the vector is the difference between the x coordinates of the end point and the start point, the y component is the difference between the y coordinates of the end point and the start point, and the z component is the difference between the z coordinates of the end point and the start point. The axis vector is angle calculated with the x axis, y axis and z axis of the three-dimensional coordinate system, respectively, to determine the angle between the vector and each coordinate axis, such as the angle corresponding to the projection length of the vector in the x direction and the total length of the vector, to obtain the orientation angle of the branch, such as the angle of 30° with the x axis, the angle of 45° with the y axis, and the angle of 60° with the z axis. The spatial relationship between different branches is calculated. For two adjacent branches, the shortest distance between their axes is calculated, and the two closest points on the two axes are found. The distance between the two points is the shortest distance and the angle between the two axis vectors. These parameters are recorded as spatial distribution characteristics.

[0035] The liver surface and vascular structure data obtained by intraoperative OCT scanning are also in the form of three-dimensional coordinate points, but due to factors such as instrument interference and tissue movement in the surgical environment, there may be noise points, such as isolated coordinate points with a distance too far from the surrounding points. Gaussian filtering is used for noise reduction. For each coordinate point, a neighborhood with a radius of 0.2 mm is defined with the coordinate point as the center, and the average values of the x, y and z coordinates of all coordinate points in the neighborhood are calculated. The average values are used to replace the coordinates of the original coordinate points. Through this smoothing process, the influence of isolated noise points on the data is reduced. The process is repeated twice, and the number of repetitions is adjusted according to the noise intensity. The stronger the noise, the more the number of repetitions, but not more than 3 times, to avoid data distortion caused by excessive smoothing. The OCT data noise reduction is completed.

[0036] The liver surface morphology features and intrahepatic duct spatial distribution features in the OCT data are extracted in the same way as the digital twin feature extraction. For example, when extracting the liver surface morphology features, a 4x4 point array is fitted to a Bezier surface patch, the principal curvature, Gaussian curvature and normal vector of each sampling point are calculated, and the area and volume of the convex and concave regions are counted. When extracting the intrahepatic duct features, measurement points are taken at 1mm intervals along the axis, the cross-sectional diameter, strike angle, and the shortest distance and angle between branches are calculated, ensuring that the types and dimensions of the OCT features and the digital twin features are completely consistent, laying the foundation for subsequent registration and matching.

[0037] First, the feature correspondence relationship is established, and the liver surface convex region features, area, volume, and position coordinates of the digital twin are one-to-one corresponding to the liver surface convex region features of the OCT data. The bile duct branch diameter and strike angle of the digital twin are one-to-one corresponding to the bile duct branch diameter and strike angle of the OCT data. Similarly, the corresponding relationship of the blood vessel branch features is established. For each set of corresponding features, the similarity is calculated. For example, if the area of a convex region in the digital twin is , and the area of the corresponding convex region in the OCT data is , the similarity = 1- | - | / max ( , ), the similarity value ranges from 0 to 1, and the closer to 1, the higher the similarity. For the strike angle, the similarity = 1- | - | / 90°, is the digital twin angle, is the OCT angle, and 90° is the maximum reasonable angle difference, ensuring that the greater the angle deviation, the lower the similarity. The average value of the similarity of all corresponding features is calculated. If the average value is ≥0.9 (this threshold is set according to the accuracy requirements of the surgery, and the higher the accuracy requirement, the higher the threshold), it is determined that the registration and matching are satisfactory. If the average value is <0.9, the digital twin feature parameters need to be adjusted. For example, if the diameter similarity of a certain bile duct branch is low (such as the digital twin diameter 2.0mm, and the OCT diameter 1.8mm), the axis coordinates of the branch are fine-tuned, the cross-sectional range is expanded or reduced, the diameter is recalculated, and the feature similarity is ≥0.9. If the position deviation of a certain convex region on the liver surface is large, the coordinates of the region in the digital twin are translated as a whole, the position of the convex region is recalculated, and the distance between the convex region and the OCT convex region is ≤0.1mm. Adjust each feature parameter repeatedly, calculate the overall similarity average value once for each adjustment, until the average value is ≥0.9. The feature correspondence relationship between the digital twin and the OCT data at this time is recorded as the registration and matching result.

[0038] According to the registration matching result, the dynamic transformation parameters of the digital twin body space to the real organ space are calculated to establish the dynamic spatial mapping relationship between the image data and the physical organ structure, specifically including: according to the registration matching result, the feature points in the digital twin body space are determined, such as the liver surface feature convex point, the bile duct tree branch end point, the blood vessel network bifurcation point, and the corresponding feature point coordinates in the OCT data representation, and the transformation relationship between the digital twin body space coordinates and the real organ space coordinates is established based on these corresponding feature point coordinates by using coordinate transformation methods such as affine transformation and thin plate spline transformation. For example, affine transformation, through the corresponding feature point coordinates, linear equations about translation, rotation, and scaling parameters are established, and the translation amount, the offset of the digital twin body space coordinates relative to the real organ space coordinates in the x, y, and z directions, the rotation angle, the angle of rotation of the digital twin body space around the x, y, and z axes, and the scaling ratio are solved. The size ratio of the digital twin body space and the real organ space in the x, y, and z directions; thin plate spline transformation calculates the nonlinear mapping relationship between the corresponding feature points to obtain the complex transformation parameters that adapt to the non-rigid deformation of the organ, such as the local coordinate offset caused by the elastic deformation of the liver tissue.

[0039] The calculated dynamic transformation parameters, translation amount, rotation angle, scaling ratio, and nonlinear transformation parameters are integrated to form a dynamic transformation relationship. This relationship can realize that the coordinates of any point in the digital twin body space are input, and the corresponding coordinates of the point in the real organ space are calculated through the transformation parameters; conversely, the coordinates of the real organ space can also be input to calculate the corresponding coordinates of the digital twin body space. Combined with the previously established three-dimensional elastic mechanics representation of the liver tissue, when the real organ is deformed by external force such as surgical instrument operation, the coordinate change after the organ deformation is obtained through mechanical calculation, and the dynamic transformation parameters are updated to make the transformation relationship reflect the coordinate mapping between the digital twin body space and the real organ space in real time, and finally the dynamic spatial mapping relationship between the image data and the physical organ structure is established.

[0040] By detailed calculation of three-dimensional morphological data, the geometric structure and spatial topology are constructed, which can accurately restore the real spatial morphology and connection relationship of liver parenchyma volume, bile duct tree and intraliver vascular network. When the three-dimensional elastic mechanics representation is generated by combining the elastic modulus distribution parameters with the geometric structure, the stiffness coefficient and elastic deformation ability of each mechanical calculation unit are accurately calculated, so that the digital twin can truly reflect the mechanical property difference of different regions of liver tissue, reduce the surgical risk caused by inaccurate representation, in the feature extraction stage, the surface fitting, curvature calculation and other methods are used to fully capture the detailed features of liver surface morphology and intraliver pipeline distribution; In the registration and matching stage, strict similarity measurement and matching threshold are adopted to ensure that the features of the digital twin are highly consistent with the real features of the OCT in the operation, effectively reduce the registration error, the calculation of dynamic transformation parameters and the establishment of dynamic spatial mapping relationship can respond to the position and shape change of the real organ in the operation in real time, through the rapid update of the transformation parameters, the digital twin can always maintain spatial consistency with the real organ, meet the real-time navigation and positioning requirements in the operation.

[0041] In a preferred embodiment of the present application, according to the dynamic spatial mapping relationship, the spatial pose of the surgical instrument is tracked in real time by the electromagnetic sensing device, and the intraoperative fluorescence imaging video stream is synchronously fused, the displacement field and soft tissue deformation tensor generated by the respiratory motion of the liver lobe are calculated, which can include: The original space signal of the surgical instrument collected by the electromagnetic sensing device in real time is converted to the space coordinate system corresponding to the real-time digital twin to obtain the position data of the surgical instrument in the digital twin space by using a dynamic space mapping relationship, specifically including: determining the core parameters of the dynamic space mapping relationship, which is established by a calibration experiment in advance and contains the translation parameters, rotation parameters and scaling parameters between the electromagnetic sensing device coordinate system and the digital twin space coordinate system. In real-time calculation, the original space signal of the surgical instrument collected by the electromagnetic sensing device is first obtained, which contains the three-dimensional position coordinates of the instrument in the electromagnetic sensing coordinate system, corresponding to the distance values of the x-axis, y-axis and z-axis directions and the three-dimensional attitude angles, corresponding to the rotation angle values around the x-axis, y-axis and z-axis. Then, coordinate conversion calculation is performed according to the dynamic space mapping relationship. For the position coordinates, the original x-axis coordinate value is added to the x-axis translation parameter in the mapping relationship to obtain the preliminarily adjusted x-axis coordinate. Then, the preliminarily adjusted x-axis coordinate is subjected to trigonometric function operation with the x-axis rotation parameter. The cosine value and the sine value are calculated according to the rotation angle, then the preliminary coordinate is multiplied by the cosine value and the sine value respectively, and finally the two results are added to obtain the rotated x-axis coordinate. Finally, the rotated x-axis coordinate is multiplied by the x-axis scaling parameter to obtain the x-axis position data of the surgical instrument in the digital twin space coordinate system. The same calculation steps are adopted to perform translation, rotation and scaling operations on the original y-axis and z-axis coordinate values to obtain the y-axis and z-axis position data in the digital twin space coordinate system in turn, and finally the conversion of the original space signal to the digital twin space position data is completed.

[0042] Based on the position data, the real-time position and attitude of the surgical instrument in the digital twin space, i.e. six-degree-of-freedom pose information, is solved, specifically including: based on the three-dimensional position data (x, y, z axis coordinates) in the digital twin space obtained in the previous step, first determine the position information part, directly take the coordinate values of the three axes as the real-time position parameters of the surgical instrument in the digital twin space, and clearly the specific parking point of the instrument in the space, then solve the attitude information, first obtain the original attitude signal collected by the electromagnetic sensing device, which contains the initial rotation angles around the x, y and z axes, and combine the coordinate system characteristics of the digital twin space to correct and calculate the initial rotation angles, first calculate the correction coefficient corresponding to each rotation axis, the correction coefficient is determined according to the three-dimensional parameters of the digital twin, and is adjusted through the difference between the size ratio and the coordinate system direction and the actual space, for example, if the z-axis direction of the digital twin is opposite to the z-axis direction of the actual electromagnetic sensor, the correction coefficient is-1, then multiply the initial rotation angle and the corresponding correction coefficient to obtain the corrected rotation angle, then, according to the three corrected rotation angles, the attitude vector of the surgical instrument is calculated, taking the position point in the digital twin space as the origin, unit vectors are established along the directions corresponding to the three rotation angles respectively, the calculation of the unit vector is realized by dividing the direction vector corresponding to the rotation angle by its module length, and the module length is calculated as the square root of the sum of squares of each component of the direction vector, the three unit vectors together constitute the attitude information of the surgical instrument, combined with the previously determined position information, the six-degree-of-freedom pose information of the surgical instrument is finally solved.

[0043] The six-degree-of-freedom pose information is time-synchronized and spatially fused with the real-time video stream of the hepatobiliary region acquired by the intraoperative fluorescence imaging device to generate an enhanced fluorescence image with spatial position markers, which specifically includes: in terms of time synchronization calculation, first, the acquisition time stamp of the six-degree-of-freedom pose information is obtained, and the system time corresponding to each pose data is generated, which is accurate to the millisecond level and the frame time stamp of the intraoperative fluorescence imaging video stream, and the system time corresponding to each video frame is generated, which is also accurate to the millisecond level, then a time matching rule is established, the time difference between each pose time stamp and each video frame time stamp is calculated, the value of the pose time stamp is subtracted from the value of the frame time stamp, and the absolute value is taken, the video frame time stamp with the smallest time difference from each pose time stamp is found, and the pose information and the corresponding video frame are marked as a time-synchronized group, if there is a time difference exceeding a preset threshold, such as 50 milliseconds, linear interpolation is used to supplement the pose data, according to the time stamps and pose parameters of the two adjacent pose information before and after, the pose change rate in the time interval is calculated, and the change rate is the difference between the two pose parameters divided by the time difference, then according to the time difference between the target video frame time stamp and the previous pose time stamp, the previous pose parameter is added to the product of the change rate and the time difference to obtain the supplemented pose data corresponding to the video frame, ensuring that each video frame has synchronized pose information, in terms of spatial fusion calculation, first, the mapping relationship between the pixel coordinate system of the fluorescence imaging video stream and the digital twin space coordinate system is determined, the actual space coordinate parameters corresponding to each pixel point in the video frame are obtained through calibration experiments, for example, first, place a marker point with known space coordinates in the fluorescence imaging field of view, record the pixel coordinates of the marker point in the video frame, then calculate the conversion formula of the pixel coordinates and the actual space coordinates through linear fitting, then, substitute the position parameter in the synchronized six-degree-of-freedom pose information into the pixel coordinate conversion formula to calculate the pixel position of the surgical instrument in the fluorescence imaging video frame, after that, add a spatial position marker to the corresponding pixel position of the video frame, first determine the graphic style of the marker, then adjust the direction of the marker according to the attitude parameter in the pose information, for example, if the instrument rotates 30 degrees around the x-axis, the long axis direction of the marker also rotates 30 degrees synchronously, superimpose the adjusted marker on the corresponding pixel position of the video frame to generate an enhanced fluorescence image with spatial position markers.

[0044] Based on enhanced fluorescence image and six degrees of freedom pose information, the periodic displacement field of liver parenchyma caused by respiratory motion is analyzed, and the deformation gradient tensor of the liver soft tissue region contacted by the surgical instrument is calculated, including: the six degrees of freedom pose data and the intraoperative fluorescence imaging video stream are spatiotemporally synchronized and aligned to generate a fusion data stream; using the dynamic spatial mapping relationship, the six degrees of freedom pose of the surgical instrument in the fusion data stream is mapped to the real-time digital twin body coordinate system to obtain the motion trajectory of the surgical instrument in the digital twin body space; based on the motion trajectory of the surgical instrument, the real-time contact position between the instrument tip and the liver surface and the intraliver vascular structure represented by the digital twin body is calculated to establish the dynamic spatial correspondence between the instrument tip point and the surface point of the liver anatomical structure; based on the dynamic spatial correspondence, the contact point position information associated with the main blood vessels and biliary structures in the liver is extracted, and the displacement of the corresponding pipeline structure in the digital twin body is analyzed to calculate the rigid displacement vector; based on the contact point position associated with the liver parenchyma soft tissue and the displacement information of the adjacent liver region in the dynamic spatial correspondence, the deformation mode of the liver soft tissue surface in the digital twin body is analyzed, and the dynamic deformation gradient tensor is calculated, specifically including: for the analysis and calculation of the periodic displacement field of liver parenchyma, the feature points of liver parenchyma region are extracted from the enhanced fluorescence image, the points easy to identify such as liver parenchyma edge, blood vessel branch are selected, the pixel coordinates of each feature point in the video frame are determined through image recognition algorithm, then the pixel coordinates of each feature point are converted into three-dimensional coordinates in the digital twin space according to the mapping relationship between the previously established pixel coordinate system and the digital twin space coordinate system, to obtain the spatial position data of each feature point at different time frames, then the periodicity of displacement is analyzed, the position change of each feature point in the continuous time frame is calculated, the position coordinates of the latter frame are subtracted from the position coordinates of the former frame to obtain the displacement components in x, y and z axis directions, a plurality of displacement component data are recorded, covering at least one complete respiratory cycle, usually 3-5 seconds, the time series analysis is performed on each group of displacement component data, the period of displacement component change with time is calculated, the time interval of repeated appearance of displacement component value is found, for example, the time difference between the maximum values of adjacent two times, to determine the periodicity of respiratory motion, according to the periodicity, the displacement components at different time frames are classified according to the period stage, the average displacement component of all feature points in each period stage is calculated, the displacement components of each feature point in the stage are summed respectively, and then divided by the number of feature points, to finally form the periodic displacement field of liver parenchyma caused by respiratory motion, including displacement distribution at different period stages and different spatial positions.

[0045] For the calculation of the deformation gradient tensor of the liver soft tissue region caused by the contact of the surgical instrument, the liver soft tissue region contacted by the surgical instrument is first determined, the overlapping relationship between the space position marker of the surgical instrument in the enhanced fluorescence image and the liver parenchyma region is determined, the range of the contact region is determined, and a certain radius of the region is determined according to the size of the instrument and the elastic properties of the liver tissue, such as a 5mm radius circular region. A plurality of feature points in the region are extracted, the number of which is not less than 10 and are uniformly distributed in the contact region. Then, the digital twin space coordinates of the feature points before contact, when the surgical instrument does not contact the liver tissue, and after contact, when the instrument contacts the tissue and causes deformation, are obtained. The pre-contact coordinates are obtained by converting the feature points in the preoperative or pre-contact image, and the post-contact coordinates are obtained by converting the feature points in the enhanced fluorescence image during contact. The displacement vector of each feature point is calculated by subtracting the pre-contact coordinates from the post-contact coordinates to obtain the displacement values in the x, y and z axis directions. Then, a calculation matrix of the deformation gradient tensor is established, taking the contact region as the research object, establishing a local coordinate system with the contact point as the origin, and establishing coordinate axes along the normal and tangent directions of the liver tissue surface. The displacement vector of each feature point is decomposed into three axis directions of the local coordinate system to obtain the displacement components. According to the position coordinates (xlocal, ylocal, zlocal) of the feature points in the local coordinate system and the corresponding displacement components (ulocal, vlocal, wlocal), the components of the deformation gradient tensor are calculated, such as the partial derivative component of u to x in the tensor. The ulocal displacement difference and the xlocal position difference of two adjacent feature points (adjacent in the xlocal direction) are calculated, and the displacement difference is divided by the position difference to obtain the partial derivative component of u to x. The same method is used to calculate the partial derivatives of u to y, u to z, v to x, etc. Finally, all 9 tensor components are combined to form the deformation gradient tensor of the liver soft tissue region.

[0046] The original space signal is converted to the digital twin space coordinate system through precise translation, rotation and scaling operations, ensuring that the position data of the surgical instrument completely matches the spatial dimensions of the digital twin, avoiding position deviation caused by coordinate system differences, and reducing instrument misoperation caused by inaccurate position perception. The detailed position and attitude calculation process can fully obtain the spatial state of the surgical instrument, not only determining the parking position of the instrument, but also accurately grasping the inclination angle, orientation and other attitude information of the instrument. Through the attitude information, it is determined whether the pointing direction of the instrument tip is towards the target surgical area, avoiding surgical path deviation caused by attitude misjudgment. Time synchronization calculation eliminates the time delay between the pose information and the fluorescence image through timestamp matching and linear interpolation, ensuring that the observed image is completely synchronized with the actual position of the instrument, avoiding visual lag caused by time difference.

[0047] In a preferred embodiment of the present application, according to the displacement field and the deformation tensor, the structure and morphology of the digital twin are dynamically updated, and the obstacle avoidance path and safe needle insertion trajectory of the surgical instrument operating in the liver are calculated in real time, which can include: The rigid displacement vector and the dynamic deformation gradient tensor are combined to construct a boundary condition set for describing the dynamic deformation of the liver and gallbladder region, which specifically includes: first, collect the previously calculated rigid displacement vector and dynamic deformation gradient tensor, the rigid displacement vector contains the translation amount of the liver tissue in x, y, z three directions, and the dynamic deformation gradient tensor contains 9 components, which describes the deformation rate of each point of the liver tissue in different directions, then determine the classification standard of the boundary condition, take the rigid displacement vector as the overall motion boundary condition, and take the deformation gradient tensor as the local deformation boundary condition, for the overall motion boundary condition, calculate the displacement value of the centroid of each liver lobe, which is obtained by the average value of the displacement vectors of all feature points of the liver lobe, and compare it with the rigid displacement vector, if the deviation is within 0.5 millimeters, the rigid displacement vector is directly included in the boundary condition set; if the deviation exceeds 0.5 millimeters, the rigid displacement vector is corrected, the centroid displacement value of the liver lobe is subtracted from the original rigid displacement vector to obtain the correction amount, and then the original vector is added to the correction amount.

[0048] For the local deformation boundary condition, first screen the components with absolute value greater than 0.01 in the deformation gradient tensor, which represent significant deformation, then label the corresponding spatial region for each significant component, determine the spatial region through the coordinate range of the feature points, such as the region with x coordinate of 50-100 millimeters and y coordinate of 30-70 millimeters, finally, arrange the overall motion boundary condition and the local deformation boundary condition with labeled spatial region in time sequence to form a complete boundary condition set, each condition contains corresponding time stamp and spatial range parameters.

[0049] Input the boundary condition set into the liver and gallbladder tissue deformation simulation process to obtain the spatial coordinate data and tissue physical property data of each anatomical structure in the liver represented by the updated real-time digital twin, which specifically includes: first, prepare the initial data required for liver and gallbladder tissue deformation simulation, in addition to the previously constructed boundary condition set, the initial anatomical structure data of liver and gallbladder tissue is also needed, including the initial spatial coordinate data of each anatomical structure in the liver, such as liver lobule, liver sinusoid, bile duct, blood vessel, etc., which can be obtained through computed tomography (CT), magnetic resonance imaging (MRI) and other medical imaging techniques, then through image segmentation and three-dimensional reconstruction technology, the image data is converted into specific spatial coordinate values, at the same time, the initial data of physical properties of liver and gallbladder tissue is also needed, such as the density, elastic modulus, Poisson's ratio, viscosity, etc. of the tissue, which can be obtained by testing liver and gallbladder tissue samples in the laboratory, for liver and gallbladder tissue of different individuals, the specific situation needs to be adjusted to ensure the accuracy and pertinence of the data.

[0050] Next, the boundary condition set is input into the liver and gallbladder tissue deformation simulation system, and the simulation system will follow the preset simulation process. First, each boundary condition in the boundary condition set is analyzed to understand its constraints and effects on liver and gallbladder tissue deformation. Then, according to the physical property data of the liver and gallbladder tissue, the physical architecture of tissue deformation is established to simulate the deformation process of the tissue under external force. During the simulation calculation process, the system will gradually calculate the spatial position changes of each anatomical structure in the liver based on the initial anatomical structure data and the dynamic deformation described by the boundary condition set. For each anatomical structure, it will be decomposed into multiple small units, and each small unit will be deformed according to the physical architecture rules. For example, for a liver lobule unit, the system will calculate the displacement in the x, y, and z directions of each time step according to the corresponding rigid displacement vector and dynamic deformation gradient tensor in the boundary condition. The size of the time step is determined according to the simulation accuracy requirement, and a smaller time interval such as 0.01 seconds is usually selected to ensure the continuity and accuracy of the simulation. Then, the spatial coordinates of the unit are updated according to the displacement.

[0051] At the same time, the change of the physical properties of the tissue is also calculated in real time during the calculation of the spatial coordinate change. Since the deformation of the tissue will cause changes in its internal stress and strain, which in turn affect the physical properties of the tissue, for example, when the tissue is stretched, its elastic modulus may change slightly. The system will update the physical property data of each small unit according to the preset physical property change rule and the calculated stress and strain values. The simulation process will continue until the set simulation time is reached or the specific termination condition is met. After the simulation is completed, the system will integrate and integrate the spatial coordinate data and physical property data of all small units to obtain the overall spatial coordinate data and tissue physical property data of each anatomical structure in the liver. These data represent the updated real-time digital twin.

[0052] Based on the updated intrahepatic anatomical structure spatial coordinate data, the real-time spatial relationship mapping between the tip of the surgical instrument and the important structures such as intrahepatic bile duct, blood vessels and liver surface is established, which specifically includes: first, the real-time spatial coordinates of the tip of the surgical instrument are determined, the electromagnetic sensing device is used to track the surgical instrument in real time, and the electromagnetic sensing device calculates the real-time coordinate values of the tip of the surgical instrument in three-dimensional space according to the electromagnetic induction relationship between the electromagnetic sensing device and the surgical instrument. Specifically, the electromagnetic sensing device emits electromagnetic waves of a specific frequency, the tip of the surgical instrument is provided with an induction coil, the induction coil receives the electromagnetic wave signal and feeds back the signal to the electromagnetic sensing device, and the electromagnetic sensing device calculates the real-time coordinate values of the tip of the surgical instrument on the x, y and z coordinate axes according to the received signal strength, phase and other information, and combines its own position parameters, and constantly updates these coordinate values at a certain time interval, such as 0.005 seconds, to ensure real-time, then the real-time spatial coordinate data of the important structures in the liver, bile duct, blood vessels and liver surface and the like are obtained, which are the related parts in the updated intrahepatic anatomical structure spatial coordinate data obtained by the liver and bile duct tissue deformation simulation, and the coordinate data corresponding to the important structures such as bile duct, blood vessels and liver surface need to be selected from the whole data and classified and arranged according to different structure types, such as all the coordinate data of the bile duct is classified as a class, all the coordinate data of the blood vessels is classified as another class, and the coordinate data of the liver surface is classified as a class.

[0053] The reference coordinate system of the spatial relationship mapping is determined. In order to ensure the comparability of the spatial coordinates of the tip of the surgical instrument and the important structures in the liver, a unified reference coordinate system needs to be established, which usually takes the patient's body as the reference, selects specific anatomical landmarks such as suprasternal fossa and pubic symphysis as reference points, establishes a three-dimensional rectangular coordinate system, and converts the real-time spatial coordinates of the tip of the surgical instrument and the real-time spatial coordinates of the important structures in the liver into the unified reference coordinate system. During the conversion process, the coordinates of the reference points and the original coordinates are related, and the coordinate transformation operations such as translation and rotation are used to unify the coordinate data from different sources into the same coordinate system, then the spatial relationship parameters between the tip of the surgical instrument and each important structure in the liver are calculated, for each important structure, its key feature points or feature lines, feature surfaces are selected to represent the structure, for example, for the bile duct, the central axis of the bile duct can be selected as the feature line; for the blood vessels, multiple feature points on the wall contour of the blood vessels can be selected; for the liver surface, multiple feature surfaces on the surface can be selected.

[0054] When calculating the distance between the tip of the surgical instrument and the feature line segment, a plurality of points on the feature line segment are selected first, and the straight-line distance between the tip of the surgical instrument and each selected point is calculated, and then the minimum value among these distances is taken as the distance between the tip of the surgical instrument and the feature line segment. When calculating the distance between the tip of the surgical instrument and the feature point, the straight-line distance is directly calculated according to the coordinate values of the two points in the three-dimensional coordinate system, that is, the square root of the sum of the squares of the coordinate differences of the two points in the x, y and z directions. When calculating the distance between the tip of the surgical instrument and the feature surface, the equation of the feature surface is determined first, which is calculated through the coordinates of a plurality of points on the feature surface, and then the perpendicular distance between the tip of the surgical instrument and the feature surface is calculated according to the point-to-plane distance calculation method, that is, according to the coordinates of the point and the coefficients of the plane equation, the perpendicular distance between the point and the plane is calculated through specific calculation steps.

[0055] The real-time coordinates of the tip of the surgical instrument, the types of the important structures in the liver, the corresponding feature points / line segments / surfaces and the calculated spatial distances and other information are sorted according to the time sequence and the structure types to form a real-time updated spatial relationship mapping table. Whenever the coordinates of the tip of the surgical instrument or the coordinates of the important structures in the liver change, the corresponding spatial distances will be recalculated, and the contents in the mapping table will be updated, so as to realize the real-time spatial relationship mapping between the tip of the surgical instrument and the important structures such as the bile ducts, blood vessels and liver surface in the liver.

[0056] According to the real-time spatial relationship mapping, the real-time minimum spatial distance of the surgical instrument tip to the nearest intrahepatic key pipeline structure is calculated, specifically including: first, the range of the intrahepatic key pipeline structure is determined, mainly including the main bile ducts in the liver, such as the left hepatic duct, the right hepatic duct, the common hepatic duct, etc. and the main blood vessels, such as the hepatic artery, the portal vein, the hepatic vein and their main branches, etc. From the real-time spatial relationship mapping table established in the front, the spatial relationship data related to these key pipeline structures is screened out, including the feature point, feature line segment information of each key pipeline structure and the distance data of the surgical instrument tip to these feature points, feature line segments, then, for each key pipeline structure, all distance data of the surgical instrument tip to different feature points, feature line segments of the structure are collected, for example, for the left hepatic duct, there are distances of the surgical instrument tip to multiple feature points thereof, and distances to different points on the central axis (feature line segment) and other multiple distance values, then, all distance data corresponding to each key pipeline structure are compared to find the minimum value, which is the minimum distance of the surgical instrument tip to the key pipeline structure, for example, the distance data corresponding to the left hepatic duct has 5mm, 3.5mm, 4.2mm, etc. After comparison, 3.5mm is the minimum distance of the surgical instrument tip to the left hepatic duct, then, the minimum distances corresponding to all key pipeline structures are summarized to obtain a set of minimum distance data, such as 3.5mm to the left hepatic duct, 4.1mm to the right hepatic duct, 2.8mm to the hepatic artery, 3.2mm to the portal vein, etc. Finally, the set of minimum distance data is compared again to find the minimum value, which is the real-time minimum spatial distance of the surgical instrument tip to the nearest intrahepatic key pipeline structure, for example, in the above summarized minimum distance data, 2.8mm is the minimum, so the real-time minimum spatial distance of the surgical instrument tip to the nearest intrahepatic key pipeline structure is 2.8mm, and with the movement of the surgical instrument and the deformation of the intrahepatic structure, the above steps are repeatedly repeated to update the minimum spatial distance value in real time.

[0057] Based on the real-time minimum spatial distance and the real-time feedback of the motion speed and direction of the surgical instrument by the electromagnetic sensing device, a real-time obstacle avoidance path and a safe needle insertion trajectory of the surgical instrument in the liver parenchyma are dynamically generated, including: based on the spatial distribution of the liver intravascular structure and the blood vessels represented by the updated digital twin, the dynamic spatial obstacle distribution of the important structure in the liver around the surgical instrument is calculated; according to the dynamic spatial obstacle distribution, combined with the real-time position and motion direction data of the surgical instrument collected by the electromagnetic navigation device, the potential interference state of the instrument motion trajectory and the intravascular structure in the liver is dynamically evaluated and predicted to obtain the state evaluation result; based on the state evaluation result, the safe and smooth real-time motion trajectory of the surgical instrument avoiding the important vascular and biliary structures in the liver is calculated and generated as the obstacle avoidance path and needle insertion guide; specifically including: first, the real-time minimum spatial distance data and the real-time feedback of the motion speed and direction of the surgical instrument by the electromagnetic sensing device are obtained, and the real-time minimum spatial distance data is the distance value from the tip of the surgical instrument to the nearest key intravascular structure in the liver calculated and updated continuously; the motion speed data of the surgical instrument is calculated by the electromagnetic sensing device through monitoring the displacement of the surgical instrument in unit time, for example, in 0.01 seconds, the surgical instrument moves 0.1 mm in x direction, 0.08 mm in y direction and 0.05 mm in z direction, the actual motion speed and direction of the surgical instrument are calculated by calculating the ratio of displacement to time, and the motion direction data is obtained by monitoring the pointing change of the surgical instrument in three-dimensional space, which is described by parameters such as azimuth angle and elevation angle.

[0058] A safety distance threshold is set, for example, through statistical analysis, when the distance between the surgical instrument and the key intravascular structure in the liver is greater than or equal to 3 mm, the structure will not be damaged, so 3 mm can be used as the safety distance threshold, at the same time, the safety distance threshold is also adjusted according to the specific type of surgery, such as liver biopsy and liver tumor ablation, and the individual situation of the patient, such as the thickness and position distribution of the intravascular structure in the liver, to ensure safety and smooth operation of the surgery, then the relationship between the real-time minimum spatial distance and the safety distance threshold is judged, if the real-time minimum spatial distance is greater than or equal to the safety distance threshold, it means that the current motion state of the surgical instrument will not threaten the key intravascular structure in the liver, at this time, according to the target position of the surgery, such as the position of the tumor and the motion direction of the current surgical instrument, the subsequent path and trajectory are planned according to the original motion trend, only the path and trajectory are slightly adjusted according to the real-time deformation of the intravascular structure in the liver and the motion data of the surgical instrument, to ensure that the surgical instrument can accurately move towards the target position.

[0059] If the real-time minimum spatial distance is less than the safety distance threshold, it means that the current movement direction of the surgical instrument may approach or even touch the key pipeline structure in the liver, which poses a safety risk. At this time, the movement direction and speed of the surgical instrument need to be adjusted to generate an obstacle avoidance path. First, the spatial position and morphology of the nearest key pipeline structure are analyzed to determine the range of the area that needs to be avoided. For example, the nearest key pipeline structure is the hepatic artery, which is located on the left side of the current movement direction of the surgical instrument. Therefore, the movement direction of the surgical instrument needs to be adjusted to the right to avoid the area where the hepatic artery is located. Then, the adjusted movement direction is calculated according to the range of the area that needs to be avoided and the target position of the surgery. The current position, movement speed, and distance to the target position of the surgical instrument are considered to ensure that the adjusted direction can avoid the dangerous area while minimizing the path to the target position. For example, the surgical instrument originally moves straight towards the target position, but due to the hepatic artery on the left side, it needs to be deflected by a certain angle to the right. The size of the deflection angle is determined by a comprehensive calculation of factors such as the distance to the hepatic artery, the thickness of the hepatic artery, and the distance to the target position. For example, it is calculated that a deflection angle of 15° is needed to ensure that the distance between the surgical instrument and the hepatic artery during movement always remains above the safety distance threshold.

[0060] At the same time, according to the adjusted movement direction, a suitable movement speed is calculated. If the adjusted path is relatively complex, the movement speed needs to be slowed down to ensure that the surgical instrument can move accurately according to the adjusted path and avoid not being able to respond in time to new situations that may arise due to excessive speed. If the adjusted path is relatively smooth and close to the target position, the speed can be appropriately maintained or adjusted to improve surgical efficiency. For example, when the path needs to be adjusted to bypass a relatively thin bile duct, the movement speed of the surgical instrument is reduced from 2mm / s to 1mm / s. Finally, according to the adjusted movement direction and speed, combined with the real-time deformation of the liver structure, each small movement step of the surgical instrument in the liver parenchyma is gradually planned. Connecting these steps in chronological order forms a real-time obstacle avoidance path. At the same time, according to the requirements of the surgery, such as the needle insertion angle and depth, and the planned obstacle avoidance path, the complete needle insertion trajectory of the surgical instrument from the initial position to the target position is determined. This trajectory is the safe needle insertion trajectory. During the surgery, the above steps are repeatedly performed to dynamically update the obstacle avoidance path and the safe needle insertion trajectory based on real-time data, ensuring that the surgical instrument always moves within a safe range and successfully completes the surgical operation.

[0061] By obtaining the basic data of rigid displacement vector and dynamic deformation gradient tensor in detail, and carrying out accurate data integration and mathematical relationship description, the boundary condition set constructed can accurately and comprehensively reflect the dynamic deformation of the liver and gall region, which provides a reliable input basis for the subsequent liver and gall tissue deformation simulation, avoids the problem that the simulation result deviates greatly from the actual situation due to inaccurate boundary conditions, and in the liver and gall tissue deformation simulation process, through careful initial data preparation, accurate simulation calculation and result summary and integration, the updated real-time digital twin body represented by the spatial coordinate data and tissue physical property data of each anatomical structure in the liver has high accuracy and real-time, which can truly reflect the dynamic change of the intrahepatic structure during the operation, provide clear and accurate real-time information of the intrahepatic structure, avoid operation failure caused by information lag or inaccuracy, and improve the safety and accuracy of the operation.

[0062] In a preferred embodiment of the present application, the obstacle avoidance path and the safe needle insertion trajectory are superimposed in real time on the surgeon's image field of view, and a consistency measure of the registration error is calculated based on the reference marker points embedded in the liver, and if the measure is greater than the allowable threshold, the real-time correction mechanism of the mapping relationship and the path is triggered, which can include: The real-time obstacle avoidance path and the safe needle insertion trajectory information are superimposed in real time on the intraoperative image device field of view seen by the surgeon, specifically including: the intraoperative image device, such as CT, MRI, etc., will continuously collect the original image data of the operation area, these data are presented in the form of a pixel matrix, for each original image, first denoising processing is needed, the average value of the gray value of each pixel point and the gray value of the surrounding pixel points is calculated, the pixel point gray value deviating from the average value by more than a preset range is corrected to the average value, so as to reduce the noise interference in the image, then image enhancement calculation is carried out, by calculating the dynamic range of the pixel point gray value, the area with concentrated gray value is stretched, so that the boundaries of different tissues in the image are clearer, facilitating the extraction of useful information subsequently.

[0063] In the digital twin body, the obstacle avoidance path and the safe needle insertion trajectory are initially in the form of three-dimensional coordinate point sequence, which needs to be converted into two-dimensional coordinate points matching the intraoperative image device field of view, first, the conversion relationship between the digital twin body coordinate system and the intraoperative image device coordinate system is determined, by collecting multiple reference points with clear coordinates in both coordinate systems, the rotation matrix and translation vector between the two coordinate systems are calculated, then for each three-dimensional coordinate point on the obstacle avoidance path and the safe needle insertion trajectory, it is substituted into the conversion formula, multiplied by the rotation matrix for rotation transformation, and then added to the translation vector for translation transformation, to obtain the corresponding two-dimensional coordinate point, at the same time, according to the resolution of the intraoperative image, the specific position of each two-dimensional coordinate point in the image pixel matrix is calculated, that is, by coordinate value and pixel size division operation, the pixel row and column number corresponding to the coordinate point in the image is determined.

[0064] The pixel positions corresponding to the two-dimensional coordinate points of the converted obstacle avoidance path and the safe needle insertion trajectory are fused with the pre-processed intraoperative image pixel matrix. For the pixel points passed by the trajectory, the gray value is adjusted or a specific color is assigned, such as red for the obstacle avoidance path and blue for the safe needle insertion trajectory. The new gray value or color value of these pixel points is calculated to clearly display the trajectory without affecting the observation of tissue details in the image. Finally, the fused pixel matrix is displayed in the image field of view of the operator. Based on the reference marker points embedded in the anatomical position of the liver, the three-dimensional spatial coordinates of the corresponding marker points in the real-time digital twin and the real surgical field image are extracted, specifically including: in the pre-processed real surgical field image, first determine the characteristics of the reference marker points, such as specific shape, gray value range, etc. The gray value mean, variance and shape feature parameters such as circularity and rectangularity of the pixels in each window are calculated by sliding window traversal of the entire image. Compare these parameters with the pre-set reference marker point feature parameters. When the similarity of the parameters in a certain window to the pre-set feature parameters exceeds the pre-set threshold, it is determined that the window is the area where the reference marker point may exist. Then, in this area, the coordinates of the tissue edge are calculated by edge detection algorithm, and then the center coordinates of the reference marker point are calculated according to the geometric shape of the reference marker point, which are the two-dimensional coordinates of the reference marker point in the real surgical field image. Then, combined with the imaging parameters of the intraoperative imaging device, such as focal length and detector distance, the three-dimensional coordinates of the reference marker point are calculated by the principle of triangulation. Specifically, the two-dimensional coordinates of the marker point in two images taken at different angles by the imaging device are used to establish two projection equations, and the three-dimensional coordinates of the marker point are obtained by simultaneous solution. In the real-time digital twin, the position of the reference marker point is pre-set and stored in the form of three-dimensional coordinates in the database. According to the unique identification of the marker point, the corresponding three-dimensional coordinate data is directly retrieved from the database as the three-dimensional spatial coordinates of the reference marker point in the real-time digital twin.

[0065] The spatial deviation between the two groups of marker point coordinates is calculated to generate a consistency measure representing the spatial registration accuracy of the digital twin and the real liver and gallbladder structure, specifically including: for each pair of corresponding reference marker points, i.e. one marker point in the digital twin and the corresponding marker point in the real surgical field image, the distance deviation in three-dimensional space is calculated. Let the three-dimensional coordinates of the marker point in the digital twin be ( , , ), and the three-dimensional coordinates of the corresponding marker point in the real surgical field image be ( , , ), according to the space two-point distance formula, the spatial deviation d of the pair of mark points is calculated, and all the embedded reference mark points are calculated in this way to obtain a plurality of sets of spatial deviation data, and the obtained all spatial deviation data is statistically analyzed to calculate the statistical characteristic value as a consistency measure. First, the average value of all spatial deviations is calculated, all deviation values are added, and then divided by the number of deviations to obtain the average deviation value. Then, the standard deviation of the deviation is calculated. First, the square of the difference between each deviation value and the average deviation value is calculated. After adding these square values and dividing by the number of deviations, the result is square rooted to obtain the standard deviation. The average deviation value and the standard deviation are combined to generate a consistency measure representing the spatial registration accuracy of the digital twin and the real liver and gallbladder structure, for example, in the form of average deviation value ± standard deviation to represent the measure.

[0066] The consistency measure is compared with the preset allowable threshold in real time. When the consistency measure is greater than the allowable threshold, the real-time correction mechanism of the spatial mapping relationship and the obstacle avoidance path is triggered, including re-computing and adjusting the dynamic spatial mapping relationship between the real-time digital twin and the real liver and gallbladder structure, and updating the real-time obstacle avoidance path and the safe needle insertion trajectory of the surgical instrument based on the corrected mapping relationship, specifically including: when the consistency measure is greater than the allowable threshold, first, the spatial mapping relationship between the digital twin and the real liver and gallbladder structure is re-determined, a larger number of reference points are collected, including the original reference mark points and the newly added obvious anatomical feature points, and the three-dimensional coordinates of these reference points in the digital twin and the real liver and gallbladder structure are obtained respectively. Then, the least square method is used for fitting calculation to establish the mathematical expression of the spatial mapping and determine the new spatial mapping relationship. According to the re-computed spatial mapping relationship, the original obstacle avoidance path and the safe needle insertion trajectory of the surgical instrument are updated. First, the three-dimensional coordinate point sequence of the original trajectory in the digital twin is obtained. For each coordinate point, the new three-dimensional coordinate point corresponding to the real liver and gallbladder structure coordinate system is obtained by conversion calculation according to the new spatial mapping relationship. Then, the newly obtained coordinate point sequence is smoothed to calculate the distance and angle between adjacent coordinate points, delete abnormal points such as points with large distance or angle deviation from adjacent points, and supplement coordinate points through interpolation algorithms such as linear interpolation, cubic spline interpolation, etc. to make the trajectory more smooth and continuous. At the same time, it is rechecked whether the updated trajectory has collision risk. The distance between each point on the trajectory and the surrounding tissues such as blood vessels and important organs is calculated. If the distance between a point and the tissue is less than the safety threshold, the coordinate of the point is adjusted until all points meet the obstacle avoidance requirements. Finally, the updated real-time obstacle avoidance path and safe needle insertion trajectory of the surgical instrument are obtained.

[0067] By accurately superimposing the real-time obstacle avoidance path and the safe needle insertion trajectory onto the surgeon's image field of view, the surgeon can clearly see the expected motion trajectory of the surgical instrument. Combined with detailed coordinate conversion and fusion calculation, the trajectory is highly matched with the actual surgical area, reducing operation errors caused by visual deviation, making the needle insertion position and path more consistent with the surgical planning, improving the accuracy of surgical operation. Based on the benchmark marker point, the registration error consistency measurement is calculated, which can monitor the registration of the digital twin and the real liver and gallbladder structure in real time, and find the registration deviation in time. When the deviation exceeds the threshold, the spatial mapping relationship is recalculated and the trajectory is updated to ensure that the digital twin always maintains high consistency with the real surgical scene, providing accurate navigation basis for surgical operation and improving the accuracy of surgery.

[0068] The embodiment of the present application also provides a computing device, comprising a processor and a memory storing a computer program, wherein the computer program is executed by the processor to perform the method described above. All implementation manners in the above method embodiment are applicable to this embodiment, and the same technical effects can be achieved.

[0069] The embodiment of the present application also provides a computer readable storage medium storing instructions, which, when executed on a computer, cause the computer to perform the method described above. All implementation manners in the above method embodiment are applicable to this embodiment, and the same technical effects can be achieved.

[0070] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered within the scope of protection of the present application.

Claims

1. An intraoperative real-time navigation system based on digital twinning, characterized in that, The method comprises the following steps: a data acquisition module is used to acquire three-dimensional morphological data of the liver and biliary tract by an intraoperative ultrasonic imaging device, and to collect the elastic modulus distribution parameters of the liver parenchyma and the bile duct wall; a construction registration module is used to construct a real-time digital twin of the liver and gallbladder region based on the three-dimensional morphological data and the elastic modulus distribution parameters, to perform multi-modal non-rigid registration on the digital twin and the liver surface topography and vascular structure obtained by the intraoperative optical coherence tomography, and to establish a dynamic spatial mapping relationship between the organ structure and the image data; a calculation and fusion module is used to track the spatial pose of the surgical instrument in real time through an electromagnetic sensing device according to the dynamic spatial mapping relationship, to synchronously fuse the intraoperative fluorescence imaging video stream, to calculate the displacement field and the soft tissue deformation tensor of the liver lobe caused by respiratory motion, and to calculate the obstacle avoidance path and the safe needle insertion trajectory of the surgical instrument in the liver; an update planning module is used to dynamically update the structure and morphology of the digital twin according to the displacement field and the deformation tensor, and to calculate the obstacle avoidance path and the safe needle insertion trajectory of the surgical instrument in the liver in real time; a feedback correction module is used to superimpose the obstacle avoidance path and the safe needle insertion trajectory on the image field of view of the operator in real time, to calculate the consistency measure of the registration error based on the reference marker points embedded in the liver, and to trigger the real-time correction mechanism of the mapping relationship and the path if the measure is greater than the allowable threshold.

2. The digital-twin-based intraoperative real-time navigation system of claim 1, wherein, The method comprises the following steps: based on the three-dimensional morphological data, a geometric structure and a spatial topology are constructed to describe the liver parenchyma volume, the bile duct tree topology and the intraliver vascular network; based on the geometric structure and the spatial topology, and in combination with the elastic modulus distribution parameters, a three-dimensional elastic mechanical representation of the liver tissue is generated, which is endowed with biomechanical properties; the geometric structure and the three-dimensional elastic mechanical representation are combined to form a real-time digital twin of the liver and gallbladder region, which has both anatomical structure and mechanical characteristics; feature extraction is performed on the digital twin to obtain the spatial distribution characteristics of the liver surface morphology and the intraliver pipeline, and the distribution characteristics are matched with the corresponding features of the real organ surface obtained by the intraoperative optical coherence tomography to obtain a registration matching result; based on the registration matching result, dynamic transformation parameters of the digital twin space to the real organ space are calculated to establish a dynamic spatial mapping relationship between the image data and the physical organ structure.

3. The digital-twin-based intraoperative real-time navigation system of claim 2, wherein, The method comprises the following steps: three-dimensional spatial coordinates and topological connection relationships of the intraliver bile duct tree branch points, the liver surface contour feature points and the intraliver vascular intersection points are extracted from the digital twin as a first registration feature set; the liver and gallbladder region surface topography and subcutaneous vascular structure images collected by the intraoperative optical coherence tomography are enhanced and segmented, and the spatial distribution information of the corresponding bile duct wall contour, liver surface topography and vascular branch points is extracted as a second registration feature set; Non-rigid feature matching is performed between the first set of registration features and the second set of registration features to establish a correspondence between the feature points and to evaluate spatial consistency to obtain an evaluation result; Based on the evaluation result, non-linear transformation parameters are calculated to achieve spatial alignment of the two feature sets to obtain a registration matching result.

4. The digital-twin-based intraoperative real-time navigation system of claim 3, wherein, According to the dynamic spatial mapping relationship, the spatial pose of the surgical instrument is tracked in real time through the electromagnetic sensing device, and the intraoperative fluorescent imaging video stream is synchronously fused, and the displacement field and soft tissue deformation tensor generated by the respiratory motion of the liver lobe are calculated, including: Using the dynamic spatial mapping relationship, the original spatial signal of the surgical instrument collected by the electromagnetic sensing device in real time is converted to the spatial coordinate system corresponding to the real-time digital twin to obtain the position data of the surgical instrument in the digital twin space; Based on the position data, the real-time position and attitude of the surgical instrument in the digital twin space, i.e., six-degree-of-freedom pose information, are calculated; The six-degree-of-freedom pose information is time-synchronized and spatially fused with the real-time video stream of the liver and gallbladder region obtained by the intraoperative fluorescent imaging device to generate an enhanced fluorescent image with spatial position markers; Based on the enhanced fluorescent image and the six-degree-of-freedom pose information, the periodic displacement field generated by the respiratory motion of the liver parenchyma is analyzed, and the deformation gradient tensor of the liver soft tissue region caused by the contact of the surgical instrument is calculated.

5. The digital-twin-based intraoperative real-time navigation system of claim 4, wherein, Based on the enhanced fluorescent image and the six-degree-of-freedom pose information, the periodic displacement field generated by the respiratory motion of the liver parenchyma is analyzed, and the deformation gradient tensor of the liver soft tissue region caused by the contact of the surgical instrument is calculated, including: The six-degree-of-freedom pose data is time-synchronized and spatially aligned with the intraoperative fluorescent imaging video stream to generate a fused data stream; Using the dynamic spatial mapping relationship, the six-degree-of-freedom pose of the surgical instrument in the fused data stream is mapped to the coordinate system of the real-time digital twin to obtain the motion trajectory of the surgical instrument in the digital twin space; Based on the motion trajectory of the surgical instrument, the real-time contact position between the tip of the instrument and the liver surface and the intraliver vascular structure represented by the digital twin is calculated to establish a dynamic spatial correspondence between the tip of the instrument and the surface points of the liver anatomical structure; Based on the dynamic spatial correspondence, the contact point position information associated with the main blood vessels and biliary structures in the liver is extracted, and the displacement of the corresponding pipeline structure in the digital twin is analyzed to calculate the rigid displacement vector; Based on the contact point position associated with the liver parenchyma soft tissue and the displacement information of the adjacent liver region in the dynamic spatial correspondence, the deformation pattern of the liver soft tissue surface in the digital twin is analyzed, and the dynamic deformation gradient tensor is calculated.

6. The digital-twin-based intraoperative real-time navigation system of claim 5, wherein, According to the displacement field and the deformation tensor, the structure and morphology of the digital twin are dynamically updated, and the obstacle avoidance path and safe needle insertion trajectory of the surgical instrument during operation in the liver are calculated in real time, including: The rigid displacement vector and the dynamic deformation gradient tensor are combined to construct a boundary condition set for describing the dynamic deformation of the liver and gallbladder region; The boundary condition set is input into the liver and gallbladder tissue deformation simulation process to obtain the spatial coordinate data and tissue physical property data of each anatomical structure represented by the updated real-time digital twin in the liver; Based on the updated intrahepatic anatomical structure spatial coordinate data, a real-time spatial relationship mapping between the tip of the surgical instrument and important structures such as intrahepatic bile ducts, blood vessels and liver surface is established; According to the real-time spatial relationship mapping, the real-time minimum spatial distance from the tip of the surgical instrument to the nearest intrahepatic key pipeline structure is calculated; Based on the real-time minimum spatial distance and the real-time feedback of the movement speed and direction of the surgical instrument by the electromagnetic sensing device, the real-time obstacle avoidance path and safe needle insertion trajectory of the surgical instrument in the liver parenchyma are dynamically generated.

7. The digital-twin-based intraoperative real-time navigation system of claim 6, wherein, Based on the real-time minimum spatial distance and the real-time feedback of the movement speed and direction of the surgical instrument by the electromagnetic sensing device, the real-time obstacle avoidance path and safe needle insertion trajectory of the surgical instrument in the liver parenchyma are dynamically generated, including: Based on the spatial distribution of intrahepatic pipeline structures and blood vessels represented by the updated digital twin, the dynamic spatial obstacle distribution of important intrahepatic structures around the surgical instrument is calculated; According to the dynamic spatial obstacle distribution, combined with the real-time acquisition of the position and movement direction data of the surgical instrument by the electromagnetic navigation device, the potential interference state of the instrument movement trajectory and the intrahepatic pipeline structure is dynamically evaluated and predicted to obtain the state evaluation result; Based on the state evaluation result, the safe and smooth real-time movement trajectory of the surgical instrument avoiding important intrahepatic vascular and biliary structures is calculated and generated as the obstacle avoidance path and needle insertion guide.

8. The digital-twin-based intraoperative real-time navigation system of claim 7, wherein, The obstacle avoidance path and safe needle insertion trajectory are superimposed in real time on the surgeon's image field of view, and based on the consistency measure of the registration error calculated based on the reference marker points embedded in the liver, if the measure is greater than the allowable threshold, the real-time correction mechanism of the mapping relationship and the path is triggered, including: The real-time obstacle avoidance path and safe needle insertion trajectory information is superimposed in real time on the intraoperative image device field of view seen by the surgeon; Based on the reference marker points embedded in the anatomical position of the liver, the three-dimensional spatial coordinates of the corresponding marker points in the real-time digital twin and the real image of the surgical field are extracted respectively; The spatial deviation between the two groups of marker point coordinates is calculated to generate a consistency measure representing the spatial registration accuracy of the digital twin and the real liver and biliary structure; The consistency measure is compared with the preset allowable threshold in real time, and when the consistency measure is greater than the allowable threshold, the real-time correction mechanism of the spatial mapping relationship and the obstacle avoidance path is triggered, including recalculating and adjusting the dynamic spatial mapping relationship between the real-time digital twin and the real liver and biliary structure, and updating the real-time obstacle avoidance path and safe needle insertion trajectory of the surgical instrument based on the corrected mapping relationship.

9. A computing device, comprising: It includes: One or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the system as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program which is executed by the processor to implement the system as claimed in any one of claims 1 to 8.

Images