A laparoscopic surgery augmented reality method based on finite element soft tissue deformation modeling and individualized anatomical priority rendering and related devices

Abstract

The application discloses a laparoscopic surgery augmented reality method based on finite element soft tissue deformation modeling and individualized anatomical priority rendering and related devices. The method comprises the following steps: constructing an individualized three-dimensional anatomical model based on patient images and extracting priority rendering structures according to risk levels; constructing a finite element grid to solve a stiffness equation to obtain a soft tissue deformation field; performing deformation compensation on rigid body registration based on the deformation field; color coding and rendering a three-dimensional anatomical overlay layer according to risk levels; calculating a three-dimensional distance of an instrument end in real time to perform hierarchical early warning; recording a navigation log and publishing a model version. The target registration error is controlled within 1 mm, and the rendering delay is lower than 50 ms. The application also discloses corresponding devices, electronic equipment, computer readable storage media and computer program products.

Description

Technical Field

[0001] This application relates to the fields of finite element biomechanical modeling and laparoscopic surgery augmented reality navigation technology, and in particular to an laparoscopic surgery augmented reality method and related device based on finite element soft tissue deformation modeling and individualized anatomical priority rendering. Background Technology

[0002] Augmented reality navigation in laparoscopic surgery aims to overlay preoperative three-dimensional anatomical information onto the intraoperative laparoscopic view in real time, providing the operator with anatomical positioning and risk alerts. Current laparoscopic surgical navigation primarily employs the following technical approaches: one involves reconstructing a three-dimensional anatomical model from preoperative CT and MRI images, then rigidly registering it with intraoperative laparoscopic images using feature point matching before overlaying; another involves constructing an anatomical spatial distance field from intraoperative images for instrument path planning and alerting to hazardous structures; and yet another involves tracking tissue movement across multiple intraoperative images to update navigation information. These technologies provide the fundamental means for three-dimensional modeling, registration overlay, and distance alerts in laparoscopic surgical navigation.

[0003] Based on the above technologies, there is still room for improvement in terms of non-rigid deformation of soft tissues under the action of pneumoperitoneum and instruments during surgery, as well as individualized organization of superimposed information according to surgical stage and anatomical risk. For example, soft tissue deformation compensation driven by physical mechanics models, node-by-node registration correction based on deformation field, and individualized anatomical priority rendering based on risk level all need further improvement. Summary of the Invention

[0004] To address the above technical issues, this application provides an augmented reality method and related apparatus for laparoscopic surgery based on finite element soft tissue deformation modeling and individualized anatomical priority rendering.

[0005] Firstly, an augmented reality method for laparoscopic surgery based on finite element soft tissue deformation modeling and individualized anatomical priority rendering is provided, comprising the following steps: S1. Constructing a three-dimensional anatomical model with individualized geometric parameters and risk level annotations based on patient images; extracting relevant anatomical structures for the current stage using a surgical stage classifier and sorting them according to risk level to obtain a set of priority-rendered anatomical structures; S2. Constructing a finite element mesh based on the set of priority-rendered anatomical structures; solving the finite element stiffness equation with pneumoperitoneum pressure, instrument contact force, and gravity as loads to obtain the soft tissue deformation field u; S3. Establishing an initial solution for rigid body registration between the laparoscopic image feature points and the three-dimensional anatomical model using the ICP algorithm; performing node-by-node deformation compensation on each node of the rigid body registration result based on the soft tissue deformation field u. S4. Based on the deformation compensation transformation matrix, a three-dimensional anatomical overlay layer is rendered in the endoscopic field of view. High-risk structures are rendered with red outlines, medium-risk structures with orange outlines, and low-risk structures with green outlines. Safety boundaries are marked with color-transparent buffers. S5. The three-dimensional distance between the end of the surgical instrument and the safety boundary is calculated in real time. When the distance is lower than the first-level threshold, a first-level warning is triggered. When the distance is lower than the second-level threshold, a second-level emergency alarm is triggered. The warning signal is output to the augmented reality interface. S6. The surgical navigation log is recorded with timestamps as indexes, including instrument coordinates, safety boundary coordinates, three-dimensional distance, and warning events. After passing quality gating, the updated three-dimensional anatomical model version is released to the surgical navigation system.

[0006] In a second aspect, a finite element soft tissue deformation modeling and individualized anatomical priority rendering laparoscopic surgery augmented reality device is provided, including an anatomical model construction unit, a finite element deformation calculation unit, a deformation compensation registration unit, a priority rendering unit, a three-dimensional distance warning unit, and a log version management unit, which are respectively used to perform steps S1 to S6 of the method described in the first aspect.

[0007] Thirdly, an electronic device is provided, including a processor and a memory, the memory being used to store a computer program, which, when executed by the processor, implements the method described in the first aspect.

[0008] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and the computer program, when executed by a processor, implements the method described in the first aspect.

[0009] Fifthly, a computer program product is provided, comprising a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0010] In any embodiment of this application, in step S1, the multi-organ three-dimensional segmentation model adopts the nnU-Net v2 framework and is pre-trained on a multi-institution dataset containing at least 1000 abdominal CT scans, with a segmentation Dice coefficient of not less than 0.90; the risk level labeling is reviewed and confirmed by no less than two licensed laparoscopic surgeons according to the anatomical injury risk rating standard; the priority rendering structure set is pre-configured according to the surgical type, supporting real-time dynamic adjustment during the operation.

[0011] In any embodiment of this application, in step S2, the finite element mesh uses tetrahedral elements (Tet4), and the mean of the element Jacobian determinant is not less than 0.6 to ensure mesh quality; the elastic parameter estimation adopts the organizational elastic modulus-Hounsfield value mapping relationship proposed by Krouskop et al., and the Poisson's ratio is uniformly taken as 0.47 to approximate incompressibility constraint; the solver adopts the direct method (sparse Cholesky decomposition) to ensure completion within 50ms per frame, and if it does not meet the requirement, it degenerates into a simplified elastic model to ensure real-time performance.

[0012] In any embodiment of this application, in step S3, ICP registration uses a point-to-plane distance metric (Point-to-Plane ICP) to improve the convergence speed, with a maximum of 100 iterations; the initial corresponding points are established using SIFT feature point nearest neighbor matching, and external points with a distance greater than 10mm between them are removed from the matched point pairs; deformation compensation uses trilinear interpolation to query the displacement of any spatial point in the finite element mesh node displacement field, with a compensation accuracy of not less than 0.1mm.

[0013] In any embodiment of this application, in step S4, the augmented reality rendering pipeline is implemented using OpenGL ES 3.0, supporting operation on a standard laparoscopic imaging system (resolution 1920×1080, frame rate 25 to 30fps); the safety buffer boundary uses gradient transparency (70% to 80% opacity in the center area, and 10% to 20% gradient in the edge area) to provide intuitive visual cues for depth and distance perception; the overall brightness after the rendering layers are superimposed is not less than 50% of the original laparoscopic image, ensuring the clarity of the surgeon's field of vision.

[0014] In any embodiment of this application, in step S5, the three-dimensional distance calculation uses a kd-tree data structure to establish a spatial index for the set of safe boundary nodes, and the time complexity of querying the nearest neighbor node is O(log N), ensuring that the real-time distance update is completed within 50ms; the distance threshold parameter supports online adjustment during the operation (in the paused state), and the adjustment record is written to the navigation log for postoperative review; the frequency and duration of the audio alarm signal can be configured according to hospital specifications.

[0015] In any embodiment of this application, in step S6, the navigation log adopts an append-only mode to ensure that the integrity of the log data is not affected by unexpected system interruptions; the log file is signed with a SHA-256 hash value to prevent subsequent tampering and meet the NMPA's requirements for the immutability of operation logs in the review of Class III medical device software; the model version library supports the complete saving of the last 5 versions, and the differences between versions are stored in an incremental patch format to save storage space.

[0016] Compared with existing technologies, this application has the following beneficial effects: First, the individualized three-dimensional anatomical model improves the registration accuracy by 3 to 5 mm compared to the general atlas method, a design not seen in existing laparoscopic AR navigation methods using general anatomical atlases. Second, finite element soft tissue deformation modeling reduces the virtual-to-real registration error by 3 to 5 mm compared to pure rigid body registration, a design not seen in existing laparoscopic AR navigation methods that do not employ finite element physical-driven deformation modeling. Third, deformation-compensated registration controls the target registration error to within 1 mm, a design not seen in existing AR navigation methods that do not establish node-by-node deformation-compensated registration. Fourth, graded color-coded rendering achieves a safety boundary visualization accuracy of 1 mm, a design not seen in existing navigation methods that do not use risk-level color-coded rendering. Fifth, three-dimensional distance graded early warning reduces the instrument safety margin monitoring delay to less than 50 ms, a design not seen in existing laparoscopic AR navigation methods that do not establish three-dimensional distance graded early warning. Sixth, the timestamp navigation log and model version gating release enable the system to be traced back to the millisecond level of accuracy. This design is not seen in existing AR navigation methods that lack version management and navigation logs. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be explained below.

[0018] Figure 1 This is a flowchart illustrating an augmented reality method for laparoscopic surgery based on finite element soft tissue deformation modeling and individualized anatomical priority rendering, provided as an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of the structure of an augmented reality device for laparoscopic surgery, which provides finite element soft tissue deformation modeling and individualized anatomical priority rendering, as provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings, so that those skilled in the art can fully understand the solutions of this application. In the specification, claims and related drawings of this application, the terms "first", "second" and other terms used are only used to distinguish different elements and do not imply any specific order.

[0022] In step S1, based on the patient's preoperative CT (recommended slice thickness not exceeding 1mm) and MRI images, a multi-organ 3D segmentation model is used to extract target anatomical structures such as the liver, gallbladder, blood vessels (diameter not less than 2mm), ureter, and peritoneum. A 3D anatomical model is constructed, containing individualized geometric parameters such as volume (unit ml, accuracy 0.1ml), centroid coordinates (unit mm, accuracy 0.1mm), and bounding box size (unit mm), along with high / medium / low risk level annotations. This model is then exported as a standardized 3D mesh (STL or OBJ format). A surgical stage classifier trained on the laparoscopic video is used to identify the current stage. Relevant anatomical structures and their two-hop neighborhood nodes in the knowledge graph are extracted and sorted in two dimensions: risk level (high risk priority) and surgical field occlusion prediction (low probability of occlusion priority). The number of structures to be rendered first ranges from 5 to 20.

[0023] In step S2, a tetrahedral finite element mesh (average mesh size 1 to 3 mm, total number of nodes approximately 5000 to 20000) is constructed for the prioritized anatomical structure set. Soft tissue elastic parameters are automatically estimated based on CT Hounsfield values ​​(elastic modulus 1 to 10 kPa, Poisson's ratio 0.45 to 0.49). Using pneumoperitoneal pressure (8 to 15 mmHg, read in real-time by an intra-abdominal pressure monitor), instrument contact force (acquired by a triaxial force sensor, accuracy 0.1 N, sampling rate above 100 Hz), and gravity (direction calculated from the operating table tilt angle) as boundary conditions, the finite element stiffness equation is solved using sparse Cholesky decomposition. The solution time per frame does not exceed 50 ms, with a time step of 10 to 100 ms, yielding the soft tissue deformation field u (three-dimensional displacement vector of each node, accuracy 0.1 mm).

[0024] In step S3, an initial rigid body registration solution T_rigid is established between the endoscopic image and the projection points of the 3D anatomical model using SIFT feature point detection (maximum 2000 feature points) and the ICP algorithm (point-to-surface measurement, maximum 100 iterations, convergence condition root mean square error variation less than 0.01 mm). The corresponding displacement of each node in T_rigid is queried from the soft tissue deformation field u using trilinear interpolation, and deformation compensation is performed independently to obtain the deformation compensation transformation matrix T_comp. The target registration error is controlled within 1 mm, and verification is performed using at least 5 anatomical landmarks (mean TRE not exceeding 1 mm).

[0025] In step S4, the 3D anatomical model is projected onto the laparoscopic field of view based on T_comp, and a 3D anatomical overlay layer is rendered: high-risk structures (major blood vessels, nerve bundles, etc.) are rendered with red outlines (2 to 4 pixels wide, opacity 80% to 100%) and a 10mm radius red gradient transparent buffer; medium-risk structures (bile ducts, ureters, etc.) are rendered in orange with a 5mm buffer; and low-risk structures (peritoneum, loose connective tissue) are rendered in green with a 3mm buffer. Rendering uses OpenGL ES 3.0 depth buffering to handle occlusion, with a frame rate of no less than 25fps and an end-to-end rendering latency of no more than 50ms.

[0026] In step S5, the Euclidean distance between the three-dimensional coordinates of the surgical instrument tip (with an electromagnetic tracking accuracy of 1 mm or a visual estimation accuracy of 2 mm) and the nearest safe boundary node (indexed by a kd-tree) under T_comp is calculated in real time with a calculation cycle of no more than 50 ms. If the distance is lower than the first-level threshold (5 to 10 mm), a yellow warning is triggered (the approaching area is highlighted on the interface, and the real-time distance value is displayed); if the distance is lower than the second-level threshold (1 to 5 mm), a red warning and a continuous audio alarm (500 Hz, 0.5 seconds interval) are triggered. The surgeon's end and the circulating nurse's end receive the alarms simultaneously, and each event is recorded with a millisecond timestamp.

[0027] In step S6, using millisecond-precision timestamps as indexes, the instrument end-effector 3D coordinates (mm), nearest safety boundary node coordinates (mm), 3D distance (mm), and warning event type (0 None / 1 Yellow Warning / 2 Red Warning) are recorded and appended to the local database in CSV format (UTF-8, comma-separated), supporting multi-dimensional queries and offline analysis. After passing quality gating (TRE mean not exceeding 1mm, rendering frame rate not less than 25fps, warning delay not exceeding 50ms), the updated 3D anatomical model version (semantic version number + timestamp) is published to the surgical navigation system model repository, retaining the 5 most recent historical versions for rollback support.

[0028] Please see Figure 2 The finite element soft tissue deformation modeling and individualized anatomical priority rendering laparoscopic surgery augmented reality device includes: an anatomical model construction unit 11, a finite element deformation calculation unit 12, a deformation compensation registration unit 13, a priority rendering unit 14, a three-dimensional distance warning unit 15, and a log version management unit 16, which are respectively used to execute steps S1 to S6.

[0029] Please see Figure 3 The electronic device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24, which are connected via a bus. The memory 22 is used to store computer program code, and the processor 21 implements the method when executing the instructions.

[0030] Clinical Application Example 1: Taking laparoscopic cholecystectomy (Calot's triangle anatomical stage) as an example, in this example, the system performs individualized 3D modeling and augmented reality overlay on the cystic duct (high risk, rendered in red), cystic artery (high risk), and common hepatic duct (high risk). The finite element deformation field solution under a pneumoperitoneum pressure of 12 mmHg takes 43 ms. After deformation compensation, the average TRE is 0.87 mm, a 79% reduction compared to the average TRE of 4.2 mm for pure rigid body registration. The closest approach distance of the instruments during surgery is 3.2 mm (triggering a red warning), promptly prompting the surgeon to adjust the operating direction. Experimental results show that the registration error after deformation compensation does not exceed 1 mm, the rendering frame rate is 28 fps, and the warning response delay is 38 ms, all meeting the performance requirements specified in the method steps of this application.

[0031] Clinical Application Example 2: Taking laparoscopic right hemicolectomy (mesenteric vessel management stage) as an example, in this example, the system simultaneously performs priority rendering of the ureter (high risk), superior mesenteric artery and its branches (high risk), and verifies the rendering priority mechanism in a complex scene with dual-risk structures in the same frame. The system automatically places the key anatomical structures of the current exposure stage at the front of the rendering queue to ensure that core safety information is always visible. Experimental results show that the average safety margin throughout the process is 6.8 mm (higher than the 5 mm threshold), no secondary warning is triggered, the model version release passes all quality gating, and all meet the performance index requirements specified in the method steps of this application.

[0032] The method described in this application can be implemented based on hardware, software, or a combination of hardware and software.

Claims

1. An augmented reality method for laparoscopic surgery based on finite element soft tissue deformation modeling and individualized anatomical priority rendering, characterized in that, The steps include: S1. Construct a three-dimensional anatomical model with individualized geometric parameters and risk level annotations based on patient images, extract relevant anatomical structures for the current stage using a surgical stage classifier and sort them by risk level to obtain a set of anatomical structures to be rendered first; S2. Based on the set of anatomical structures to be rendered first, a finite element mesh is constructed. The finite element stiffness equation is solved with pneumoperitoneum pressure, instrument contact force and gravity as loads to obtain the soft tissue deformation field u. S3. Using the ICP algorithm, an initial solution for rigid body registration is established between the feature points of the endoscopic image and the three-dimensional anatomical model. Based on the soft tissue deformation field u, node-by-node deformation compensation is performed on each node of the rigid body registration result. The deformation compensation amount is determined by the displacement value of the corresponding node in the soft tissue deformation field, and the deformation compensation transformation matrix is ​​obtained. S4. Based on the deformation compensation transformation matrix, render a three-dimensional anatomical overlay layer in the laparoscopic field of view. High-risk structures are rendered with red outlines, medium-risk structures are rendered with orange outlines, and low-risk structures are rendered with green outlines. Safety boundaries are marked with color-transparent buffers. S5. Calculate the three-dimensional distance between the surgical instrument tip and the safety boundary in real time. If the distance is lower than the first-level threshold, trigger a first-level warning. If the distance is lower than the second-level threshold, trigger a second-level emergency alarm and output the warning signal to the augmented reality interface. S6. Record instrument coordinates, safety boundary coordinates, 3D distances, and warning events as surgical navigation logs using timestamps as indexes. After passing quality gating, release updated 3D anatomical model versions to the surgical navigation system.

2. The method according to claim 1, characterized in that, In step S1, the individualized geometric parameters include volume, centroid coordinates, and bounding box size; the risk level is divided into three levels: high, medium, and low; and the number of priority rendering structures ranges from 5 to 20.

3. The method according to claim 1, characterized in that, In step S2, the finite element mesh size ranges from 1 to 3 mm; the soft tissue elastic parameters are automatically estimated based on the Hounsfield values ​​of CT images, and the elastic modulus ranges from 1 to 10 kPa; the solution time for each frame does not exceed 50 ms.

4. The method according to claim 1, characterized in that, In step S3, the convergence condition of the ICP algorithm is that the root mean square registration error is less than 0.5 mm; the deformation compensation amount is determined by the displacement value of the corresponding node in the deformation field; and the target registration error is controlled within 1 mm.

5. The method according to claim 1, characterized in that, In step S4, the augmented reality rendering frame rate is no less than 25fps; the radius of the high-risk safety buffer is 10mm, the medium-risk buffer is 5mm, and the low-risk buffer is 3mm; the rendering transparency ranges from 50% to 80%.

6. The method according to claim 1, characterized in that, In step S5, the first-level distance threshold ranges from 5 to 10 mm, triggering a yellow warning; the second-level distance threshold ranges from 1 to 5 mm, triggering a red warning and an audio alarm; the navigation log is stored in CSV format, supporting offline analysis of postoperative accuracy.

7. A finite element soft tissue deformation modeling and individualized anatomical priority rendering augmented reality device for laparoscopic surgery, characterized in that, include: Anatomical model construction unit (11) is used to perform step S1 as described in claim 1; Finite element deformation calculation unit (12) is used to perform step S2 as described in claim 1; deformation compensation registration unit (13) is used to perform step S3 as described in claim 1; priority rendering unit (14) is used to perform step S4 as described in claim 1; three-dimensional distance warning unit (15) is used to perform step S5 as described in claim 1; The log version management unit (16) is used to perform step S6 as described in claim 1.

8. An electronic device, characterized in that, include: A processor and a storage unit for storing computer program code, the code containing computer instructions, wherein when the processor executes these instructions, the electronic device performs the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program containing program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in any one of claims 1 to 6.

Images