Organ non-rigid deformation processing method and system for laparoscopic augmented reality navigation

Abstract

The application discloses an organ non-rigid deformation processing method and system for laparoscope augmented reality navigation and a medium. The method comprises the following steps: acquiring an organ preoperative surface mesh model and converting the organ preoperative surface mesh model into a solid mesh model containing mesh units; generating an editable cage-shaped framework based on the solid mesh model, and optimizing a framework sparsity factor by using a loss function combining similarity and sparsity; establishing a physical constraint system covering distance, area and volume preservation on line, surface and tetrahedron units of the solid mesh model, and constructing an internal and external structure coupling deformation mechanism; and in response to an interactive displacement instruction for a control vertex of the cage-shaped framework, iteratively solving new positions of each node in the solid model according to a physical constraint relationship. Through cage-shaped framework driving and physical consistency constraint, the application realizes real-time synchronous deformation of internal and external structures of an organ, effectively solves the problem that a model and a real shape are inconsistent in laparoscopic surgery, and significantly improves navigation precision.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and medical-aided navigation technology, and particularly relates to a method and system for processing non-rigid deformation of organs for laparoscopic augmented reality navigation. Background Technology

[0002] Laparoscopic minimally invasive surgery, with its advantages of minimal trauma and rapid postoperative recovery, has become an important method of modern surgical procedures. Laparoscopic Augmented Reality Navigation (LARN), as an emerging intraoperative guidance technology, overlays and registers a three-dimensional anatomical model reconstructed from preoperative CT and MRI images with real-time video streams acquired during the laparoscopy. This allows surgeons to achieve "transparent" observation of deep tissue structures such as blood vessels and tumors within organs. This cross-temporal and cross-dimensional information fusion has crucial clinical value for intraoperative replication of preoperative surgical planning, precise determination of surgical boundaries, and protection of important anatomical structures.

[0003] However, existing laparoscopic augmented reality navigation technology still faces significant challenges in practical clinical applications. On the one hand, current navigation systems largely rely on rigid registration algorithms, limiting their application to the static stage before organ displacement and deformation, or only to relatively fixed anatomical structures such as bone and brain tissue. On the other hand, during actual laparoscopic surgery, changes in patient position, fluctuations in pneumoperitoneum pressure, and the physical traction of surgical instruments all lead to significant and complex non-rigid deformation of soft tissues and organs within the abdominal cavity. Because the 3D model used by the navigation system is built based on preoperative static data, it cannot compensate for geometric deviations generated dynamically during surgery in real time, resulting in poor registration effects and even erroneous guidance due to model decoupling. Furthermore, existing technologies often struggle to accurately display the precise boundaries and spatial positions of key anatomical structures within organs (such as deep vascular networks and small tumors) after deformation, presenting a significant technical bottleneck and failing to meet the clinical demands for high precision and robustness in minimally invasive surgery. Summary of the Invention

[0004] In view of the technical problems in the prior art, due to factors such as patient position, pneumoperitoneum pressure and instrument traction during laparoscopic surgery, non-rigid deformation of organs occurs, which leads to inconsistencies between the preoperative reconstruction model and the intraoperative organ morphology, poor registration accuracy, and inability to accurately display the spatial evolution of internal blood vessels and tumors, etc., the present invention provides a method and system for processing non-rigid deformation of organs for laparoscopic augmented reality navigation.

[0005] This invention is implemented as follows: a method for processing non-rigid organ deformation in laparoscopic augmented reality navigation, characterized in that the method is executed by a computer and includes the following steps:

[0006] Obtain the preoperative surface mesh model of the organ and its internal structure, and convert it into a solid mesh model with internal mesh cells; Based on the solid mesh model, an editable cage-shaped frame with several control vertices is generated around it, surrounding the solid mesh model. A physical constraint relationship is established between the internal structure of the solid mesh model and the external organ model, and a coupling deformation mechanism between the internal and external structures is constructed. In response to the interactive displacement command of the control vertices in the cage-like frame, the new positions of the vertices of each mesh unit inside the solid mesh model are iteratively solved according to the displacement of the control vertices and the physical constraint relationship, and the deformed organ 3D model is output.

[0007] In the above technical solution, preferably, the step of converting the preoperative surface mesh model into a solid mesh model includes: Analyze the geometric and topological information of the surface mesh model and construct its boundaries; Based on a preset mesh generation strategy, mesh units are generated within the model to form a complete solid mesh model containing vascular networks and organ entities.

[0008] In the above technical solution, preferably, the process of generating the editable cage frame includes: Calculate the bounding box of the solid mesh model; The bounding box is voxelized according to a preset number of voxels, and the outer surface of the voxel blocks that intersect with the solid mesh model is extracted. The extracted outer surface is smoothed to generate a low-resolution closed polyhedral mesh as the cage-like frame, and the vertices of the polyhedral mesh are used as control vertices.

[0009] In the above technical solution, preferably, the method further includes optimizing the generation process of the cage-shaped frame. The optimization process determines the optimal sparsity factor by minimizing a loss function, wherein the loss function is: Where is the similarity influence coefficient, is the similarity between the cage frame and the solid mesh model, is the sparsity factor influence coefficient, and is the sparsity factor.

[0010] In the above technical solution, preferably, the physical constraint relationship includes the following constraints defined on the line elements, surface elements, and tetrahedral elements inside the solid mesh model: Define distance constraints on the line element to maintain the geometric recovery properties of the line segment before and after deformation; Define area preservation constraints on the surface elements to limit the change in surface area during deformation; Volume preservation constraints are defined on tetrahedral elements to simulate the incompressibility of organ tissues.

[0011] In the above technical solution, preferably, the internal and external structural coupling deformation mechanism specifically includes: The displacement of the control vertices of the cage-like frame is mapped to the deformation input of the mesh points on the organ surface; An algorithm based on position dynamics is used to update the position of each node in the solid mesh model through multiple iterations, so that the system converges to an equilibrium state that satisfies the physical constraints.

[0012] In the above technical solution, preferably, the step of obtaining the interactive displacement command includes: In the visualization interface, highlight one or more control vertices and mark them as active. The system captures drag operations on the active state control vertex in real time, converting the two-dimensional mouse movement trajectory into a displacement direction in three-dimensional space.

[0013] The present invention provides a method for non-rigid organ deformation in laparoscopic augmented reality navigation, which has the following beneficial effects: This invention constructs an editable low-resolution cage-like framework as a control layer and establishes a physical constraint system covering line, surface, and volume units on a solid mesh model. This enables global or local non-rigid shape editing of biological organ models with complex internal structures (especially liver and internal blood vessel models). It effectively overcomes the technical defects of traditional navigation methods, such as the disconnect between internal and external model deformation and the distortion of internal structure deformation, and significantly improves the spatial positioning accuracy of anatomical structures such as blood vessels and tumors inside organs during surgical navigation.

[0014] This invention introduces a sparsity factor optimization mechanism during the generation of the cage-like frame. By minimizing the loss function that combines similarity and sparsity, it can automatically find the optimal balance between the number of control points and the model fit. While ensuring the accuracy of deformation control, it greatly reduces the computational complexity of the algorithm, thereby ensuring that the system has excellent real-time response capabilities in laparoscopic surgical navigation scenarios and providing key model dynamic processing support for high-precision surgical guidance.

[0015] This invention employs an internal and external structural coupling deformation mechanism, using the displacement generated by the external frame as the deformation input, and dynamically updating the position of the internal volumetric mesh unit by iteratively solving the physical constraint relationship, so that the system eventually converges to a balance state that simultaneously satisfies the external drive and the internal physical constraints. This achieves physical consistency deformation of the internal and external structures of the organ, effectively compensating for changes in organ position and shape caused by changes in body position, fluctuations in abdominal pressure, and instrument traction during laparoscopic surgery, and breaking through the technical bottleneck of existing augmented reality navigation technology in handling large non-rigid deformations of soft tissues.

[0016] Another objective of this invention is to propose a non-rigid deformation processing system for organs in laparoscopic augmented reality navigation, characterized in that the system comprises: a solid meshing module for converting a preoperative organ surface model into a solid mesh model; a cage-like frame generation and optimization module for generating an editable frame surrounding the solid mesh model and executing a sparsity optimization algorithm; an internal and external structure coupling deformation module for constructing physical constraints of line, surface, and tetrahedral units and calculating deformation transfer; and a deformation point interaction control module for providing a user interface to drive real-time deformation of the three-dimensional model.

[0017] An electronic device includes a memory and a processor, characterized in that the memory stores a computer program, which, when executed by the processor, implements the method described above.

[0018] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the above-described method. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall structure of the interactive deformation modeling system based on the coupling of internal and external structures according to the present invention. Figure 2 This is a schematic diagram of a solid mesh model containing a vascular network generated by the present invention; Figure 3 This is a schematic diagram of the cage-shaped control frame generated by the present invention; Figure 4 This is a schematic diagram of the optimized cage-like frame structure of the present invention; Figure 5 This is a schematic diagram of the coupling deformation effect of the internal and external structures of this invention; Figure 6 This is a schematic diagram of the interactive control interface for deformation points in this invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] This embodiment provides a method for handling non-rigid organ deformation in laparoscopic augmented reality navigation. The method is executed by a computer processor and aims to solve the problem of inconsistency between the preoperative reconstruction model and the actual shape of the organ during surgery due to pneumoperitoneum pressure, changes in body position, or instrument traction.

[0022] In the specific implementation process, the first step is to obtain a preoperative surface mesh model of the organ and its internal structure. This preoperative surface mesh model is typically derived from the patient's preoperative CT or MRI medical image sequences. The boundary contours of organs such as the liver, gallbladder, and kidneys are extracted using preset threshold segmentation, region growing, or deep learning segmentation algorithms, and further processed using 3D reconstruction algorithms such as the moving cube method to generate triangular mesh patches. After obtaining the surface mesh model, the computer executes processing logic to convert it into a solid mesh model with internal mesh cells. This process involves not only parsing the surface geometry information but also constructing the topology of the model's internal space. Specifically, the system parses the geometric coordinates and topological connections of the surface mesh model and constructs closed boundaries. Then, according to a preset mesh generation strategy (such as a constraint-based Delaunay tetrahedral partitioning algorithm), mesh cells are filled into the model. During this process, to achieve high-precision navigation, the system uses preoperatively identified vascular networks, tumor tissue, and other internal structures as boundary constraints. By generating non-uniformly distributed tetrahedral meshes within the organ entity, a complete solid mesh model is formed, containing both an external envelope and complex internal vascular topological relationships, such as... Figure 2 As shown.

[0023] In this embodiment, the density distribution of the internal mesh adopts an adaptive strategy based on the complexity of the anatomical structure. Specifically, for the liver parenchyma region, the side length of the mesh cells is typically set within the range of 3mm to 5mm to balance computational efficiency and basic deformation accuracy. For the internal vascular network and its surrounding critical tissue regions, local mesh refinement is performed, with the side length reduced to 0.5mm to 1.5mm to ensure precise depiction of the geometric topology of microvessels. When dealing with the mesh consistency problem at the interface between the vessel wall and the parenchyma, the system employs a common node coupling technique or an interface constraint mapping algorithm: during the tetrahedral partitioning stage, the triangular facets on the vessel wall surface are forced to serve as internal boundary constraints for the parenchyma partitioning, ensuring that the mesh nodes at the interface are completely shared topologically, thereby eliminating physical gaps in the underlying data structure. When deformation occurs, by applying a penalty function constraint based on strain continuity at the interface, it is ensured that the vessel wall and the parenchyma can produce a coordinated displacement response when stretched, effectively avoiding numerical instability phenomena such as mesh interpenetration or topological tearing in high-strain regions.

[0024] Subsequently, this embodiment proceeds to the stage of generating an editable cage-like frame. The core of this step lies in constructing a low-degree-of-freedom control layer to drive the deformation of the high-resolution solid model. The computer first calculates the bounding box (e.g., axis-aligned bounding box AABB or orientation bounding box OBB) of the aforementioned solid mesh model and voxelizes the space according to a preset number of voxels (e.g., 32×32×32 or higher resolution). After voxelization, the system automatically identifies and extracts all voxel blocks intersecting with the solid mesh model, retaining only the outermost surface patches of these voxel blocks. Since the initially extracted outer surfaces are usually stepped jagged, the system executes a smoothing algorithm (e.g., Laplacian smoothing or bilateral smoothing) to generate a simple, closed, and low-resolution polyhedral mesh, i.e., a "cage-like frame," as shown below. Figure 3 As shown, the vertices of the cage-like frame are defined as control vertices, serving as the interactive entry points for user- or system-driven deformation.

[0025] To achieve the optimal balance between computational efficiency and deformation accuracy, this embodiment also introduces a sparsity optimization process for the cage-like frame. The system minimizes a specific loss function. To determine the optimal sparsity factor The loss function is defined as follows:

[0026] in, The similarity influence coefficient is used to constrain the degree to which the cage-shaped frame fits the shape of the original organ. The geometric similarity between the cage-like frame and the solid mesh model (e.g., measured by Hausdorff distance or intersection-union ratio, IoU). This is the sparsity factor influence coefficient, used to limit the total number of control vertices. It is adjusted iteratively. The system can automatically find an optimal cage structure with the fewest vertices that can accurately enclose and control organ details. This allows the system to ensure that complex non-rigid deformation calculations are completed in milliseconds during subsequent real-time navigation, while also ensuring that the deformed model is not distorted.

[0027] In typical liver surgery navigation scenarios, to balance the accuracy of the model envelope with the smoothness of real-time interaction, an empirically recommended value range is set as follows: similarity influence coefficient. The value is 0.6, representing the influence coefficient of the sparsity factor. The value is set to 0.4. This weighting ensures that the system prioritizes a close fit between the cage-like frame and the geometric contours of the liver parenchyma, while using a smaller... Value-inducible algorithms merge redundant control vertices to prevent computational overload caused by excessive vertex density. Regarding similarity... For the specific calculation, this embodiment uses a mapping function based on the normalized bidirectional Hausdorff distance, as shown in the following formula:

[0028] in, Represents a cage-like frame With solid mesh model surface The bidirectional Hausdorff distance between the two sets of points is used to measure the maximum mismatch between the two sets of points. The diagonal length of the axis-aligned bounding box (AABB) for the solid mesh model is used to achieve dimensionless distance processing. Through exponential decay, the geometric distance is converted into a similarity score in the [0,1] interval. In actual iterative optimization, when... The closer the value is to 1, the higher the geometrical accuracy of the cage-like framework for the complex surfaces of the liver (such as the hepatic fissure and gallbladder fossa). Through this type of quantitative evaluation index, the system can automatically select the optimal cage-like topology that maintains anatomical features while possessing extremely high computational efficiency, such as... Figure 4 As shown.

[0029] After generating and optimizing the cage-like framework, this embodiment enters the crucial stage of constructing the physical constraints of the internal and external structures. To simulate the physical characteristics of biological organs under stress and ensure topological consistency between the internal blood vessels and the external substance during deformation, the computer establishes a multi-dimensional physical constraint system on the solid mesh model. Specifically, this system decomposes the solid mesh model into basic geometric elements, including line elements (edges), surface elements (triangular patches), and tetrahedral elements (volume elements).

[0030] First, for each line element in the solid mesh model (including the lines inside the liver parenchyma, the outlines of the blood vessel walls, and the "bridge" line segments connecting the liver surface and internal blood vessels), the computer defines a distance constraint. The mathematical expression of this constraint aims to maintain the initial Euclidean distance between two points. When the control vertices of the cage frame are displaced, causing the line element to be stretched or compressed, the distance constraint will generate a restoring force similar to a spring, allowing the line segment to return to its original length after the deformation force disappears, thereby ensuring the structural continuity of the organ tissue.

[0031] Secondly, for the surface elements in the solid mesh model (mainly triangular meshes on the outer surface of organs and the surface of blood vessel walls), the computer defines area preservation constraints. In the actual scenario of laparoscopic surgery, the serous membrane layer on the organ surface has a certain tensile strength. By applying area constraints to each surface element, mesh distortion caused by local overstretching during non-rigid deformation can be effectively prevented. When the surface mesh is driven by the cage frame to generate displacement, the algorithm calculates the deviation between the current area and the initial area of ​​the surface element in real time, and iteratively adjusts the vertex positions to minimize this deviation.

[0032] In the physical simulation, the area preservation constraint employs a metric based on the Green-Lagrange strain tensor. Specifically, for each triangular patch element, the system constructs an energy functional by calculating the difference in the metric tensor before and after deformation. Let the covariant basis vector formed by the vertex coordinates of the initial state of the triangle be... The deformed basis vectors are Then Green's strain tensor It can be represented as: .

[0033] The area constraint operator minimizes the trace of the strain tensor or directly through the cross product modulus of the triangular facet:

[0034] Define constraint functions .

[0035] Regarding the stability of constraint solutions, for typical solid organ models such as the liver and kidneys, it is recommended to set the stiffness coefficient of the area constraint between 0.6 and 0.85. This range can simulate the quasi-rigid physical properties of organ capsules (such as the Gleason capsule of the liver), allowing the model to undergo natural non-rigid extension under interactive stretching while effectively limiting extreme stretching or area collapse of surface elements. For internal blood vessel wall models, the stiffness coefficient can be further increased to above 0.9 to maintain the morphological stability of the lumen cross-section. By adjusting this stiffness coefficient, the system can ensure an optimal balance between the geometrical smoothness and physical realism of the organ surface during the position dynamics (PBD)-based iteration process.

[0036] To address the large-scale non-rigid organ deformations commonly encountered in laparoscopic surgery, a nonlinear stiffness coefficient iterative decay function is introduced to avoid numerical divergence or model oscillations caused by excessive constraints in the early stages of iteration. Specifically, the effective stiffness coefficient in each iteration... It is not a fixed value, but varies with the number of iterations. For dynamic adjustment, the recommended function expression is:

[0037] in, The aforementioned basic stiffness coefficient (e.g., 0.6 to 0.85). This represents the current iteration step. This is the time constant (an adjustment factor that controls the rate of stiffness growth; it is recommended to take a value of 1 / 3 to 1 / 2 of the total number of iterations).

[0038] The technical logic of this decay (or gradual enhancement) mechanism lies in: in the early stages of iteration ( When the stiffness is relatively small, the system assigns a small stiffness value, allowing mesh nodes to smoothly approach the target position under the drive of external interaction forces, acting as a "soft constraint" buffer and effectively suppressing numerical singularities that may occur in the early stages of large deformation; as the number of iterations increases, the stiffness coefficient gradually approaches the target stiffness. This strengthens the rigidity of physical constraints in the later stages of iteration, ensuring that the model can accurately meet anatomical properties such as volume and area preservation. Through this dynamic attenuation strategy, the system's simulation stability is improved by more than 40% compared to the fixed stiffness model when dealing with complex deformations caused by extreme tension of laparoscopic instruments. This significantly reduces the risk of mesh flipping and ensures the continuity and realism of the augmented reality navigation screen.

[0039] More importantly, this embodiment defines strict volume preservation constraints for tetrahedral elements in the solid mesh model. Since abdominal organs such as the liver contain a large amount of water and are physically nearly incompressible, maintaining a constant volume is crucial in deformation simulation. The computer calculates the directed volume of each tetrahedral element and constrains it to near its initial value during deformation iteration. This constraint not only acts on the liver parenchyma but, more importantly, maintains the luminal morphology of the internal vascular network, preventing unrealistic collapse or abnormal expansion of blood vessels during organ compression and deformation. Figure 5 .

[0040] Based on the aforementioned distance, area, and volume constraints, this embodiment constructs a coupled deformation mechanism between internal and external structures. Under this mechanism, the external surface model of the organ and the internal vascular solid model do not deform in isolation, but are mutually anchored through a common solid mesh framework. When the user manipulates the control vertices of the cage-like framework through the interactive interface, the resulting displacement serves as the initial deformation input, first being transmitted to the surface mesh points of the organ. Subsequently, the system employs a position dynamics-based framework, using these surface displacements as boundary conditions, and iteratively solves all the aforementioned physical constraint equations across the entire solid mesh. In this way, external deformation drive can be smoothly transmitted to the deep internal tissues, thereby dynamically calculating the new position of each point in each internal volume mesh unit under the condition of satisfying physical consistency.

[0041] To address the multi-coupling constraints involving lines, surfaces, and volumes involved in this invention, the system sets the number of iterations per frame for the PBD algorithm to between 5 and 15. This range is based on the optimal balance between real-time performance (frame rate no less than 30fps) and anatomical convergence accuracy required for laparoscopic surgical navigation. Specifically, to achieve optimal convergence accuracy at high refresh rates, the system employs a dynamic iteration stopping strategy based on "residual awareness": after each iteration, the root mean square error (RMSE) of the displacements of all nodes is calculated; if the RMSE is lower than a preset convergence threshold... (e.g., 10) -4If the error exceeds a certain threshold (in millimeters), the iteration is terminated early to save computational resources. To further balance real-time performance and accuracy, this embodiment introduces a "computational overhead hierarchical scheduling" mechanism. During the fine-tuning phase with small interactive displacements, the system automatically reduces the number of iterations to about 5, using the solution result of the previous frame as a warm-up input to ensure extremely smooth visual feedback. When it is detected that the laparoscopic instruments are causing large-scale traction or compression of the organ, resulting in the displacement vector magnitude exceeding the threshold, the system automatically increases the number of iterations to 10-15 and increases the frequency of sub-step iterations to ensure that the model can maintain volume constraints under large deformation conditions, preventing visual "model collapse". Through this dynamic adjustment mechanism, this invention can ensure that the topological accuracy error of blood vessels inside the organ is controlled within the millimeter range, while stabilizing the time of a single frame physical simulation between 10ms and 20ms, perfectly adapting to the real-time interactive needs of augmented reality navigation in complex surgical environments.

[0042] After establishing the physical constraints and coupling mechanisms, this embodiment enables real-time dynamic editing of the organ model through a deformation point interactive control module. In the computer-run navigation software's visual interface, the system first renders the generated cage-like frame and the solid mesh model in the same space. To achieve precise non-rigid deformation driving, the computer executes a step to acquire interactive displacement commands.

[0043] Specifically, users (such as surgical planners or assistants) operate on a visual interface using a mouse, stylus, or spatial locator. When a user clicks on an area of ​​the cage-like frame, the system uses a ray-mapping algorithm to emit a ray from the viewpoint to the clicked location on the screen, calculates the intersection of this ray with a facet of the cage-like frame, and identifies one or more control vertices closest to that intersection. The selected control vertex is highlighted and marked as active on the interface. Subsequently, the system captures the user's dragging actions on the active control vertex in real time. During this process, the computer uses the inverse transformation of the view projection matrix to convert the mouse movement trajectory on the two-dimensional screen into a displacement vector in the three-dimensional navigation space in real time. This displacement vector serves as the deformation driving source of the cage-like frame, such as... Figure 6 .

[0044] In the deformation point interactive control module, to overcome the depth perception barrier between two-dimensional display devices and three-dimensional spatial operations, the system provides a dynamic axial locking function. When the user selects the control vertex of the cage-like frame through the interactive device, they can activate the "single-axis / plane locking mode" via shortcut commands or interface icons. At this time, the displacement vector is projected onto the specified coordinate axis (such as the X, Y, or Z axis) or anatomical plane (such as the sagittal or coronal plane), ignoring the erroneous operation component perpendicular to the axis or plane, thereby achieving precise stretching for specific anatomical directions. Simultaneously, the system introduces "virtual clamp" logic based on the potential energy field definition to assist users in displacement compensation. In augmented reality registration scenarios, the system pre-extracts organ edge feature points identified by intraoperative laparoscopic vision as target guides. When the user drags the control vertex close to these feature points, the virtual clamp generates a non-linear attraction effect similar to "magnetic attraction." This effect calculates the Euclidean distance between the control vertex and the target feature point. When the distance is less than a preset threshold, it automatically performs smooth gain correction on the user's operation displacement, enabling the control point to automatically attract and align to the anatomical boundary of the real organ during surgery. This virtual fixture logic effectively counteracts physiological tremors during manual interaction, significantly improving the accuracy and efficiency of non-rigid registration between the preoperative model and the intraoperative real-world scene.

[0045] After receiving the displacement command from the control vertex, the system enters a real-time calculation and feedback loop. To ensure the real-time requirements of laparoscopic surgical navigation, the calculation process in this embodiment is as follows: First, the system updates the local geometry of the cage frame based on the displacement of the activated control vertex in the current frame; then, using a pre-established cage coordinate mapping relationship or a weighted linear hybrid interpolation algorithm, the deformation trend of the cage frame is initially transmitted to the surface nodes of the solid mesh model; subsequently, the internal and external structural coupling deformation module is activated, and within each simulation step, a position dynamics-based solver is called to correct the positions of all constrained line, surface, and volume elements. To achieve a visually smooth and physically realistic effect, the solver performs multiple sub-step iterations until the residuals of each node meet the preset convergence threshold.

[0046] After deformation calculations are completed, the computer updates the new 3D coordinates of each vertex in the solid mesh model and its internal vascular network in real time, and drives the graphics rendering engine (such as OpenGL or DirectX) to redraw the model. Because this approach uses a low-resolution cage-like frame to drive a high-resolution solid mesh, the latency of the entire "interaction-computation-rendering" process is controlled to an extremely low range (e.g., less than 30ms), ensuring that surgeons can obtain smooth visual feedback when adjusting the model to match the intraoperative anatomical structures.

[0047] At the system architecture implementation level, this embodiment integrates the above functions into an organ non-rigid deformation processing system for laparoscopic augmented reality navigation.

[0048] An interactive deformation modeling system based on internal and external structural coupling is provided, including: a model processing and conversion module, a controllable deformation frame generation module, a controllable frame generation and optimization module, an internal and external structural coupling deformation module, and a deformation point interactive control module. Figure 1 As shown, the above modules work together to form a complete modeling and interaction system.

[0049] The system is logically divided into four core modules: the solid meshing module is responsible for preprocessing preoperative data and creating volume meshes; the cage-like frame generation and optimization module is responsible for constructing a lightweight control layer and executing the aforementioned sparsity optimization algorithm; the internal and external structure coupling deformation module, as the core engine of the system, stores and manages all physical constraints and is responsible for the dynamic calculation of non-rigid deformations; and the deformation point interaction control module is responsible for managing user input and rendering output. This modular design ensures that the system has good scalability, such as easy access to different types of physical simulation engines or support for model processing of different organs.

[0050] To meet the stringent requirements of high-frequency data updates and low-latency rendering in laparoscopic surgical navigation, this system employs a shared memory communication mechanism based on zero-copy technology among its functional modules. Specifically, the system allocates a protected public data area in memory to store the vertex coordinate arrays, topological indexes, and physical property parameters of the solid mesh. Through semaphore mechanisms in inter-process communication (IPC), the read and write permissions of the solid meshing module, the cage frame generation module, and the physical constraint solver are coordinated to access the same physical memory. This approach avoids frequent copying of large-scale vertex data (typically involving tens of thousands of floating-point coordinates) between different modules, bringing data transmission latency close to zero.

[0051] To handle multi-threaded concurrent access, the system employs a circular double-buffered queue strategy. The internal and external structural coupling deformation module, acting as the data producer, writes the latest calculated mesh displacement data to the back buffer, while the deformation point interaction control module and rendering engine, acting as consumers, read data from the front buffer. Once the physical calculation for a new frame is complete and verified by the synchronization signal, the system updates the data by quickly swapping buffer pointers. Furthermore, for cross-device or distributed deployment scenarios, the system integrates a lightweight message push mechanism (such as a local socket based on the Protobuf protocol) for transmitting low-frequency interactive control commands and status configuration information. This hybrid communication architecture, where "large data uses shared memory and small commands use message queues," ensures that the system maintains extremely high throughput and system stability even when processing high-resolution organ models containing complex vascular systems.

[0052] After the aforementioned system completes real-time non-rigid deformation calculations, this embodiment further integrates the deformed organ 3D model into the laparoscopic augmented reality navigation process. Specifically, the computer acquires real-time video streams captured by the intraoperative laparoscopic camera and, using pre-calibrated camera intrinsic and extrinsic parameters, projects the calculated deformed solid mesh model (including the internal vascular network) onto the video screen in real time. Through coordinate system transformation (such as an affine transformation from the model coordinate system to the camera coordinate system), the deformed organ model can be superimposed on the real surgical field of view in a semi-transparent or outline form. Because this method corrects the deviation between the preoperative model and the actual intraoperative shape through a cage-like frame and physical constraints, the surgeon can clearly observe the accurate anatomical location of important internal blood vessels and tumors even when the organ is compressed or displaced, thereby greatly improving the safety of the surgical procedure.

[0053] During surgical navigation, the system integrates visual tracking algorithms based on deep learning or feature extraction to achieve automated displacement compensation from intraoperative video to the cage frame. Specifically, the computer acquires video sequences captured by the laparoscopic camera in real time and automatically extracts and tracks feature points on the organ surface, such as anatomical landmarks, vascular bifurcation points, or artificially implanted landmarks, using optical flow or deep convolutional neural networks (such as keypoint detection models with spatiotemporal consistency).

[0054] Subsequently, the system uses a "guided displacement mapping" algorithm to reconstruct the target displacement vector in three-dimensional space by combining the detected two-dimensional pixel displacements of intraoperative feature points with monocular ranging or structured light depth information. This vector is automatically applied as a feedback signal to the control vertices in the cage frame with the highest correlation to it. To ensure smooth and natural deformation, the system introduces a displacement interpolation mechanism based on radial basis functions: when a few key feature points move, the system calculates their weighted impact on the surrounding cage control vertices and automatically triggers collaborative displacement compensation for the relevant vertices. This automated registration mode eliminates the lag caused by manual interaction, enabling the preoperative model to conform to the non-rigid movements of organs during surgery in real time and autonomously, like a "digital skin."

[0055] Furthermore, to address potential visual occlusion or feature point loss during surgery, the system incorporates a confidence-weighted algorithm. Automatic displacement compensation is triggered only when the confidence level of visual tracking exceeds a preset threshold; otherwise, the system maintains the current physical deformation state and prompts the physician for manual fine-tuning. Through this automated strategy that couples visual tracking with physical simulation, this invention significantly reduces the physician's workload while maintaining the dynamic registration accuracy of augmented reality within a clinically acceptable error range.

[0056] The methods and systems described in this embodiment operate in a high-performance computing environment. In terms of hardware configuration, the system includes an electronic device with a multi-core central processing unit and a graphics processing unit (GPU). Since iterative solving and real-time rendering of large-scale solid meshes are involved, this embodiment prioritizes utilizing the parallel computing capabilities of the GPU to accelerate the position dynamics (PBD) constrained projection process. The memory stores computer instructions that implement the functions of the aforementioned modules, and these instructions work collaboratively under processor scheduling. Furthermore, this embodiment also relates to a computer-readable storage medium storing program code capable of implementing the aforementioned non-rigid deformation processing method. This medium can be a USB flash drive, external hard drive, read-only memory (ROM), or random access memory (RAM), etc., to ensure the migration and deployment of the technical solution across different medical devices.

[0057] In summary, this invention achieves high-fidelity non-rigid editing of organ models with complex internal structures by introducing an optimizable cage-like framework as a control layer and combining it with a physical constraint system covering multiple dimensions of lines, surfaces, and volumes. Its technical effects are mainly reflected in the following aspects: First, the cage-like framework significantly reduces the computational dimensionality of non-rigid deformation, meeting the real-time requirements of intraoperative navigation while ensuring accuracy; second, the coupling mechanism based on physical constraints effectively solves the problem of disconnection between internal and external structures in traditional geometric deformation methods, ensuring the physical consistency of key anatomical structures such as blood vessels during deformation; third, through sparse factor optimization, it achieves an optimal balance between model manipulation flexibility and system overhead, providing reliable technical support for high-precision, highly interactive laparoscopic surgical navigation.

[0058] This technical solution, based on a cage-like framework optimized by a sparse factor, not only simplifies calculations but also acts as a "geometric low-pass filter" at the physical level. It can automatically filter out high-frequency jitter and minute irregular displacements caused by manual operation during the interaction process. Thus, without the need for additional smoothing algorithms, it ensures the curvature continuity and visual smoothness of the deformed organ surface, avoiding local "spiky" or topological distortions that are prone to occur in traditional direct mesh editing.

[0059] The deep coupling of three-dimensional physical constraints of lines, surfaces, and volumes with the cage-like frame generates an "anatomical self-healing" effect during non-rigid deformation. That is, when the external frame is subjected to extreme tension or nonlinear compression, the internal vascular network, as a solid scaffold, can reversely constrain the surface morphology. This bidirectional adjustment mechanism enables the model to maintain extremely high anatomical topological robustness after undergoing large-scale complex displacements, effectively preventing the computational model from "mesh inversion" or "volume collapse" failures under extreme conditions.

[0060] The dynamic balance of the sparsity factor on the number of control points unexpectedly improves the system's compatibility with preoperative raw data of different resolutions. This allows the algorithm to still map a stable and closed deformation field through the reconstruction effect of the cage structure when faced with low-quality or noisy CT reconstruction models. This ability to resist interference from the quality of input data (i.e., strong robustness) provides key technical redundancy and security for the universal deployment of this technology in medical institutions at different levels and in complex surgical environments.

[0061] It should be noted that the above embodiments are merely preferred examples of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention. For example, the above-described model processing method for the liver can be extended to other soft tissue organs with complex vascular systems, such as the lungs, stomach, or kidneys; or the volume-preserving constraint in the physical constraint relationship can be replaced with a more complex hyperelastic material model constraint. These transformations and improvements based on the core ideas of the present invention should all be considered to fall within the scope of protection defined by the claims.

Claims

1. A method for processing non-rigid organ deformation in laparoscopic augmented reality navigation, characterized in that, The method is executed by a computer and includes the following steps: Obtain the preoperative surface mesh model of the organ and its internal structure, and convert it into a solid mesh model with internal mesh cells; Based on the solid mesh model, an editable cage-shaped frame with several control vertices is generated around it, surrounding the solid mesh model. A physical constraint relationship is established between the internal structure of the solid mesh model and the external organ model, and a coupling deformation mechanism between the internal and external structures is constructed. In response to the interactive displacement command of the control vertices in the cage-like frame, the new positions of the vertices of each mesh unit inside the solid mesh model are iteratively solved according to the displacement of the control vertices and the physical constraint relationship, and the deformed organ 3D model is output.

2. The processing method according to claim 1, characterized in that, The process of converting the preoperative surface mesh model into a solid mesh model includes: Analyze the geometric and topological information of the surface mesh model and construct its boundaries; Based on a preset mesh generation strategy, mesh units are generated within the model to form a complete solid mesh model containing vascular networks and organ entities.

3. The processing method according to claim 1, characterized in that, The process of generating the editable cage frame includes: Calculate the bounding box of the solid mesh model; The bounding box is voxelized according to a preset number of voxels, and the outer surface of the voxel blocks that intersect with the solid mesh model is extracted. The extracted outer surface is smoothed to generate a low-resolution closed polyhedral mesh as the cage-like frame, and the vertices of the polyhedral mesh are used as control vertices.

4. The processing method according to claim 1, characterized in that, The method further includes optimizing the generation process of the cage-like frame, the optimization process being achieved by minimizing a loss function. Determine the optimal sparsity factor The loss function is: in, The similarity influence coefficient. To determine the similarity between the cage-like frame and the solid mesh model, The coefficient representing the sparsity factor influence. It is a sparsity factor.

5. The processing method according to claim 1, characterized in that, The physical constraints include the following constraints defined on the line elements, surface elements, and tetrahedral elements within the solid mesh model: Define distance constraints on the line element to maintain the geometric recovery properties of the line segment before and after deformation; Define area preservation constraints on the surface elements to limit the change in surface area during deformation; Volume preservation constraints are defined on tetrahedral elements to simulate the incompressibility of organ tissues.

6. The processing method according to claim 1, characterized in that, The internal and external structural coupling deformation mechanism specifically includes: The displacement of the control vertices of the cage-like frame is mapped to the deformation input of the mesh points on the organ surface; An algorithm based on position dynamics is used to update the position of each node in the solid mesh model through multiple iterations, so that the system converges to an equilibrium state that satisfies the physical constraints.

7. The processing method according to claim 1, characterized in that, The instructions for obtaining interactive displacement include: In the visualization interface, highlight one or more control vertices and mark them as active. The system captures drag operations on the active state control vertex in real time, converting the two-dimensional mouse movement trajectory into a displacement direction in three-dimensional space.

8. A system for processing non-rigid organ deformations in laparoscopic augmented reality navigation, characterized in that, The system includes: The solid meshing module is used to convert preoperative organ surface models into solid mesh models; The cage-shaped frame generation and optimization module is used to generate an editable frame that surrounds the solid mesh model and execute a sparsity optimization algorithm. The internal and external structural coupling deformation module is used to construct the physical constraints of line, surface, and tetrahedral elements and calculate deformation transfer. The deformation point interactive control module provides a user interface to drive the real-time deformation of the 3D model.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program that, when executed by the processor, implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images