Individualized virtual laparoscopic surgery robot platform and device

Abstract

The invention discloses an individualized virtual laparoscopic surgery robot platform and device, and belongs to the technical field of medical robots. The platform is constructed based on clinical specific patient disease features and individualized anatomical features, and comprises an individualized three-dimensional bionic model reconstruction module which generates an individualized three-dimensional bionic model according to clinical images and constructs a specific robot patient; the mechanical arm array module is used for simulating the operation of a laparoscopic surgery; the virtual operation feedback module is used for providing real-time force feedback and visual feedback; and the interactive updating and storage export module is used for switching the visual angle of the virtual bionic organ in real time through a high-resolution and focal-length-adjustable virtual laparoscope lens, interactively cooperating with the above parts, operating the mechanical arm array module and controlling the updating of the individualized three-dimensional bionic model. The virtual laparoscopic surgery robot platform for individualized cases has a unique multi-module interaction and system integration strategy, and highly real and individualized surgery simulation is achieved.

Description

Technical Field

[0001] This application relates to the technical field of medical robots, specifically an individualized virtual laparoscopic surgery robot platform and device. Background Art

[0002] As a minimally invasive surgical method, laparoscopic surgery has advantages such as small trauma and fast recovery. However, the complexity of the surgery requires highly precise surgical planning and training. Existing simulators for laparoscopic surgery planning and training lack individualized functions and can only perform simulation operations in the same abdominal cavity scenario, unable to make personalized adjustments according to the specific anatomical structure and pathological characteristics of the patient. This means that the training and planning may not fully simulate the complex situations encountered in actual surgery, limiting the potential of the simulator in improving surgical accuracy and safety. Moreover, many existing simulators lack a real-time feedback mechanism and cannot simulate the impact of surgical operations on the patient's internal environment. This feedback is crucial for training doctors to make quick decisions and adapt to unforeseen situations during surgery. The inability to provide individualized real-time feedback and sufficient interactivity limits its application in surgical training and rehearsal.

[0003] The Chinese patent application document with the publication number CN112989449A discloses a tactile force feedback simulation interaction method and device with optimized motion stiffness, which simulates two three-dimensional geometric models with coincident initial centroid positions, including a static model and a motion model; moves the motion model to continuously collide with the static model, and based on the local contact force optimization principle, assigns a predetermined vector direction to each collision and separation between the models, and further combines the penetration depth distance calculation method to calculate the minimum distance in the predetermined vector direction when the models collide and separate each time; constructs a dynamic control model associated with elastic stiffness, damping constant, and the minimum distance, and substitutes the minimum distance into the dynamic control model to obtain the magnitude of the force applied to the motion model each time. By introducing a variable virtual stiffness non-linear optimization algorithm for different geometric models, stable rigid body model tactile force feedback simulation interaction operations are realized. However, this method still has some deficiencies in real-time calculation and feedback, as well as the overall coordination control of the system platform, and cannot meet the requirements of virtual laparoscopic surgery simulation.

[0004] Therefore, there is an urgent need to develop a new virtual laparoscopic surgery robot platform to meet the real-time calculation and feedback requirements of virtual laparoscopic surgery simulation. Summary of the Invention

[0005] This application aims to solve at least one of the technical problems in the related technologies to some extent. For this purpose, this application provides an individualized virtual laparoscopic surgery robot platform and device, and its unique multi-module interaction and system integration strategy realizes highly realistic and individualized surgical simulation.

[0006] To achieve the above object, in a first aspect, the present application provides a virtual laparoscopic surgery robot platform, including:

[0007] An individualized three-dimensional bionic model reconstruction module, configured to generate an individualized three-dimensional bionic model according to clinical medical imaging data;

[0008] A robotic arm array module, including at least three independent robotic arms, configured to simulate the operations of laparoscopic surgery;

[0009] A virtual surgery feedback module, configured to cooperate with the robotic arm array module to provide real-time force feedback and visual feedback;

[0010] An interaction module, configured to control the coordinated cooperation of the individualized three-dimensional bionic model reconstruction module, the robotic arm array module, and the virtual surgery feedback module, update the individualized three-dimensional bionic model accordingly according to the operations of the robotic arm array module, and control the virtual surgery feedback module to update the visual feedback result.

[0011] Preferably, the robotic arm array module configures an adaptive gain adjustment strategy for torque control and speed control for each robotic arm, and the adaptive gain adjustment strategy can enable the PID controller of each robotic arm to dynamically adjust the gain parameters according to the real-time error and system state.

[0012] Preferably, the formula expression of the adaptive gain adjustment strategy is as follows:

[0013]

[0014] wherein, u(t) is the control input, the torque or speed adjustment signal; t is the time point, e(t) is the error between the set point and the current position, K p (t), K i (t), K d (t) are the proportional, integral, and differential gain parameters respectively, defined with time and system state as independent variables as follows:

[0015]

[0016] K i (t) = K i0 + a i · f i (e(t), ∫e(t)dt, S(t));

[0017]

[0018] wherein, K p0 , K i0 , K d0is the initial gain corresponding to proportional, integral, and derivative; a p , a i , a d are the adjustment coefficients corresponding to proportional, integral, and derivative, used to control the amplitude of adaptive adjustment; f p , f i , f d is the adaptive function; S(t) represents the current state information, which is the virtual surgical scene parameter.

[0019] Preferably, the robotic arm array module installs joint torque sensors at the joints of each robotic arm to capture angle changes, installs piezoelectric sensors to capture the force when contacting the virtual organ, and installs acceleration sensors to monitor the acceleration of the movement. The robotic arm includes at least three rotational degrees of freedom and one linear degree of freedom; the robotic arm array module applies the Kalman filtering algorithm to fuse data from different sensors. Among them, the update formula of the Kalman filter is:

[0020] x k|k = x k|k-1 + K k (Z k - H k x k|k-1 );

[0021] x k|k is the state estimate updated at the k-th time point, x k|k -1 is the estimated value at the previous time point, Z k is the observed value, H k is the observation model matrix, K k is the Kalman gain. The sensor data is integrated into the state estimate through the observation model matrix H k for dynamic system state estimation of the movement of the robotic arm.

[0022] Preferably, the robotic arm array module calculates the force when the end of the robotic arm contacts the virtual organ using the force model calculation formula. The force model calculation formula is:

[0023] F = k·Δx;

[0024] Among them, F is the contact force, Δx is the deformation amount, and k is the elastic constant. The value of k is dynamically adjusted according to the physical properties of different virtual organs;

[0025] Adopts a force feedback algorithm based on impedance control to enable the operator to feel the force matching the interaction in the virtual environment. The force feedback algorithm formula is expressed as:

[0026]

[0027] Among them, F dis the desired force, x d , are the desired position, velocity, and acceleration, respectively, and x, are the actual position, velocity, and acceleration, and K d , B d , M d are the control stiffness, damping, and mass parameters, respectively.

[0028] Preferably, an affine transformation is used to convert the physical coordinate system and the virtual environment coordinate system of the robotic arm:

[0029]

[0030] where (x, y, z) is the position of the end point of the robotic arm in the physical coordinate system, and (x′, y′, z′) is the position in the virtual coordinate system after conversion. The conversion matrix includes the rotation angle R, the scaling S, and the translation T. Among them, t x , t y , t z are the translation amounts along the x, y, and z axes, respectively.

[0031] Preferably, the virtual laparoscopic surgical robot platform further includes a system integration module. The system integration module uses timestamp and buffer management technologies to align data streams from different sources and realizes synchronous processing of data of the individualized three-dimensional bionic model reconstruction module, the robotic arm array module, and the virtual surgery feedback module.

[0032] Preferably, the system integration module deploys a GPU-based computing unit to support high-speed data processing and complex signal operations;

[0033] The system integration module sets an embedded real-time operating system to manage the priorities and scheduling of tasks to ensure that critical tasks with high priorities are completed within time limits. The critical tasks with high priorities include signal processing and feedback control; the scheduling algorithm of the embedded real-time operating system is set to cyclic scheduling.

[0034] Preferably, the individualized three-dimensional bionic model reconstruction module uses the Loop subdivision algorithm to refine the individualized three-dimensional bionic model. Without changing the topological structure of the tissue organs, the smoothness and details of the abdominal organ model are improved by increasing the number of vertices and faces; and, the QEM algorithm is used for local mesh optimization processing.

[0035] Preferably, the steps of using the QEM algorithm for local mesh optimization processing include:

[0036] Introduce a weight factor to improve the QEM algorithm. Among them, the weight factor is set based on the surgical target position, organ importance, and bionic material properties;

[0037] For a liver surgery scenario, Q is set as a triangular mesh of the liver, with its vertex set V and edge set E. For each vertex v in V, its error metric is expressed as follows:

[0038]

[0039] where K i is the planar error matrix associated with vertex V; w i is the weight factor; i is all the nodes associated with vertex v; this formula is to perform a weighted sum of all the planar error matrices associated with vertex v, and the weight factor w i Achieve external control over mesh refinement for different surgical scenarios.

[0040] Preferably, the setting strategy of the weight factor w i includes: assigning higher weights to the liver and adjacent organs to reduce the simplification of these regions; using smaller weights for the organs and vascular tissues distal to the surgical target to obtain a more simplified tissue mesh; and, according to the physical property differences between organs and soft tissues, setting weights lower than those of organs for soft tissue regions.

[0041] Preferably, the individualized three-dimensional bionic model reconstruction module further includes adding physical parameters based on real biomechanics to the individualized three-dimensional bionic model to simulate the real physical behaviors of different tissues; the physical parameters include elastic modulus, density, and friction coefficient. The elastic modulus of the liver is set to 0.5 - 1.5 kPa, and the elastic modulus of muscle tissue is set to 8 - 12 kPa; when performing parameter mapping, the voxelization method is used to map the physical parameters to each grid cell of the three-dimensional model, and the corresponding physical parameters are calculated and assigned at the grid nodes through trilinear interpolation to ensure the continuity of the physical properties of the individualized three-dimensional bionic model throughout the volume.

[0042] In a second aspect, the present application provides an individualized virtual surgical operation device, including the virtual laparoscopic surgery robot platform as described above.

[0043] Based on the above technical solutions, the virtual laparoscopic surgery robot platform and device of the present application, compared with the prior art, at least have one of the following beneficial effects:

[0044] 1. The present application sets up an individualized three-dimensional bionic model reconstruction module, a robotic arm array module, a virtual surgery feedback module, and an interaction module. Through a unique multi-module interaction and system integration strategy, it realizes a highly realistic and individualized surgical simulation, and can meet the requirements of virtual laparoscopic surgery simulation in real-time calculation and feedback as well as the overall coordination control of the system platform.

[0045] 2. The robotic arm array module of this application is provided with an adaptive gain adjustment strategy for torque control and speed control. Through the PID controller with adaptive gain adjustment, the precise control of the robotic arm in complex surgical scenarios is ensured, enhancing the authenticity and precision of the surgery.

[0046] 3. The interaction module of this application realizes the precise mapping and deep coupling of the robotic arm movements and the individualized three-dimensional bionic model, ensuring a highly realistic human-machine interaction. The system integration module ensures the efficient communication and real-time calculation of each module, and enhances the overall coordination and stability of the platform by adopting timestamp and buffer management technologies to align data streams from different sources and setting up an embedded real-time operating system.

[0047] 4. This application introduces weight factors into the traditional QEM algorithm to achieve the retention of details for different tissues and organs. These weight factors can be set based on the surgical target position, organ importance, and material properties. For example, in the scenario of liver surgery, the liver can be set as an important organ to increase the weight coefficient; the important organs around the liver can be set as secondary targets with a weaker weight coefficient; and a weaker coefficient can be adopted for the surrounding soft tissues. This can greatly optimize the number of three-dimensional meshes in the laparoscopic surgery scenario, improving the virtual surgery operation efficiency and real-time performance.

[0048] Other features and advantages of this application will be described in the subsequent specification. Moreover, some of them will become apparent from the specification, or the objectives and other advantages of this application can be realized and obtained through the structures specifically pointed out in the written specification and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a module connection diagram of the individualized virtual laparoscopic surgery robot platform of this application;

[0050] Figure 2 It is a module connection diagram of the individualized three-dimensional bionic model reconstruction module in this application;

[0051] Figure 3 It is a module connection diagram of the robotic arm array module in this application;

[0052] Figure 4 It is a module connection diagram of the virtual surgery feedback module in this application;

[0053] Figure 5 It is a module connection diagram of the interaction module in this application;

[0054] Figure 6 It is a physical diagram of the individualized virtual laparoscopic surgery robot platform of this application;

[0055] Figure 7Schematic diagram of virtual surgery for the individualized virtual laparoscopic surgery robot platform in this application. Detailed implementation manners

[0056] To make the objectives, technical solutions and advantages of this application clearer and more understandable, the following further elaborates on this application in detail in combination with specific embodiments and with reference to the accompanying drawings.

[0057] The terms used in the embodiments of this invention are merely for the purpose of describing specific embodiments and are not intended to limit the embodiments of this invention. The singular forms "a", "the" and "said" used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0058] Aiming at the deficiencies of the prior art, the objective of this invention is to provide an individualized virtual laparoscopic surgery robot platform, expecting to achieve highly realistic and individualized surgical simulations and improve the operation efficiency and real-time performance of virtual surgery through a multi-module interaction and system integration strategy.

[0059] The basic idea of this invention is to set up an individualized three-dimensional bionic model reconstruction module, a robotic arm array module, a virtual surgery feedback module, and an interaction module. Among them, the interaction module is used to control the coordinated cooperation of the individualized three-dimensional bionic model reconstruction module, the robotic arm array module, and the virtual surgery feedback module, update the corresponding individualized three-dimensional bionic model according to the operations of the robotic arm array module, and control the virtual surgery feedback module to update the visual feedback results. Through the above unique multi-module interaction and system integration strategy, highly realistic and individualized surgical simulations are achieved.

[0060] Embodiment 1

[0061] In order to develop an efficient and highly realistic virtual laparoscopic surgery robot platform, the inventor of this invention conducted in-depth research on related technologies and proposed an individualized virtual laparoscopic surgery robot platform.

[0062] Specifically, as Figure 1 shown, an individualized virtual laparoscopic surgery robot platform is provided, including:

[0063] An individualized three-dimensional bionic model reconstruction module, used to generate an individualized three-dimensional bionic model according to clinical medical imaging data;

[0064] A robotic arm array module, including at least three independent robotic arms, used to simulate the operations of laparoscopic surgery;

[0065] A virtual surgery feedback module, used to cooperate with the robotic arm array module to provide real-time force feedback and visual feedback;

[0066] An interaction module is used to control the coordinated cooperation of the individualized three-dimensional bionic model reconstruction module, the robotic arm array module, and the virtual surgery feedback module. It updates the individualized three-dimensional bionic model accordingly based on the operations of the robotic arm array module, and controls the virtual surgery feedback module to update the visual feedback results.

[0067] In this application, by setting up the individualized three-dimensional bionic model reconstruction module, the robotic arm array module, the virtual surgery feedback module, and the interaction module, and adopting a unique multi-module interaction and system integration strategy, highly realistic and individualized surgical simulations are achieved.

[0068] As Figure 2 shown, it is the module connection diagram of the individualized three-dimensional bionic model reconstruction module in this application. Among them, the individualized three-dimensional bionic model reconstruction module is used to generate an individualized three-dimensional bionic model based on clinical medical imaging data. This module is responsible for creating an accurate three-dimensional static geometric organ model from the original individualized three-dimensional CT images, and attaching preset dynamic parameters of different tissues and organs to construct a three-dimensional dynamic bionic organ model, providing an accurate target for subsequent virtual laparoscopic surgery simulations. This individualized three-dimensional bionic model reconstruction module is divided into two main parts:

[0069] 1. Reconstruction of individualized three-dimensional geometric model

[0070] (1) Data preprocessing: Provide clear and accurate data input for subsequent image analysis and model reconstruction. Adjust the window width and window level according to the absorption characteristics of the CT rays by the target tissue to enhance the image contrast. For example, the window width of a liver CT scan can be set to 150 - 200 HU, and the window level to 30 - 60 HU. Use median filtering to denoise the image and standardize the image.

[0071] (2) Organ segmentation: This solution combines artificial intelligence automatic segmentation and manual segmentation to outline human tissue and organs with biological differences. Use big data of clinical CT images as a sample set to complete the construction of a pre-trained nnU-Net model for performing the initial segmentation of organs. Considering that insufficient training samples will lead to a decrease in the accuracy of lesion segmentation, then for lesion areas such as tumors, artificial assistance correction is adopted to ensure the accuracy of organ segmentation. Apply post-processing methods such as morphological operations (such as dilation and erosion) to refine the segmentation results and improve the continuity and smoothness of the edges of abdominal tissue and organs.

[0072] (3) 3D geometric model reconstruction: Apply the Marching Cubes algorithm to extract the isosurface of the 3D model. For each voxel, use the gray value of the corresponding original 3D CT image as the threshold to determine whether the voxel is inside or outside the model surface, and then create vertices on the boundary of the voxel to form a triangular mesh. Optimize all the meshes, including removing isolated elements, smoothing the surface, and mesh simplification, to improve the processing efficiency and visual quality of the 3D model and provide high-quality geometric samples for subsequent organ bionic models.

[0073] 2. Reconstruction of individualized 3D bionic models

[0074] (1) Geometric model subdivision and optimization: Use the Loop subdivision algorithm to refine the geometric model structure to ensure that without changing the topological structure of the tissue organs, the smoothness and details of the abdominal organ model are improved by increasing the number of vertices and faces. Use the Quadric Error Metrics (QEM) algorithm as a local mesh optimization technique to reduce unnecessary details (such as overly large soft tissue areas, broken blood vessel reconstructions, etc.) and optimize the computational efficiency of the model. At the same time, maintain high resolution in the target feature regions (such as lesions, around the liver, tumors) to ensure the accuracy of laparoscopic surgery simulation.

[0075] To meet the need for dynamically adjusting the mesh refinement degree during virtual laparoscopic surgery simulation, the present invention adopts a weight factor improvement strategy to improve the QEM algorithm and achieve detail retention for different tissue organs. In traditional QEM, the error metric is usually calculated through the quadratic error matrix of vertices. To more precisely control details in specific regions (such as the liver, soft tissue, or blood vessels), weight factors are introduced. These weight factors can be set based on the surgical target location, organ importance, and bionic material properties, and can handle various individualized laparoscopic surgery environments. For a liver surgery scenario, Q is set as the liver triangular mesh, with its vertex set V and edge set E. For each vertex v in V, its error metric is expressed as:

[0076]

[0077] where, K i is the plane error matrix related to vertex V; w i is the weight factor; i is all the nodes related to vertex v. This formula is to perform a weighted sum of all the plane error matrices related to vertex v, and the weight factor w can achieve external control of mesh refinement for different surgical scenarios. In the present invention, w iIt can be defined according to the following aspects: 1) Assign higher weights to the liver and adjacent organs to reduce the simplification of these regions; use smaller weights for the organs and vascular tissues distal to the surgical target to obtain a more simplified tissue mesh; 2) According to the physical property differences between organs and soft tissues, set weights significantly lower than those of organs for soft tissue regions.

[0078] (2) Physical parameter attachment: Attach physical parameters based on real biomechanics to the three-dimensional bionic model, such as elastic modulus, density, and friction coefficient, to simulate the real physical behavior of different tissues. First, define the physical parameters of various tissues according to the literature and experimental data. For example, the elastic modulus of the liver can be set within the range of 0.5 - 1.5 kPa, while that of muscle tissue is 8 - 12 kPa. Perform parameter mapping, and use the voxelization method to map the physical parameters to each grid cell of the three-dimensional model. Calculate and assign these parameters at the grid nodes through trilinear interpolation to ensure the continuity of the physical properties of the model throughout the volume.

[0079] Among them, the Loop subdivision algorithm is a technique in computer graphics used to smooth and refine three-dimensional geometric models. It adds new vertices and faces at each vertex of the model to increase the detail and smoothness of the model. The Loop subdivision algorithm calculates the positions of the new vertices, which are usually weighted averages based on the positions of the original vertices and adjacent vertices. This calculation method ensures that the positions of the new vertices can transition smoothly, thereby reducing the jaggedness of the mesh. Although the mesh is refined, the topological structure of the model (i.e., the connection method of vertices, edges, and faces) remains unchanged. This means that the basic shape and features of the model are retained, and only the detail and smoothness are enhanced. After applying the Loop subdivision algorithm, the surface of the abdominal organ model will become smoother and more detailed, which helps to more realistically simulate and display the morphological characteristics of abdominal organs. For surgical simulation and training, this can provide a more realistic surgical experience.

[0080] Finally, transfer the constructed three-dimensional geometric model to other models such as the virtual surgery feedback module through the communication protocol for further processing.

[0081] This application improves the accuracy of organ segmentation by combining artificial intelligence automatic segmentation and manual segmentation techniques. By using the Marching Cubes algorithm and the improved QEM algorithm to optimize the three-dimensional geometric model, a high-quality three-dimensional organ model is provided.

[0082] Such as Figure 3As shown, it is the module connection diagram of the robotic arm array module in this application. Among them, the robotic arm operation array is the core control unit of the robotic arm array, responsible for sending operation instructions to control the actions of the robotic arms. It includes three independent robotic arms, and each robotic arm has specific degrees of freedom, all of which include at least three rotational degrees of freedom and one linear degree of freedom for performing different operations. The rotational degrees of freedom of the robotic arm allow the joints to rotate within a plane, while the linear degree of freedom allows the joints to move in a straight line, such as for telescopic motion. The PID controller is used to automatically adjust the actions of the robotic arms according to the error signal to achieve precise control. The high-precision encoder is used to accurately measure the position and speed of the robotic arm joints and provide feedback for control. The sensor signal integration is responsible for collecting and integrating signals from different sensors for further processing and analysis. The angle sensor measures the angular changes of the robotic arm joints and provides position feedback. The force sensor detects the force when the robotic arm contacts the environment or an object and is used to achieve force control and avoid collisions. The acceleration sensor measures the acceleration of the robotic arm joints and helps to control the dynamic response and stability. The communication protocol transfer is responsible for transmitting data and instructions between the robotic arm array module and other system modules to ensure the accurate transmission of information. All components of the robotic arm array module work together, enabling the robotic arm array module to precisely perform complex operations, such as simulating the actions of surgical tools in virtual laparoscopic surgery. Through the feedback of the high-precision encoder and various sensors, the PID controller can adjust the actions of the robotic arms in real time to ensure the smoothness and precision of the operations. The communication protocol transfer ensures the coordinated operation of the entire system.

[0083] Among them, the robotic arm array module configures an adaptive gain adjustment strategy for torque control and speed control for each robotic arm to ensure the smoothness and precision of the operation. The adaptive gain adjustment strategy enables the PID controller of each robotic arm to dynamically adjust the gain parameters according to the real-time error and system state.

[0084] Specifically, the formula expression of the adaptive gain adjustment strategy is as follows:

[0085]

[0086] Among them, u(t) is the control input, the torque or speed adjustment signal; t is the time point, e(t) is the error between the set point and the current position, and K p (t), K i (t), K d (t) are the proportional, integral, and differential gain parameters respectively, which are defined with time and system state as independent variables as follows:

[0087]

[0088] K iu(t) = K i0 + a i · f i (e(t), ∫e(t)dt, S(t))

[0089]

[0090] where K p0 , K i0 , K d0 are the initial gains corresponding to proportional, integral, and derivative; a p , a i , a d are the adjustment coefficients corresponding to proportional, integral, and derivative, used to control the amplitude of adaptive adjustment; f p , f i , f d are adaptive functions; S(t) represents the current state information, which is the parameter of the virtual surgical scene.

[0091] The adaptive gain adjustment strategy introduced in this application allows the PID controller of the robotic arm to dynamically adjust the gain parameters according to the real-time error and system state. This means that the robotic arm can automatically optimize its performance in different operation stages (such as cutting, knotting, rapid movement, and fine operation). At the same time, the introduced adaptive gain adjustment can speed up the system's response speed to errors, especially in cases where rapid adjustment is required (such as emergency avoidance, abnormal robotic arm trajectory). In complex surgical scenarios, real-time adjustment of gain parameters can ensure that the system is always in the optimal control state.

[0092] Preferably, the robotic arm array module is equipped with joint torque sensors at the joints of each robotic arm to capture angle changes, piezoelectric sensors are installed to capture the force when contacting the virtual organ, and acceleration sensors are installed to monitor the acceleration of the movement. The robotic arm has at least three rotational degrees of freedom and one linear degree of freedom; the Kalman filtering algorithm is applied to fuse data from different sensors. Among them, the update formula of the Kalman filter is:

[0093] x k|k = x k|k-1 + K k (Z k - H k x k|k-1 )

[0094] x k|k is the state estimate updated at the k-th time point, x k|k-1 is the estimated value at the previous time point, Z k is the observed value, which contains the actual measurement data of the sensor, H k is the observation model matrix, K kis the Kalman gain, and the sensor data is integrated into the state estimation through the observation model matrix H k for dynamic system state estimation of the actions of the robotic arm.

[0095] Preferably, the robotic arm array module calculates the force when the end of the robotic arm contacts the virtual organ using the force model calculation formula, and the force model calculation formula is:

[0096] F = k·Δx

[0097] where F is the contact force, Δx is the deformation amount, k is the elastic constant, and the value of k is dynamically adjusted according to the physical properties of different virtual organs;

[0098] The force feedback algorithm based on impedance control is adopted to enable the operator to feel the force matching the interaction in the virtual environment, and the force feedback algorithm formula is expressed as:

[0099]

[0100] where F d is the desired force, x d , are the desired position, velocity, and acceleration respectively, x, are the actual position, velocity, and acceleration, and K d , B d , M d are the control stiffness, damping, and mass parameters respectively.

[0101] As Figure 4 shown, it is the module connection diagram of the virtual surgery feedback module in this application; this virtual surgery feedback module provides real-time force feedback, simulates the physical interaction when the robotic arm contacts the virtual organ, and enhances the realism of the surgery simulation. It provides high-definition visual feedback, displays the individualized abdominal cavity environment and virtual organs, and enhances the immersion of the surgery simulation. Among them, the force feedback unit adopts a force sensing design, and a high-precision piezoelectric sensor is assembled at the end of the robotic arm to capture the force when contacting the virtual organ. The force model calculation formula uses F = k·Δx to calculate the contact force, where F is the contact force, Δx is the deformation amount, k is the elastic constant, and the value of k can be dynamically adjusted according to the physical properties (such as elastic modulus) of different virtual organs. Implement the force feedback algorithm based on impedance control to ensure that the operator feels the force matching the interaction in the virtual environment. The force feedback algorithm can be expressed as:

[0102]

[0103] where F d is the desired force, x d , are the desired position, velocity, and acceleration respectively, x, are the actual position, velocity, and acceleration, K d , B d , M d are the stiffness, damping, and mass parameters of the control respectively.

[0104] The visual feedback module therein uses a medical-grade high-resolution display to ensure the clarity and details of the images. Advanced image rendering technologies, such as ray tracing or real-time rendering algorithms, are applied to provide realistic visual effects. The graphics processing unit (GPU) is used to accelerate the rendering process to ensure real-time performance. A user-friendly interaction interface is developed to support functions such as surgical record and risk point prompt. Touch screen or voice recognition technology is used to improve the interactivity of the interface. The interface design should consider usability and accessibility to ensure that all users can quickly understand and operate.

[0105] Such as Figure 5 shown, is the module connection diagram of the interaction module in this application, which is the central control unit of the system. This module is responsible for coordinating the operations of all other modules, processing the signals from the robotic arm operation and the deformation signals of the bionic model, ensuring the accurate transmission and feedback of information, and at the same time supporting complex data processing and real-time computing requirements. Among them, an action mapping algorithm is designed to achieve the precise mapping of the physical actions of the robotic arm to the actions of the virtual model. Affine transformation is used to convert the physical coordinate system of the robotic arm and the coordinate system of the virtual environment. where (x, y, z) is the position of the end point of the robotic arm in the physical coordinate system, and (x′, y′, z′) is the position in the virtual coordinate system after conversion. The transformation matrix includes the rotation angle R, the scaling S, and the translation T. Where t x , t y , t z are the translation amounts along the x, y, and z axes respectively.

[0106] Model coupling is to achieve the deep coupling of the three-dimensional bionic model and the robotic arm array module, and adjust the deformation of the bionic model in real time according to the action signals of the robotic arm to achieve highly realistic human-machine interaction. It involves (1) signal capture and processing: high-precision encoders are installed on each joint of the robotic arm to capture the joint angles and speeds in real time. The Kalman filter is used to smooth and predict the motion trajectory of the robotic arm. (2) Model response algorithm: A real-time bionic model response system based on a physics engine is developed. The finite element method (FEM) is used to simulate the dynamic response of the model. For example, when the robotic arm contacts the bionic model, the stress distribution and deformation at the contact point are calculated to update the shape of the model.

[0107] Preferably, the virtual laparoscopic surgery robot platform further includes a system integration module, which uses timestamp and buffer management technologies to align data streams from different sources and achieve synchronous processing of data from the individualized three-dimensional bionic model reconstruction module, the robotic arm array module, and the virtual surgery feedback module. To ensure that data from the deformation sensor, the robotic arm sensor, and different functional modules can be synchronously processed. The virtual laparoscopic robot platform has extremely high requirements for operation real-time performance. Therefore, the present invention uses timestamp and buffer management technologies to align data streams from different sources. The extended Kalman filter (EKF) is applied to fuse data from the robotic arm sensor and the bionic model sensor to improve the accuracy of state estimation. A low-pass filter is further designed to reduce the influence of sensor noise and environmental interference.

[0108] Preferably, the system integration module deploys a GPU-based computing unit to support high-speed data processing and complex signal operations; the system integration module sets an embedded real-time operating system to manage the priorities and scheduling of tasks to ensure that critical tasks with high priorities are completed within time limits. The critical tasks with high priorities include signal processing and feedback control; the scheduling algorithm of the embedded real-time operating system is set to cyclic scheduling to ensure the response time and predictability of the system. An efficient data stream architecture is designed to reduce data transmission latency and avoid data loss. The platform has verified the compatibility and stability of multiple modules: the ROS (Robot Operating System) architecture is used to standardize the communication between different modules of the overall system. Comprehensive system tests are carried out, including unit tests, integration tests, and performance tests, to ensure that all modules of the virtual laparoscopic surgery robot can work together, run stably, and meet the design requirements. The dynamic configuration function between modules is realized, allowing the system to automatically reconfigure the connections and settings between modules according to the current workload and performance metrics, improving the flexibility and scalability of the system.

[0109] Embodiment 2

[0110] This application provides an individualized virtual surgery operation device, including the virtual laparoscopic surgery robot platform described in Embodiment 1 above. As Figure 6 and Figure 7 shown, Figure 6 is the physical diagram of the individualized virtual laparoscopic surgery robot platform in this application, Figure 7This is a schematic diagram of performing virtual surgery on the individualized virtual laparoscopic surgical robot platform in this application. The virtual laparoscopic surgical robot platform has the following functions: setting an individualized three-dimensional bionic model reconstruction module, a robotic arm array module, a virtual surgery feedback module, and an interaction module. Through a unique multi-module interaction and system integration strategy, it realizes highly realistic and individualized surgical simulation. The robotic arm array module is set with an adaptive gain adjustment strategy for torque control and speed control. Through an adaptive gain-adjusted PID controller, it ensures the precise control of the robotic arm in complex surgical scenarios, improving the authenticity and precision of the surgery. Its interaction module realizes the precise mapping and deep coupling of the robotic arm movements and the individualized three-dimensional bionic model, ensuring a highly realistic human-machine interaction. The system integration module ensures the efficient communication and real-time calculation of each module by adopting timestamp and buffer management technologies to align data streams from different sources and setting an embedded real-time operating system, improving the overall coordination and stability of the platform. It introduces weight factors into the traditional QEM algorithm to retain details of different tissues and organs. These weight factors can be set based on the surgical target position, organ importance, and material properties. For example, in the liver surgery scenario, the liver can be set as an important organ to increase the weight coefficient; the important organs around the liver can be set as secondary targets with a weaker weight coefficient; and a much weaker coefficient can be adopted for the surrounding soft tissues. This can greatly optimize the number of three-dimensional meshes in the laparoscopic surgery scenario, improving the operation efficiency and real-time performance of the virtual surgery. An affine transformation is used to achieve the precise mapping of the physical actions of the robotic arm to the virtual model actions. A real-time bionic model response system based on a physics engine uses the finite element method (FEM) to simulate the dynamic response of the model. The above technical improvements work together to overcome the problems of existing simulators in terms of individualization, real-time feedback, interactivity, technical integration, data utilization, and surgical complexity simulation, providing a more precise, efficient, and practical virtual laparoscopic surgical robot platform.

[0111] An adaptive gain function is introduced as a coefficient to improve the traditional PID controller and realize the control process of the robotic arm. The adaptive gain function dynamically adjusts the gain parameters according to the real-time error and system state, and can automatically optimize its performance in different operation stages (such as cutting, knotting, rapid movement, and fine operation). This is very important for accelerating the system's response speed to errors and dealing with emergency scenarios (such as emergency avoidance and abnormal robotic arm trajectories). Especially in complex surgical scenarios, on the premise of meeting the real-time interaction of the laparoscopic surgical robot, the gain parameters can be adjusted in real time to ensure that the system is always in the optimal control state.

[0112] In the process of constructing a virtual laparoscopic surgical robot platform, the collaborative work of each module is indispensable, forming an innovative overall system. The individualized three-dimensional organ reconstruction module ensures the accuracy of organ segmentation by combining artificial intelligence and manual segmentation techniques, and provides a high-quality three-dimensional geometric model through the Marching Cubes algorithm and improved QEM optimization technology. This provides an accurate target for subsequent surgical simulations. The robotic arm array module ensures the precise control of the robotic arms in complex surgical scenarios through a PID controller with adaptive gain adjustment, enhancing the authenticity and accuracy of the surgery. The virtual surgery feedback module provides real-time force feedback and high-definition visual feedback, enhancing the immersion of the surgical simulation. The real-time robotic arm control and virtual organ model interaction module achieves an accurate mapping and deep coupling between the robotic arm movements and the bionic model, ensuring a highly realistic human-machine interaction. The multi-module interaction and system integration module ensures the efficient communication and real-time calculation of each module through the ROS architecture and real-time operating system, enhancing the overall coordination and stability of the system. If any one module is missing, the overall performance of the system will be significantly affected. For example, without the individualized three-dimensional organ reconstruction module, even if the other modules are working properly, the surgical simulation will lose its meaning due to the lack of an accurate organ model and cannot provide a realistic surgical environment. The innovation of this system lies in the tight coupling and mutual dependence among its modules, forming an inseparable whole and ensuring the efficiency and accuracy of virtual laparoscopic surgical simulations.

[0113] The above describes specific embodiments of the present invention. Additionally, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0114] In the description of the embodiments of the present invention, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present invention. In the embodiments of the present invention, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in the embodiments of the present invention and the features of different embodiments or examples.

[0115] In addition, the terms "first" and "second" are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features, excluding any order. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature and are used to distinguish each other. In the description of the embodiments of the present invention, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0116] Any process or method description shown in a flowchart or otherwise described herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of the embodiments of the present invention includes additional implementations where functions may be performed in a substantially simultaneous manner or in an order opposite to that shown or discussed, according to the functions involved, which should be understood by those skilled in the technical field of the embodiments of the present invention.

[0117] The above are only the preferred embodiments of the embodiments of the present invention and are not intended to limit the embodiments of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present invention shall be included within the scope of protection of the embodiments of the present invention.

Claims

1. A virtual laparoscopic surgery robot platform, characterized in that: include: Individualized 3D bionic model reconstruction module, used to generate individualized 3D bionic models based on clinical medical imaging data; A robotic arm array module, including at least three independent robotic arms, for simulating the operation of laparoscopic surgery; A virtual surgery feedback module, used to cooperate with the robotic arm array module to provide real-time force feedback and visual feedback; The interactive module is used to control the coordinated cooperation among the individualized three-dimensional bionic model reconstruction module, the robotic arm array module and the virtual surgery feedback module, control the individualized three-dimensional bionic model to be updated accordingly according to the operation of the robotic arm array module, and control the virtual surgery feedback module to update the visual feedback result.

2. The virtual laparoscopic surgery robot platform according to claim 1, characterized in that: The robotic arm array module is configured with an adaptive gain adjustment strategy for torque control and speed control for each robotic arm, and the adaptive gain adjustment strategy enables the PID controller of each robotic arm to dynamically adjust the gain parameters according to the real-time error and system status.

3. The virtual laparoscopic surgery robot platform according to claim 2, characterized in that: The formula of the adaptive gain adjustment strategy is as follows: Where u(t) is the control input, torque or speed adjustment signal; t is the time point, e(t) is the error between the set point and the current position, K p (t), K i (t), K d (t) are the proportional, integral and differential gain parameters, respectively, with time and system state as independent variables, and are defined as follows: Among them, K p0 , K i0 , K d0 is the initial gain corresponding to the proportional, integral and differential; a p 、a i 、a d is the adjustment coefficient corresponding to the proportion, integration and differentiation, which is used to control the amplitude of adaptive adjustment; f p 、f i 、f d is an adaptive function; S(t) represents the current state information and is the parameter of the virtual surgery scene.

4. The virtual laparoscopic surgery robot platform according to claim 3, characterized in that: The robotic arm array module is equipped with a joint torque sensor at each joint of the robotic arm to capture angle changes, a piezoelectric sensor to capture the force when in contact with the virtual organ, and an acceleration sensor to monitor the acceleration of the action; the robotic arm includes at least three rotational degrees of freedom and one linear degree of freedom; the robotic arm array module uses the Kalman filter algorithm to fuse data from different sensors, wherein the update formula of the Kalman filter is: x k|k =x k|k-1 +K k (Z k -H k x k|k-1 ); x k|k is the updated state estimate at the kth time point, x k|k-1 is the estimated value at the previous time point, Z k is the observed value, H k is the observation model matrix, K k is the Kalman gain, the sensor data is passed through the observation model matrix H k It is integrated into the state estimation to perform dynamic system state estimation of the robot arm's motion.

5. The virtual laparoscopic surgery robot platform according to claim 1, characterized in that: The mechanical arm array module uses a force model calculation formula to calculate the force when the end of the mechanical arm contacts the virtual organ. The force model calculation formula is: F = k·Δx; Among them, F is the contact force, Δx is the deformation, k is the elastic constant, and the k value is dynamically adjusted according to the physical properties of different virtual organs; The force feedback algorithm based on impedance control is used to make the operator feel the force that matches the interaction in the virtual environment. The force feedback algorithm formula is expressed as: Among them, F d is the desired force, x d , are the desired position, velocity, and acceleration, respectively, x, is the actual position, velocity and acceleration, K d , B d 、M d are the controlled stiffness, damping and mass parameters respectively.

6. The virtual laparoscopic surgery robot platform according to claim 1, characterized in that: Affine transformation is used to convert the physical coordinate system of the robot arm and the virtual environment coordinate system: Among them, (x, y, z) is the position of the end point of the robot arm in the physical coordinate system, (x′, y′, z′) is the position in the virtual coordinate system after transformation, and the transformation matrix contains the rotation angle R, scaling S and translation T. Among them, t x , t y , t z are the translations along the x, y, and z axes respectively.

7. The virtual laparoscopic surgery robot platform according to claim 1, characterized in that: The virtual laparoscopic surgical robot platform also includes a system integration module, which uses timestamp and buffer management technology to align data streams from different sources to achieve synchronous processing of data from an individualized three-dimensional bionic model reconstruction module, a robotic arm array module, and a virtual surgery feedback module.

8. The virtual laparoscopic surgery robot platform according to claim 7, characterized in that: The system integration module deploys a GPU-based computing unit to support high-speed data processing and complex signal operations; The system integration module sets an embedded real-time operating system to manage the priority and scheduling of tasks to ensure that high-priority critical tasks are completed within a time limit. The high-priority critical tasks include signal processing and feedback control. The scheduling algorithm of the embedded real-time operating system is set to cyclic scheduling.

9. The virtual laparoscopic surgery robot platform according to any one of claims 1 to 8, characterized in that: The individualized three-dimensional bionic model reconstruction module uses the Loop subdivision algorithm to refine the individualized three-dimensional bionic model, and improves the smoothness and details of the abdominal organ model by increasing the number of vertices and facets without changing the topological structure of the tissue and organ; and uses the QEM algorithm to perform local mesh optimization processing.

10. The virtual laparoscopic surgery robot platform according to claim 9, characterized in that: The steps of local mesh optimization using the QEM algorithm include: The QEM algorithm is improved by introducing weight factors, where the weight factors are set based on the surgical target location, organ importance, and bionic material properties; For a liver surgery scenario, Q is set as a liver triangular mesh with a vertex set V and an edge set E. For each vertex v in V, the error metric is expressed as follows: Among them, K i is the plane error matrix associated with vertex V; w i is the weight factor; i is all nodes related to vertex v; the formula is the weighted sum of all plane error matrices related to vertex v, and the weight factor w i Implement external control of mesh refinement for different surgical scenarios.

11. The virtual laparoscopic surgery robot platform according to claim 10, characterized in that: The weight factor w i The setting strategies include: assigning higher weights to the liver and adjacent organs to reduce the simplification of these areas; using smaller weights for organs and vascular tissues distal to the surgical target to obtain a simpler tissue mesh; and, based on the differences in physical properties between organs and soft tissues, setting a lower weight for soft tissue areas than for organs.

12. The virtual laparoscopic surgery robot platform according to claim 11, characterized in that: The individualized three-dimensional bionic model reconstruction module also includes adding physical parameters based on real biomechanics to the individualized three-dimensional bionic model to simulate the real physical behavior of different tissues; the physical parameters include elastic modulus, density and friction coefficient, the elastic modulus of the liver is set to 0.5-1.5kPa, and the elastic modulus of muscle tissue is set to 8-12kPa; when performing parameter mapping, the physical parameters are mapped to each grid unit of the three-dimensional model using a voxelization method, and the corresponding physical parameters are calculated and allocated on the grid nodes through cubic linear interpolation to ensure the continuity of the physical properties of the individualized three-dimensional bionic model over the entire volume.

13. An individualized virtual surgical operation device, characterized in that: Including the virtual laparoscopic surgery robot platform as described in claim 12.

Images