Self-adaptive force feedback method and device based on individualized virtual laparoscopic surgery and medium

Abstract

The invention discloses a self-adaptive force feedback method and device based on individualized virtual laparoscopic surgery and a medium, and belongs to the technical field of self-adaptive force feedback calculation. The self-adaptive force feedback method comprises the following steps: detecting whether a touch pen point of a virtual surgical instrument is in contact with an abdominal cavity tissue / organ model or not; wherein the outermost layer of the abdominal cavity tissue / organ model is coated with a surface film, and when the touch pen point of the virtual surgical instrument is in contact with the surface film, it is judged that the touch pen point is in contact with the corresponding abdominal cavity tissue / organ model; if the touch pen point of the virtual surgical instrument is in contact with the abdominal cavity tissue / organ model, identifying the type of the contacted abdominal cavity tissue / organ model, and switching to force feedback parameter setting of the corresponding abdominal cavity tissue / organ model for force feedback calculation; and detecting whether a touch pen point of the virtual surgical instrument penetrates through the surface film and enters the abdominal cavity tissue / organ model or not, if so, performing force feedback calculation based on the switched force feedback parameter setting, and performing scene rendering. According to the method, the force feedback parameters can be adaptively switched according to the types of different tissues and organs, and the quality and authenticity of a virtual operation are improved.

Description

Technical Field

[0001] The present application relates to the field of adaptive force feedback computing technology, and specifically to an adaptive force feedback method, device, and medium based on individualized virtual laparoscopic surgery. Background Art

[0002] With the progress of modern clinical medicine and the growing demand for precision medical services from the public, digital, individualized, precise and minimally invasive laparoscopic surgery has become an urgent clinical need and a frontier area of ​​medical technology development. However, the current laparoscopic field in the world generally faces the problems of a single system training model, a lack of individualized training modules for different lesions, a lack of adaptive force feedback modules for different tissues and organs, and a lack of authenticity in surgical simulation, which is not conducive to preoperative simulation training for clinicians.

[0003] The Chinese patent application document with publication number CN112528529A discloses a laparoscopic naked-eye simulation reality method, which proposes a finite element simulation contact collision detection method and a soft tissue material failure criterion for an organ body model; experiments and analysis are conducted on the influence of inflation pressure, respiratory movement, and external factors such as pulling, squeezing, and cutting of surgical instruments on the deformation of the organ body model, the magnitude of the interaction force between different factors and tissues is determined, the force threshold for contact collision and rupture is set, and a contact collision detection method for loads and organ tissues based on force response is designed, and the finite element model of the complete organ is simulated, optimized, and verified. The soft tissue of the organ will deform or rupture under the action of load, and the core of the rupture problem is to determine the failure criteria of different organ tissues. This scheme studies the tissue failure criteria of the organ based on the material distortion energy theory, that is, when the distortion energy density of the organ body model tissue reaches its limit value, the material is destroyed, and the Abaqus software subroutine Vumat is used to define the soft tissue hyperelastic material properties and failure criteria. However, this method cannot achieve fast and accurate switching of adaptive force feedback parameters between different tissues and organs.

[0004] It can be seen that there is an urgent need to develop a new adaptive force feedback method based on virtual laparoscopic surgery to effectively improve the processing speed and accuracy of adaptive force feedback switching and enhance the quality and authenticity of virtual surgery. Summary of the invention

[0005] The present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the present application provides an adaptive force feedback method, device, and medium based on personalized virtual laparoscopic surgery, which can adaptively switch force feedback parameters according to different types of tissues and organs, thereby improving the quality and authenticity of virtual surgery.

[0006] To achieve the above objectives, in a first aspect, the present application provides an adaptive force feedback method based on virtual laparoscopic surgery, comprising:

[0007] Detecting whether the tactile pen tip of the virtual surgical instrument is in contact with the abdominal tissue / organ model; wherein the outermost layer of the abdominal tissue / organ model is covered with a surface film, and when the tactile pen tip of the virtual surgical instrument is in contact with the surface film, it is determined that it is in contact with the corresponding abdominal tissue / organ model;

[0008] If the tactile pen tip of the virtual surgical instrument comes into contact with the abdominal tissue / organ model, the type of the abdominal tissue / organ model in contact is identified, and the force feedback parameter setting of the corresponding abdominal tissue / organ model is switched to perform force feedback calculation;

[0009] Detect whether the tactile pen tip of the virtual surgical instrument penetrates the surface membrane and enters the abdominal tissue / organ model. If so, perform force feedback calculation based on the switched force feedback parameter settings and perform scene rendering.

[0010] Preferably, the process of modeling the abdominal tissue / organ model includes:

[0011] Acquiring clinical medical imaging data, and preprocessing the clinical medical imaging data;

[0012] The preprocessed clinical medical image data is input into the Mamba-CNN joint segmentation model for segmentation processing, and the tissue and organ segmentation image is output; wherein the Mamba-CNN joint segmentation model uses a CNN network in the low-order part of the encoder to capture local and detail features, and uses a Mamba model in the high-order part of the encoder to extract global dependency features;

[0013] Based on three-dimensional image reconstruction technology, an abdominal tissue / organ model of the corresponding tissues and organs is constructed.

[0014] Preferably, the surface membrane is generated by expanding the tissue and organ segmentation image to wrap the abdominal tissue / organ model; the distance from the surface membrane to the abdominal tissue / organ model is changed by adjusting the expansion coefficient of the surface membrane, and the minimum unit size and number constituting the surface membrane are adjusted by changing the downsampling parameters.

[0015] Preferably, the surface membrane is provided with an expansion coefficient adjustment strategy, and the expansion coefficient adjustment strategy is configured as follows: when the acceleration of the virtual surgical instrument movement increases, the expansion coefficient of the surface membrane increases, and when the acceleration of the virtual surgical instrument movement decreases, the expansion coefficient of the surface membrane decreases; wherein the expansion coefficient represents the degree of outward expansion centered on the expansion structure core.

[0016] Preferably, the step of generating the surface film by dilating the tissue and organ segmentation image comprises:

[0017] Acquire tissue and organ segmentation images;

[0018] Performing expansion processing on the tissue and organ segmentation image with the expansion structure core as the center;

[0019] The surface rendering reconstruction method is used to reconstruct the expanded tissue and organ surface in three dimensions, generate a surface membrane, and calibrate the three-dimensional spatial coordinate range of the surface membrane.

[0020] Preferably, the surface rendering and reconstruction method adopts one of a contour method, a marching cube method, and an isosurface method.

[0021] Preferably, the surface film is provided with a spatial resolution adjustment strategy, wherein the spatial resolution represents the number of triangular meshes per unit area of ​​the surface film, and the more triangular meshes there are, the higher the spatial resolution of the surface film; the spatial resolution adjustment strategy is configured as follows: when the minimum vertical distance between the tactile pen tip of the virtual surgical instrument and the surface film is less than a first threshold, the spatial resolution of the surface film is improved by increasing the downsampling parameters.

[0022] Preferably, the step of detecting whether the tactile pen tip of the virtual surgical instrument is in contact with the abdominal tissue / organ model comprises:

[0023] Collect the three-dimensional spatial coordinates of virtual surgical instruments in real time;

[0024] Based on the three-dimensional spatial coordinates of the virtual surgical instrument and the three-dimensional spatial coordinate range of the surface membrane of each abdominal tissue / organ model, a collision detection algorithm is used to perform collision detection between the tactile pen tip of the virtual surgical instrument and the surface membrane of all abdominal tissue / organ models;

[0025] If the tactile pen tip of the virtual surgical instrument collides with the surface membrane of any abdominal tissue / organ model, it is determined that the tactile pen tip of the virtual surgical instrument is in contact with the abdominal tissue / organ model.

[0026] Preferably, the step of identifying the type of abdominal tissue / organ model that is contacted comprises:

[0027] Record the three-dimensional coordinate position where the tactile pen tip of the virtual surgical instrument contacts the abdominal tissue / organ model;

[0028] Determining the relationship between the three-dimensional coordinate position and the three-dimensional space coordinate range of the surface membrane of each abdominal tissue / organ model;

[0029] If the three-dimensional coordinate position falls within the three-dimensional space coordinate range of the surface membrane of the corresponding abdominal tissue / organ model, it is determined that the virtual surgical instrument tactile pen tip is in contact with the corresponding abdominal tissue / organ model.

[0030] Preferably, the collision detection algorithm adopts one of the OBB algorithm, the SAT algorithm and the FCL triangle surface detection algorithm.

[0031] In a second aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described adaptive force feedback methods.

[0032] In a third aspect, the present application provides a computer-readable storage medium, comprising a computer program, which, when executed on an electronic device, enables the electronic device to execute any of the above-described adaptive force feedback methods.

[0033] In a fourth aspect, the present application provides a personalized virtual surgical operation device, including the above-mentioned electronic device.

[0034] Based on the above technical solution, it can be seen that the adaptive force feedback method based on virtual laparoscopic surgery of the present application has at least one of the following beneficial effects compared with the prior art:

[0035] 1. In order to ensure the sensitivity of the virtual surgical instrument to generate force feedback when it contacts the abdominal tissue / organ model, the present invention adds a layer of refined surface film on the basis of volume rendering of tissues and organs as the boundary of adaptive switching force feedback parameters. The refined surface film has a high spatial resolution, which helps to reduce the misjudgment of tissue and organ collision detection.

[0036] 2. The present invention adjusts the expansion coefficient of the surface film according to the acceleration of the virtual surgical instrument movement, and adjusts the spatial resolution of the surface film according to the minimum vertical distance between the tactile pen tip of the virtual surgical instrument and the surface film, which greatly improves the calculation speed and collision detection and recognition accuracy.

[0037] Other features and advantages of the present application will be described in the subsequent description, and in part will become apparent from the description, or it may be understood through the implementation of the present application that the objects and other advantages of the present application can be realized and obtained through the written description and the structures particularly pointed out in the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the adaptive force feedback method based on virtual laparoscopic surgery in this application;

[0039] Figure 2 A module connection diagram of the individualized virtual surgical operation device in this application;

[0040] Figure 3 A connection diagram of a virtual surgical instrument signal acquisition module in an individualized virtual surgical operation device in this application;

[0041] Figure 4 A connection diagram of a force feedback parameter acquisition module in an individualized virtual surgical operation device in this application;

[0042] Figure 5 A connection diagram of an adaptive parameter adjustment module and a force feedback calculation and scene rendering module in the individualized virtual surgical operation device in this application;

[0043] Figure 6 A schematic diagram of expanding a tissue and organ segmentation image to generate a surface membrane in this application. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0045] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0046] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an adaptive force feedback method based on virtual laparoscopic surgery, aiming to overcome the problems in the current virtual laparoscopic system that the elastic deformation parameters for different tissues and organs are the same, the force feedback mode is single, and there is a lack of surgical simulation authenticity.

[0047] The basic idea of ​​the present invention is to collect a large amount of offline data of force feedback and puncture depth or stretching length for training based on the actual test of force feedback of different tissues and organs by clinicians, and to construct a force feedback prediction model for different tissues and organs, so that when the virtual surgical instruments in the virtual laparoscopic system come into contact with different tissues and organs, they can identify the tissues and organs in real time and adaptively switch their elastic deformation and force feedback, thereby improving the realism of the virtual surgery. At the same time, in order to ensure the sensitivity of the virtual surgical instruments to generate force feedback when they come into contact with tissues and organs, the present application adds a layer of refined surface film on the basis of volume rendering of tissues and organs, as the boundary of adaptively switching force feedback parameters. In addition, the refined surface film has a high spatial resolution, which helps to reduce the misjudgment of tissues and organs by contacts. In order to improve the calculation speed, the present invention adjusts the spatial resolution of the surface film according to the volume size of different tissues and organs.

[0048] Embodiment 1

[0049] In order to develop an accurate adaptive force feedback method, the inventors conducted in-depth research on artificial intelligence and force feedback technology, and proposed an adaptive force feedback method based on virtual laparoscopic surgery.

[0050] Specifically, Figure 1 As shown, an adaptive force feedback method based on virtual laparoscopic surgery is provided, comprising the following steps:

[0051] Detecting whether the tactile pen tip of the virtual surgical instrument is in contact with the abdominal tissue / organ model; wherein the outermost layer of the abdominal tissue / organ model is covered with a surface film, and when the tactile pen tip of the virtual surgical instrument is in contact with the surface film, it is determined that it is in contact with the corresponding abdominal tissue / organ model;

[0052] If the tactile pen tip of the virtual surgical instrument comes into contact with the abdominal tissue / organ model, the type of the abdominal tissue / organ model in contact is identified, and the force feedback parameter setting of the corresponding abdominal tissue / organ model is switched to perform force feedback calculation;

[0053] Detect whether the tactile pen tip of the virtual surgical instrument penetrates the surface membrane and enters the abdominal tissue / organ model. If so, perform force feedback calculation based on the switched force feedback parameter settings and perform scene rendering.

[0054] Preferably, the process of modeling the abdominal tissue / organ model includes:

[0055] Acquiring individualized clinical medical image data, and preprocessing the individualized clinical medical image data;

[0056] The pre-processed individualized clinical medical image data is input into the Mamba-CNN joint segmentation model for segmentation processing, and the tissue and organ segmentation image is output; wherein the Mamba-CNN joint segmentation model uses a CNN network in the low-order part of the encoder to capture local and detail features, and uses a Mamba model in the high-order part of the encoder to extract global dependency features;

[0057] Based on three-dimensional image reconstruction technology, an abdominal tissue / organ model of the corresponding tissues and organs is constructed.

[0058] Preferably, the surface membrane is generated by expanding the tissue and organ segmentation image to wrap the abdominal tissue / organ model; the distance from the surface membrane to the abdominal tissue / organ model is changed by adjusting the expansion coefficient of the surface membrane, and the minimum unit size and number constituting the surface membrane are adjusted by changing the downsampling parameters.

[0059] Preferably, the surface membrane is provided with an expansion coefficient adjustment strategy, and the expansion coefficient adjustment strategy is configured as follows: when the acceleration of the virtual surgical instrument movement increases, the expansion coefficient of the surface membrane increases, and when the acceleration of the virtual surgical instrument movement decreases, the expansion coefficient of the surface membrane decreases; wherein the expansion coefficient represents the degree of outward expansion centered on the expansion structure core.

[0060] Preferably, the step of generating the surface film by dilating the tissue and organ segmentation image comprises:

[0061] Acquire tissue and organ segmentation images;

[0062] Performing expansion processing on the tissue and organ segmentation image with the expansion structure core as the center;

[0063] The surface rendering reconstruction method is used to reconstruct the expanded tissue and organ surface in three dimensions, generate a surface membrane, and calibrate the three-dimensional spatial coordinate range of the surface membrane.

[0064] Specifically, when the virtual surgical instrument collides with the tissue organ based on the spatial position obtained by the acquisition signal, the specific tissue organ is automatically identified, and the force feedback parameter of the tissue organ is switched to calculate the force feedback. In order to improve the sensitivity of adaptively switching the force feedback parameters of different tissue organs and calculating the force feedback, the present application innovatively introduces a surface film as the boundary of parameter adjustment. The surface film is generated by expanding the segmented three-dimensional image and wrapping the tissue organ. By changing the expansion coefficient inf, the distance from the surface film to the tissue organ can be changed. By changing the downsampling parameter ds, the minimum unit size and number of the surface film can be adjusted to achieve the improvement of the calculation speed on the basis of ensuring that the misjudgment rate of the contact point to the tissue organ is small. The present invention uses the collision recognition of the virtual surgical instrument and the surface film as a signal for switching the force feedback parameter. Before constructing the surface film, the three-dimensional space boundaries of different tissues and organs will be automatically detected by artificial intelligence technology, and a three-dimensional model based on voxels will be constructed to calibrate the three-dimensional space coordinate range of different tissue organs, and also prepare for subsequent simulation of cutting, burning and other operations. When the virtual surgical instrument passes through the surface membrane and enters the three-dimensional space of the tissue and organ, the force feedback is calculated based on the switched force feedback parameters, and the scene is rendered. When rendering the scene, the application uses a multi-level rendering method to represent the process of the virtual surgical instrument passing through the surface membrane to operate on the tissue and organ to improve the rendering speed and performance, that is, when the virtual surgical instrument is close to the surface membrane, a small number of triangle models are used for simplification, and when the virtual surgical instrument penetrates the surface membrane to operate on the tissue and organ, a large number of triangle models are used for detailed scene details to provide a sense of reality visually. By adaptively switching the force feedback parameters for different tissues and organs in real time, the feedback force of different tissues and organs can be obtained, thereby enhancing the sense of reality of the virtual surgery.

[0065] Preferably, the surface rendering and reconstruction method adopts one of the contour method, marching cubes method and opaque cube method.

[0066] Preferably, the surface film is provided with a spatial resolution adjustment strategy, wherein the spatial resolution represents the number of triangular meshes per unit area of ​​the surface film, and the more triangular meshes there are, the higher the spatial resolution of the surface film; the spatial resolution adjustment strategy is configured as follows: when the minimum vertical distance between the tactile pen tip of the virtual surgical instrument and the surface film is less than a first threshold, the spatial resolution of the surface film is improved by increasing the downsampling parameters.

[0067] Preferably, the step of detecting whether the tactile pen tip of the virtual surgical instrument is in contact with the abdominal tissue / organ model comprises:

[0068] Collect the three-dimensional spatial coordinates of virtual surgical instruments in real time;

[0069] Based on the three-dimensional spatial coordinates of the virtual surgical instrument and the three-dimensional spatial coordinate range of the surface membrane of each abdominal tissue / organ model, a collision detection algorithm is used to perform collision detection between the tactile pen tip of the virtual surgical instrument and the surface membrane of all abdominal tissue / organ models;

[0070] If the tactile pen tip of the virtual surgical instrument collides with the surface membrane of any abdominal tissue / organ model, it is determined that the tactile pen tip of the virtual surgical instrument is in contact with the abdominal tissue / organ model.

[0071] Preferably, the step of identifying the type of abdominal tissue / organ model that is contacted comprises:

[0072] Record the three-dimensional coordinate position where the tactile pen tip of the virtual surgical instrument contacts the abdominal tissue / organ model;

[0073] Determining the relationship between the three-dimensional coordinate position and the three-dimensional space coordinate range of the surface membrane of each abdominal tissue / organ model;

[0074] If the three-dimensional coordinate position falls within the three-dimensional space coordinate range of the surface membrane of the corresponding abdominal tissue / organ model, it is determined that the virtual surgical instrument tactile pen tip is in contact with the corresponding abdominal tissue / organ model.

[0075] Specific as Figure 6 As shown, it is a schematic diagram of expanding the tissue and organ segmentation image to generate a surface film in this application, wherein Figure 6 (a) shows the liver boundary after segmentation by the joint segmentation model. Figure 6 (b) shows the schematic diagram of the expanded liver. Figure 6 (c) Schematic diagram of the liver showing the construction of the surface membrane after expansion. Figure 6(d) shows a schematic diagram of the liver after surface membrane downsampling. Since the surface membrane wraps the tissues and organs, before the virtual surgical instrument operates the abdominal tissue / organ model, it first collides with the surface membrane. In order to improve the sensitivity of adaptively switching the force feedback parameters of different tissues and organs and calculating the force feedback, the present application uses the collision of the virtual surgical instrument with the surface membrane as a signal for tissue and organ identification and starting to switch the force feedback parameters. The three-dimensional spatial coordinates of the virtual surgical instrument are collected in real time. According to the calibrated three-dimensional spatial range of the surface membranes of different tissues and organs, a collision detection algorithm is used to perform collision detection between the tactile pen tip of the virtual surgical instrument and the surface membranes of various tissues and organs. If a collision occurs with a certain tissue or organ, the system identifies the tissue or organ and starts to switch the force feedback parameters. The collision detection algorithm used can be OBB, SAT algorithm, FCL-triangle face detection, etc.; when the tactile pen tip of the virtual surgical instrument continues to enter the calibrated three-dimensional spatial boundary range of a specific tissue or organ, it means that the virtual surgical instrument has collided with the tissue or organ, and the force feedback is calculated based on the switched force feedback parameters and the following modules, and cutting, burning and other operations are performed. The higher the spatial resolution of the surface film, the lower the error rate in identifying different tissues and organs when colliding with the surface film, especially for adjacent tissues and organs.

[0076] In order to ensure high sensitivity in adaptively adjusting force feedback parameters, low error rate in tissue and organ identification, and improve calculation speed at the same time, the present invention performs downsampling reconstruction on the acquired surface film, and the downsampling ratio is ds. After downsampling, the number of triangular meshes constituting the surface film is ds*N, where N represents the number of triangular meshes before downsampling. When the minimum vertical distance between the tactile pen tip of the virtual surgical instrument and the surface film is less than a first threshold, the downsampling ratio ds is executed to perform downsampling processing, thereby improving the spatial resolution of the surface film, that is, increasing the number of triangular meshes constituting the surface film.

[0077] When the tactile pen tip of the virtual surgical instrument penetrates the surface membrane and enters the three-dimensional spatial boundary of a specific tissue or organ, the force feedback parameters obtained above are brought into the corresponding physical deformation model, and the elastic deformation and force feedback of each radiation layer in the three dimensions of X, Y, and Z in three-dimensional space are calculated in real time and transmitted to the surgical instrument signal acquisition module. At the same time, the spatial coordinates of the deformed data nodes are updated and visual feedback in the virtual scene is provided.

[0078] It should be noted that, in step S2, the force feedback parameter settings of the corresponding abdominal tissue / organ model are switched to perform force feedback calculations. The force feedback parameter settings of different abdominal tissue / organ models can be collected and calculated in the following manner.

[0079] In order to depict and collect the motion trajectory of the handle in actual laparoscopic surgery, the virtual surgical instrument uses a spherical coordinate system to establish the spatial movement and position expression of the actual handle, including the horizontal rotation angle α and vertical rotation angle β collected when the handle rotates, the diameter R collected when the handle is pulled, and the spin angle θ collected when the handle is shearing, ligating, burning and other operations. In order to capture the action signals of the virtual surgical instrument in real time, a high-precision encoder is installed at the joint of the virtual surgical instrument to capture the angle change, and a force sensor is used to monitor the contact force, and an accelerometer is installed to monitor the acceleration of the action, and these signals are accurately transmitted to the central control unit for processing. The specific force feedback acquisition calculation process is as follows:

[0080] a. For different tissues and organs, multiple experienced clinicians were used as test subjects, and force sensors and strain gauges were used to measure the depth of action, such as the puncture depth or the stretching length L. 测试 The force F required 测试 , the overall longitudinal deformation l 测试 With the contact point as the center, set m marking points from the inside to the outside and record the deformation r in the tangent direction i and the longitudinal deformation y i , where i∈{1,2,…m}. The force sensor can be a resistive strain sensor, a capacitive sensor, etc.;

[0081] b. Construct an initial physical deformation model for different tissues and organs. The physical deformation model used is a spring-mass model based on a hexagonal network topology. When a force is applied, the tissue deformation is centered at the contact point and spreads to the surrounding areas in a radial concentric hexagonal structure. The specific formula is as follows:

[0082]

[0083] α i =2i-1 (4)

[0084]

[0085] Where F represents the total force acting on the tissue or organ, m is the number of radiation layers, which is initialized to 2; Δy i is the spring deformation of the i-th layer particle, i.e., the longitudinal deformation; K i is the spring elastic coefficient in different radiation layers; l i represents the longitudinal spring deformation of the i-th layer of particles; r represents the radius of the hexagonal network topology, which is initialized to 1; α i The physical deformation model used in this application may also be other spring-mass models.

[0086] c. Based on the artificial intelligence model, the model shown in step b is trained according to the offline data collected in step a to obtain the optimal elastic coefficient K i , and calculate the error ∈ = F 测试 -F. The training model used can be artificial intelligence and machine learning methods such as CNN.

[0087] d. Increase the number of radiation layers m, and repeat steps b and c until the error ∈ is less than the set threshold or the number of radiation layers m reaches the set threshold, and determine the optimal number of radiation layers m and the optimal elastic coefficient K i and the optimal network radius r, thereby determining the physical deformation model of specific tissues and organs;

[0088] e. Traverse all organs and tissues, calculate and save the force feedback parameters for different organs and tissues, including the optimal number of radiation layers m and the optimal elastic coefficient K i and the optimal network radius r for subsequent retrieval.

[0089] It should be noted that the process of training the above-mentioned Mamba-CNN joint segmentation model includes: preprocessing and data enhancement of the preset clinical imaging data of abdominal organs, and performing geometric transformation, brightness and contrast intensity transformation, and elastic deformation on the data to increase the data set and enhance the robustness and generalization of the artificial intelligence model;

[0090] The training set and the test set are divided, and the Mamba-CNN joint segmentation model is trained on the training set and tested on the test set to achieve automatic segmentation of different tissues and organs. The selected artificial intelligence model is the Mamba-CNN joint segmentation algorithm. The CNN network is used in the low-order part of the encoder to capture local and detail features, and the Mamba model is used in the high-order part of the encoder to extract global dependency features. In the low-order part, CNN is used to capture local and detail features. The convolutional layer can automatically learn and extract local features in the image, such as edges, textures, etc. In the high-order part, the Mamba model is used to extract global dependency features. The Mamba model can capture the global dependency relationship of time or spatial sequences in the image data.

[0091] It should be noted that the termination condition of model training is that the rate of decrease of loss in multiple consecutive training cycles is less than the preset threshold.

[0092] Using 3D image reconstruction technology, 3D models of different tissues and organs are constructed based on voxels, and smoothing and other processing are performed to calibrate the 3D spatial coordinate range of different tissues and organs and calculate the volume of different tissues and organs. The size of voxels is adjusted to improve the calculation speed while ensuring that the reconstructed object is not distorted;

[0093] The above-mentioned surface rendering reconstruction method is used to perform three-dimensional surface reconstruction on the expanded abdominal tissue / organ model, and the specific process of generating the surface membrane includes:

[0094] The tissue and organ segmentation image is expanded, the expansion structure kernel is K, and the expansion coefficient is inf, which indicates the degree of expansion outward from the expansion structure kernel. The larger the inf, the greater the image expansion degree, and the greater the boundary distance of the tissue and organ before and after expansion. The selected expansion coefficient is related to the acceleration of the action in real laparoscopic surgery. The greater the acceleration, the larger the selected expansion coefficient is to ensure the sensitivity of adaptive switching; the surface rendering reconstruction method is used to perform three-dimensional surface reconstruction of the expanded tissue and organ, which is used as the boundary of collision detection, and the three-dimensional spatial coordinate range of the surface film is calibrated. The selected surface rendering reconstruction method can be the contour method, Marching cubes, Opaque Cube and other methods, and the number of triangular meshes constituting the surface film is initialized. The larger the number, the higher the spatial resolution of the surface film.

[0095] Embodiment 2

[0096] This embodiment provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the adaptive force feedback method described in the first embodiment, and implements the following functions: in order to ensure the sensitivity of the virtual surgical instrument to generate force feedback when contacting the abdominal tissue / organ model, a layer of refined surface film is added on the basis of volume rendering of the tissue and organ as the boundary of the adaptive switching force feedback parameter. The refined surface film has a high spatial resolution, which helps to reduce the misjudgment of the collision detection of the tissue and organ. The expansion coefficient of the surface film is adjusted according to the acceleration of the action of the virtual surgical instrument, and the spatial resolution of the surface film is adjusted according to the minimum vertical distance between the tactile pen tip of the virtual surgical instrument and the surface film, which greatly improves the calculation speed and the accuracy of collision detection and recognition, and improves the quality and authenticity of the virtual surgery. Specifically, the electronic device can be a controller in a specific device.

[0097] Embodiment 3

[0098] Based on the same technical concept, the embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the steps of the above-mentioned adaptive force feedback method. And realize the following functions: In order to ensure the sensitivity of the virtual surgical instrument to generate force feedback when contacting the abdominal tissue / organ model, a layer of refined surface film is added on the basis of volume rendering of the tissue and organ as the boundary of the adaptive switching force feedback parameters. The refined surface film has a high spatial resolution, which helps to reduce the misjudgment of the collision detection of the tissue and organ. The expansion coefficient of the surface film is adjusted according to the acceleration of the virtual surgical instrument action, and the spatial resolution of the surface film is adjusted according to the minimum vertical distance between the tactile pen tip of the virtual surgical instrument and the surface film, which greatly improves the calculation speed and the accuracy of collision detection and recognition, and improves the quality and authenticity of the virtual surgery.

[0099] Embodiment 4

[0100] like Figure 2 As shown, the present application provides an individualized virtual surgical operation device, comprising:

[0101] The virtual surgical instrument signal acquisition module collects the motion trajectory of the virtual surgical instrument handle during virtual laparoscopic surgery, including the spatial position and movement signal acquisition of the handle, the horizontal and vertical rotation angles, the handle pulling diameter, the handle operation spin angle, the contact force, the motion acceleration and other signals, and transmits the collected signals to the central control processing unit for processing;

[0102] The force feedback parameter acquisition module is used to obtain the test deformation of different tissues and organs based on clinical experience, initialize the physical deformation model, then train the physical deformation model for different tissues and organs to determine the optimal parameters, and finally save the force feedback parameters of different tissues and organs for subsequent switching of force feedback parameter settings;

[0103] The adaptive parameter adjustment module and the force feedback calculation and scene rendering module are used to implement the adaptive force feedback method in the first embodiment.

[0104] Specific as Figure 3-Figure 5As shown in the figure, in the virtual surgical instrument signal acquisition module, the virtual surgical instrument uses a spherical coordinate system to establish the spatial movement and position expression of the actual handle, including the horizontal rotation angle α and vertical rotation angle β collected when the handle rotates, the diameter R collected when the handle is pulled, and the spin angle θ collected when the handle is sheared, ligated, burned, etc. In order to capture the action signals of the virtual surgical instrument in real time, a high-precision encoder is installed at the joint of the virtual surgical instrument to capture the angle change, and a force sensor is used to monitor the contact force, and an accelerometer is installed to monitor the acceleration of the action, and these signals are accurately transmitted to the central control unit for processing.

[0105] The force feedback parameter acquisition module uses force sensors and strain gauges to measure the depth of action, such as the puncture depth or the stretch length L, for different tissues and organs, with multiple experienced clinicians as test subjects. 测试 The force F required 测试 , the overall longitudinal deformation l 测试 With the contact point as the center, set m marking points from the inside to the outside and record the deformation r in the tangent direction i and the longitudinal deformation y i , where i∈{1,2,…m}. The force sensor can be a resistive strain sensor, a capacitive sensor, etc.;

[0106] Then, for different tissues and organs, an initial physical deformation model is constructed. The physical deformation model used is a spring-mass model based on a hexagonal network topology structure. When a force is applied, the tissue deformation is centered on the contact point and spreads to the surrounding areas in a radial concentric hexagonal structure.

[0107] Based on the artificial intelligence model, the initial physical deformation model is trained according to the collected offline data to obtain the optimal elastic coefficient. The training model used can be an artificial intelligence and machine learning method such as CNN.

[0108] Determine the optimal number of radiation layers, the optimal elastic coefficient and the optimal network radius, thereby determining the physical deformation model of specific tissues and organs;

[0109] Finally, all organs and tissues are traversed, and the force feedback parameters for different organs and tissues are calculated and saved, including the optimal number of radiation layers, the optimal elastic coefficient, and the optimal network radius, for subsequent retrieval.

[0110] The adaptive parameter adjustment module and the force feedback calculation and scene rendering module first construct abdominal tissue / organ models of different tissues and organs based on clinical medical imaging data, and calibrate their three-dimensional spatial coordinate ranges; then, the tissue and organ segmentation images are expanded and the surface is reconstructed in three dimensions, which is used as the boundary of collision detection, and the three-dimensional spatial coordinate range of the surface membrane is calibrated; then, the movement information of the virtual surgical instrument handle in the virtual laparoscopic surgery collected by the virtual surgical instrument signal acquisition module is obtained, and it is calculated whether the tactile pen tip of the virtual surgical instrument is in contact with the surface membrane, and whether it penetrates the surface membrane and enters the abdominal tissue / organ model; if the tactile pen tip of the virtual surgical instrument is in contact with the surface membrane, the corresponding tissue and organ are identified and its force feedback parameters are called; if the tactile pen tip of the virtual surgical instrument penetrates the surface membrane and enters the abdominal tissue / organ model, force feedback calculation is performed based on the switched force feedback parameter settings, and scene rendering is performed.

[0111] In order to ensure the sensitivity of the virtual surgical instrument to generate force feedback when contacting the abdominal tissue / organ model, the present application adds a layer of refined surface film on the basis of volume rendering of tissues and organs as the boundary of adaptive switching force feedback parameters. The refined surface film has a high spatial resolution, which helps to reduce the misjudgment of tissue and organ collision detection. On the other hand, the expansion coefficient of the surface film is adjusted according to the acceleration of the virtual surgical instrument movement, and the spatial resolution of the surface film is adjusted according to the minimum vertical distance between the tactile pen tip of the virtual surgical instrument and the surface film, which greatly improves the calculation speed and collision detection recognition accuracy, and improves the quality and authenticity of virtual surgery.

[0112] The above describes specific embodiments of the present invention. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0113] In the description of the embodiments of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present invention. In the embodiments of the present invention, the schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in the embodiments of the present invention and the features of the different embodiments or examples, without contradiction.

[0114] In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features, and do not include any ordering. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features and are used to distinguish each other. In the description of the embodiments of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0115] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred implementation of the embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0116] The above description is only a preferred embodiment of the embodiment of the present invention and is not intended to limit the embodiment of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiment of the present invention should be included in the scope of protection of the embodiment of the present invention.

Claims

1. An adaptive force feedback method based on virtual laparoscopic surgery, characterized in that: include: Detecting whether the tactile pen tip of the virtual surgical instrument is in contact with the abdominal tissue / organ model; wherein the outermost layer of the abdominal tissue / organ model is covered with a surface film, and when the tactile pen tip of the virtual surgical instrument is in contact with the surface film, it is determined that it is in contact with the corresponding abdominal tissue / organ model; If the tactile pen tip of the virtual surgical instrument comes into contact with the abdominal tissue / organ model, the type of the abdominal tissue / organ model in contact is identified, and the force feedback parameter setting of the corresponding abdominal tissue / organ model is switched to perform force feedback calculation; Detect whether the tactile pen tip of the virtual surgical instrument penetrates the surface membrane and enters the abdominal tissue / organ model. If so, perform force feedback calculation based on the switched force feedback parameter settings and perform scene rendering.

2. The adaptive force feedback method according to claim 1, characterized in that: The process of modeling the abdominal tissue / organ model includes: Acquiring clinical medical imaging data, and preprocessing the clinical medical imaging data; The preprocessed clinical medical image data is input into the Mamba-CNN joint segmentation model for segmentation processing, and the tissue and organ segmentation image is output; wherein the Mamba-CNN joint segmentation model uses a CNN network in the low-order part of the encoder to capture local and detail features, and uses a Mamba model in the high-order part of the encoder to extract global dependency features; Based on three-dimensional image reconstruction technology, an abdominal tissue / organ model of the corresponding tissues and organs is constructed.

3. The adaptive force feedback method according to claim 2, characterized in that: The surface membrane is generated by expanding the tissue and organ segmentation image to wrap the abdominal tissue / organ model; the distance from the surface membrane to the abdominal tissue / organ model is changed by adjusting the expansion coefficient of the surface membrane, and the minimum unit size and number constituting the surface membrane are adjusted by changing the downsampling parameters.

4. The adaptive force feedback method according to claim 3, characterized in that: The surface membrane is provided with an expansion coefficient adjustment strategy, and the expansion coefficient adjustment strategy is configured as follows: when the acceleration of the virtual surgical instrument movement increases, the expansion coefficient of the surface membrane increases, and when the acceleration of the virtual surgical instrument movement decreases, the expansion coefficient of the surface membrane decreases; wherein the expansion coefficient represents the degree of outward expansion centered on the expansion structure core.

5. The adaptive force feedback method according to claim 4, characterized in that: The step of generating the surface film by dilating the tissue and organ segmentation image comprises: Acquire tissue and organ segmentation images; Performing expansion processing on the tissue and organ segmentation image with the expansion structure core as the center; The surface rendering reconstruction method is used to reconstruct the expanded tissue and organ surface in three dimensions, generate a surface membrane, and calibrate the three-dimensional spatial coordinate range of the surface membrane.

6. The adaptive force feedback method according to claim 5, characterized in that: The surface rendering and reconstruction method adopts one of the contour method, the marching cube method and the isosurface method.

7. The adaptive force feedback method according to claim 5, characterized in that: The surface film is provided with a spatial resolution adjustment strategy, wherein the spatial resolution represents the number of triangular meshes per unit area of ​​the surface film. The more triangular meshes there are, the higher the spatial resolution of the surface film is. The spatial resolution adjustment strategy is configured as follows: when the minimum vertical distance between the tactile pen tip of the virtual surgical instrument and the surface film is less than a first threshold, the spatial resolution of the surface film is improved by increasing a downsampling parameter.

8. The adaptive force feedback method according to claim 5, characterized in that: The steps of detecting whether the tactile pen tip of the virtual surgical instrument is in contact with the abdominal tissue / organ model include: Collect the three-dimensional spatial coordinates of virtual surgical instruments in real time; Based on the three-dimensional spatial coordinates of the virtual surgical instrument and the three-dimensional spatial coordinate range of the surface membrane of each abdominal tissue / organ model, a collision detection algorithm is used to perform collision detection between the tactile pen tip of the virtual surgical instrument and the surface membrane of all abdominal tissue / organ models; If the tactile pen tip of the virtual surgical instrument collides with the surface membrane of any abdominal tissue / organ model, it is determined that the tactile pen tip of the virtual surgical instrument is in contact with the abdominal tissue / organ model.

9. The adaptive force feedback method according to claim 8, characterized in that: Steps for identifying the type of abdominal tissue / organ model that was exposed include: Record the three-dimensional coordinate position where the tactile pen tip of the virtual surgical instrument contacts the abdominal tissue / organ model; Determining the relationship between the three-dimensional coordinate position and the three-dimensional space coordinate range of the surface membrane of each abdominal tissue / organ model; If the three-dimensional coordinate position falls within the three-dimensional space coordinate range of the surface membrane of the corresponding abdominal tissue / organ model, it is determined that the virtual surgical instrument tactile pen tip is in contact with the corresponding abdominal tissue / organ model.

10. The adaptive force feedback method according to claim 9, characterized in that: The collision detection algorithm adopts one of the OBB algorithm, the SAT algorithm and the FCL triangle surface detection algorithm.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the adaptive force feedback method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a computer or a processor, the computer or the processor executes the adaptive force feedback method according to any one of claims 1 to 10.

13. An individualized virtual surgical operation device, characterized in that: Comprising the electronic device as claimed in claim 11.

Images