Bronchoscope intervention analog simulation method and system

Abstract

The invention discloses a bronchoscope intervention analog simulation method and system.The method comprises the steps that an analog simulation platform used for simulating the bronchoscope intervention process is built in advance, the analog simulation platform is connected with a handle rocker and a controller, and a trachea three-dimensional model and a bronchoscope tail end three-dimensional model are built and input into the analog simulation platform; physical parameters and collision constraint parameters are set for the trachea three-dimensional model and the bronchoscope tail end three-dimensional model respectively, so that a virtual trachea and a virtual bronchoscope tail end which can simulate collision are constructed; an electric signal output by the handle rocker is obtained to obtain a real target posture of the tail end of the real bronchoscope, and the controller determines a first driving amount used for controlling the tail end of the virtual bronchoscope according to the current virtual posture of the tail end of the virtual bronchoscope and the real target posture of the tail end of the real bronchoscope; and the analog simulation platform controls the tail end of the virtual bronchoscope to move in the virtual trachea according to the first driving quantity. The method can be applied to bronchoscope intervention simulation training and controller precision verification.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a bronchoscopic intervention simulation method and system. Background Art

[0002] Minimally invasive surgery has an increasingly important position in modern surgery due to its advantages such as less trauma and faster recovery. However, minimally invasive surgery requires extremely high operating accuracy and proficiency of doctors. Before doctors actually operate bronchoscope robots to perform minimally invasive surgery, they need to adapt to the remote control operation process of bronchoscope robots through repeated practice. In addition, with the continuous development of surgical robot technology, surgical path planning and bronchoscope control algorithms have become the key to achieving automatic navigation and description of lung nodules. The process of verifying these algorithms also requires a lot of experimental resources, which increases R&D costs.

[0003] There are two mainstream methods for surgical robot algorithm verification and minimally invasive surgery training. One is the more traditional animal or human model experiment on a real platform, which has problems such as high cost, limited resources, and inability to repeat training. Limited by the scarcity of non-reusable medical devices (bronchoscopes, biopsy needles, ablation needles) and laboratories, this training process consumes a lot of human resources and funds.

[0004] The other is a simulation experiment based on ROS simulation environment and MATLAB / Simulink, which verifies surgical robot algorithms and conducts minimally invasive surgery training through simulation experiments. It is reproducible and can be repeatedly trained. However, the existing controller design method based on MATLAB cannot provide a real physical engine and has restrictions. As a result, the controller designed by the established soft robot model cannot fully consider the disturbance in the real environment and the complexity of nonlinear software modeling, resulting in a large deviation between the ideal control effect and the real effect, and cannot achieve a good display effect.

[0005] In addition, in the current surgical robot algorithm verification process, the algorithm's convergence speed and overshoot rate need to be tested through a bending test on the end of the bronchoscope, which will significantly reduce the service life of the bronchoscope. Moreover, the process of verifying these algorithms consumes a lot of experimental resources, increasing R&D costs. Even because of the loss of human models and bronchoscope consumables, the training cost is much higher than the traditional method, so it is not adopted.

[0006] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application, nor does it necessarily provide technical guidance. In the absence of clear evidence that the above content has been disclosed before the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0007] The purpose of the present invention is to provide a bronchoscopic intervention simulation method and system, which can be applied to bronchoscopic intervention simulation training and verification of controller accuracy.

[0008] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A bronchoscopic intervention simulation method comprises the following steps:

[0010] A simulation platform for simulating a bronchoscope intervention process is pre-built, the simulation platform is connected to a handle rocker and a controller respectively, the handle rocker is configured to output an electrical signal corresponding to a target posture of the bronchoscope end, and the controller is configured to determine a driving amount for controlling the bronchoscope end to reach the target posture according to a current posture and a target posture of the bronchoscope end;

[0011] Constructing a tracheal three-dimensional model according to the structure of a real trachea, establishing a bronchoscope end three-dimensional model according to the structure of a real bronchoscope end, inputting the tracheal three-dimensional model and the bronchoscope end three-dimensional model into the simulation platform, and setting physical parameters and collision constraint parameters for the tracheal three-dimensional model and the bronchoscope end three-dimensional model respectively, so as to construct a virtual trachea and a virtual bronchoscope end capable of simulating collision in the simulation platform;

[0012] Acquiring an electrical signal output by the handle rocker to obtain a real target posture of the real bronchoscope terminal, and the controller determining a first driving amount for controlling the virtual bronchoscope terminal according to the current virtual posture of the virtual bronchoscope terminal and the real target posture of the real bronchoscope terminal;

[0013] The simulation platform controls the movement of the distal end of the virtual bronchoscope in the virtual trachea according to the first driving amount.

[0014] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the controller determines a first driving amount for controlling the virtual bronchoscope terminal according to the current virtual posture of the virtual bronchoscope terminal and the real target posture of the real bronchoscope terminal, comprising the following steps:

[0015] Determine the coordinate system where the end of the virtual bronchoscope is located as a virtual coordinate system, determine the coordinate system where the end of the real bronchoscope is located as a real coordinate system, and determine a mapping relationship between the virtual coordinate system and the real coordinate system;

[0016] According to the mapping relationship, mapping the real target posture of the real bronchoscope end to the virtual coordinate system to obtain a first virtual target posture of the virtual bronchoscope end in the virtual coordinate system;

[0017] The controller determines the first driving amount according to the current virtual posture of the virtual bronchoscope end in the virtual coordinate system and the first virtual target posture;

[0018] or,

[0019] The controller determines a first driving amount for controlling the virtual bronchoscope terminal according to the current virtual posture of the virtual bronchoscope terminal and the real target posture of the real bronchoscope terminal, comprising the following steps:

[0020] Determine the coordinate system where the end of the virtual bronchoscope is located as a virtual coordinate system, determine the coordinate system where the end of the real bronchoscope is located as a real coordinate system, and determine a mapping relationship between the virtual coordinate system and the real coordinate system;

[0021] According to the mapping relationship, mapping the current virtual posture of the virtual bronchoscope end to the real coordinate system to obtain the current real posture of the real bronchoscope end in the real coordinate system;

[0022] The controller determines the first driving amount according to the current real posture of the real bronchoscope tip in the real coordinate system and the real target posture.

[0023] Furthermore, any one of the above-mentioned technical solutions or a combination of multiple technical solutions further includes the following steps:

[0024] Based on the first driving amount, the simulation platform controls the end of the virtual bronchoscope to move to a first posture in the virtual trachea;

[0025] The control accuracy of the controller is determined according to the first posture of the virtual bronchoscope tip and the first virtual target posture; or, according to the mapping relationship, the first posture is mapped to the real coordinate system to obtain the first real posture of the real bronchoscope tip in the real coordinate system, and the control accuracy of the controller is determined according to the first real posture of the real bronchoscope tip and the first real target posture.

[0026] Furthermore, any one of the above-mentioned technical solutions or a combination of multiple technical solutions further includes the following steps:

[0027] Determining a current posture of the virtual bronchoscope tip and a second virtual target posture of the virtual bronchoscope tip;

[0028] The controller determines a second driving amount according to the current posture of the virtual bronchoscope tip and the second virtual target posture;

[0029] Based on the second driving amount, the simulation platform controls the end of the virtual bronchoscope to move to a second posture in the virtual trachea;

[0030] Determining the simulation accuracy of the simulation platform according to the second posture of the virtual bronchoscope end and the second virtual target posture;

[0031] If the simulation accuracy does not meet the preset requirements, the simulation platform is optimized until the simulation accuracy meets the preset requirements; if the simulation accuracy meets the preset requirements, the simulation platform is applied to simulate bronchoscopic intervention.

[0032] Furthermore, any one of the above-mentioned technical solutions or a combination of multiple technical solutions further includes:

[0033] For the end of the virtual bronchoscope, a continuum dynamics model MΔv=dt(f(x,t)) is constructed, wherein M is the mass matrix of the end of the virtual bronchoscope, x is the position of the end of the virtual bronchoscope, Δv is the velocity change of the end of the virtual bronchoscope, and f(x,t) represents the external force function of the end of the virtual bronchoscope;

[0034] The simulation platform is based on the continuum dynamics model MΔv=dt(f(x,t)) and utilizes a forward and inverse differential equation solver or a linear solver to simulate the movement process and bending effect of the virtual bronchoscope tip in the virtual trachea.

[0035] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the controller is configured with an automatic target aiming function, and the automatic target aiming function includes: when the distance between the end of the bronchoscope and the target is not greater than a preset distance threshold, the controller controls the end of the bronchoscope to automatically aim at the target;

[0036] The automatic aiming target function is simulated by using the simulation platform in the following manner:

[0037] Determining a virtual target point on the virtual trachea, and determining the coordinates of the virtual target point;

[0038] Determining the coordinates of the end of the virtual bronchoscope, and determining the distance from the end of the virtual bronchoscope to the virtual target point according to the coordinates of the end of the virtual bronchoscope and the coordinates of the virtual target point;

[0039] When the distance is not greater than a preset distance threshold, the controller determines a third driving amount according to the coordinates of the virtual bronchoscope tip, the current posture and the coordinates of the virtual target, and the simulation platform controls the movement of the virtual bronchoscope tip according to the third driving amount.

[0040] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the first driving amount is the driving amount of the guide wire, and the guide wire is configured to control the posture of the end of the real bronchoscope;

[0041] Based on the discrete PID controller of linear stepper motor control, the first driving quantity [u 1 ,u 2 ,u 3 ,u 4 ] T Determined by the following formula:

[0042] e i (k) = sd i -s i (k)

[0043] u i (k) = K p *e i (k)+K i *(e i (k)+e i (k-1))+K d *(e i (k)-e i (k-1))

[0044] Among them, u i (k)=[u 1 ,u 2 ,u 3 ,u 4 ] T , i is an integer from 1 to 4, e i (k) represents the wire state error at time k, sd i Indicates the guidewire state quantity corresponding to the desired posture, s i (k) represents the guide wire state corresponding to the current posture, e i (k-1) represents the wire state error at time k-1, K p is the scale parameter, K i is the integration parameter, K d is the differential parameter.

[0045] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the first driving amount [u 1 ,u 2 ,u3 ,u 4 ] T The following steps are also included:

[0046] The internal force of the guidewire at each step is calculated by the following formula: f(x j )≈f(x j-1 )+K(x j-1 )dx, where f(x j ) is the position x of the guide wire j The internal stiffness force at j-1 ) is the position x of the guide wire j-1 Internal stiffness force at x j and x j-1 are two adjacent nodes on the guidewire, K(x j-1 ) is the position x of the guidewire j-1 The corresponding tangent stiffness matrix, K(x j-1 ) is the position x of the guidewire j-1 The corresponding tangent stiffness matrix at ;

[0047] The guide wire is kept in static balance in each step by the following formula: j-1 )dx=p+f(x j-1 )+J T λ, where p represents the external force on the guidewire, including gravity, and J T represents the transpose of the Jacobian matrix, λ represents the Lagrange multiplier;

[0048] Projecting the finite element model of the guidewire into the constraint space includes:

[0049] By setting λ = 0 and solving the equation -K(x j-1 )dx=p+f(x j-1 )+J T λ to find the free displacement x of the guide wire free , and the vector between the actual posture and the desired posture of the virtual bronchoscope tip is

[0050] Project the mechanics into the constraint space to get the smallest possible projection space for the inverse problem: δ e The vector representing the position of the virtual bronchoscope tip based on the constraint, J a T represents the transpose of the Jacobian matrix of the driving guide wire, λ a represents the Lagrange multiplier driving the guide wire;

[0051] At the end of the time step, the value of the constraint response is used to correct the final configuration, thereby obtaining the relationship between the deformation amount of the virtual bronchoscope tip and the pull wire length of the guide wire: x = x free +K -1 J a T λ a , x is the end position of the virtual bronchoscope.

[0052] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the simulation platform is configured with a first-view display module and a global-view display module; wherein,

[0053] The first viewing angle display module is configured to display a moving picture of the virtual bronchoscope tip in the virtual trachea along the moving direction of the virtual bronchoscope tip in the virtual trachea;

[0054] The global viewing angle display module is configured to display the virtual trachea in perspective and to globally display a moving picture of the tip of the virtual bronchoscope in the virtual trachea.

[0055] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, it also includes using SOFA to construct the simulation platform, and configuring the trachea three-dimensional model and the bronchoscope end three-dimensional model as a software structure in the SOFA;

[0056] Physical parameters and collision constraint parameters are set for the trachea three-dimensional model and the bronchoscope terminal three-dimensional model, respectively, wherein the physical parameters of the trachea three-dimensional model include one or more of the mass, density, Young's modulus, Poisson's ratio, friction coefficient and damping coefficient in the tracheal cavity and elastic modulus of the tracheal inner wall; the physical parameters of the bronchoscope terminal include the mass, density, Young's modulus, Poisson's ratio, friction coefficient and damping coefficient of the bronchoscope terminal; the collision constraint parameters include that the maximum driving speed cannot exceed 15.7 mm / s, and the maximum traction length cannot exceed 14 mm.

[0057] Furthermore, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the three-dimensional model of the bronchoscope end is constructed in the following manner:

[0058] Constructing a first bronchoscope end three-dimensional model of the bronchoscope end using three-dimensional software;

[0059] Performing meshing processing on the first bronchoscope terminal three-dimensional model to obtain a second bronchoscope terminal three-dimensional model that can be recognized by the SOFA, wherein the second bronchoscope terminal three-dimensional model is configured as the bronchoscope terminal three-dimensional model;

[0060] and / or,

[0061] It also includes constructing the trachea three-dimensional model in the following manner:

[0062] Pre-acquire CT images including the lung airways;

[0063] Segmenting the tracheal structure in the CT image using an image segmentation technique, and performing three-dimensional reconstruction on the tracheal structure to obtain a first tracheal three-dimensional model;

[0064] Performing meshing processing on the first trachea three-dimensional model to obtain a second trachea three-dimensional model that can be recognized by the SOFA, wherein the second trachea three-dimensional model is configured as the trachea three-dimensional model;

[0065] and / or,

[0066] An effector is set at the end of the virtual bronchoscope, and the effector is a sampling point for reading the coordinates and direction of the end of the virtual bronchoscope in real time. The coordinates p of the end of the virtual bronchoscope are read by the effector. t =[x t ,y t ,z t ] and direction d t =[α, β, γ], and determine the current virtual posture of the virtual bronchoscope tip according to the coordinates and direction of the virtual bronchoscope tip.

[0067] Furthermore, based on any one of the technical solutions or a combination of multiple technical solutions mentioned above, the simulation platform is used to verify the control accuracy of the controller. If the control accuracy of the controller meets the preset control accuracy requirements, the controller is applied to the bronchoscope robot.

[0068] According to another aspect of the present invention, there is provided a bronchoscope intervention simulation system, comprising a pre-built simulation platform, the simulation platform being connected to a handle rocker and a controller respectively, the handle rocker being configured to output an electrical signal corresponding to a target posture of the bronchoscope end, the controller being configured to determine a driving amount for controlling the posture of the bronchoscope end according to a current posture and a target posture of the bronchoscope end;

[0069] The bronchoscopic intervention simulation system is configured to perform bronchoscopic intervention simulation using the bronchoscopic intervention simulation method described in any one of the above technical solutions or a combination of multiple technical solutions.

[0070] The beneficial effects brought by the technical solution provided by the present invention are as follows:

[0071] a. The present invention constructs a high-simulation simulation environment through a pre-established simulation platform, which can simulate the visual effects, collision effects, etc. in the real intervention process with high simulation, and can replace traditional animal experiments and human model experiments, reducing the consumption of experimental resources and greatly reducing training costs. At the same time, the automatic setting function of the simulation platform system greatly shortens the operation preparation time, improves the training efficiency, and realizes low-cost and high-efficiency training;

[0072] b. The bronchoscope intervention simulation method provided by the present invention can not only be used for simulation training of bronchoscope intervention process, but also can be used to verify and improve the accuracy of the controller in the bronchoscope robot system, and can also be used to verify and improve the simulation accuracy of the simulation platform itself;

[0073] c. The bronchoscopic intervention simulation method provided by the present invention connects the simulation platform with the bronchoscopic robot remote control device, the joystick and the controller, which not only improves the integration of the robot product, but also realizes multi-degree-of-freedom operation, and can meet the needs of users for accurate training of complex surgical skills;

[0074] d. The bronchoscopic intervention simulation method provided by the present invention provides the possibility of repeated practice: doctors can practice repeatedly in the same operation scenario until they fully master the bronchoscopic operation skills. This simulation-based training system breaks through the one-time operation limitation of traditional training methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 A schematic diagram of a flow chart of a bronchoscopic intervention simulation method provided by an exemplary embodiment of the present invention;

[0077] Figure 2 A schematic diagram of the structure of a virtual trachea provided for an exemplary embodiment of the present invention;

[0078] Figure 3 A schematic diagram of the structure of a virtual bronchoscope tip provided for an exemplary embodiment of the present invention;

[0079] Figure 4 A comparison diagram of a bending posture of a real bronchoscope tip and a 45° bending of a virtual bronchoscope tip provided for an exemplary embodiment of the present invention;

[0080] Figure 5 A comparison diagram of a bending posture of a real bronchoscope tip and a 90° bending of a virtual bronchoscope tip provided for an exemplary embodiment of the present invention;

[0081] Figure 6 A schematic diagram of a simulation interface of a first perspective and a global perspective provided for an exemplary embodiment of the present invention;

[0082] Figure 7 A simulation schematic diagram of a virtual bronchoscope tip being located near a virtual target point provided for an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0083] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0084] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0085] In addition to the above-mentioned deficiencies in the prior art, although the existing interventional surgery simulators can provide certain training functions, they also have obvious deficiencies in simulation and operational feel, including: (1) Most bronchoscope controller simulation test schemes cannot reflect the elastic deformation of the bronchoscope during bending; (2) Some lung bronchoscope intervention simulations have no collision settings, which makes it easy for the bronchoscope to penetrate the lung model; (3) In some simulation schemes, the bronchoscope lung intervention simulation completely follows the tracheal centerline trajectory, and the trajectory is too ideal to reflect the actual tracheal navigation scene, which greatly reduces the training effect.

[0086] Therefore, the present invention aims to provide a highly simulated, low-cost, and high-efficiency bronchoscope controller testing scenario to solve the problem of large loss of the bronchoscope body during bronchoscope controller testing; in addition, the present invention provides a safe and efficient training platform for surgeons by constructing a highly realistic surgical environment and simulating the collision and visual effects between the bronchoscope and the tracheal wall in real surgery, so as to improve the success rate of the operation and ensure the safety of patients during the operation.

[0087] The symbols designed in this invention are defined as follows:

[0088] (1) Bending angle of the bronchoscope tip: θ;

[0089] (2) Deflection angle of the bronchoscope tip:

[0090] (3) Target bending angle of the bronchoscope tip: θ d ;

[0091] (4) Target deflection angle of the bronchoscope tip:

[0092] (5) Spatial coordinates of the end of the bronchoscope: p t =[x t ,y t ,z t ];

[0093] (6) Bronchoscope tip direction vector: d t =[α,β,γ];

[0094] (7) The state quantities of the four guide wires corresponding to the actual posture: s = [s 1 ,s 2 ,s 3 ,s 4 ];

[0095] (8) The state quantities of the four guide wires corresponding to the desired posture: sd = [sd 1 ,sd 2 ,sd 3 ,sd 4 ];

[0096] (9) Arc length of the end of the bronchoscope: l;

[0097] (10) Cross-sectional radius of the end of the bronchoscope: R 1 ;

[0098] (11) Plane coordinates of the joystick: p joystick =[x joystick ,y joystick ];

[0099] (12) Bronchoscope robot execution end coordinates: p tip =[x tip ,y tip ,z tip ].

[0100] In one embodiment of the present invention, a bronchoscopic intervention simulation method is provided. Figure 1 , the method comprises the following steps:

[0101] A simulation platform for simulating a bronchoscope intervention process is pre-built, the simulation platform is connected to a handle rocker and a controller respectively, the handle rocker is configured to output an electrical signal corresponding to a target posture of the bronchoscope end, and the controller is configured to determine a driving amount for controlling the posture of the bronchoscope end according to a current posture and a target posture of the bronchoscope end;

[0102] A tracheal three-dimensional model is constructed according to the structure of a real trachea, and a bronchoscope end three-dimensional model is constructed according to the structure of a real bronchoscope end. The tracheal three-dimensional model and the bronchoscope end three-dimensional model are input into the simulation platform, and physical parameters and collision constraint parameters are set for the tracheal three-dimensional model and the bronchoscope end three-dimensional model respectively, so as to construct a virtual trachea and a virtual bronchoscope end capable of simulating collision in the simulation platform; it should be noted that the virtual trachea is not limited to a virtual trachea constructed based on a human tracheal structure, but also includes a virtual trachea constructed based on an animal tracheal structure;

[0103] Acquiring an electrical signal output by the handle rocker to obtain a real target posture of the real bronchoscope terminal, and the controller determining a first driving amount for controlling the virtual bronchoscope terminal according to the current virtual posture of the virtual bronchoscope terminal and the real target posture of the real bronchoscope terminal;

[0104] The simulation platform controls the movement of the distal end of the virtual bronchoscope in the virtual trachea according to the first driving amount.

[0105] It should be noted that the handle rocker and the controller are both preset components of the bronchoscope robot system for controlling the end of a real bronchoscope. In order to be able to truly simulate the bronchoscope intervention process and verify the accuracy and performance of the bronchoscope robot system, including the accuracy of the controller. The simulation platform in this application is electrically connected to the handle rocker through an I / O interface to obtain the electrical signal output by the handle rocker. The controller can be electrically connected to the simulation platform through another I / O interface, or the controller can be integrated into the simulation platform as a module.

[0106] In this embodiment, there are two ways for the controller to determine the first driving amount for controlling the virtual bronchoscope tip according to the current virtual posture of the virtual bronchoscope tip and the real target posture of the real bronchoscope tip.

[0107] One method is: determining the coordinate system where the virtual bronchoscope end is located as a virtual coordinate system, determining the coordinate system where the real bronchoscope end is located as a real coordinate system, and determining a mapping relationship between the virtual coordinate system and the real coordinate system; according to the mapping relationship, mapping the real target posture of the real bronchoscope end to the virtual coordinate system to obtain a first virtual target posture of the virtual bronchoscope end in the virtual coordinate system; the controller determines the first driving amount according to the current virtual posture of the virtual bronchoscope end in the virtual coordinate system and the first virtual target posture.

[0108] After obtaining the first driving amount in this way, based on the first driving amount, the simulation platform controls the virtual bronchoscope tip to move to a first posture in the virtual trachea. The control accuracy of the controller is determined according to the first posture of the virtual bronchoscope tip and the first virtual target posture. If the control accuracy of the controller meets the preset control accuracy requirements, it is determined that the controller can be applied to the bronchoscopic surgery system. If the control accuracy of the controller does not meet the preset control accuracy requirements, it is necessary to optimize and improve the controller, including optimizing algorithms, optimizing parameters, etc. until the control accuracy of the controller meets the preset control accuracy requirements, and then apply the controller to the bronchoscopic surgery system.

[0109] Another method is: determining the coordinate system where the virtual bronchoscope end is located as a virtual coordinate system, determining the coordinate system where the real bronchoscope end is located as a real coordinate system, and determining a mapping relationship between the virtual coordinate system and the real coordinate system; according to the mapping relationship, mapping the current virtual posture of the virtual bronchoscope end to the real coordinate system to obtain the current real posture of the real bronchoscope end in the real coordinate system; the controller determines the first driving amount according to the current real posture of the real bronchoscope end in the real coordinate system and the real target posture.

[0110] After obtaining the first driving amount in this way, based on the first driving amount, the simulation platform controls the virtual bronchoscope end to move to a first posture in the virtual trachea. According to the mapping relationship, the first posture is mapped to the real coordinate system to obtain the first real posture of the real bronchoscope end in the real coordinate system, and the control accuracy of the controller is determined according to the first real posture and the first real target posture of the real bronchoscope end. If the control accuracy of the controller does not meet the preset control accuracy requirements, it is necessary to optimize and improve the controller, including optimizing algorithms, optimizing parameters, etc. until the control accuracy of the controller meets the preset control accuracy requirements, and then apply the controller to the bronchoscope surgery system.

[0111] In one embodiment of the present invention, the bronchoscopic intervention simulation method further comprises determining and improving the simulation accuracy of the simulation platform by:

[0112] Determining a current posture of the virtual bronchoscope tip and a second virtual target posture of the virtual bronchoscope tip;

[0113] The controller determines a second driving amount according to the current posture of the virtual bronchoscope tip and the second virtual target posture;

[0114] Based on the second driving amount, the simulation platform controls the end of the virtual bronchoscope to move to a second posture in the virtual trachea;

[0115] Determining the simulation accuracy of the simulation platform according to the second posture of the virtual bronchoscope end and the second virtual target posture;

[0116] If the simulation accuracy does not meet the preset requirements, the simulation platform is optimized until the simulation accuracy meets the preset requirements; if the simulation accuracy meets the preset requirements, the simulation platform is applied to simulate the bronchoscopic intervention process.

[0117] In one embodiment of the present invention, the construction process of the simulation platform, virtual trachea and virtual bronchoscope end is as follows.

[0118] The present invention uses the open source C++ library SOFA physics engine, OpenGL computer graphics rendering technology, etc. to build the simulation platform, and constructs a highly realistic bronchoscopic surgery simulation environment. The system can realistically restore the bronchoscopic surgery scene and provide a virtual environment for bronchoscopic controller testing. The SOFA physics engine realizes bending, deformation, and collision effects for the virtual bronchoscope, and OpenGL renders the lung environment. In addition, multiple IO interfaces are designed for different controllers and joysticks to connect for simulation and testing, avoiding excessive use of physical bronchoscopes and resulting in loss.

[0119] It should be noted that, in other embodiments, a simulation system based on Unity3D or Unreal Engine can be used to replace the SOFA physics engine. Unity and Unreal engines can provide similar 3D scene construction and real-time feedback functions through physical plug-ins. Although the accuracy of these two engines in simulating the movement of soft robots is not as good as SOFA, they can be used as alternatives in some simple scenarios.

[0120] The bronchoscope end three-dimensional model is constructed in the following manner:

[0121] A first bronchoscope end 3D model of the bronchoscope end is constructed using 3D software. For example, according to the design drawing of the bronchoscope, a 3D structural model of the bronchoscope end is created using software such as CAD / SolidWorks. It should be noted that the 3D structural model of the bronchoscope end includes a soft structure of the bronchoscope end and a plurality of guide wires, usually 4 guide wires, arranged inside the soft structure and used to pull the bronchoscope end to change the posture of the bronchoscope end.

[0122] The first bronchoscope end three-dimensional model is meshed to obtain a second bronchoscope end three-dimensional model in an obj data format that can be recognized by the SOFA, and the second bronchoscope end three-dimensional model is configured as the bronchoscope end three-dimensional model. In a specific embodiment of the present invention, the bronchoscope end three-dimensional model is constructed as follows Figure 3 shown.

[0123] The trachea three-dimensional model is constructed in the following manner:

[0124] Pre-acquire CT images including the lung airways;

[0125] Segmenting the tracheal structure in the CT image including a three-dimensional surface model of the lung airways and bronchi using an image segmentation technique, and performing three-dimensional reconstruction on the tracheal structure to obtain a first three-dimensional tracheal model;

[0126] The first trachea three-dimensional model is meshed by using a triangular mesh and a tetrahedral mesh to obtain a second trachea three-dimensional model in an obj data format that can be recognized by the SOFA, and the second trachea three-dimensional model is configured as the trachea three-dimensional model. In a specific embodiment of the present invention, the trachea three-dimensional model constructed is as follows Figure 2 shown.

[0127] In this embodiment, the tracheal three-dimensional model and the bronchoscope end three-dimensional model are configured as software structures in the SOFA, and physical parameters and collision constraint parameters are set for the tracheal three-dimensional model and the bronchoscope end three-dimensional model respectively.

[0128] In order to simplify the bending calculation of the bronchoscope end, the first bronchoscope end three-dimensional model is preferably meshed with triangular meshes and tetrahedral meshes by QTetraMesher, so that the continuum structure is converted into a finite mesh model to obtain the second bronchoscope end three-dimensional model, and the bending drive is performed by setting the stretching guide wire. Using the finite element method to split and discretize the soft structure can obtain more accurate deformation and position data.

[0129] At the same time, the software structure calculation characteristics of the SOFA physics engine are used to set the physical parameters of the third model of the second bronchoscope tip, including the mass, density, Young's modulus, Poisson's ratio, friction coefficient and damping coefficient of the bronchoscope tip. Four deformable guide wires are set as bronchoscope bending drives, and the model's Young's modulus, Poisson's ratio and other parameters are set to achieve control of the model's deformation performance (in a specific embodiment of the present invention, the Young's modulus and Poisson's ratio of the guide wire are set to 206 Gpa and 0.28). At the same time, external forces are set to simulate gravity and disturbances. Based on the above design, the bending state of the bronchoscope in the real environment can be basically restored in the computer. Figure 3 and Figure 4 As shown, the virtual bronchoscope end in the simulation platform provided by the present invention simulates the changes of the real bronchoscope end very accurately. It can be seen that the simulation platform provided by the present application has a very high simulation accuracy.

[0130] The control objectives to be achieved based on the constructed kinematic model are the bending angle θ and the deflection angle of the bronchoscope tip when it is bent. The expected length change of the guide wire in the robot and the posture of the end of the bronchoscope satisfy the constant curvature model relationship:

[0131]

[0132] Among them, s 1 、s 2 、s 3 、s 4is the state quantity of the four guide wires used to drive the posture change of the bronchoscope end, κ represents the curvature of the bronchoscope end, l represents the arc length of the bronchoscope end, R 1 represents the cross-sectional radius of the end of the bronchoscope, Indicates the bending angle.

[0133] An effector is set at the end of the virtual bronchoscope, and the effector is a sampling point for reading the position and direction of the end of the virtual bronchoscope in real time. The effector is configured to obtain the coordinates p of the end of the virtual bronchoscope in real time. t =[x t ,y t ,z t ] and direction d t =[α,β,γ], where x t ,y t ,z t The data respectively represent the coordinates of the sampling points corresponding to the effector in the virtual coordinate system, and α, β, and γ represent the Euler angles. The real-time bending angle θ and deflection angle of the end of the virtual bronchoscope can be calculated based on the data. Here, the reference direction vector z is set to [0, 0, 1], then the bending angle θ and the deflection angle The calculation formula is as follows:

[0134]

[0135] The physical parameters of the tracheal three-dimensional model include one or more of the mass, density, Young's modulus, Poisson's ratio, friction coefficient and damping coefficient in the tracheal cavity, and elastic modulus of the tracheal inner wall. The collision constraint parameters include that the maximum driving speed cannot exceed 15.7 mm / s, and the maximum traction length cannot exceed 14 mm.

[0136] The bending angle θ of the end of the virtual bronchoscope is (i.e., the current virtual posture) is input into the controller, and the controller calculates the virtual bronchoscope end according to the bending angle θ and the deflection angle The target bending angle θ corresponding to the target virtual posture of the end of the virtual bronchoscope d Deflection angle with target The driving amount required to make the virtual bronchoscope terminal reach the desired posture, ie, the target virtual posture, is calculated and inputted into the virtual bronchoscope terminal.

[0137] Among them, the target bending angle θ corresponding to the target virtual posture d Deflection angle with target Determined according to the electrical signal output by the joystick. In one embodiment of the present invention, the joystick is accessed through the IO interface provided by the pygame development library to input the electrical signal output by the joystick into the simulation platform. Based on the mapping relationship between the joystick and the target posture of the end of the bronchoscope, it is ensured that the joystick and the button can achieve corresponding motion effects. The buttons on the side of the handle are used to control the advancement and retraction of the bronchoscope, and the joystick controls the bending deformation of the end of the bronchoscope robot. The maximum bending angle of the end of the bronchoscope is limited to 90 degrees, and the resulting mapping relationship between the joystick and the bending angle is as follows:

[0138]

[0139]

[0140] Among them, x joystick With y joystick is the output value of the joystick, x joystick With y joystick The horizontal coordinate and the vertical coordinate of the joystick have a corresponding relationship. When the absolute value of the output value of the joystick is less than or equal to 0.1, θ is directly set to 0. This setting is used to set the joystick dead zone.

[0141] For the end of the virtual bronchoscope, a continuum dynamics model MΔv=dt(f(x,t)) is constructed, wherein M is the mass matrix of the end of the virtual bronchoscope, x is the degree of freedom of the end of the virtual bronchoscope, Δv is the velocity change of the end of the virtual bronchoscope, and f(x,t) represents the external force function of the end of the virtual bronchoscope;

[0142] The simulation platform is based on the continuum dynamics model MΔv=dt(f(x,t)) and uses a forward and inverse differential equation solver or a linear solver to simulate the motion process and bending effect of the virtual bronchoscope tip in the virtual trachea. Specifically, the corresponding forward and inverse differential equation solver or linear solver of the SOFA software simulation platform is used to calculate the motion process and bending effect of the virtual bronchoscope tip in the lungs.

[0143] The simulation platform is configured with a first perspective display module and a global perspective display module. The first perspective display module is configured to display the motion picture of the virtual bronchoscope tip in the virtual trachea along the motion direction of the virtual bronchoscope tip in the virtual trachea. The global perspective display module is configured to display the virtual trachea in perspective and to globally display the motion picture of the virtual bronchoscope tip in the virtual trachea. Figure 6 The left side illustrates the advancement process of the bronchoscope in the lung airway based on SOFA displayed by the first viewing angle display module, as shown in FIG. Figure 6The right side illustrates the global status of the SOFA-based pulmonary airway bronchoscope displayed by the second viewing angle display module.

[0144] In one embodiment of the present invention, the controller is configured with an automatic aiming target function, and the automatic aiming target function includes: when the distance between the end of the bronchoscope and the target is not greater than a preset distance threshold, the controller controls the end of the bronchoscope to automatically aim at the target. When the end of the bronchoscope approaches the target, the manual remote control input is terminated.

[0145] In this embodiment, the automatic aiming target function is simulated by using the simulation platform in the following manner:

[0146] Determining a virtual target point on the virtual trachea, and determining the coordinates of the virtual target point;

[0147] Determining the coordinates of the end of the virtual bronchoscope, and determining the distance from the end of the virtual bronchoscope to the virtual target point according to the coordinates of the end of the virtual bronchoscope and the coordinates of the virtual target point;

[0148] When the distance is not greater than a preset distance threshold, the controller determines a third driving amount according to the coordinates of the virtual bronchoscope tip, the current posture and the coordinates of the virtual target, and the simulation platform controls the movement of the virtual bronchoscope tip according to the third driving amount.

[0149] According to the coordinates of the virtual target point (target lung node) in the initial lung model, i.e., the virtual trachea, and the coordinates of the end of the virtual bronchoscope read in real time in the software [x tip ,y tip ,z tip ], using geometric methods to solve the expected value of the target posture of the soft robot and the corresponding expected length change [s 1 ,s 2 ,s 3 ,s 4 ] T Finally, the automatic aiming function of the bronchoscope tip is realized using an inverse solver or a control algorithm such as PID or model predictive control.

[0150] In this embodiment, the process of using a general quadratic programming inverse solver and a PID controller to solve the desired length change and driving amount of the guide wire is as follows.

[0151] The finite element model (FEM) of the guidewire is projected into the constraint space. The finite element model of the guidewire is a set of unit combinations that are connected only at the nodes, transmit forces only at the nodes, and are constrained only at the nodes. The nonlinear geometric deformation is converted into an iterative linearization problem. In each step of the simulation, the linearized internal force of the guidewire is calculated:

[0152] f(x j )≈f(x j-1 )+K(x j-1 )dx, where f(x j ) is the position x of the guide wire j The internal stiffness force at j-1 ) is the position x of the guide wire j-1 Internal stiffness force at x j and x j-1 are two adjacent nodes on the guidewire, K(x j-1 ) is the tangent stiffness matrix that depends on the actual position of the node, K(x j-1 ) is the position x of the guidewire j-1 Correspondence Tangent Stiffness Matrix .

[0153] The guidewire is kept in static balance in each step by the following formula:

[0154] -K(x j-1 )dx=p+f(x j-1 )+J T λ, where p represents the external force on the guidewire, including gravity, J T represents the transpose of the Jacobian matrix and λ represents the Lagrange multiplier.

[0155] Projecting the finite element model of the guidewire into the constraint space includes:

[0156] (1) By setting λ = 0 and solving the equation -K(x j-1 )dx=p+f(x j-1 )+J T λ to find the free displacement x of the guide wire free , that is, how the system would move without constraints, and the vector between the actual and desired positions of the effector is

[0157] (2) Project the mechanics into the constraint space to obtain the minimum possible projection space of the inverse problem: δ e The vector representing the position of the virtual bronchoscope tip based on the constraint, J a Trepresents the transpose of the Jacobian matrix of the driving guide wire, λ a represents the Lagrange multiplier driving the guide wire;

[0158] (3) At the end of the time step, the value of the constraint response is used to correct the final configuration, thereby obtaining the relationship between the deformation amount of the virtual bronchoscope tip and the wire pulling length of the guide wire: x = x free +K -1 J a T λ a , x is the end position of the virtual bronchoscope.

[0159] Among them, step (2) is the core step in the inverse solver, J e K -1 J a T It is a quadratic convex function, which involves the optimization method to solve the minimum value. The general formula for solving the quadratic programming problem is:

[0160]

[0161] The traditional motor control algorithm controls the traction length based on the input torque, while the output of the linear stepper motor and the controllable objects of the simulation platform are the change amount and change rate of the guide wire. In actual operation, considering the limitations of the motor speed mode, it is preferred to use the position output mode.

[0162] Based on the discrete PID controller controlled by the linear stepper motor, the driving amount of the guide wire (including the first driving amount and the second driving amount) is expressed by the following formula:

[0163] e i (k) = sd i -s i (k)

[0164] u i (k) = K p *e i (k)+K i *(e i (k)+e i (k-1))+K d *(e i (k)-e i (k-1))

[0165] Among them, e i (k) represents the wire state error at time k, sd i Indicates the guidewire state quantity corresponding to the desired posture, s i (k) represents the guide wire state corresponding to the current posture, e i(k-1) represents the wire state error at time k-1, K p is the scale parameter, K i is the integration parameter, K d is the differential parameter.

[0166] Finally, it is observed whether the driving amount can drive the virtual bronchoscope terminal to reach the target virtual posture after being input into the virtual bronchoscope terminal. Figure 4 This is a comparison of the bending posture of the real bronchoscope end and the 45° bending of the virtual bronchoscope end. Figure 5 This is a comparison diagram of the bending posture of the real bronchoscope end and the 90° bending of the virtual bronchoscope end. It can be seen that the virtual bronchoscope end can basically restore the bending effect of the real bronchoscope end, so the simulation platform can be used to test the controller.

[0167] The above-mentioned bronchoscope intervention simulation method provided by the present invention not only has high simulation, can reliably and accurately simulate the bronchoscope operation process, is used for the training of bronchoscope operation, but also can test the performance of the bronchoscope controller, can reduce the number of tests of the bronchoscope entity, and reduce the loss of the bronchoscope. As described in the above embodiment, the bending state of the virtual bronchoscope is estimated by the position coordinates and direction of the virtual bronchoscope end, and the real-time coordinates and direction can be directly obtained by setting the tracking point at the end of the virtual bronchoscope in the Sofa physics engine. A controller (such as a PID controller, a fuzzy logic controller) interface is provided, and the controller is used to calculate the driving amount required to change the posture of the bronchoscope end, and then the driving amount is input into the virtual bronchoscope end to drive the virtual guide wire, and the deformation state of the virtual bronchoscope end is updated, so that the control effect of the controller can be obtained without using the real object, greatly reducing the loss of the real bronchoscope.

[0168] The bending state of the virtual bronchoscope is estimated based on the coordinates of the end of the bronchoscope, and the controller is used to update the bronchoscope drive amount (the length of the guidewire required to pull the wire). The virtual bronchoscope is input and the bending state of the bronchoscope is updated to achieve the performance test of the bronchoscope bending controller without using a physical bronchoscope; by importing the human lung model and manipulating the external joystick, the bronchoscope is controlled to move forward, backward, and bend in a collision lung model in a simulated environment. At the same time, the interactive interface will provide the first-person perspective of the bronchoscope and the global perspective of the lung perspective to assist doctors in completing the exploration and training of various branches of the lung.

[0169] In addition, when the end of the bronchoscope is about to reach the nodule position, i.e. the target point, it can be switched to the automatic aiming simulation mode, and the end of the bronchoscope will automatically aim at the target point. The aiming accuracy depends on the performance of the controller used for the test. In this case, a PID controller is used. Based on the above scheme, a safe and efficient training platform is provided for surgeons to help them master the operation skills of the bronchoscope robotic surgery system.

[0170] The bronchoscopic intervention simulation method provided by the present invention has the following advantages:

[0171] Highly realistic simulation environment: Through computer graphics, topology, etc., a highly realistic surgical environment is constructed, which can simulate the visual effects and collision effects in real surgery with high simulation;

[0172] Multi-degree-of-freedom remote control equipment: It provides a simulation platform interface for the bronchoscopic surgical robot remote control equipment, which not only improves the integration of robot products, but also realizes multi-degree-of-freedom operation and accurately trains complex surgical skills;

[0173] Low-cost and efficient training: The simulation system replaces traditional animal experiments and human model experiments, which reduces the consumption of experimental resources and greatly reduces the training cost. At the same time, the system's automatic setting function greatly shortens the operation preparation time and improves the training efficiency.

[0174] Possibility of repeated practice: Doctors can practice repeatedly in the same operation scenario until they fully master the bronchoscopic operation skills. This simulation-based training system breaks through the one-time operation limitation of traditional training methods.

[0175] In one embodiment of the present invention, a bronchoscope intervention simulation system is provided, comprising a pre-built simulation platform, wherein the simulation platform is connected to a handle joystick and a controller respectively, wherein the handle joystick is configured to output an electrical signal corresponding to a target posture of the bronchoscope tip, and the controller is configured to determine a driving amount for controlling the posture of the bronchoscope tip according to the current posture and target posture of the bronchoscope tip.

[0176] In this embodiment, the bronchoscopic intervention simulation system is configured to perform bronchoscopic intervention simulation using the bronchoscopic intervention simulation method described in any one of the above embodiments or a combination of multiple embodiments.

[0177] It should be noted that the above-mentioned bronchoscopic intervention simulation system embodiment and bronchoscopic intervention simulation method embodiment are based on the same inventive concept, and the entire content of the bronchoscopic intervention simulation method is incorporated into the bronchoscopic intervention simulation system embodiment by reference.

[0178] It should be noted that, in this article, relational terms such as are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0179] The above is only a specific implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A bronchoscopic intervention simulation method, characterized in that: The following steps are involved: A simulation platform for simulating a bronchoscope intervention process is pre-built, the simulation platform is connected to a handle rocker and a controller respectively, the handle rocker is configured to output an electrical signal corresponding to a target posture of the bronchoscope end, and the controller is configured to determine a driving amount for controlling the bronchoscope end to reach the target posture according to a current posture and a target posture of the bronchoscope end; Constructing a tracheal three-dimensional model according to the structure of a real trachea, establishing a bronchoscope end three-dimensional model according to the structure of a real bronchoscope end, inputting the tracheal three-dimensional model and the bronchoscope end three-dimensional model into the simulation platform, and setting physical parameters and collision constraint parameters for the tracheal three-dimensional model and the bronchoscope end three-dimensional model respectively, so as to construct a virtual trachea and a virtual bronchoscope end capable of simulating collision in the simulation platform; Acquiring an electrical signal output by the handle rocker to obtain a real target posture of the real bronchoscope terminal, and the controller determining a first driving amount for controlling the virtual bronchoscope terminal according to the current virtual posture of the virtual bronchoscope terminal and the real target posture of the real bronchoscope terminal; The simulation platform controls the movement of the distal end of the virtual bronchoscope in the virtual trachea according to the first driving amount.

2. The bronchoscopic intervention simulation method according to claim 1, characterized in that: The controller determines a first driving amount for controlling the virtual bronchoscope terminal according to the current virtual posture of the virtual bronchoscope terminal and the real target posture of the real bronchoscope terminal, comprising the following steps: Determine the coordinate system where the end of the virtual bronchoscope is located as a virtual coordinate system, determine the coordinate system where the end of the real bronchoscope is located as a real coordinate system, and determine a mapping relationship between the virtual coordinate system and the real coordinate system; According to the mapping relationship, mapping the real target posture of the real bronchoscope end to the virtual coordinate system to obtain a first virtual target posture of the virtual bronchoscope end in the virtual coordinate system; The controller determines the first driving amount according to the current virtual posture of the virtual bronchoscope end in the virtual coordinate system and the first virtual target posture; or, The controller determines a first driving amount for controlling the virtual bronchoscope terminal according to the current virtual posture of the virtual bronchoscope terminal and the real target posture of the real bronchoscope terminal, comprising the following steps: Determine the coordinate system where the end of the virtual bronchoscope is located as a virtual coordinate system, determine the coordinate system where the end of the real bronchoscope is located as a real coordinate system, and determine a mapping relationship between the virtual coordinate system and the real coordinate system; According to the mapping relationship, mapping the current virtual posture of the virtual bronchoscope end to the real coordinate system to obtain the current real posture of the real bronchoscope end in the real coordinate system; The controller determines the first driving amount according to the current real posture of the real bronchoscope tip in the real coordinate system and the real target posture.

3. The bronchoscopic intervention simulation method according to claim 2, characterized in that: The following steps are also included: Based on the first driving amount, the simulation platform controls the end of the virtual bronchoscope to move to a first posture in the virtual trachea; The control accuracy of the controller is determined according to the first posture of the virtual bronchoscope tip and the first virtual target posture; or, according to the mapping relationship, the first posture is mapped to the real coordinate system to obtain the first real posture of the real bronchoscope tip in the real coordinate system, and the control accuracy of the controller is determined according to the first real posture of the real bronchoscope tip and the first real target posture.

4. The bronchoscopic intervention simulation method according to claim 1, characterized in that: The following steps are also included: Determining a current posture of the virtual bronchoscope tip and a second virtual target posture of the virtual bronchoscope tip; The controller determines a second driving amount according to the current posture of the virtual bronchoscope tip and the second virtual target posture; Based on the second driving amount, the simulation platform controls the end of the virtual bronchoscope to move to a second posture in the virtual trachea; Determining the simulation accuracy of the simulation platform according to the second posture of the virtual bronchoscope end and the second virtual target posture; If the simulation accuracy does not meet the preset requirements, optimizing the simulation platform until the simulation accuracy meets the preset requirements; If the simulation accuracy meets the preset requirements, the simulation platform is applied to simulate the bronchoscopic intervention process.

5. The bronchoscopic intervention simulation method according to claim 1, characterized in that: Also includes: For the end of the virtual bronchoscope, a continuum dynamics model MΔv=dt(f(x,t)) is constructed, wherein M is the mass matrix of the end of the virtual bronchoscope, x is the position of the end of the virtual bronchoscope, Δv is the velocity change of the end of the virtual bronchoscope, and f(x,t) represents the external force function of the end of the virtual bronchoscope; The simulation platform is based on the continuum dynamics model MΔv=dt(f(x,t)) and utilizes a forward and inverse differential equation solver or a linear solver to simulate the movement process and bending effect of the virtual bronchoscope tip in the virtual trachea.

6. The bronchoscopic intervention simulation method according to claim 1, characterized in that: The controller is configured with an automatic target aiming function, and the automatic target aiming function includes: when the distance between the end of the bronchoscope and the target is not greater than a preset distance threshold, the controller controls the end of the bronchoscope to automatically aim at the target; The automatic aiming target function is simulated by using the simulation platform in the following manner: Determining a virtual target point on the virtual trachea, and determining the coordinates of the virtual target point; Determining the coordinates of the end of the virtual bronchoscope, and determining the distance from the end of the virtual bronchoscope to the virtual target point according to the coordinates of the end of the virtual bronchoscope and the coordinates of the virtual target point; When the distance is not greater than a preset distance threshold, the controller determines a third driving amount according to the coordinates of the virtual bronchoscope tip, the current posture and the coordinates of the virtual target, and the simulation platform controls the movement of the virtual bronchoscope tip according to the third driving amount.

7. The bronchoscopic intervention simulation method according to claim 1, characterized in that: The first driving amount is the driving amount of the guide wire, and the guide wire is configured to control the posture of the end of the real bronchoscope; Based on the discrete PID controller of linear stepper motor control, the first driving quantity [u1,u2,u3,u4] T Determined by the following formula: and i (k)=sd i -s i (k) u i (k)=K p *e i (k)+K i *(e i (k)+e i (k-1))+K d *(e i (k)-e i (k-1)) Among them, u i (k) = [u1,u2,u3,u4] T , i is an integer from 1 to 4, e i (k) represents the wire state error at time k, sd i Indicates the guidewire state quantity corresponding to the desired posture, s i (k) represents the guide wire state corresponding to the current posture, e i (k-1) represents the wire state error at time k-1, K p is the scale parameter, K i is the integration parameter, K d is the differential parameter.

8. The bronchoscopic intervention simulation method according to claim 7, characterized in that: Determine the first driving amount [u1,u2,u3,u4] T The following steps are also included: The internal force of the guidewire at each step is calculated by the following formula: f(x j )≈f(x j-1 )+K(x j-1 )dx, where f(x j ) is the position x of the guide wire j The internal stiffness force at j-1 ) is the position x of the guide wire j-1 Internal stiffness force at x j and x j-1 are two adjacent nodes on the guidewire, K(x j-1 ) is the tangent stiffness matrix that depends on the actual position of the node, K(x j-1 ) is the position x of the guidewire j-1 The corresponding tangent stiffness matrix at ; The guide wire is kept in static balance in each step by the following formula: j-1 )dx=p+f(x j-1 )+J T λ, where p represents the external force on the guidewire, including gravity, and J T represents the transpose of the Jacobian matrix, λ represents the Lagrange multiplier; Projecting the finite element model of the guidewire into the constraint space includes: By setting λ = 0 and solving the equation -K(x j-1 )dx=p+f(x j-1 )+J T λ to find the free displacement x of the guide wire free , and the vector between the actual posture and the desired posture of the virtual bronchoscope tip is Project the mechanics into the constraint space to get the smallest possible projection space for the inverse problem: δ e The vector representing the position of the virtual bronchoscope tip based on the constraint, J a T represents the transpose of the Jacobian matrix of the driving guide wire, λ a represents the Lagrange multiplier driving the guide wire; At the end of the time step, the value of the constraint response is used to correct the final configuration, thereby obtaining the relationship between the deformation amount of the virtual bronchoscope tip and the pull wire length of the guide wire: x = x free +K -1 J a T λ a , x is the end position of the virtual bronchoscope.

9. The bronchoscopic intervention simulation method according to claim 1, characterized in that: The simulation platform is configured with a first-view display module and a global-view display module; wherein, The first viewing angle display module is configured to display a moving picture of the virtual bronchoscope tip in the virtual trachea along the moving direction of the virtual bronchoscope tip in the virtual trachea; The global viewing angle display module is configured to display the virtual trachea in perspective and to globally display a moving picture of the tip of the virtual bronchoscope in the virtual trachea.

10. The bronchoscopic intervention simulation method according to claim 1, characterized in that: It also includes using SOFA to construct the simulation platform, and configuring the trachea three-dimensional model and the bronchoscope end three-dimensional model as a software structure in the SOFA; Physical parameters and collision constraint parameters are set for the trachea three-dimensional model and the bronchoscope terminal three-dimensional model, respectively, wherein the physical parameters of the trachea three-dimensional model include one or more of the mass, density, Young's modulus, Poisson's ratio, friction coefficient and damping coefficient in the tracheal cavity and elastic modulus of the tracheal inner wall; the physical parameters of the bronchoscope terminal include the mass, density, Young's modulus, Poisson's ratio, friction coefficient and damping coefficient of the bronchoscope terminal; the collision constraint parameters include that the maximum driving speed cannot exceed 15.7 mm / s, and the maximum traction length cannot exceed 14 mm.

11. The bronchoscopic intervention simulation method according to claim 10, characterized in that: It also includes constructing the bronchoscope end three-dimensional model by the following method: Constructing a first bronchoscope end three-dimensional model of the bronchoscope end using three-dimensional software; Performing meshing processing on the first bronchoscope terminal three-dimensional model to obtain a second bronchoscope terminal three-dimensional model that can be recognized by the SOFA, wherein the second bronchoscope terminal three-dimensional model is configured as the bronchoscope terminal three-dimensional model; and / or, It also includes constructing the trachea three-dimensional model in the following manner: Pre-acquire CT images including the lung airways; Segmenting the tracheal structure in the CT image using an image segmentation technique, and performing three-dimensional reconstruction on the tracheal structure to obtain a first tracheal three-dimensional model; Performing meshing processing on the first trachea three-dimensional model to obtain a second trachea three-dimensional model that can be recognized by the SOFA, wherein the second trachea three-dimensional model is configured as the trachea three-dimensional model; and / or, An effector is set at the end of the virtual bronchoscope, and the effector is a sampling point for reading the coordinates and direction of the end of the virtual bronchoscope in real time. The coordinates p of the end of the virtual bronchoscope are read by the effector. t =[x t ,y t ,z t ] and direction d t =[α, β, γ], and determine the current virtual posture of the virtual bronchoscope tip according to the coordinates and direction of the virtual bronchoscope tip.

12. The bronchoscopic intervention simulation method according to any one of claims 1 to 11, characterized in that: The simulation platform is used to verify the control accuracy of the controller. If the control accuracy of the controller meets the preset control accuracy requirement, the controller is applied to the bronchoscope robot.

13. A bronchoscopic intervention simulation system, characterized in that: A pre-built simulation platform is included, the simulation platform is connected to a handle rocker and a controller respectively, the handle rocker is configured to output an electrical signal corresponding to a target posture of the bronchoscope end, and the controller is configured to determine a driving amount for controlling the posture of the bronchoscope end according to a current posture and a target posture of the bronchoscope end; The bronchoscopic intervention simulation system is configured to perform bronchoscopic intervention simulation using the bronchoscopic intervention simulation method as described in any one of claims 1 to 12.

Images