A method for determining a haptic feedback force in a virtual clamp-based bronchial intervention robot-assisted control

Abstract

The application discloses a method for determining haptic feedback force in bronchial intervention robot-assisted control based on a virtual clamp, and the steps are as follows: S1, a three-dimensional model of a bronchial airway tree is constructed; S2, a bronchial center line is generated, and three-dimensional discrete points of a target branch are extracted to be fitted into a three-dimensional parameter curve; S3, a virtual clamp guide pipeline with the three-dimensional parameter curve as a central axis is constructed, and a pipeline wall thickness area is defined as a transition area; S4, a surgery simulation platform into which the three-dimensional model of the bronchial airway tree is imported is built; S5, actual speed and actual position of an actuator end are obtained; S6, whether the actuator end is located in the transition area in the virtual clamp guide pipeline is judged, and a pipeline boundary error compensation amount and a corresponding pipeline boundary error compensation force are obtained; S7, a desired force of operation is obtained; and S8, haptic feedback force is obtained and acts on an operator; the method can guide the clamp to move on a specified route all the time, reduces the operation strength of the operator, and improves operation precision.

Description

Technical Field

[0001] This invention relates to the field of virtual technology, and in particular to a method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper. Background Technology

[0002] Lung cancer, as one of the cancers with the highest incidence and mortality rates, threatens human health. With the development of chest CT scan technology, more and more lung nodules of different sizes can be detected. Determining the nature of lung nodules can improve the accuracy of lung cancer diagnosis to some extent, thereby enabling earlier treatment and reducing mortality. Currently, the main techniques for diagnosing the nature of lung nodules include bronchoscopy, CT-guided percutaneous lung biopsy, and thoracoscopic surgical resection biopsy. However, these techniques have shortcomings in terms of accuracy and safety. Furthermore, the two existing bronchoscopic robotic systems lack tactile feedback, posing a risk of bronchial damage during interventional procedures.

[0003] Virtual fixture control algorithms provide tactile feedback to the operator through haptic force feedback devices, assisting the operator in moving the actuator along a predetermined trajectory and avoiding prohibited areas. This effectively reduces the operator's psychological burden and improves operational accuracy. However, the simple constraint definitions of traditional virtual fixtures do not incorporate medical characteristics, and simple linear error compensation forces may cause the haptic force feedback device to lock up, failing to meet the needs of surgical operations. Furthermore, position control alone is insufficient to meet the control requirements of dynamic interaction. Summary of the Invention

[0004] The purpose of this invention is to provide a method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper, which solves the above-mentioned technical problems.

[0005] Therefore, the technical solution of the present invention is as follows:

[0006] A method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper, comprising the following steps:

[0007] S1. Process the lung CT image data to obtain a three-dimensional model of the bronchial airway tree;

[0008] S2. Generate the bronchial centerline based on the 3D model of the bronchial airway tree obtained in step S1, extract the 3D discrete points of the target branch in the bronchial centerline, and fit them into a 3D parametric curve.

[0009] S3. Based on the three-dimensional parameter curve obtained in step S2, construct a virtual clamp-guided pipe with the three-dimensional parameter curve as the central axis, and define the pipe wall thickness region as the transition region.

[0010] S4. Use the CHAI3D library to build a surgical simulation platform with an imported 3D model of the bronchial airway tree to establish virtual units of the bronchial model; use the GLFW open-source image library and the CHAI3D library to complete the dynamic rendering of the surgical environment; connect the completed surgical simulation platform to a tactile feedback device and complete gravity compensation.

[0011] S5. Use the surgical simulation platform to collect the operation data of the tactile feedback device to obtain the actual speed and position of the actuator end when the operator operates the main hand of the tactile feedback device;

[0012] S6. By calculating the distance between the actuator end and the nearest point on the three-dimensional parameter curve in real time, determine whether the actuator end is located in the transition area of ​​the virtual clamp-guided pipeline. When the actuator end moves to the transition area, generate the pipeline boundary error compensation amount and the corresponding pipeline boundary error compensation force.

[0013] S7. Take the position of the point closest to the actuator end on the three-dimensional parametric curve as the desired position, and calculate the desired force of the operation based on the actual position and actual velocity of the actuator end.

[0014] S8. The expected force and the pipeline boundary error compensation force are added together to obtain the tactile feedback force, which is then applied to the operator.

[0015] Further, the specific implementation steps of step S1 are as follows: import the lung CT image data into the medical imaging software Mimics, and construct a three-dimensional model of the bronchial airway tree based on the coronal view of the lungs using the medical imaging software Mimics, including: 1) clicking the Segment Airway module, using the pop-up pen to mark two points from top to bottom in the tracheal apex cavity region of the coronal view of the lungs, so that the software can automatically construct the airway model; 2) based on the constructed airway model, clicking the calculate part module to generate the airway tree model; smoothing the airway tree model to obtain the three-dimensional model of the bronchial airway tree.

[0016] Furthermore, in step S2, the bronchial centerline is generated using the FitCenterline module in the medical imaging software Mimics.

[0017] Further, in step S2, the extracted three-dimensional discrete points are interpolated and fitted using a quasi-uniform B-spline curve equation. The steps are as follows:

[0018] B-spline curve equation The expression is:

[0019] , ,

[0020] In the formula, P i It is the i-th three-dimensional discrete point in the bronchial centerline, where i ranges from 0 to n, and n is the total number of three-dimensional discrete points; B i，k (s) represents the basis functions of the i-th k-th B-spline; s represents the parameters of the B-spline curve equation; s i It is a set of continuously changing values ​​of a non-decreasing sequence called the node vector, with the first and last values ​​being 0 and 1 respectively. The repetition of the first and last values ​​is the same as the order k of the spline. The repetition is calculated as the curve order + 1, and the remaining nodes are uniformly distributed.

[0021] Furthermore, in step S3, the outer diameter R of the pipe guided by the virtual fixture is 1mm to 5mm, and the pipe wall thickness is 0.1 to 0.2 times the outer diameter of the pipe.

[0022] Furthermore, the specific implementation steps of step S6 are as follows:

[0023] S601. Using Newton's iteration method, calculate the parameter s and position X corresponding to the point closest to the actuator end on the path of the three-dimensional parametric curve. r ;

[0024] The function for Newton's iteration method is:

[0025] ,

[0026] ,

[0027] In the formula, It is a point The distance function between the three-dimensional parametric curve and the curve. for Regarding parameter s in s k The first derivative value at that point, for Regarding parameter s in s k The second derivative value at;

[0028] Position X r The calculation formula is: = ;

[0029] S602, The position of the nearest point on the parameter curve obtained from S601. Calculate the distance ΔX and its value between the actual position of the actuator end and the nearest point on the curve. The formula for calculating the distance between two points is as follows:

[0030] ,

[0031] ;

[0032] S603. Determine whether the actuator end is located in the transition zone within the pipeline based on the distance value:

[0033] when When this happens, the actuator end does not move to the transition area, and there is no pipeline boundary error compensation force.

[0034] when When the actuator tip moves into the transition zone inside the pipe, it generates a pipe boundary error compensation amount. Thus, the pipeline boundary error compensation force F is obtained. d。

[0035] Furthermore, the specific implementation steps of step S7 are as follows: The actual speed of the actuator end effector in step S5... The distance ΔX obtained from step S602 is substituted into the impedance control calculation model to obtain the desired force F applied to the operator. r The expression for the impedance control calculation model is as follows: In the formula, D is the damping component and K is the stiffness component.

[0036] Compared with existing technologies, this method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper extracts the bronchial centerline from a 3D model of the tracheal tree created from lung CT images. It then fits a specified branch on the bronchial centerline into a parametric curve and expands it into a duct-guided virtual gripper. This allows the virtual gripper to incorporate the medical characteristics of the airway tree, guiding the actuator end effector to always move along a designated path. Furthermore, by combining this method with impedance control, it meets the control requirements for compliant robot interaction tasks, reducing operator workload while improving operational accuracy. Attached Figure Description

[0037] Figure 1 This is a flowchart of the method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper according to the present invention.

[0038] Figure 2 This is an image illustration of the process of importing lung CT image data into the medical imaging software Mimics in step S101 of the method for determining tactile feedback force in the bronchial intervention robot-assisted control based on virtual gripper of the present invention.

[0039] Figure 3 This is a schematic diagram showing two points marked from top to bottom in step S102 of the method for determining tactile feedback force in the bronchial intervention robot-assisted control based on virtual gripper of the present invention.

[0040] Figure 4This is a schematic diagram of the bronchial airway tree three-dimensional model obtained in step S103 of the method for determining tactile feedback force in the bronchial intervention robot assisted control based on virtual gripper of the present invention.

[0041] Figure 5 This is a schematic diagram of the bronchial centerline obtained in step S201 of the method for determining tactile feedback force in the bronchial intervention robot-assisted control based on virtual gripper of the present invention.

[0042] Figure 6 This is a schematic diagram of the three-dimensional parameter curve obtained in step S203 of the method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper according to the present invention.

[0043] Figure 7 This is a cross-sectional view of the pipe obtained in step 3 of the method for determining tactile feedback force in the bronchial intervention robot-assisted control based on virtual gripper of the present invention. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention.

[0045] See Figure 1 The specific implementation steps of the method for determining tactile feedback force in the assisted control of the bronchial intervention robot based on virtual gripper are as follows:

[0046] S1. Process the lung CT image data to obtain and save the 3D model of the bronchial airway tree;

[0047] Specifically, the implementation steps of step S1 are as follows:

[0048] S101. Import the lung CT image data into the medical imaging software Mimics, and display the lung CT image data in sagittal, coronal, horizontal, and three-dimensional views respectively; for example... Figure 2 As shown, Figure a is a coronal view of the lungs, Figure b is a horizontal view of the lungs, Figure c is a sagittal view of the lungs, and Figure d is a three-dimensional view of the lungs.

[0049] S102. Based on the coronal view of the lungs, a three-dimensional model of the bronchial airway tree was constructed using the medical imaging software Mimics, including: 1) See Figure 3 1) Click the Segment Airway module, and use the pop-up pen to mark two points from top to bottom in the airway region at the top of the trachea in the coronal view of the lungs, so that the software can build an airway model by itself; 2) Based on the built airway model, click the calculate part module to generate an airway tree model.

[0050] S103. Select the airway tree model constructed in step S102, click the smooth module to smooth it, and obtain a 3D model of the bronchial airway tree, as shown below. Figure 4 As shown;

[0051] S104. Save the 3D model of the bronchial airway tree obtained in step S103 as an STL file. Specifically, select the bronchial airway tree in the target box of the right toolbar of Mimics, and select to export the STL model of the bronchial airway tree to complete the saving of the model.

[0052] S2. Generate the bronchial centerline based on the 3D model of the bronchial airway tree obtained in step S1, extract the 3D discrete points of the target branch in the bronchial centerline, and fit them into a 3D parametric curve.

[0053] Specifically, the implementation steps of step S2 are as follows:

[0054] S201. Based on the 3D model of the bronchial airway tree obtained in step S1, the bronchial centerline is generated using the FitCenterline module in the medical imaging software Mimics, as shown below. Figure 5 As shown;

[0055] S202. On the bronchial centerline generated in step S201, select the target branch. Since the target branch is composed of several three-dimensional discrete points, extracting the target branch means extracting the three-dimensional discrete points of the target branch; save the extracted three-dimensional discrete points of the target branch.

[0056] S203. The three-dimensional discrete points extracted in step S202 are interpolated and fitted using the quasi-uniform B-spline curve equation to obtain the three-dimensional parametric curve.

[0057] The specific implementation steps of step S203 are as follows:

[0058] B-spline curve equation The expression is:

[0059] , ,

[0060] In the formula, P i It is the i-th three-dimensional discrete point in the bronchial centerline, where i ranges from 0 to n, and n is the total number of three-dimensional discrete points; B i，k (s) represents the basis functions of the i-th k-th B-spline; s represents the parameters of the B-spline curve equation; s i It is a set of continuously changing values ​​in a non-decreasing sequence called the node vector, with the first and last values ​​being 0 and 1 respectively. The repetition of the first and last nodes is the same as the order k of the spline, calculated as the curve order + 1. The remaining nodes are uniformly distributed; where si The value range is 0 to 1;

[0061] In practical applications, step S203 can be implemented using MATLAB or C++. After step S203, the three-dimensional discrete points extracted in step S202 are interpolated and fitted to obtain a three-dimensional parametric curve composed of dense points, such as... Figure 6 As shown.

[0062] S3. Based on the three-dimensional parameter curve obtained in step S2, construct a virtual fixture-guided pipeline with the three-dimensional parameter curve as its central axis. (See [link]) Figure 7 ;

[0063] In this embodiment, the outer diameter R of the virtual clamp guiding the pipe is 2.5 mm, and the pipe wall thickness is... The value is 0.2R, or 0.5mm. The purpose of constructing this virtual clamp guide pipe is to monitor the movement deviation of the actuator end effector. That is, the three-dimensional parameter curve is the theoretical travel path of the actuator end effector. The actuator end effector can only move freely within the virtual clamp guide pipe. When the actuator end effector moves to the pipe wall thickness area (i.e., the transition area), a feedback force is applied to the tactile force feedback device operated by the operator, so that the actuator end effector returns to the travel path.

[0064] S4. Use the CHAI3D library to build a surgical simulation platform with an imported 3D model of the bronchial airway tree to establish virtual units of the bronchial model; use the GLFW open-source image library and the CHAI3D library to complete the dynamic rendering of the surgical environment; connect the completed surgical simulation platform to a tactile feedback device, and use the gravity compensation module provided by the tactile feedback device to compensate for the gravity of the tactile feedback device.

[0065] In step S4, the construction of the surgical simulation platform, the dynamic rendering of the surgical environment, and gravity compensation are all implemented using existing technologies. Specifically, the construction of the surgical simulation platform includes setting bronchial texture mapping, bronchial glossiness, ambient light color, diffuse reflection color, and specular reflection color. In this embodiment, the haptic force feedback device specifically uses the omega6 haptic force feedback device. This haptic force feedback device has 6 degrees of freedom and is a parallel structure, consisting of a motor and a sensing device. The main hand is the operator's operating end. By controlling the force feedback main hand, the operator drives the joint movement of the device mechanism to rotate the motor. The angular displacement information of the motor rotation is acquired by the sensing device and sent to the surgical simulation platform. The virtual device in the simulation platform performs corresponding movements in the environment based on the received actuator end position information.

[0066] S5. Use the surgical simulation platform to collect the operation data of the tactile feedback device to obtain the actual speed and position of the actuator end when the operator operates the main hand of the tactile feedback device;

[0067] Specifically, the actual velocity and position of the actuator end effector are related to the operator of the haptic force feedback device. Therefore, in this step, the operator moves the force feedback hand of the haptic force feedback device, and the actual position X and actual velocity of the actuator end effector are calculated by the function provided by the haptic force feedback device. Furthermore, through debugging, in actual operation, the master arm can transmit motion commands to the slave arm using force feedback.

[0068] S6. Using Newton's iteration method, calculate the parameter s corresponding to the point closest to the actuator end on the three-dimensional parameter curve, and calculate the actual position corresponding to parameter s; by calculating the distance between the actuator end of the tactile force feedback device and the nearest point on the three-dimensional parameter curve, determine whether the actuator end is located in the transition area inside the pipeline, so that when the actuator end moves to the transition area, the pipeline boundary error compensation amount and the corresponding pipeline boundary error compensation force are generated.

[0069] Specifically, the implementation steps of step S6 are as follows:

[0070] S601. Using Newton's iteration method, calculate the parameter s and position X corresponding to the point on the three-dimensional parametric curve closest to the actuator end. r ;

[0071] The function of Newton's iteration method is as follows:

[0072] ,

[0073] ,

[0074] In the formula, It is a point The distance function between the three-dimensional parametric curve and the curve. for Regarding parameter s in s k The first derivative value at that point, for Regarding parameter s in s k The second derivative value at;

[0075] Location The calculation method is as follows: = ;

[0076] S602. The position of the nearest point on the parameter curve obtained in step S601. Calculate the actual position of the actuator end effector and the nearest point on the curve. The distance △X between them and the distance value The distance and the formula for calculating the distance value are as follows:

[0077] ,

[0078] ,

[0079] In the formula, the actual position of the actuator end is obtained by the tactile force feedback device; This represents the Euclidean distance between two points.

[0080] S603. Based on the distance value obtained in step S602, determine whether the actuator end is located in the transition area within the pipe:

[0081] when When this happens, the actuator end does not move to the transition area, and there is no pipeline boundary error compensation force.

[0082] when When the actuator tip moves into the transition zone inside the pipe, it generates a pipe boundary error compensation amount. Thus, the pipeline boundary error compensation force is obtained. .

[0083] S7. Take the position of the point closest to the actuator end on the three-dimensional parametric curve as the desired position, and calculate the desired force F based on the actual position and velocity of the actuator end. r ;

[0084] Specifically, the implementation steps of step S7 are as follows:

[0085] The actual speed at the actuator end The distance ΔX obtained from step S602 is substituted into the impedance control calculation model to obtain the desired force F applied to the operator. r ;

[0086] The expression for the impedance control calculation model is:

[0087] ,

[0088] In the formula, D is the damping component, which is an inherent property of the tactile force feedback device; K is the stiffness component, which is taken as 30 N / m in this embodiment; and the actual speed of the actuator end effector is... Obtained through function calculation provided by the haptic force feedback device;

[0089] S8, Expectation F r Compensation force F for pipeline boundary error d The summation produces a tactile feedback force that is applied to the operator.

Claims

1. A method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper, characterized in that, The steps are as follows: S1. Process the lung CT image data to obtain a three-dimensional model of the bronchial airway tree; S2. Generate the bronchial centerline based on the 3D model of the bronchial airway tree obtained in step S1, extract the 3D discrete points of the target branch in the bronchial centerline, and fit them into a 3D parametric curve. S3. Based on the three-dimensional parameter curve obtained in step S2, construct a virtual clamp-guided pipe with the three-dimensional parameter curve as the central axis, and define the pipe wall thickness region as the transition region. S4. Use the CHAI3D library to build a surgical simulation platform with an imported 3D model of the bronchial airway tree to establish virtual units of the bronchial model; use the GLFW open-source image library and the CHAI3D library to complete the dynamic rendering of the surgical environment; connect the completed surgical simulation platform to a tactile feedback device and complete gravity compensation. S5. Use the surgical simulation platform to collect the operation data of the tactile feedback device to obtain the actual speed and position of the actuator end when the operator operates the main hand of the tactile feedback device; S6. By calculating the distance between the actuator end and the nearest point on the three-dimensional parameter curve in real time, determine whether the actuator end is located in the transition area of ​​the virtual clamp-guided pipeline. When the actuator end moves to the transition area, generate the pipeline boundary error compensation amount and the corresponding pipeline boundary error compensation force. S7. Take the position of the closest point on the three-dimensional parametric curve to the actuator end as the desired position, and calculate the desired force of the operation based on the actual position and actual velocity of the actuator end. S8. The expected force and the pipeline boundary error compensation force are added together to obtain the tactile feedback force, which is then applied to the operator.

2. The method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper according to claim 1, characterized in that, The specific implementation steps of step S1 are as follows: import lung CT image data into the medical imaging software Mimics, and construct a three-dimensional model of the bronchial airway tree based on the coronal view of the lungs using the medical imaging software Mimics, including: 1) clicking the Segment Airway module, using the pop-up pen to mark two points from top to bottom in the tracheal apex cavity region of the coronal view of the lungs, so that the software can automatically construct the airway model; 2) based on the constructed airway model, click the calculate part module to generate the airway tree model; smooth the airway tree model to obtain the three-dimensional model of the bronchial airway tree.

3. The method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper according to claim 1, characterized in that, In step S2, the bronchial centerline is generated using the FitCenterline module in the medical imaging software Mimics.

4. The method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper according to claim 1, characterized in that, In step S2, the extracted three-dimensional discrete points are interpolated and fitted using a quasi-uniform B-spline curve equation. The steps are as follows: B-spline curve equation The expression is: ， ， In the formula, P i It is the i-th three-dimensional discrete point in the bronchial centerline, where i ranges from 0 to n, and n is the total number of three-dimensional discrete points; B i，k (s) represents the basis functions of the i-th k-th B-spline; s represents the parameters of the B-spline curve equation. s i It is a set of continuously changing values ​​of a non-decreasing sequence called the node vector, with the first and last values ​​being 0 and 1 respectively. The repetition of the first and last values ​​is the same as the order k of the spline. The repetition is calculated as the curve order + 1, and the remaining nodes are uniformly distributed.

5. The method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper according to claim 1, characterized in that, In step S3, the outer diameter R of the pipe guided by the virtual fixture is 1mm to 5mm, and the pipe wall thickness is 0.1 to 0.2 times the outer diameter of the pipe.

6. The method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper according to claim 1, characterized in that, The specific implementation steps of step S6 are as follows: S601. Using Newton's iteration method, calculate the parameter s and position X corresponding to the point closest to the actuator end on the path of the three-dimensional parametric curve. r ; The function for Newton's iteration method is: ， ， In the formula, It is a point The distance function between the three-dimensional parametric curve and the curve. for Regarding parameter s in s k The first derivative value at that point, for Regarding parameter s in s k The second derivative value at; Position X r The calculation formula is: = ; S602, The position of the nearest point on the parameter curve obtained from S601. Calculate the distance ΔX and its value between the actual position of the actuator end and the nearest point on the curve. The formula for calculating the distance between two points is as follows: ， ； S603. Determine whether the actuator end is located in the transition zone within the pipeline based on the distance value: when When the actuator end does not move to the transition area, there is no pipeline boundary error compensation force. when When the actuator tip moves into the transition zone inside the pipe, it generates a pipe boundary error compensation amount. Thus, the pipeline boundary error compensation force F is obtained. d .

7. The method for determining tactile feedback force in the assisted control of a bronchial intervention robot based on a virtual gripper according to claim 1, characterized in that, The specific implementation steps of step S7 are as follows: The actual speed of the actuator end effector in step S5... The distance ΔX obtained from step S602 is substituted into the impedance control calculation model to obtain the desired force F applied to the operator. r The expression for the impedance control calculation model is as follows: In the formula, D is the damping component and K is the stiffness component.

Images