A method and apparatus for motion control based on visual guidance

Abstract

The application discloses a motion control method and device based on visual guidance, and comprises the following steps: acquiring image information of a human body cavity; pre-processing the image; obtaining a cavity center of the human body cavity according to the pre-processed image information; obtaining a motion control strategy of a bronchoscope according to the cavity center; and adjusting the position and posture of the bronchoscope by a trachea cannula robot according to the motion control strategy, so that the bronchoscope moves towards the cavity center. The technical scheme of the application improves the efficiency and safety of a trachea cannula operation.

Description

Technical Field

[0001] This invention belongs to the field of medical robot technology, and relates to a motion control method and device based on vision guidance, and further relates to motion control of endotracheal intubation by operating a fiberoptic bronchoscope using an endotracheal intubation robot based on visual feedback. Background Technology

[0002] Endotracheal intubation is an advanced airway management technique. When a patient is under anesthesia, a flexible fiberoptic bronchoscope is inserted through the nasal cavity into the trachea to provide optimal conditions for airway patency, ventilation, oxygen supply, and airway suction. It is a crucial measure for rescuing patients with respiratory dysfunction. The operation of the fiberoptic bronchoscope is considered a key factor in the success of endotracheal intubation. However, even for experienced surgeons, manually operating a flexible fiberoptic bronchoscope is a challenge, requiring the surgeon to simultaneously coordinate multiple movements of the bronchoscope to complete the procedure. Misoperation of the bronchoscope can easily lead to surgical complications such as bleeding or asphyxiation. Manual operation of the fiberoptic bronchoscope requires both hands. One hand holds the proximal end of the bronchoscope, equipped with a display screen, to control the rotation of the distal end of the catheter, while the other hand advances or rotates the catheter to the desired position before inserting the trachea along the catheter. This procedure demands smooth operation throughout, short completion time, and no secondary injury to the patient. Currently, due to the long and narrow structure of the patient's upper airway and the fact that the monocular camera at the end of the fiberoptic bronchoscope only provides the surgeon with a partial view, the procedure is less successful. Doctors need to quickly identify various anatomical features of the human body. Furthermore, classic flexible endoscopes require doctors to combine multiple movements with limited degrees of freedom to rotate, push, or pull the endoscope's tip. This necessitates that doctors accurately, stably, and rapidly manipulate the bronchoscope while simultaneously quickly identifying the body's anatomical features. Therefore, the overall surgery is quite challenging, increasing the cognitive and operational burden on the doctor.

[0003] To improve the stability, accuracy, and speed of endoscopic procedures, traditional optimization methods employ a combination of semi-automatic and master-slave teleoperation to reduce the burden on surgeons. However, this approach still suffers from drawbacks such as system complexity, reliance on surgeon intervention, and the robot's lack of image perception capabilities throughout the entire surgical process. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a motion control method and device based on visual guidance. The endoscope obtains the position of the cavity center according to visual feedback technology; based on the position of the cavity center, a fuzzy PD motion controller is designed to generate robot control commands. The robot continuously adjusts the state of the bronchoscope according to the feedback error, so that it moves along the cavity midline, thereby completing the intubation surgery.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention provides a motion control method based on visual guidance, comprising:

[0007] Step S1: Obtain image information of human body cavities;

[0008] Step S2: Preprocess the image;

[0009] Step S3: Obtain the cavity center of the human body cavity based on the preprocessed image information;

[0010] Step S4: Based on the cavity center, obtain the motion control strategy for the bronchoscope;

[0011] Step S5: The endotracheal intubation robot adjusts the position and orientation of the bronchoscope according to the motion control strategy, so that the bronchoscope moves toward the center of the cavity.

[0012] Preferably, in step S1, a bronchoscope is used to obtain image information of the human body cavity.

[0013] Preferably, in step S2, an adaptive threshold segmentation and target detection algorithm are used to segment the cavity region, and the centroid of the cavity region is calculated as the cavity center.

[0014] Preferably, in step S3, a velocity-based motion controller is designed using the cavity center position as feedback information, and the motion controller parameters are adjusted online using a fuzzy algorithm.

[0015] Preferably, the motion controller sends the calculated feed speed, rotation speed, and end-bending speed of the bronchoscope to the endotracheal intubation robot for motor drive, controlling the movement of the bronchoscope to complete the intubation operation.

[0016] The present invention also provides a motion control device based on vision guidance, comprising:

[0017] The acquisition module is used to acquire image information of human body cavities;

[0018] A preprocessing module is used to preprocess the image;

[0019] The first processing module is used to obtain the cavity center of the human body cavity based on the preprocessed image information;

[0020] The second processing module is used to obtain the motion control strategy of the bronchoscope based on the cavity center.

[0021] The motion control module is used by the endotracheal intubation robot to adjust the position and posture of the bronchoscope according to the motion control strategy, so that the bronchoscope moves toward the center of the cavity.

[0022] Preferably, the acquisition module uses a bronchoscope to acquire image information of human body cavities.

[0023] Preferably, the first processing module is used to segment the cavity region using an adaptive threshold segmentation and target detection algorithm, and calculate the centroid of the cavity region as the cavity center.

[0024] Preferably, the first processing module is used to design a speed-based motion controller with the cavity center position as feedback information, and to adjust the motion controller parameters online using a fuzzy algorithm.

[0025] Preferably, the motion control module sends the calculated feed speed, rotation speed, and end-bending speed of the bronchoscope to the endotracheal intubation robot via the motion controller for motor drive, thereby controlling the movement of the bronchoscope to complete the intubation operation.

[0026] This invention utilizes the displacement deviation between the image center and the cavity center to control the endotracheal intubation robot in real time, adjusting the position and orientation of the bronchoscope to ensure its accurate movement along the cavity center. This invention improves the efficiency and safety of endotracheal intubation surgery.

[0027] This invention's technical solution, based on image intensity analysis and HOG feature extraction, achieves cavity center detection. Adaptive control of the endoscope (fiberoptic bronchoscope) tip is achieved through closed-loop feedback, enabling automatic manipulation of the endoscope. Fuzzy logic algorithms are used to update the control gain online, improving the stability of flexible endoscope control. The advantage of this method is that it allows the endotracheal intubation robot to autonomously adjust the fiberoptic bronchoscope's state under monocular vision alone, avoiding direct friction and collision between the bronchoscope and human tissue structures, thus improving surgical efficiency and safety. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the motion control method based on vision guidance of the present invention.

[0029] Figure 2 This is a schematic diagram of a vision-guided motion control device. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Example 1:

[0033] like Figure 1 As shown, this invention provides a vision-guided motion control method for autonomous cannulation in minimally invasive surgeries such as gastroscopy, comprising:

[0034] Step S1: Obtain image information of human body cavities;

[0035] Step S2: Preprocess the image;

[0036] Step S3: Obtain the cavity center of the human body cavity based on the preprocessed image information;

[0037] Step S4: Based on the cavity center, obtain the motion control strategy for the bronchoscope;

[0038] Step S5: The endotracheal intubation robot adjusts the position and orientation of the bronchoscope according to the motion control strategy, so that the bronchoscope moves toward the center of the cavity.

[0039] In one embodiment of the present invention, in step S1, the internal tissue structure of the human body cavity is closely related to individual differences, individual developmental level, individual physical condition, and the posture of the endoscope tip. An endoscope (fiberoptic bronchoscope) is used to enter the human body cavity through a natural opening (i.e., the nasal cavity) to acquire image information of the human body cavity.

[0040] In one embodiment of the present invention, in step S2, each frame of the image information is subjected to grayscale conversion and contrast stretching; the grayscale value of the image after contrast stretching is inverted; and then the image after grayscale inversion is binarized to obtain a binarized image.

[0041] In one embodiment of the present invention, step S3, obtaining the cavity center, specifically involves: performing threshold segmentation on the preprocessed image; establishing evaluation criteria to screen multiple isolated dark regions caused by lighting conditions and the complex structure of the cavity, thereby obtaining the optimal dark region; for the glottis, a special structure in the detection image that simultaneously contains both the trachea and esophagus—a target recognizer—using SVM to extract histogram oriented gradient features (HOG) to accurately identify the glottis position to assist in determining the optimal dark region. The cavity center is detected by calculating the centroid of the optimal dark region.

[0042] Furthermore, a threshold segmentation algorithm is used to process the image, binarizing it, with the threshold S set as follows:

[0043] S = I″ max -20 (where I″) max (Represents the highest grayscale value in an image). Multiple isolated dark regions, caused by lighting conditions and the complex structure of the cavity, are filtered for their centers using an algorithm. This algorithm is as follows: with Among them, P min and P mean These represent the minimum and average gray values ​​within each isolated dark region of the grayscale image, respectively, while A represents the number of pixels within each region. This algorithm can yield O... * The region with the largest value is the dark region with a relatively large grayscale value in the entire image. For the glottis, there are two cavity centers. Both cavities are relatively large, leading to the trachea and esophagus respectively. Using the filtering algorithm mentioned above might not correctly identify the cavities. To address this issue, an SVM-based detector is constructed, and the algorithm for starting this detector is S′. * =0.68A1-A2, where A1 and A2 represent the areas of the two largest regions among all regions (A1>A2), when S′ * When the value is greater than 0, the detector is activated. A bounding box is used to define the glottis, and then the cavity center is found within the box. Finally, after determining the optimal region, the algorithm... We obtain the two-dimensional coordinates P′ that can be assumed to be the center of the cavity in the image coordinate system. l =[p′ x ,p′ y ]. Where [x,y] represents the O selected above. * The two-dimensional coordinates of all pixels within the dark region in the image coordinate system.

[0044] In one embodiment of the present invention, in step S4, the displacement error of the regional image center is calculated based on the cavity center; a velocity-based motion control strategy is constructed using PD, and the motion controller parameters are adjusted in real time using a fuzzy algorithm; the feedback error is input, and the motion controller inputs the velocities of the three sub-motions of the bronchoscope, including the feed speed, rotation speed, and end-bending speed. A hybrid control method of fuzzy PID and jogging is used to adjust the attitude of the bronchoscope end-effector so that the detected cavity center aligns with the image center, coordinating with the feed of the bronchoscope. The fuzzy logic-based motion control strategy enables the bronchoscope to complete the attitude adjustment of its end-effector even without a kinematic model suitable for flexible mechanisms.

[0045] Furthermore, step S4 specifically involves: First, calculating the center of the cavity. and the center of the lens Pixel error E = [e x ,e y ] T Analysis shows that the adjustment of the bronchoscope's direction can be considered as a combination of two basic movements, namely, rotational motion w. z and bending motion w y The error between the two is defined as:

[0046]

[0047] The two-dimensional image coordinate system E = [e x ,e y ] T Further conversion to the polar coordinate system results in error E′=[e′ d ,e′ θ ] T ,e′ d and e′ θ The horizontal and axial errors are respectively corrected through the robot's vertical and rotational movements. Then, a proportional-derivative (PD) controller is used to construct a mapping from the error e′ to the rotational speed w. z and bending speed w y The posture adjustment of the end of the bronchoscope is shown below:

[0048]

[0049]

[0050] in, These are the stiffness gain coefficients of two motion control systems, and That is the damping gain coefficient. and They are e′ respectively d and e′ θ The derivative of . Furthermore, the control coefficients of the controller are updated in real time throughout the control process to achieve better control performance. A fuzzy algorithm is used to adjust the aforementioned PD gain, i.e. and Based on PD control, the required rotational and bending angles of the bronchoscope tip are calculated and sent to the robot's motor controller. In short, based on visual feedback, the position and orientation of the bronchoscope tip are adjusted to achieve smooth bronchoscope steering.

[0051] In one embodiment of the present invention, in step S5, the endotracheal intubation robot sends the speed output by the controller as the control command to the robot's motor driver, and the motor responds to the control command; the endotracheal intubation robot adjusts the position and posture of the bronchoscope so that it moves toward the center of the cavity.

[0052] Example 2:

[0053] like Figure 2 As shown, the present invention provides a motion control device based on vision guidance, characterized in that it includes:

[0054] The acquisition module is used to acquire image information of human body cavities;

[0055] A preprocessing module is used to preprocess the image;

[0056] The first processing module is used to obtain the cavity center of the human body cavity based on the preprocessed image information;

[0057] The second processing module is used to obtain the motion control strategy of the bronchoscope based on the cavity center.

[0058] The motion control module is used by the endotracheal intubation robot to adjust the position and posture of the bronchoscope according to the motion control strategy, so that the bronchoscope moves toward the center of the cavity.

[0059] As one embodiment of the present invention, the acquisition module uses a bronchoscope to acquire image information of human body cavities.

[0060] As one embodiment of the present invention, the first processing module is used to segment the cavity region using an adaptive threshold segmentation and target detection algorithm, and calculate the centroid of the cavity region as the cavity center.

[0061] As one embodiment of the present invention, the first processing module is used to design a speed-based motion controller with the cavity center position as feedback information, and to adjust the motion controller parameters online using a fuzzy algorithm.

[0062] In one embodiment of the present invention, the motion control module sends the calculated feed speed, rotation speed and end-bending speed of the bronchoscope to the endotracheal intubation robot via the motion controller for motor drive, thereby controlling the movement of the bronchoscope to complete the intubation operation.

[0063] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A motion control device based on vision guidance, characterized in that, include: The acquisition module is used to acquire image information of human body cavities; A preprocessing module is used to preprocess the image; The first processing module is used to obtain the cavity center of the human body cavity based on the preprocessed image information; The second processing module is used to obtain the motion control strategy of the bronchoscope based on the cavity center. The motion control module is used by the endotracheal intubation robot to adjust the position and posture of the bronchoscope according to the motion control strategy, so that the bronchoscope moves toward the center of the cavity. The acquisition module uses a bronchoscope to acquire image information of human body cavities; The first processing module is used to segment the cavity region using an adaptive threshold segmentation and target detection algorithm, and calculate the centroid of the cavity region as the cavity center; The second processing module is used to design a velocity-based motion controller with the cavity center position as feedback information, and to adjust the motion controller parameters online using a fuzzy algorithm; The motion control module sends the calculated feed speed, rotation speed and end-bending speed of the bronchoscope to the endotracheal intubation robot via the motion controller to drive the motor and control the movement of the bronchoscope to complete the intubation operation. The second processing module performs the following: First, calculate the cavity center. and the center of the lens pixel error Analysis shows that the adjustment of the bronchoscope's direction can be considered as a combination of two basic movements: rotational motion. and bending motion The error between the two is defined as: Two-dimensional image coordinate system Error converted to polar coordinates , and The horizontal and axial errors are respectively corrected through the robot's vertical and rotational movements; a proportional-derivative (PD) controller is used to correct these errors. Construct a mapping; to rotational speed and The posture adjustment of the end of the bronchoscope is shown below: in, These are the stiffness gain coefficients of two motion control systems, and That is the damping gain coefficient. and They are respectively and The derivative; A fuzzy algorithm is used to adjust the PD gain, i.e. ( , ) and( , Based on PD control, the required rotational and bending motion angles of the bronchoscope tip are calculated and sent to the robot's motor controller.

Images