Flexible Instrument Control Method, System, Electronic Device, Storage Medium and Robot

Abstract

The present invention relates to the technical field of medical devices, and provides a flexible instrument control method, system, electronic device, storage medium, and robot. The flexible instrument is configured with a virtual navigation module for generating virtual navigation data, and an image acquisition module is further configured at the end of the flexible instrument for acquiring real images. The flexible instrument control method includes: controlling the registration of the virtual navigation module based on the real images; obtaining the traveling position and state of the flexible instrument in the target cavity system; obtaining virtual navigation data based on the traveling position and state; and controlling the end of the flexible instrument to travel to the target position in the target cavity system based on the virtual navigation data. The flexible instrument control method and robot of the present invention are used to solve the defects in the prior art that the preparation and use process of using a robot device to implement the operation of a flexible instrument through a cavity system is time-consuming, the use operation is complex, and the registration efficiency and accuracy are relatively low.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a flexible instrument control method, system, electronic device, storage medium and robot. Background Art

[0002] When operating a traditional flexible instrument through a cavity system, generally an operator holds the flexible instrument by hand and stands beside the target object to perform inspections and operations of preset actions. During the process of the flexible instrument entering the anatomical structure through the cavity system, the operator generally needs to hold the control end of the flexible instrument with one hand and hold the flexible instrument at the entrance end of the cavity system with the other hand, and use both hands and a single arm to control the attitude of the end of the flexible instrument, so as to control the flexible instrument to travel in the cavity system and finally reach the target anatomical structure site. During the operation process, the operator also needs to observe the visualization interface to understand the position of the end of the flexible instrument in the cavity system in real time, so as to judge and adjust the attitude of the end of the flexible instrument. This puts forward relatively high requirements for the operator's hand-eye coordination operation ability, and the operator's work intensity is relatively high.

[0003] To solve the above problems, currently a robotic device is used to operate the flexible instrument, changing the manual direct operation method to an indirect operation method. Operating the robotic device to control the flexible instrument requires registering the flexible instrument control robotic arm of the robotic device and the anatomical position of the target object. In the prior art, there are manual registration methods and automatic registration methods. Among them, manual registration requires the operator to spend a long time training and learning the usage method of the robotic device in advance, and the usage operation is complex; the algorithm of automatic registration is complex, and the registration process requires additional detection equipment and positioning devices. The placement preparation of the registration equipment is time-consuming, the preparation and implementation process takes a long time, the operation is complex, and there are deviations and other influences in the positioning devices, so the registration efficiency and accuracy are relatively low. Summary of the Invention

[0004] The present invention provides a flexible instrument control method, system, electronic device, storage medium and robot, so as to solve the defects that the preparation and use process time of using a robotic device to implement the operation of a flexible instrument through a cavity system in the prior art is long, the usage operation is complex, and the registration efficiency and accuracy are relatively low.

[0005] In a first aspect, the present invention provides a flexible instrument control method. The flexible instrument is configured with a virtual navigation module for generating virtual navigation data, and an image acquisition module is further configured at the end of the flexible instrument for acquiring real images. The control method of the flexible instrument includes:

[0006] Based on the real images, control the registration of the virtual navigation module.

[0007] Obtain the traveling position and state of the flexible instrument within the target channel system;

[0008] Based on the traveling position and state, obtain virtual navigation data;

[0009] Based on the virtual navigation data, control the end of the flexible instrument to travel to the target position within the target channel system.

[0010] According to the flexible instrument control method provided by the present invention, the controlling the registration of the virtual navigation module based on the real image includes:

[0011] Determine that the end of the flexible instrument reaches the target feature position of the target channel system;

[0012] Obtain the real image of the target feature position in real time and the virtual image generated by the virtual navigation module;

[0013] Based on the real image and the virtual image, control the registration of the virtual navigation module so that the registered virtual image is consistent with the real image.

[0014] According to the flexible instrument control method provided by the present invention, the controlling the registration of the virtual navigation module based on the real image and the virtual image so that the registered virtual image is consistent with the real image includes:

[0015] Control to calibrate the real image and obtain the first feature information of the target feature in the calibrated real image;

[0016] Control to calibrate the virtual image and obtain the second feature information of the target feature in the calibrated virtual image;

[0017] Based on the first feature information and the second feature information, determine the registration parameters;

[0018] Based on the registration parameters, control the virtual navigation module to perform registration so that the registered second feature information is consistent with the first feature information.

[0019] According to the flexible instrument control method provided by the present invention, the target feature includes a branch channel; the controlling to calibrate the real image and obtain the first feature information of the target feature in the calibrated real image includes:

[0020] Obtain the first fitting graph of the profiles of the sub-channels of the branch channel in the real image;

[0021] Based on the first fitting graph, determine the first calibration parameter;

[0022] Based on the first calibration parameter, control and adjust the end attitude of the flexible instrument to calibrate the perspective of the real image to the target perspective;

[0023] Obtain the first feature information, where the first feature information includes the cross-sectional area of the first fitting graph of each sub-channel contour of the branch channel in the real image under the target perspective.

[0024] According to the flexible instrument control method provided by the present invention, the determining the first calibration parameter based on the first fitting graph includes:

[0025] Obtain the geometric center information of the first fitting graph of each sub-channel contour and the position information of the connection line of each geometric center;

[0026] Based on the geometric center information and the position information of the connection line of the geometric centers, determine the first calibration parameter;

[0027] Wherein, the first calibration parameter includes an angle offset and a position offset. The angle offset is the included angle between the connection line of the geometric centers and the connection line of the geometric centers of the target graph in the real image under the target perspective, and the position offset is the distance between the symmetric center point surrounded by each geometric center and the center point of the real image under the target perspective.

[0028] According to the flexible instrument control method provided by the present invention, the controlling and calibrating the virtual image and obtaining the second feature information of the target feature in the calibrated virtual image includes:

[0029] Obtain the second fitting graph of each sub-channel contour of the branch channel in the virtual image;

[0030] Based on the second fitting graph, determine the second calibration parameter;

[0031] Based on the second calibration parameter, control and adjust the virtual image perspective of the virtual navigation module to calibrate the perspective of the virtual image to the target perspective;

[0032] Obtain the second feature information, where the second feature information includes the cross-sectional area of the second fitting graph of each sub-channel contour of the branch channel in the virtual image under the target perspective.

[0033] According to the flexible instrument control method provided by the present invention, the determining the registration parameter based on the first feature information and the second feature information includes:

[0034] According to the cross-sectional area of the first fitting graph and the cross-sectional area of the second fitting graph, based on a preset relational expression, obtain the depth adjustment parameter of the virtual image; the registration parameter includes the depth adjustment parameter.

[0035] According to the flexible device control method provided by the present invention, the obtaining of virtual navigation data based on the travel position and state includes:

[0036] Determining the current travel position and state of the flexible device;

[0037] Based on the planned path and the current travel position and state, acquiring the virtual navigation data;

[0038] Among them, the virtual navigation data includes a virtual image and an operation guidance prompt, the virtual image displays the virtual cavity image corresponding to the current travel position and state, and the operation guidance prompt is used to prompt the target adjustment action of the flexible instrument at the current travel position and state.

[0039] According to the flexible instrument control method provided by the present invention, the control of the end of the flexible instrument to move to the target position in the target cavity system based on the virtual navigation data includes:

[0040] When the end of the flexible instrument reaches the branch cavity position, the flexible instrument is controlled to perform the target adjustment action according to the operation guidance prompt to control the end of the flexible instrument to enter the target sub-cavity in the branch cavity.

[0041] According to the flexible instrument control method provided by the present invention, before controlling the registration of the virtual navigation module based on the real image, the method further includes:

[0042] Acquire a digital three-dimensional model of the target cavity system;

[0043] The planned path is determined based on the digitized three-dimensional model and the target position.

[0044] According to the flexible instrument control method provided by the present invention, the step of determining a planned path based on the digital three-dimensional model and the target position includes:

[0045] In the digital three-dimensional model, from the entrance of the target cavity system to the target position, the branch cavities of each level of the target cavity system are marked according to preset rules;

[0046] Based on the markings, determining at least one planned path;

[0047] An action threshold of the target position is determined according to the target shape and size of the target position.

[0048] According to the flexible instrument control method provided by the present invention, the step of obtaining the moving position and state of the flexible instrument in the target cavity system includes:

[0049] Obtain the length information of the flexible instrument entering the target channel system and the end state information of the flexible instrument;

[0050] Based on the length information, the end state information, and the planned path, determine the traveling position and state of the flexible instrument within the target channel system.

[0051] According to the flexible instrument control method provided by the present invention, a sensitive measurement element is further configured at the end of the flexible instrument, and the control method of the flexible instrument further includes:

[0052] Obtain the contact force between the end of the flexible instrument and the inner wall of the target channel system in real time;

[0053] When the contact force is greater than the first threshold, control to send a prompt message;

[0054] When the contact force is greater than the second threshold, control the flexible instrument to move in a restricted mode, where the second threshold is greater than the first threshold.

[0055] In a second aspect, the present invention further provides a control system, including:

[0056] A first control module, configured to control the registration of the virtual navigation module based on the real image;

[0057] A first acquisition module, configured to acquire the traveling position and state of the flexible instrument within the target channel system;

[0058] A second acquisition module, configured to acquire virtual navigation data based on the traveling position and attitude information;

[0059] A second control module, configured to control the end of the flexible instrument to travel to a target position within the target channel system based on the virtual navigation data;

[0060] Wherein, the flexible instrument is configured with the virtual navigation module, the virtual navigation module is used to generate the virtual navigation data, and an image acquisition module is further configured at the end of the flexible instrument, and the image acquisition module is used to acquire the real image.

[0061] In a third aspect, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the flexible instrument control method as described in any one of the above.

[0062] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the flexible instrument control method as described in any one of the above.

[0063] In a fifth aspect, the present invention further provides a robot, including a virtual navigation module and the above-mentioned electronic device, and the virtual navigation module is configured to be capable of generating virtual navigation data.

[0064] In the flexible instrument control method provided by the present invention, during the process of manipulating a flexible instrument to enter a target cavity system, first, a real image inside the target cavity system is acquired by an image acquisition module, and the virtual navigation module is registered by using the real image in combination with image recognition technology to achieve intraoperative real-time image registration. The registration process does not require additional detection devices and components, is simple to operate, saves time and effort, and improves the efficiency and accuracy of registration. Then, the virtual navigation module obtains virtual navigation data according to the traveling position and state of the flexible instrument in the target cavity system, and the traveling position and state of the flexible instrument in the target cavity system are tracked by means of the virtual navigation module. The operator observes the real image inside the target cavity system and the virtual image of the virtual navigation module to confirm the traveling position and state of the flexible instrument in the target cavity system. By watching the visual operation prompts given by the virtual navigation module, the virtual navigation module guides the flexible instrument to the target position, enabling the operator to complete the corresponding operation adjustment according to the virtual navigation data, and finally manipulating the flexible instrument to accurately reach the target position, realizing intraoperative visual operation guidance, without the need for additional learning and training, and effectively improving the implementation efficiency and accuracy of flexible instrument manipulation.

[0065] The flexible instrument control method of the present invention can be applied to a surgical robot for performing surgery through a natural human cavity, and can effectively solve the defects in the prior art that the preparation and use process of using a surgical robot device for performing surgery through a natural human cavity is time-consuming, the use operation is complex, and the efficiency and accuracy of registration are relatively low. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0067] Figure 1 is a working schematic diagram of the robot provided by an embodiment of the present invention;

[0068] Figure 2 is a schematic diagram of the composition and working process of the robot provided by an embodiment of the present invention;

[0069] Figure 3 is a working schematic diagram of the flexible instrument provided by an embodiment of the present invention;

[0070] Figure 4It is a schematic diagram of the operation process of the flexible instrument provided by an embodiment of the present invention;

[0071] Figure 5 It is a schematic diagram of the process of the flexible instrument control method provided by the present invention;

[0072] Figure 6 It is a schematic diagram before real image calibration provided by an embodiment of the present invention;

[0073] Figure 7 It is a schematic diagram after real image calibration provided by an embodiment of the present invention;

[0074] Figure 8 It is a schematic diagram of the principle of depth correction of the virtual navigation image provided by an embodiment of the present invention;

[0075] Figure 9 It is a schematic diagram of marking the target cavity system provided by an embodiment of the present invention;

[0076] Figure 10 It is a schematic diagram of virtual navigation operation guidance prompt provided by an embodiment of the present invention;

[0077] Figure 11 It is a schematic diagram of the structure of the control system provided by the present invention;

[0078] Figure 12 It is a schematic diagram of the structure of the electronic device provided by the present invention.

[0079] Reference numerals:

[0080] 100: Flexible instrument; 1: Interface box; 2: Flexible body; 21: First flexible body; 22: Second flexible body;

[0081] 200: Support device; 300: Robotic arm device; 301: Robotic arm assembly; 302: Flexible instrument drive device; 400: Host; 401: Main robotic arm; 500: Operating table; 600: Display system; 700: Virtual navigation module; 800: Entrance; 900: Introduction device;

[0082] 1110: First control module; 1120: First acquisition module; 1130: Second acquisition module; 1140: Second control module;

[0083] 1210: Processor; 1220: Communication interface; 1230: Memory; 1240: Communication bus. Detailed implementation manners

[0084] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the protection scope of the present invention.

[0085] In the description of the embodiments of the present invention, it should be noted that unless otherwise clearly specified and limited, the terms "first" and "second" are used for numbering the product components for clear description and do not represent any substantial difference. Terms such as "upper", "lower", "left", and "right" are only used to represent relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0086] It should be noted that unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0087] The flexible instrument control method of the present invention is applicable to a robot system for manipulating a flexible instrument to control the flexible instrument to enter a target position through a target channel system. Among them, the flexible instrument can be any instrument that can be manipulated to perform a preset action and has the characteristics of being slender and flexible; the robot system can be any robot that can manipulate the above flexible instrument. In the embodiments of the present invention, a surgical robot and a medical flexible instrument, such as a bronchoscope, are taken as examples to better illustrate the beneficial effects of the flexible instrument control method, control system, electronic device, non-transitory computer-readable storage medium and robot of the present invention.

[0088] The following will be combined with Figures 1-12 Describe the flexible instrument control method, control system, electronic device, non-transitory computer-readable storage medium and robot of the present invention.

[0089] Figure 1 is the operation schematic diagram of the robot provided by the embodiment of the present invention. As Figure 1 shown, some embodiments of the robot include a flexible instrument 100, a support device 200, a robotic arm device 300, a host 400, an operation console 500, a display system 600 and a virtual navigation module 700.

[0090] Among them, the host 400 includes a main robotic arm 401, a main control device, a data processing system, a detection system, a robotic arm drive system, etc.; the operation console 500 includes a controller and a communication system, which is connected to the main control device of the host 400 through a communication method, so as to transmit the action and position signals of the controller to the main control device. The main control device converts the above signals into control data for driving the flexible instrument 100 to move, and transmits the control data to the robotic arm drive system and the robotic arm device 300. The main control device also receives data from the data processing system. The data processing system is used to process the sensing data obtained by the detection system and the data of the robotic arm drive system, so as to ensure the safe and accurate control of the main control device.

[0091] The host 400 is connected to the robotic arm device 300 through the main robotic arm 401, and the robotic arm drive system drives the main robotic arm 401 to move; the main robotic arm 401 can drive the robotic arm device 300 to move up and down along the vertical direction, and the main robotic arm 401 can also drive the robotic arm device 300 to rotate around the vertical axis, so that the robotic arm device 300 can accurately align with the entrance 800 of the target channel system. The flexible instrument 100 is installed on the robotic arm device 300. The robotic arm device 300 transmits the power of the micro servo motor to the flexible instrument 100 according to the control signal, and drives the flexible instrument 100 to perform linear, rotational, and end bending movements, so as to realize that the flexible instrument 100 enters the target channel system through the entrance 800 of the target channel system, and adjusts the traveling posture in the target channel system and finally reaches the target position.

[0092] The end of the flexible instrument 100 is configured with an image acquisition module, such as an endoscope. The operator watches the image of the target channel system collected by the image acquisition module presented by the display system 600 and the visual operation prompt and the three-dimensional digital image of the target channel system presented in the virtual navigation module 700, and manipulates the main robotic arm 401 through the controller on the operation console 500 (which can be a bedside operation console or a remote operation console) to align the robotic arm device 300 with the entrance 800 of the target channel system, and controls the robotic arm device 300 to drive the flexible instrument 100 to enter the target position in the target channel system through the entrance 800 of the target channel system, or drive the flexible instrument 100 to move out of the target channel system. During the movement of the flexible instrument 100, the flexible instrument 100 is supported by the support device 200.

[0093] In some embodiments, the flexible instrument 100 has a proximal end, a distal end, and at least one lumen extending between the proximal end and the distal end. The flexible instrument 100 is used to build a target working channel between the entrance 800 of the target channel system and the target position, and the working instrument can reach the target position through the target working channel, so as to perform the target operation on the target position.

[0094] Among them, the target channel system can be the natural human body channels of a patient, such as the human tracheal tree, esophagus, gastrointestinal tract system, etc.; the entrance 800 of the target channel system includes the oral cavity, nasal cavity, etc. The target position can be the anatomical structure position of the lesion of the patient. The flexible instrument 100 can be a medical instrument such as a bronchoscope, gastroscope, colonoscope, etc.; the working instrument can be a medical instrument such as an inspection, sampling and treatment tool, and the inspection, sampling and treatment tool (including interventional instruments such as surgical, diagnostic, treatment or biopsy instruments) reaches the lesion position through the target working channel built by the flexible instrument 100, and the position and operation of the inspection, sampling and treatment tool are carried out under the fluoroscopic examination of the fluoroscopic device, so as to perform operations such as imaging detection, biopsy sampling and energy treatment surgery on the lesion.

[0095] Figure 2 It is a schematic diagram of the composition and operation process of the robot provided by the embodiment of the present invention. As Figure 2 shown, when the operator operates the robotic arm device 300 to control the flexible instrument 100 to enter the target channel system, the image acquisition module obtains the image inside the target channel system in real time as it travels inside the target channel system with the flexible instrument 100; the image acquisition module sends the image inside the target channel system to the display system 600, and the operator observes the traveling position and state of the flexible instrument 100 inside the target channel system in real time through the display system 600. The image acquisition module also synchronously sends the image data inside the target channel system to the detection system, and the detection system sends the real-time data inside the target channel system after calculation and processing to the virtual navigation module 700. The virtual navigation module 700 automatically or manually registers the navigation system image according to the real-time data inside the target channel system, so as to achieve the effect that the position and angle information of the navigation system image are consistent with the real-time image of the image acquisition module. The operator can more intuitively enter the target channel under the guidance of the planned path by watching the visual navigation prompt presented in the virtual navigation module 700.

[0096] During the entry process, in addition to the guiding function, the virtual navigation module 700 also has the function of prompting and guiding the attitude adjustment position and angle of the flexible instrument 200 inside the target channel system. The operator observes the real-time image of the flexible instrument 200 and the prompt information given by the virtual navigation module 700 to confirm the position of the end of the flexible instrument 200 inside the target channel system, and adjusts the flexible instrument 200 to enter the next-level target channel until it reaches the target position. After the end of the flexible instrument 100 reaches the target position, the working instrument is inserted through the target working channel built by the flexible instrument 100, so as to perform the target operation on the target position. The working instrument can also be within the field of view of the image acquisition module of the flexible instrument 100 to ensure the safety and accuracy of the target operation.

[0097] Figure 3It is a working schematic diagram of the flexible instrument provided by an embodiment of the present invention. As Figure 3 shown, the flexible instrument 100 is characterized by being slender and flexible, and it is difficult to maintain a straight shape in the natural state. Especially during the process of the flexible instrument 100 moving into the target channel system, when the section of the flexible instrument 100 outside the target channel system is not effectively supported, when the linear movement of the flexible instrument 100 encounters resistance, or when a rotation operation is performed on the flexible instrument 100 or the end of the flexible instrument 100 is bent and adjusted, the flexible instrument 100 is prone to irregular bending and torsion phenomena, resulting in poor positioning accuracy of the flexible instrument 100. The determined shape is the basis for the robotic arm device 300 to accurately control the flexible instrument 100. To ensure that the section of the flexible instrument 100 outside the target channel system maintains a straight shape, when the flexible instrument 100 enters or exits the target channel system, the support device 200 needs to effectively support the flexible instrument 100 along the length direction of the flexible instrument 100.

[0098] The robotic arm device 300 includes a robotic arm assembly 301 and a flexible instrument driving device 302. One end of the support device 200 is connected to the movable end of the robotic arm assembly 301, and the other end of the support device 200 is connected to the fixed end of the robotic arm assembly 301. It is defined that the movable end of the robotic arm assembly 301 is far from the target channel system, and the fixed end of the robotic arm assembly 301 is close to the target channel system. The flexible instrument driving device 302 is arranged at the movable end of the robotic arm assembly 301. The head end of the flexible instrument 100 is connected to the flexible instrument driving device 302 through an interface device, and the tail end of the flexible instrument 100 passes through the fixed end of the robotic arm assembly 301. The support device 200 can support the flexible instrument 100 along the length direction. One end of the support device 200 is connected to the flexible instrument driving device 302, and the other end is connected to the fixed end of the robotic arm assembly 301. The support device 200 can undergo contraction deformation or extension deformation as the robotic arm assembly 301 moves.

[0099] Figure 3 The arrow direction in [[ ]] indicates the working direction, and the working direction is the moving direction of the movable end of the robotic arm assembly 301 relative to the fixed end. The movable end of the robotic arm assembly 301 approaches or moves away from the fixed end of the robotic arm assembly 301 along the arrow direction.

[0100] When the movable end of the robotic arm assembly 301 moves in the working direction towards or away from the fixed end of the robotic arm assembly 301, the support device 200 can contract or extend as the robotic arm assembly 301 moves. The support device 200 is formed with a support channel for supporting the flexible instrument 100, and the flexible instrument 100 is inserted through the support channel. The flexible instrument 100 moves synchronously with the support device 200 in the working direction. The support device 200 can maintain a straight shape in the working direction, thereby ensuring that the flexible instrument 100 can maintain a straight shape during movement.

[0101] When the movable end of the robotic arm assembly 301 moves in the working direction towards the fixed end of the robotic arm assembly 301, the support device 200 is gradually compressed and shortened in the length direction. The support device 200 always maintains a straight shape in the working direction. Thus, the flexible instrument 100 can stably enter the target position within the target channel system along the support channel from the entrance 800 of the target channel system. When the movable end of the robotic arm assembly 301 moves in the working direction away from the fixed end of the robotic arm assembly 301, the support device 200 is gradually stretched and lengthened in the length direction. The support device 200 always maintains a straight shape in the working direction. Thus, the flexible instrument 100 can stably move out of the target channel system along the support channel to the outside of the target channel system.

[0102] The flexible instrument 100 has a working channel inside. The flexible instrument 100 enters the target position within the target channel system along the support channel. The working instrument reaches the target position along the working channel and performs a target operation on the target position. During the process of the flexible instrument 100 moving along the support channel towards the target position within the target channel system, the flexible instrument driving device 302 transmits the power of the micro servo motor to the flexible instrument 100, driving the flexible instrument 100 to rotate or perform a bending action at the end, so that the flexible instrument 100 can adjust the end attitude in the target channel system and finally reach the target position.

[0103] During the process of the flexible instrument 100 moving along the support channel towards the target position within the target channel system, or during the process of moving from the target channel system to the outside of the target channel system along the support channel, the support device 200 can always maintain a straight shape, so that a partial section of the flexible instrument 100 located outside the target channel system can be stably supported, effectively avoiding irregular bending and twisting of the flexible instrument 100 during movement, rotation or end bending. The flexible instrument 100 maintains a straight shape during movement, which is conducive to the flexible instrument 100 accurately moving to the target position within the target channel system or moving out of the target channel system.

[0104] Figure 4It is a schematic diagram of the operation process of the flexible instrument provided by an embodiment of the present invention. As Figure 4 shown, the detection system includes a sensing module and a video module. The flexible instrument 100 includes an interface box 1 and a flexible body 2. The flexible body 2 is connected to the interface box 1, and the interface box 1 is connected to the flexible instrument driving device 302, so as to install the flexible instrument 100 on the robotic arm device 300 of the robot. The control adjustment signal sent by the operator through the operation console 500 is converted into a driving control signal of the micro servo motor of the flexible instrument driving device 302 by the main control device, and the rotational movement of the micro servo motor drives the flexible body 2 connected through the interface box 1. The end posture of the flexible body 2 is controlled by the combination of three degrees of freedom of movement: up and down bending at the end, left and right rotation, and forward and backward movement.

[0105] In some embodiments, the flexible body 2 may include a first flexible body 21 and a second flexible body 22. As the first flexible body 21 enters deeper into the target channel system, the inner diameter of the channel of the target channel system becomes thinner and thinner. Due to the limitation of the outer diameter size of the first flexible body 21, the first flexible body 21 generally can only reach a pre-specified first preset position, and then the second flexible body 22 with a thinner outer diameter size enters through the first working channel of the first flexible body 21. The second flexible body 22 does not have a visual function. The position of the second flexible body 22 in the target channel system can be detected by a sensor provided at the end of the second flexible body 22. The sensor can be a shape sensor or a magnetic positioning sensor; the detection data is sent to the detection system, and the detection system processes the weak optoelectronic signal of the sensor into a digital signal that can be stably transmitted, and then calculates through the data processing system to convert the digital signal of the sensor into data with a certain physical meaning. The operator compares the position data of the second flexible body 22 through the virtual navigation module 700 to determine whether the second preset position has been reached; when the operator determines that the position of the second flexible body 22 has reached the previously set second preset position, a working instrument can be inserted into the second working channel of the second flexible body 22, and the relative position between the working instrument and the target position is generally confirmed through a medical fluoroscopy device.

[0106] Figure 5 It is a schematic diagram of the flow of the flexible instrument control method provided by the present invention. As Figure 5 shown, the flexible instrument control method provided by the present invention includes the following steps:

[0107] Step S10, based on the real image, control the registration of the virtual navigation module;

[0108] Step S20, obtain the traveling position and state of the flexible instrument in the target channel system;

[0109] Step S30, based on the above traveling position and state, obtain virtual navigation data;

[0110] Step S40: Based on the virtual navigation data, control the end of the flexible instrument to travel to the target position within the target channel system;

[0111] Wherein, the flexible instrument is configured with a virtual navigation module for generating virtual navigation data; the end of the flexible instrument is further configured with an image acquisition module for acquiring real images.

[0112] In this embodiment, the flexible instrument is used in conjunction with the virtual navigation module. During the process of controlling the flexible instrument to travel along the planned path within the target channel system and adjusting its posture, the virtual navigation module can generate virtual navigation data based on the real-time judgment of the positional relationship between the flexible instrument and the planned path, and the virtual navigation data presents a visual virtual navigation screen. The virtual navigation screen not only presents the travel path within the target channel system in the form of continuous lines, but also gives the operator prompts in the form of numbers and graphics according to the next target position to be traveled, such as the rotation angle of the end of the flexible instrument, the bending angle of the end, etc. Thus, the virtual navigation module gives the operator intraoperative visual and graphical operation guidance prompt information to guide the posture adjustment position and angle of the flexible instrument within the target channel system. The operator adjusts the flexible instrument to enter the next-level target channel according to the visual navigation prompts presented in the virtual navigation module until reaching the target position, enabling the operator to complete the corresponding operation adjustment according to the prompt information without the need for additional learning and training, effectively improving the implementation efficiency and accuracy of the flexible instrument control.

[0113] Meanwhile, the end of the flexible instrument is further configured with an image acquisition module, such as an endoscope. During the process of the image acquisition module traveling within the target channel system along with the flexible instrument, it can acquire real images within the target channel system in real time.

[0114] Before the virtual navigation module outputs intraoperative operation guidance prompt information, it is necessary to register the virtual navigation module with the actual position of the flexible instrument. According to the real images within the target channel system, through image recognition, automatic or manual registration is performed on the virtual images of the virtual navigation module and the real images of the flexible instrument, so that the perspective of the virtual navigation module is consistent with the perspective of the flexible instrument, achieving the effect that the position and angle information of the virtual images of the virtual navigation module are consistent with the real images of the image acquisition module. The flexible instrument can more intuitively enter the target channel under the guidance of the planned path. The registration process does not require additional detection equipment and components, solving the problems of time-consuming and laborious preparation for placing existing registration equipment and the influence caused by deviations of positioning devices, and improving the efficiency and accuracy of registration through real-time vision combined with image recognition technology.

[0115] The flexible instrument control method of the present invention, during the process of manipulating the flexible instrument into the target cavity system, first obtains the real image inside the target cavity system through the image acquisition module, and uses the real image combined with image recognition technology to register the virtual navigation module, realizing real-time intraoperative image registration. The registration process does not require additional detection equipment and components, is simple to operate, saves time and effort, and improves the efficiency and accuracy of registration. Then, the virtual navigation module obtains virtual navigation data according to the traveling position and state of the flexible instrument in the target cavity system, and uses the virtual navigation module to track the traveling position and state of the flexible instrument in the target cavity system. The operator observes the real image inside the target cavity system and the virtual image of the virtual navigation module to confirm the traveling position and state of the flexible instrument in the target cavity system. By watching the visual operation prompt given by the virtual navigation module, the virtual navigation module guides the flexible instrument to the target position, enabling the operator to complete the corresponding operation adjustment according to the virtual navigation data, and finally manipulating the flexible instrument to accurately reach the target position, realizing visual operation guidance during the operation, without the need for additional learning and training, effectively improving the implementation efficiency and accuracy of flexible instrument manipulation.

[0116] The flexible instrument control method of the present invention can be applied to the operation of a surgical robot for performing surgery through a natural human cavity, and can effectively solve the defects in the prior art that the preparation and use process of using a surgical robot device to perform surgery through a natural human cavity is time-consuming, the use operation is complex, and the efficiency and accuracy of registration are relatively low.

[0117] In some embodiments, the flexible instrument control method of the present invention can be applied to a surgical robot for performing surgery through a natural body cavity. With the aid of a virtual navigation module, the movement position of a flexible medical instrument, such as a bronchoscope, within the natural body cavity is tracked. The virtual navigation module guides the flexible medical instrument towards the target anatomical structure site and accurately reaches the target anatomical structure site. During the process of the flexible medical instrument traveling along the surgical planned path, the virtual navigation screen of the virtual navigation module not only presents the traveling path of the flexible medical instrument in the form of continuous lines, but also automatically gives prompts to the operator in the form of numbers and graphics according to the target position of the next step. For example, the rotation angle of the end of the flexible medical instrument, the bending angle of the end, etc., enabling the operator to complete the corresponding operation adjustments according to the prompt information of the intraoperative visual operation guidance without the need for additional learning and training, effectively improving the implementation efficiency and surgical accuracy of the surgery through the natural body cavity. At the same time, through an image acquisition module, such as an endoscope, a real image within the natural body cavity is obtained, and the virtual navigation module is registered by combining the real image with image recognition technology to achieve real-time intraoperative image registration. The registration process does not require additional detection devices and components, which can reduce the preparation and implementation time of the surgery through the natural body cavity. With intuitive operation under image navigation, it has the effects of quick preparation, convenient use, and easy operation, significantly improving the accuracy and stability of the surgery and improving the working conditions of doctors.

[0118] In some embodiments, to improve the convenience of the operator in operating the robot, the adjustment method of the controller on the operating table is consistent with the operation prompt graphics displayed on the navigation screen of the virtual navigation module, facilitating the operator to maintain consistent manipulation actions according to the graphic prompt information on the navigation screen.

[0119] Specifically, based on the real image, controlling the registration of the virtual navigation module specifically includes the following steps:

[0120] Step S100, determining that the end of the flexible instrument reaches the target feature position of the target cavity system;

[0121] Step S200, obtaining the real image of the target feature position and the virtual image generated by the virtual navigation module in real time;

[0122] Step S300, based on the real image and the virtual image, controlling the registration of the virtual navigation module so that the registered virtual image is consistent with the real image.

[0123] In this embodiment, the virtual navigation module displays three-dimensional images of the target channel system and the target position in the form of three-dimensional visualization images, and displays the planned path and the virtual channel image formed by the planned path on the three-dimensional image of the target channel system; the virtual navigation module also determines the positional relationship between the perspective of the flexible instrument and the planned path in real time according to the traveling position and state of the flexible instrument, outputs a virtual image of the virtual channel image corresponding to the traveling position and state of the flexible instrument, and correspondingly outputs a visual operation guidance prompt on the virtual image.

[0124] The virtual navigation module is registered by using real images combined with image recognition technology. The real image from the target perspective is used as the target image, and the virtual image perspective of the virtual navigation module is adjusted to be consistent with the perspective of the image acquisition module at the end of the flexible instrument, so as to adjust the positions and angles of the virtual image and the real image to be consistent. The flexible instrument can more intuitively enter the target channel under the guidance of the planned path, effectively improving the implementation efficiency and accuracy of the flexible instrument operation.

[0125] Specifically, when registering the virtual navigation module, first determine that the end of the flexible instrument reaches the target feature position of the target channel system. The target feature position refers to the position where the target feature is located in the target channel system, and the target feature refers to the feature that is inherent in the target channel system and has a specific recognition pattern. For example, if the target channel system is the tracheal tree, the target feature may be the carina of the trachea. Then, the real image of the target feature position is obtained in real time through the image acquisition module at the end of the flexible instrument, and the virtual image of the target feature position is generated in real time through the virtual navigation module; finally, the target features in the virtual image and the target image can be recognized through image recognition, and the position and angle differences of the target features in the virtual image and the real image are compared, and the virtual image perspective of the virtual navigation module is adjusted to be consistent with the perspective of the image acquisition module of the flexible instrument based on the differences, thereby completing the registration of the virtual navigation module. The position and angle information of the registered virtual image are consistent with the real image.

[0126] In this embodiment, the virtual navigation module is registered by using the real image of the target feature position, which improves the accuracy of image recognition and realizes the real-time image registration of the virtual navigation module by using real images combined with image recognition technology. The registration is accurate and reliable, and no additional detection equipment and components are required during the registration process. It is fast to prepare and simple and convenient to use.

[0127] In some specific embodiments, the target channel system is the airway, i.e., the tracheal tree, the target feature is the carina of trachea, and the target feature position is the position of the carina of trachea at the first-level branch of the airway, i.e., the first bifurcation position of the airway; the flexible instrument is a bronchoscope. After the end of the medical device reaches the first bifurcation position of the natural body cavity and the position is confirmed, the virtual image of the virtual navigation module and the real image of the endoscope can be automatically or manually compared. When there are position and angle differences between the images presented by the virtual image and the real image, the depth data and angle of the virtual image are automatically adjusted or manually adjusted to be consistent with the position and angle of the real image. The judgment and comparison of the two are displayed in digital form on the visualization interface. After the adjustment is consistent, the system completes the registration of the virtual image and the real image.

[0128] Specifically, based on the real image and the virtual image, the virtual navigation module is controlled for registration so that the registered virtual image is consistent with the real image, which specifically includes the following steps:

[0129] Step S310, control to calibrate the real image, and obtain the first feature information of the target feature in the calibrated real image;

[0130] Step S320, control to calibrate the virtual image, and obtain the second feature information of the target feature in the calibrated virtual image;

[0131] Step S330, determine the registration parameters based on the first feature information and the second feature information;

[0132] Step S340, based on the registration parameters, control the virtual navigation module for registration so that the registered second feature information is consistent with the first feature information.

[0133] In this embodiment, after obtaining the real image and the virtual image of the target feature position, first, calibrate the real image, adjust the viewing angle of the flexible instrument to the determined target viewing angle, and acquire the real image under the target viewing angle, so as to obtain the calibrated real image; the target feature in the calibrated real image has a determined position and angle, so that the first feature information of the target feature in the calibrated real image can be conveniently obtained. Then, calibrate the virtual image, adjust the viewing angle of the virtual image to be the same as that of the real image, and obtain the second feature information of the target feature in the calibrated virtual image, so as to obtain the first feature information and the second feature information of the target feature under the same viewing angle, and compare the differences between the virtual image and the real image more accurately. It should be noted that the first feature information and the second feature information of the target feature in this embodiment are actually the names of the same feature information of the target feature in different images; by comparing the first feature information and the second feature information, the differences between the target features in the virtual image and the real image can be determined, so that the registration parameters of the virtual navigation module can be adjusted. Finally, based on the registration parameters, adjust the positions and angles of the virtual image and the real image to be consistent, complete the registration of the virtual navigation module, and the second feature information after registration is consistent with the first feature information.

[0134] Specifically, the target feature includes a branched channel; controlling the calibration of the real image and obtaining the first feature information of the target feature in the calibrated real image specifically includes the following steps:

[0135] Step S311, obtain the first fitting graph of the profiles of each sub-channel of the branched channel in the real image;

[0136] Step S312, determine the first calibration parameter based on the first fitting graph;

[0137] Step S313, based on the first calibration parameter, control and adjust the end state of the flexible instrument to make the viewing angle of the calibrated real image be the target viewing angle;

[0138] Step S314, obtain the first feature information, where the first feature information includes the cross-sectional area of the first fitting graph of the profiles of each sub-channel of the branched channel in the real image under the target viewing angle.

[0139] In this embodiment, after obtaining the real image, first, image processing technology is applied to process the real image. The graphic contours of the sub-channels of the branched channel are extracted through image recognition and fitted into the contour boundaries of known shapes, so as to obtain the first fitted graph of the sub-channel contours of the branched channel in the real image. By comparing the differences between each first fitted graph and the target graph in the real image from the target perspective, the first calibration parameter is determined. The first calibration parameter is the adjustment amount required to adjust the end perspective of the flexible instrument to the target perspective. According to the first calibration parameter, the end state of the flexible instrument is adjusted, so as to calibrate the perspective of the real image to the target perspective. Finally, a real image is collected from the target perspective, and image processing is performed on the real image. The cross-sectional area of the first fitted graph of the sub-channel contours of the branched channel in the real image from the target perspective is obtained through image recognition, so as to obtain the first characteristic information of the branched channel.

[0140] In some specific embodiments, for real-time virtual image registration, the operator controls the bronchoscope to enter the target channel system through the operating console. For example, the target channel system is an airway system, and the branched channel is at the first-level branch of the main airway. When the bronchoscope reaches the carina, that is, the position of the first-level branch of the main airway, through the epiglottis from the entrance of the target channel system, a real image and a virtual image are obtained. Image processing technology is applied to process the image at the branched channel obtained, and the graphic contours of the right and left branched channels are extracted and fitted into circular contour boundaries, so as to obtain the first fitted graphs of the right and left channel contours.

[0141] Specifically, based on the first fitted graph, determining the first calibration parameter specifically includes the following steps:

[0142] Step S3121, obtaining the geometric center information of the first fitted graph of each sub-channel contour and the position information of the connection line of each geometric center;

[0143] Step S3122, determining the first calibration parameter based on the geometric center information and the position information of the connection line of the geometric centers;

[0144] Among them, the first calibration parameter includes an angular offset and a position offset. The angular offset is the included angle between the connection line of the geometric centers and the connection line of the geometric centers of the target graph in the real image from the target perspective; the position offset is the distance between the symmetric center point surrounded by each geometric center and the center point of the real image from the target perspective.

[0145] In this embodiment, after obtaining the first fitting graph of the profiles of each sub-channel before calibration, first calculate the geometric center coordinates of the first fitting graph of the profiles of each sub-channel, and calculate the position information of the connection lines of each geometric center. The included angle between the connection lines of each geometric center and the connection line of the geometric centers of the target graph in the real image under the target perspective can be determined through the position information of the connection lines of each geometric center, so as to determine the angle offset; the distance between the symmetric center point surrounded by each geometric center and the center point of the real image under the target perspective can be determined through each geometric center coordinate, so as to determine the position offset; the angle offset and the position offset are used to determine the first calibration parameter.

[0146] In some specific embodiments, as Figure 6 shown, first calculate the graph center coordinates of the first fitting graphs of the right and left channel profiles in the real image before calibration and the included angle θ between the connection line of the centers and the horizontal line of the real image; determine the angle offset for adjusting the horizontal angle of the perspective of the flexible instrument according to the included angle between the connection line of the centers and the horizontal line of the real image. Adjust the end state of the flexible instrument according to the angle offset and the position offset, and adjust the perspective of the bronchoscope to achieve calibration of the real image. As Figure 7 shown, in the calibrated real image, the connection line of the graph centers of the right and left channel profiles coincides with the horizontal axis of the image, and the midpoint of the connection line of the centers of the first fitting graphs of the right and left channel profiles is located at the center position of the image, that is, the intersection of the horizontal x-axis and the vertical y-axis, so as to achieve the perspective calibration of the image acquisition module of the flexible instrument. The target features in the real image collected by the image acquisition module of the flexible instrument are centered and horizontal. Finally, calculate the cross-sectional areas S1 and S2 of the first fitting graphs of the right and left channel profiles in the real image under the target perspective to obtain the first feature information.

[0147] For example, when the flexible instrument is a bronchoscope, after installation, the perspective of the bronchoscope is calibrated so that the image collected by the bronchoscope is centered and in a horizontal position.

[0148] Specifically, control the calibrated virtual image, and obtain the second feature information of the target feature in the calibrated virtual image, which specifically includes the following steps:

[0149] Step S321, obtain the second fitting graph of the profiles of each sub-channel of the branch channel in the virtual image;

[0150] Step S322, determine the second calibration parameter based on the second fitting graph;

[0151] Step S323, based on the second calibration parameter, control and adjust the perspective of the virtual image of the virtual navigation module to calibrate the perspective of the virtual image to the target perspective;

[0152] Step S324, obtain the second feature information, where the second feature information includes the cross-sectional areas of the second fitting graphs of the profiles of each sub-channel of the branch channel in the virtual image under the target perspective.

[0153] In this embodiment, after obtaining the virtual image, the same image processing technology as that for the real image is applied to process the virtual image. The graphic contours of the sub-channels of the branched channel are extracted through image recognition and fitted into a contour boundary with the same shape as that in the real image, so as to obtain a second fitted graphic of the sub-channel contours of the branched channel in the virtual image. By comparing the differences between each second fitted graphic and the target image in the real image from the target perspective, a second calibration parameter is determined. The second calibration parameter is the adjustment amount required to adjust the perspective of the virtual image to the target perspective. The perspective of the virtual image of the virtual navigation module is adjusted to be consistent with the real image according to the second calibration parameter, so as to calibrate the perspective of the virtual image to the target perspective. Finally, image processing is performed on the virtual image from the target perspective, and the cross-sectional areas of the second fitted graphics of the sub-channel contours of the branched channel in the virtual image from the target perspective are obtained through image recognition, so as to obtain the second characteristic information of the branched channel.

[0154] In some specific embodiments, the virtual image is processed, the graphic contours of the right and left channels are extracted and fitted into a circular contour boundary, and the cross-sectional areas S1nav and S2nav of the second fitted graphics of the right and left channel contours in the calibrated virtual image are calculated to obtain the second characteristic information.

[0155] Specifically, based on the first characteristic information and the second characteristic information, registration parameters are determined, which specifically include the following steps:

[0156] Step S331, based on a preset relational expression, obtain the depth adjustment parameter of the virtual image according to the cross-sectional area of the first fitted graphic and the cross-sectional area of the second fitted graphic. The registration parameter includes the depth adjustment parameter.

[0157] In this embodiment, Figure 8 is a schematic diagram of the principle of depth registration of the virtual image of the virtual navigation module. As Figure 8 shown, for the channel image obtained at the target feature position in the target channel system, when the depth of the perspective for obtaining the image from the target feature is different, there are also differences in the cross-sectional areas of the fitted graphics of the target feature on the channel image. Therefore, the cross-sectional area of the first fitted graphic and the cross-sectional area of the second fitted graphic are respectively the cross-sectional areas of the fitted graphics of the same target feature in the real image and the virtual image. Through the differences between the cross-sectional area of the first fitted graphic and the cross-sectional area of the second fitted graphic, the depth difference between the perspective of the virtual image and the perspective of the real image along the target channel system can be correspondingly converted, so that the depth value of the perspective adjustment of the virtual navigation module required to make the cross-sectional area of the first fitted graphic and the cross-sectional area of the second fitted graphic equal can be obtained.

[0158] Specifically, after adjusting the virtual image angle of the virtual navigation module to make the viewing angle of the virtual image consistent with the viewing angle of the image acquisition module of the flexible instrument, the cross-sectional area of the first fitting graph and the cross-sectional area of the second fitting graph obtained are imported into a preset relational formula. The preset relational formula is used to convert the difference between the cross-sectional area of the first fitting graph and the cross-sectional area of the second fitting graph into the depth adjustment parameter of the virtual navigation module, so as to obtain the depth adjustment parameter of the virtual image, and further determine the registration parameter of the virtual navigation module.

[0159] Among them, the preset relational formula for the depth adjustment parameter of the virtual image is: ΔA = f(S - Snav); in the formula, S is the cross-sectional area of the first fitting graph, which is also the cross-sectional area of the cavity image where the end of the flexible instrument is to enter; Snav is the cross-sectional area of the second fitting graph, which is also the cross-sectional area of the cavity image where the virtual navigation is to enter; ΔA is the depth adjustment parameter of the virtual image, which is also the depth value of the viewing angle adjustment of the virtual navigation module required to make Snav equal to S.

[0160] Finally, control and adjust the depth value of the virtual navigation module according to the depth adjustment parameter, so as to realize the registration of the virtual navigation module.

[0161] Specifically, based on the traveling position and state of the flexible instrument in the target cavity system, virtual navigation data is obtained, which specifically includes the following steps:

[0162] Step S301, determine the current traveling position and state of the flexible instrument;

[0163] Step S302, based on the planned path and the current traveling position and state of the flexible instrument, obtain virtual navigation data;

[0164] Among them, the virtual navigation data includes a virtual image and an operation guidance prompt. The virtual image displays the virtual cavity image corresponding to the current traveling position and state of the flexible instrument; the operation guidance prompt is used to prompt the target adjustment action of the flexible instrument in the current traveling position and state.

[0165] In this embodiment, during the process of controlling the flexible instrument to travel along the planned path and adjust its posture within the target channel system, the current traveling position and state of the flexible instrument within the target channel system are first determined. The virtual navigation module continuously judges the relationship between the current traveling position and state of the flexible instrument and the planned path, and generates virtual navigation data in real time according to the current traveling position and state of the flexible instrument within the target channel system. The virtual navigation data presents a visual virtual image. The virtual image presents the traveling path within the target channel system in the form of continuous lines, and displays the virtual channel image corresponding to the current traveling position and state of the flexible instrument. The virtual navigation data also gives the operator operation guidance prompts in the form of numbers and graphics according to the next traveling position of the planned path, such as the rotation angle of the end of the flexible instrument, the bending angle of the end, etc. The virtual navigation module gives the operator intraoperative visual operation guidance prompt information to guide the posture adjustment position and angle of the flexible instrument within the target channel system. The operator adjusts the flexible instrument to enter the next-level target channel by watching the virtual image and the visual operation guidance prompt presented in the virtual navigation module until reaching the target position, enabling the operator to complete the corresponding operation adjustment according to the prompt information without the need for additional learning and training, effectively improving the implementation efficiency and accuracy of the flexible instrument control.

[0166] Specifically, based on the virtual navigation data, controlling the end of the flexible instrument to travel to the target position within the target channel system specifically includes the following steps:

[0167] Step S401, when the end of the flexible instrument reaches the position of the branch channel, according to the operation guidance prompt, control the flexible instrument to perform the target adjustment action to control the end of the flexible instrument to enter the target sub-channel in the branch channel.

[0168] In this embodiment, when it is determined that the end of the flexible instrument reaches the position of the branch channel within the target channel system, the virtual navigation data will give operation guidance prompts for the traveling position and angle of the flexible instrument according to the presented planned path, guiding the operator to make the target adjustment action according to the operation guidance prompt to adjust the flexible instrument to enter the target sub-channel in the branch channel. According to the operation guidance prompt, the robot can automatically or manually control the flexible instrument by the operator to perform the target adjustment action, thereby controlling the end of the flexible instrument to enter the target sub-channel in the branch channel. This embodiment can reduce the preparation and implementation time of the flexible instrument control, and perform intuitive operations under image navigation, having the effects of fast preparation, convenient use, and easy operation.

[0169] In some specific embodiments, to provide convenience for the operator to operate the robot, the adjustment method of the operation console is consistent with the operation guidance prompt graphics displayed on the virtual image. The controller of the flexible instrument can be a trackball, a touch screen, a touchpad, etc. with a 360-degree azimuth adjustment function, which is convenient for the operator to keep consistent manipulation actions according to the graphic prompt information on the virtual image navigation screen.

[0170] Further, before controlling the registration of the virtual navigation module based on the real image, the following steps are further included:

[0171] Step S00, obtaining a digital three-dimensional model of the target cavity system;

[0172] Step S01, determining a planned path based on the digital three-dimensional model and the target position.

[0173] In this embodiment, before implementing the control to enable the flexible instrument to enter the target position through the target cavity system to perform corresponding operations, first, according to the digital three-dimensional model of the target cavity system and the target position, a planned path for entering the target position through the target cavity system is planned, and multiple planned paths can be planned; then the above planned path data is imported into the robot to assist the operator in intuitive operations through the visualization interface. The virtual navigation module displays the digital three-dimensional model of the target cavity system and the target area in the form of a three-dimensional visualization image, and displays the planned path and the virtual cavity image formed by the planned path on the three-dimensional image of the target cavity system, and also displays the virtual navigation image of the traveling position and angle prompt information of the flexible instrument.

[0174] For example, the digital three-dimensional model of the target cavity system is an image of a three-dimensional pulmonary bronchial tree.

[0175] In some specific embodiments, a computed tomography (CT) or magnetic resonance imaging (MRI) is performed on the surgical site (e.g., lungs) of the patient to obtain a CT image or MRI image of the surgical site of the patient, and then the CT image or MRI image data is imported into the preoperative planning system, and the image processing technology is used to segment the surgical site image in the imported CT image or MRI image data to obtain the surgical site image data, and then further processed to generate a digital three-dimensional lung bronchial tree. The preoperative planning system extracts, three-dimensionally reconstructs, and visualizes the target area data in the image data. After the surgical robot imports the planned path data, the assistant doctor performs intuitive surgical operations through a visual interface, and the digital images of three-dimensional tissues, organs, and target lesions are displayed on the visual interface. The doctor first selects the surgical path, and the surgical path can be displayed on the three-dimensional tissue and organ image, and the virtual natural cavity image composed of the surgical path is displayed. The operator is provided with a three-dimensional digital cavity model, a surgical plan path, and a virtual endoscopic navigation screen through the visual system. Then, through the human-computer interaction control controller, the medical device is operated to enter the natural cavity of the human body and move in the natural cavity of the human body according to the direction prompted by the displayed surgical path. During the process, the endoscope at the end of the medical device acquires images of the natural cavities of the human body in real time.

[0176] Specifically, based on the digital three-dimensional model and the target position, the planned path is determined, which specifically includes the following steps:

[0177] Step S011, in the digital three-dimensional model, from the entrance of the target cavity system to the target position, marking the branch cavities of the target cavity system at all levels according to preset rules;

[0178] Step S012, determining at least one planned path based on the mark;

[0179] Step S013, determining the motion threshold of the target position according to the target shape and size of the target position.

[0180] In this embodiment, from the entrance of the target cavity system to the distal end, the branch cavities of the target cavity system are marked according to the preset rules from near to far and from left to right, so that each branch of the target cavity system is automatically marked with a unique code; by observing and evaluating the digital three-dimensional images of the target cavity system and the target position, 1-3 planned paths are planned according to the above markings; the above planned path data is imported into the virtual navigation module, and the operator can select one of them as the planned path for implementation as needed. At the same time, according to the target shape and size of the target position, a safety inspection barrier can be set to determine the action threshold of the target position. The safety inspection barrier is used to limit the range of motion of the end of the flexible instrument at the target position to prevent the end of the flexible instrument from moving too much and causing damage to the inner wall of the target cavity system, thereby improving safety.

[0181] In some specific embodiments, such as Figure 9 shown, the target channel system is a three-dimensional digital organ, such as the bronchial tree. A unique code is automatically marked at each branch of the bronchial tree. The marking rule is that the first letter of the right bronchial airway is R, R1 represents the first-level branch, R2-1 represents the second-level branch numbered 1, R2-2 represents the second-level branch numbered 2, and so on. The three-dimensional digital organ starts from the natural channel entrance to the distal end, and the marking of each branch path is successively R1, L1, R2-1, R2-2, R2-3, L2-1, L2-2... Rn-1, Rn-2... Ln-1, Ln-2.

[0182] Specifically, obtaining the traveling position and state of the flexible instrument in the target channel system specifically includes the following steps:

[0183] Step S201, obtaining the length information of the flexible instrument entering the target channel system and the end state information of the flexible instrument;

[0184] Step S202, determining the traveling position and state of the flexible instrument in the target channel system based on the above length information, end state information, and planned path.

[0185] In this embodiment, a position detection sensor is provided to measure the length of the flexible instrument entering the target channel system; during the process of the flexible instrument entering the target channel system, by obtaining the length of the flexible instrument entering the target channel system, the traveling position of the flexible instrument in the target channel system can be accurately determined according to the planned path, which is beneficial for the virtual navigation module to achieve accurate navigation and improve the control accuracy.

[0186] In some embodiments, a position detection sensor can be provided on the flexible instrument driving device to detect the displacement of the flexible instrument driving device relative to the fixed end of the robotic arm assembly, so as to obtain the position of the flexible instrument movement and realize the measurement of the length of the flexible instrument entering the target channel system.

[0187] Furthermore, a limit sensor is also provided to protect the movement position of the flexible instrument driving device. When the flexible instrument driving device moves towards the target channel system and triggers the proximal limit sensor, the flexible instrument driving device immediately stops moving. Similarly, when the flexible instrument driving device moves away from the target channel system and triggers the distal limit sensor, the flexible instrument driving device immediately stops moving, thereby ensuring the safety of remote operation.

[0188] In some embodiments, such as Figure 3As shown, the robot further includes an introduction device 900. One end of the introduction device 900 is connected to the fixed end of the robotic arm assembly 301, and the other end enters the inlet 800 of the target channel system. The end of the flexible instrument 100 enters the introduction device 900 and then enters the target channel system. A detection sensor is provided at the inlet of the introduction device 900. The detection sensor is used to detect the length of the flexible instrument 100 entering the introduction device 900, and further measure the length of the flexible instrument 100 entering the target channel system. The detection sensor can be inductive or contact type. The inductive type can be a photoelectric induction device provided at the inlet of the introduction device 900. The photoelectric induction device emits invisible light, and the light irradiates the flexible instrument 100. The flexible instrument 100 is densely provided with scale lines at a certain interval. When the flexible instrument 100 passes through the introduction device 900, the photoelectric induction device measures the movement length of the flexible instrument 100 through the change of light. The contact type can be a contact type pinch feed guide wheel provided at the inlet of the introduction device 900, and a rotary encoder is provided at the shaft end of the pinch feed guide wheel. The length of the flexible instrument 100 entering the introduction device 900 is obtained by measuring the number of rotations of the pinch feed guide wheel.

[0189] Further, a sensitive measurement element is further configured at the end of the flexible instrument; the flexible instrument control method further includes the following steps:

[0190] Step S50, obtaining the contact force between the end of the flexible instrument and the inner wall of the target channel system in real time;

[0191] Step S51, controlling to send a prompt message when the contact force is greater than the first threshold;

[0192] Step S52, controlling the flexible instrument to move in a restricted mode when the contact force is greater than the second threshold, where the second threshold is greater than the first threshold.

[0193] In this embodiment, to ensure the safety of the travel adjustment of the flexible instrument in the target channel system, a sensitive measurement element is installed at the end of the flexible instrument. During the attitude adjustment of the end of the flexible instrument, the contact force between the end of the flexible instrument and the inner wall of the target channel system can be measured in real time, so as to effectively avoid damage to the inner wall of the target channel system caused by excessive attitude adjustment actions during the attitude adjustment process of the end of the flexible instrument, and ensure the safety of the flexible instrument control.

[0194] Specifically, when the contact force exceeds a preset first threshold, a prompt message is controlled to be sent to alert the operator to pay attention to the adjustment behavior of the flexible instrument. When the contact force further exceeds a preset second threshold, and the second threshold is greater than the first threshold, it indicates that the adjustment action of the flexible instrument is further increased, which may cause damage to the inner wall of the target channel system. At this time, the flexible instrument is controlled to move in a restricted mode, and the movement of the flexible instrument controlled by the operator's operation behavior will be restricted, thus effectively ensuring safety.

[0195] In some embodiments, as Figure 10 shown, the visual operation guidance prompt of the virtual navigation module includes a scale disk and a hollow arrow graphic. Among them, the solid arrow outside the scale disk represents the rotation angle of the end of the flexible instrument, and the length of the hollow arrow graphic represents the bending angle of the end of the flexible instrument. An increase in length indicates an increase in the bending angle. When the bending angle increases and exceeds the set first threshold, the color can be changed to alert the operator; the alert color can be a prominent color, such as yellow; when the contact force between the end of the flexible instrument with an excessive bending angle and the inner wall of the target channel system exceeds the set second threshold, the color of the arrow further alerts the operator through a flashing form, thereby achieving the effect of improving operation safety.

[0196] In some embodiments, during the insertion of the working instrument, especially when passing through the end bending sections of the first flexible body and the second flexible body, the postures of the first flexible body and the second flexible body will be interfered. The minute changes in the postures of the first flexible body and the second flexible body can be detected by the sensitive measurement elements installed thereon. The change amount is converted by the detection system and the data processing system through the posture maintenance algorithm model into the driving forces of the first flexible body and the second flexible body. By adjusting the driving forces of the flexible instrument, the postures of the end bending ends of the first flexible body and the second flexible body are maintained, thereby ensuring the accuracy of the target operation on the target position.

[0197] The control system provided by the present invention will be described below. The control system described below can be correspondingly referred to the control method of the flexible instrument described above.

[0198] As Figure 11As shown in the figure, the control system provided by the present invention includes a first control module 1110, a first acquisition module 1120, a second acquisition module 1130, and a second control module 1140. The first control module 1110 is configured to control the registration of the virtual navigation module based on the real image. The first acquisition module 1120 is configured to acquire the traveling position and state of the flexible instrument in the target cavity system. The second acquisition module 1130 is configured to acquire virtual navigation data based on the above traveling position and state. The second control module 1140 is configured to control the end of the flexible instrument to travel to the target position in the target cavity system based on the virtual navigation data. Wherein, the flexible instrument is configured with a virtual navigation module, and the virtual navigation module is used to generate virtual navigation data. The end of the flexible instrument is further configured with an image acquisition module, and the image acquisition module is used to acquire real images.

[0199] Figure 12 An example of the physical structure diagram of an electronic device is shown in Figure 12 As shown in the figure, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230, and a communication bus 1240. Among them, the processor 1210, the communication interface 1220, and the memory 1230 complete mutual communication through the communication bus 1240. The processor 1210 can call the logical instructions in the memory 1230 to execute the flexible instrument control method, and the method includes: controlling the registration of the virtual navigation module based on the real image; acquiring the traveling position and state of the flexible instrument in the target cavity system; acquiring virtual navigation data based on the above traveling position and state; controlling the end of the flexible instrument to travel to the target position in the target cavity system based on the virtual navigation data. Wherein, the flexible instrument is configured with a virtual navigation module, and the virtual navigation module is used to generate virtual navigation data. The end of the flexible instrument is further configured with an image acquisition module, and the image acquisition module is used to acquire real images.

[0200] It should be noted that the electronic device in this embodiment can be a server, a PC, or other devices when specifically implemented, as long as its structure includes a processor 1210, a communication interface 1220, a memory 1230, and a communication bus 1240 as shown in Figure 12 As shown in the figure, the processor 1210, the communication interface 1220, and the memory 1230 complete mutual communication through the communication bus 1240, and the processor 1210 can call the logical instructions in the memory 1230 to execute the above method. This embodiment does not limit the specific implementation form of the electronic device.

[0201] In addition, when the logical instructions in the above-mentioned memory 1230 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0202] Furthermore, the present invention also discloses a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the flexible instrument control method provided by the above-mentioned various methods. The method includes: based on a real image, controlling the registration of a virtual navigation module; obtaining the traveling position and state of a flexible instrument in a target cavity system; based on the above-mentioned traveling position and state, obtaining virtual navigation data; based on the virtual navigation data, controlling the end of the flexible instrument to travel to a target position in the target cavity system; wherein, the flexible instrument is configured with a virtual navigation module, and the virtual navigation module is used to generate virtual navigation data; the end of the flexible instrument is further configured with an image acquisition module, and the image acquisition module is used to acquire real images.

[0203] On the other hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the flexible instrument control method provided by the above-mentioned various methods. The method includes: based on a real image, controlling the registration of a virtual navigation module; obtaining the traveling position and state of a flexible instrument in a target cavity system; based on the above-mentioned traveling position and state, obtaining virtual navigation data; based on the virtual navigation data, controlling the end of the flexible instrument to travel to a target position in the target cavity system; wherein, the flexible instrument is configured with a virtual navigation module, and the virtual navigation module is used to generate virtual navigation data; the end of the flexible instrument is further configured with an image acquisition module, and the image acquisition module is used to acquire real images.

[0204] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0205] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0206] On the other hand, the present invention also provides a robot, including a virtual navigation module and the electronic device provided in the above embodiment. The virtual navigation module is configured to be able to generate virtual navigation data.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims

1. A control system for a flexible instrument, characterized in that, The flexible instrument is configured with a virtual navigation module for generating virtual navigation data, and an image acquisition module is further configured at the end of the flexible instrument for acquiring real images; The control system of the flexible instrument includes: A first control module for controlling the registration of the virtual navigation module based on the real images; A first acquisition module for acquiring the traveling position and state of the flexible instrument within the target cavity system; A second acquisition module for acquiring virtual navigation data based on the traveling position and state; A second control module for controlling the end of the flexible instrument to travel to a target position within the target cavity system based on the virtual navigation data; Wherein, the flexible instrument is configured with the virtual navigation module for generating the virtual navigation data, and an image acquisition module is further configured at the end of the flexible instrument for acquiring the real images; the controlling the registration of the virtual navigation module based on the real images specifically includes: determining that the end of the flexible instrument reaches a target feature position of the target cavity system; acquiring in real time the real image of the target feature position and the virtual image generated by the virtual navigation module; controlling the registration of the virtual navigation module based on the real image and the virtual image so that the registered virtual image is consistent with the real image; The controlling the registration of the virtual navigation module based on the real image and the virtual image so that the registered virtual image is consistent with the real image includes: controlling the calibration of the real image to obtain first feature information of the target feature in the calibrated real image; controlling the calibration of the virtual image to obtain second feature information of the target feature in the calibrated virtual image; determining registration parameters based on the first feature information and the second feature information; controlling the virtual navigation module to perform registration based on the registration parameters so that the second feature information after registration is consistent with the first feature information; the target feature includes a branched cavity; the controlling the calibration of the real image to obtain first feature information of the target feature in the calibrated real image includes: obtaining a first fitting graph of the profiles of the sub-cavities of the branched cavity in the real image; determining a first calibration parameter based on the first fitting graph; controlling and adjusting the attitude of the end of the flexible instrument based on the first calibration parameter to calibrate the viewing angle of the real image to a target viewing angle; obtaining the first feature information, where the first feature information includes the cross-sectional area of the first fitting graph of the profiles of the sub-cavities of the branched cavity in the real image at the target viewing angle; Determining the first calibration parameter based on the first fitting graph includes: obtaining the geometric center information of the first fitting graph of each sub-channel profile and the position information of the connection lines of the geometric centers; determining the first calibration parameter based on the geometric center information and the position information of the connection lines of the geometric centers; wherein, the first calibration parameter includes an angle offset and a position offset, the angle offset is the included angle between the connection line of the geometric centers and the connection line of the geometric centers of the target graph in the real image under the target viewing angle, and the position offset is the distance between the symmetric center point surrounded by the geometric centers and the center point of the real image under the target viewing angle; Controlling the calibration of the virtual image and obtaining the second feature information of the target feature in the calibrated virtual image includes: obtaining the second fitting graph of each sub-channel profile of the branch channel in the virtual image; determining the second calibration parameter based on the second fitting graph; controlling and adjusting the virtual image viewing angle of the virtual navigation module based on the second calibration parameter to calibrate the viewing angle of the virtual image to the target viewing angle; obtaining the second feature information, where the second feature information includes the cross-sectional area of the second fitting graph of each sub-channel profile of the branch channel in the virtual image under the target viewing angle; Determining the registration parameter based on the first feature information and the second feature information includes: obtaining the depth adjustment parameter of the virtual image based on the cross-sectional area of the first fitting graph and the cross-sectional area of the second fitting graph according to a preset relational expression; the registration parameter includes the depth adjustment parameter.

2. The control system of the flexible instrument according to claim 1, characterized in that Obtaining virtual navigation data based on the traveling position and state includes: Determining the current traveling position and state of the flexible instrument; Obtaining the virtual navigation data based on the planned path and the current traveling position and state; wherein, the virtual navigation data includes a virtual image and an operation guidance prompt, the virtual image displays the virtual channel image corresponding to the current traveling position and state, and the operation guidance prompt is used to prompt the target adjustment action of the flexible instrument in the current traveling position and state.

3. The control system of the flexible instrument according to claim 2, characterized in that, Controlling the end of the flexible instrument to travel to the target position in the target channel system based on the virtual navigation data includes: When the end of the flexible instrument reaches the branch channel position, according to the operation guidance prompt, controlling the flexible instrument to perform the target adjustment action to control the end of the flexible instrument to enter the target sub-channel in the branch channel.

4. The control system of the flexible instrument according to claim 2, characterized in that, Before controlling the registration of the virtual navigation module based on the real image, it further includes: Obtaining the digital three-dimensional model of the target channel system; Determining the planned path based on the digital three-dimensional model and the target position.

5. The control system of the flexible instrument according to claim 4, wherein Determining the planned path based on the digital three-dimensional model and the target position includes: In the digital three-dimensional model, from the entrance of the target channel system to the target position, marking each level of branch channel of the target channel system according to a preset rule; Determining at least one planned path based on the marking; Determine the action threshold of the target position according to the target shape and size of the target position.

6. The control system of the flexible instrument according to claim 1, characterized in that, The obtaining of the traveling position and state of the flexible instrument in the target channel system includes: Obtain the length information of the flexible instrument entering the target channel system and the end state information of the flexible instrument; Based on the length information, the end state information, and the planned path, determine the traveling position and state of the flexible instrument in the target channel system.

7. The control system of the flexible instrument according to claim 1, wherein A sensitive measurement element is further disposed at the end of the flexible instrument, and the control method of the flexible instrument includes: Obtain in real time the contact force between the end of the flexible instrument and the inner wall of the target channel system; When the contact force is greater than the first threshold, control to send a prompt message; When the contact force is greater than the second threshold, control the flexible instrument to move in a restricted mode, where the second threshold is greater than the first threshold.

8. A robot, characterized in that, It includes a virtual navigation module and a control system of the flexible instrument according to any one of claims 1-7, and the virtual navigation module is configured to be capable of generating virtual navigation data.

Images