Collision warning for medical equipment

Abstract

The medical device (100) comprises a medical device (108) having a curved section and a distal end, at least one imaging device (110) located at the distal end, at least one sensor (112), and at least one processor (201). The processor performs the following steps: receiving input data from a user input, the imaging device (110), the sensor (112), or a combination thereof; determining a bending plane and bending angle of the distal end of the medical device (108); predicting a movement and position of the curved section based on the input data; displaying an image view based on the input data; and displaying the predicted movement and position of the curved section in the image view.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority from U.S. Provisional Patent Application No. 63 / 309,410, filed February 11, 2022, which is incorporated by reference in its entirety.

[0002] The present disclosure relates generally to medical devices, and more particularly to medical devices, apparatus, methods, and storage media for providing collision warnings and better guiding users operating medical devices. [Background technology]

[0003] Minimally invasive imaging and treatment devices are used in image-guided therapy to view inside the body. These types of medical procedures can be performed with catheters, endoscopes, ablation devices, and other devices. Flexible medical tools are inserted into the patient's body and instruments are passed through the tools to examine or treat areas inside the body. A bronchoscope is an endoscopic instrument used to look inside a patient's airways. Catheters and other medical tools can be inserted through the tool channels of the bronchoscope to provide a pathway to target areas within the patient for diagnosis, treatment, etc.

[0004] Medical devices in the form of robotic or snake-like catheter assemblies typically have a rotary drive assembly that imparts rotational motion to a guidewire of the catheter. The drive assembly is releasably connected to the catheter, and a detachment mechanism may be used to disengage the drive assembly from the catheter in response to a detachment force.

[0005] There are multiple methods of visual presentation to guide device navigation during medical procedures, such as endoscopes, catheters, and other medical devices. An over-the-shoulder view configuration in virtual bronchoscopy can be used by updating the lung airway model derived from tomosynthesis fluoroscopy. A wide-angle panoramic view from virtual bronchoscopy can be used to assist in situations where the medical device may collide with the airway wall. Other endoscopic tools for minimally invasive procedures (such as catheters) may include multiple cameras to provide a full-angle 360° integrated view to fully present the surroundings.

[0006] US Patent Application Publication No. 20220202502 describes visualization and directional guidance during endoscopic procedures, providing peripheral navigation used to guide a navigation catheter within the tracheobronchial tree when the scope becomes pinched and cannot be advanced. Default views include dynamic 3D map, bronchoscopic view, local view, and tip view. A virtual panoramic view provides a wide-angle view to the user during navigation, which is useful in situations where one is facing a wall.

[0007] Currently, there are several issues with visual presentations that guide medical device navigation during medical procedures. For example, when a user manipulates a robotic catheter to approach a bend around a bifurcation in a lung airway, the catheter body may buckle. Such buckling occurs specifically when the distal curved section hits the airway wall. The distal curved section is manipulated to match the airway shape while the user is looking through the endoscopic camera. However, since the endoscopic camera is distally separated from the distal curved section due to the rigid tip between them, the user must compensate for the bend angle to avoid collision between the catheter body and the airway wall.

[0008] Although peripheral navigation views are known in the prior art that present visual aids of the current surroundings from angles beyond the endoscopic camera view, it is still difficult for users to predict potential collisions between the catheter body and the airway wall when maintaining bending commands and / or moving along the centerline of the endoscopic camera view with the rigid tip only. See, for example, U.S. Patent Nos. 5,036,852, 6,827,695, 10,420,626, 10,893,911, 11,202,652, 11,558,653, and U.S. Patent Application Publication Nos. 20130238048, 20150133959, and 20170188946.

[0009] It would be beneficial to implement a solution that overcomes these concerns and avoids collisions of medical equipment during medical procedures. Summary of the Invention

[0010] An advantage of the present disclosure is to provide collision warning and better guide users operating medical equipment.

[0011] According to some embodiments, a medical device includes a medical device having at least one bendable section, a distal end, at least one imaging device located at the distal end, at least one sensor, and at least one processor that performs the steps of receiving input data from a user input, an imaging device, a sensor, or a combination thereof, determining a bending plane and a bending angle of the distal end of the medical device, predicting a movement and a position of the at least one bendable section based on the input data, displaying an image view based on the input data, and displaying the predicted movement and position of the at least one bendable section in the image view.

[0012] According to some embodiments, a method for a medical apparatus comprising a medical device having at least one bendable section and a distal end, at least one imaging device, and at least one sensor includes: receiving input data from the imaging device and sensor, user input, the imaging device, the sensor, or a combination thereof; determining a bending plane and a bending angle of the distal end of the medical device; predicting a movement and position of the at least one bendable section based on the input data; displaying an image view based on the input data; and displaying the predicted movement and position of the at least one bendable section in the image view.

[0013] Further features of the present disclosure will become apparent from the following description of exemplary embodiments which refer to the accompanying drawings. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 illustrates a medical device according to some embodiments. [Diagram 2] FIG. 2 is a block diagram of the medical device of FIG. [Diagram 3] FIG. 3 is a block diagram of a controller, according to some embodiments. [Figure 4] 4A and 4B illustrate a catheter according to some embodiments. [Diagram 5] FIG. 5 illustrates a method according to some embodiments. [Figure 6] FIG. 6 illustrates a steerable catheter with an endoscopic camera at its distal end, according to some embodiments. [Figure 7] FIG. 7 (including FIGS. 7A, 7B, and 7C) illustrates a steerable catheter with an endoscopic camera at its distal end, according to some embodiments. [Figure 8] FIG. 8 is a top view of a distal section of a steerable catheter, according to some embodiments. [Figure 9]FIG. 9 is a top view of a distal section of a steerable catheter, according to some embodiments. [Figure 10] FIG. 10 illustrates a follow-the-leader trajectory according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Various example embodiments, features, and aspects of the present disclosure relating to medical devices, apparatus, methods, and storage media for providing collision warning and better guiding a user operating a medical device are described below with reference to the drawings, which may include different characteristics, advantages, disadvantages, performance parameters, etc.

[0016] To provide better guidance when a user manipulates a medical device such as a catheter around a bifurcation in an airway, the future position of the distal curved tip (or one or more selected points on the catheter body) can be predicted based on the current bending angle and plane by assuming a certain insertion / travel distance and rigid tip length. The predicted position is projected onto the current endoscopic camera view or virtual bronchoscope view. In conjunction with visualizing the lumen opening against the airway wall in the current view, the user is informed whether the current bending command via joystick input or other type of input is sufficient for the catheter body to pass through the opening space or will cause a collision against the airway wall. If the latter occurs, the user will adjust the bending command (particularly correct the bending angle) until the predicted position of the tip of the curved section is within the lumen opening space of the current endoscopic camera view or virtual bronchoscope view.

[0017] In the following embodiments, medical device, equipment, device or instrument configurations are described that functionally implement intravascular imaging modalities such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), IVUS (Intravascular Ultrasound), Positron Emission Tomography (PET), X-ray imaging, Angiography, Optical Coherence Tomography (OCT), Multi-Modality OCT (MMOCT), Near Infrared Autofluorescence (NIRAF), Spectral Encoded Endoscopy (SEE), combinations or hybrids thereof, for avoiding collisions of medical devices during medical procedures. The described medical device configurations may have various characteristics, advantages, disadvantages, performance parameters, etc. The present disclosure is not limited to any particular configuration.

[0018] Figure 1 illustrates a medical device 100 configured as an exemplary robotic catheter assembly or snake configuration, according to some embodiments. Figure 2 illustrates a hardware configuration of the medical device 100. The medical device 100 is not limited to these configurations, and variations and other configurations of the medical device 100 are within the scope of the present disclosure.

[0019] The robotic catheter 100 includes one or more of a handheld controller 102, a medical tool 104, an actuator 106, a medical instrument 108, an imaging device 110, a sensor 112, a detector 114, a console 116, a display 118, and a miniature display 120, and may include other elements or components. Throughout this disclosure, the medical tool 104 is referred to as a "biopsy tool" and the medical instrument 108 is referred to as a "catheter," by way of example. One or more of various other types of tools, instruments, configurations, or arrangements, such as, for example, a snake-like robotic catheter, an endoscope, a sheath, a guidewire, a needle, a probe, a forceps, and the like, are also within the scope of this disclosure.

[0020] The robotic catheter 100 of some embodiments can achieve its functions using one or more processes, techniques, algorithms, etc. that can avoid collisions of medical equipment while providing better work efficiency to physicians during medical procedures.

[0021] The controller 102 has a housing with an elongated handle or handle section that can be grasped by hand, and one or more input devices, such as levers, buttons, or other input devices that allow a user (such as a physician) to send commands to the medical device 100 to move the catheter 108. The controller 102 executes software, computer instructions, algorithms, etc., such that a user can complete all operations using the handheld controller 102 by holding the handheld controller 102 in one hand.

[0022] The medical tool 104 may be a biopsy tool or other type of tool. The actuator 106 may include one or more motors to drive each section of the catheter 108. The controller 102, the medical device 108, the console 116 and other elements are interconnected to the actuator 106. The controller 102 includes at least one processor and is configured to control the medical device 108 via the actuator 106 and to control the actuator 106 according to manipulation by a user.

[0023] The medical device 108 may be a catheter or another type of medical device. The imaging device 110 may be a mechanical, digital or electronic device configured to record, store or transmit visual images, such as a camera, camcorder, or movie camera. The sensor 112 may be an electromagnetic tracking sensor (EM tracking sensor) and is attached to the tip of the catheter 108. The detector 114 detects the position of the EM tracking sensor 112 and outputs the detected position information to the controller 102 and / or the console 116. The controller 102 receives catheter tip position information directly from the tracking sensor 112 or the detector 114.

[0024] The console 114 executes software, computer instructions, algorithms, etc., to control the display of navigation screens on the display 118 and other types of images or information on the miniature display 120. The console 116 can generate a three-dimensional (3D) model of the patient's internal branch structures (e.g., lungs and other internal structures) based on medical images such as CT or MRI. Alternatively, the 3D model may be received by the console 114 from another device.

[0025] The console 116 obtains the catheter position information from the detector 114. Alternatively, the console 116 can obtain the catheter position information directly from the tracking sensor 112.

[0026] The console 116 executes software based on the 3D model and the catheter position information to generate a navigation screen and output it to the display 104. The navigation screen can show the current position of the catheter 108 on the 3D model. The navigation screen allows the user to recognize the current position of the catheter 110 within the branched structure.

[0027] The console 116 can perform corrections of the acquired 3D model based on the position information of the catheter 108 so as to minimize discrepancies between the position of the catheter 108 and the path mapped onto the 3D model.

[0028] Display 118 and / or miniature display 120 may be display devices configured as, for example, a monitor, an LCD (liquid crystal display), an LED (light emitting diode) display, an OLED (organic LED) display, a plasma display, an organic electroluminescent panel, etc. Based on control of the device, a navigation screen may be displayed on display 116 showing one or more images being captured, captured images, captured video recorded in storage, etc. Miniature display 120 is smaller than display 118, and they may each display similar or other types of images and / or information.

[0029] The controller 102 and / or console 116 may include one or more or a combination of levers, keys, buttons, switches, a mouse, a keyboard, etc. for controlling elements of the device 100, and each have components 200 (shown in FIG. 3 ) including one or more or a combination of a processor 201, a memory 202, sensors 203, an input / output (I / O) interface 204, a communications interface 205, a display 206, a power source 207, and may include other elements or components. The device 100 may be interconnected with medical instruments and various other equipment, and may be controlled independently, externally, or remotely by the controller 102 and / or console 116.

[0030] The processor 201 may be configured as one or more processors, control circuits, circuitry, or a combination thereof, and performs the overall control of the medical device 100. The memory 202 may store programs, software, computer instructions, information, other data, or a combination thereof. The memory 202 is used as a working memory. The processor 201 executes the programs, instructions, code, or software stored or accumulated in the memory 202 to perform various data processing, computational, algorithmic tasks, and other functions of the medical device 100.

[0031] The sensors 203 can monitor, measure or detect various types of data of the medical device 100 and transmit or send the sensor readings or data to a host over a network. The I / O interface 204 can interconnect various components with the medical device 100 to exchange data or information with the medical device 100. The I / O interface 204 can input catheter position information to the console 116 and output information to the display 118 for displaying navigation screens. The communication interface 205 can interconnect various components with the medical device 100 to facilitate communication with the medical device 100.

[0032] The display 206 corresponds to the display 118 and / or the miniature display 120 and can present a display for a user to view images, data, and other information, and can be configured as an LCD or other type of display. The controller 102 and / or the console 116 can control the display of the display 206 and the input of various settings or default information set by the input / output interface 204 and the communication interface 205, and can provide input to the medical device 100.

[0033] The power source 207 provides power to the medical device 100 and maintains a regulated power supply to the medical device 100 and may operate in a power on mode, a power off mode, and other modes. The power source 107 may include a battery housed within the medical device 100, or may include an external power source, such as line power from a power outlet or an AC power source, which may be interconnected with the medical device 100 via an AC / DC adapter and / or DC / DC converter to adapt the power voltage from the power source to one or more voltages used by the components of the medical device 100.

[0034] The components are connected to each other by a bus 208 so that the components can communicate with each other. The bus 208 connects the medical device 100 to input devices, output devices, communication devices, and other devices. The input devices are configured to allow a user to transmit information and select commands to the medical device 100 and may include one or more or a combination of a mouse with keys or buttons with alphanumeric characters, icons, pictograms, or other types of symbols, a keyboard, a touch screen, and the like. The output devices are configured to display data or images generated by the medical device 100 and may include a printer, a display, or other output configuration.

[0035] The bus 208 transmits and receives data between these connected hardware devices, or transmits commands from the processor 201 to other hardware devices. The components may be implemented by one or more physical devices that can be coupled to the processor 201 through a communication channel. For example, the controller 102 and / or the console 116 may be implemented using circuitry in the form of an ASIC (Application Specific Integrated Circuit) or the like. Alternatively, the controller 102 and / or the console 116 may be implemented as a combination of hardware and software, where the software is loaded into the processor from a memory or via a network connection. The functionality of the controller 102 and / or the console 116 may be stored in a storage medium, which may include a RAM (Random Access Memory), a ROM (Read Only Memory), a magnetic or optical disk, a diskette, a cloud storage, or the like.

[0036] The sensor 203 may include one or more or a combination of a processor, detection circuitry, memory, hardware, software, firmware, and may include other circuitry, elements, or components. The sensor 203 may be a plurality of sensors and obtains sensor information output from one or more sensors that detect the movement, current position, and operation of the components interconnected with the medical device 100. The sensor 203 may include a multi-axis acceleration sensor or accelerometer sensor and a multi-axis gyroscope sensor, may be a combination of an acceleration sensor and a gyroscope sensor, may include other sensors, and may be configured using a piezoelectric transducer, a mechanical switch, a single axis accelerometer, a multi-axis accelerometer, or other types of configurations. The sensor 203 can monitor, detect, measure, record or store physical, operational, quantified or other characteristic parameters of the medical device 100, including one or more or combinations of crash, impact, drop, descent, motion, acceleration, deceleration, velocity, rotation, temperature, pressure position, orientation, movement or other types of data of the medical device 100 along multiple axes, multiple dimensions, x-axis, y-axis, z-axis or any combination thereof, and can generate sensor readings, information, data, digital signals, electrical signals or other types of information corresponding to the detected conditions. The medical device 100 can transmit or send the sensor reading data to a remote host or server in a wireless or wired manner. The sensor 203 can be interrogated and can generate sensor reading signals or information that can be processed in real time, stored, post-processed or any combination thereof. The information or data generated by the sensor 203 can be processed, demodulated, filtered or conditioned to remove noise or other types of signals.The sensors 203 may include one or more or combinations of an acceleration sensor, a deceleration sensor or accelerometer sensor, a gyroscope sensor, a power sensor, a battery sensor, a proximity sensor, a motion sensor, a position sensor, a rotation sensor, a magnetic sensor, an air pressure sensor, an illumination sensor, a pressure sensor, an angular position sensor, a temperature sensor, an altimeter sensor, an infrared sensor, an acoustic sensor, an air monitoring sensor, a piezoelectric sensor, a strain gauge sensor, a sonic sensor, a vibration sensor, a depth sensor, and may include other types of sensors.

[0037] For example, an acceleration sensor can detect or measure the movement of the mass of a component of the medical device 100 by position or can detect the speed of the movement of a component of the medical device 100. A gyroscope sensor can detect or measure the angular velocity or angle of movement and can measure the movement of the medical device 100 up to six degrees of freedom in three-dimensional space, such as three translational degrees of freedom along Cartesian x, y, z coordinates and rotation along one or more of the yaw, pitch, roll, and horizontal axes to change orientation between the axes. Yaw is when a component of the medical device 100 rotates left or right on a vertical axis. Rotation on the front-to-back axis is called roll. Rotation from side to side is called pitch. The sensor 203 can monitor shock or drop impacts with low power consumption, dynamic range, and bandwidth to accurately detect and capture impact events and convert sensor readings into digital signals for additional or later processing. The entire shock profile can be characterized by its peak amplitude and pulse width for further analysis. The processor 201 of the medical device 100 can also query the capacity of the power source 207 and can alert the user to replace the battery when the battery capacity value falls below a predetermined threshold amount.

[0038] Examples of the acceleration sensor include a gravity sensor and a fall detection sensor. The gyroscope sensor may include an angular velocity sensor, an image stabilization sensor, a geomagnetic sensor, and the like. The position sensor may be a GPS sensor that receives data output from a global positioning system (GPS). The longitude and latitude of the current position can be obtained from information output from an access point of a radio frequency identification device (RFID) and a WiFi device and a wireless base station, and thus, for example, the detection of these can be used as a position sensor. Such a sensor may be disposed inside or outside the medical device 100.

[0039] The medical device 108, according to some embodiments, may be configured as a catheter 300, as shown in Figures 4A and 4B.

[0040] The catheter 300 is a flexible medical device having bendable sections including a proximal bendable section 310A, a middle bendable section 310B, and a distal bendable section 310C. A hollow chamber 340 runs from proximal to distal through the catheter 300 and can be used as a working channel for medical procedures. The catheter 300 includes a plurality of drive wires 342 and support wires 344, which are respectively disposed in lumens 346 surrounding the central hollow chamber 340, as shown in the cross-sectional view of FIG. 4B. Each of the proximal bendable section 310A, the middle bendable section 310B, and the distal bendable section 310C of the catheter 300 can be bent by a plurality of drive wires 342 (drive linear members) as a driving backbone. The position of the catheter 300 can be maintained by a support wire 344 (support linear member) as a passive sliding backbone. One or more lumens may be left free to facilitate the use of additional optical fibers or wires added to the catheter 300. The atraumatic tip 348 of the catheter 300 is fitted with a tracking sensor 110. At the proximal end of the catheter 300, a drive wire is connected to an actuator 106, which may include one or more motors, that actuate each section of the catheter 300 by pushing or pulling the drive wire.

[0041] The controller 102 can control the catheter 108 based on an algorithm known as a follow-the-leader (FTL) algorithm or other algorithms. By applying the FTL algorithm, the intermediate section 310B and the proximal section 310A of the catheter 108 move in a first position in the same manner as the distal section moves in a second position at or near the first position. The controller 102 and / or the console 114 can include input elements that allow a user to adjust the position of the flexible portion of the catheter 108. The input elements can be configured as a mouse, keyboard, joystick, lever, or another shape that facilitates user interaction.

[0042] The medical device 100 provides collision warnings and better guides the user in operating the medical device 108, and is further configured to predict the motion of the medical device 108 based on user input, imaging devices 110, sensors 112, or a combination thereof, and previous predictions made by the medical device 100. The user can adjust the attitude of the tip sections and can predict tip misalignment and a combination of stage motion and section articulation.

[0043] 5 illustrates a method according to some embodiments for a medical apparatus 100 comprising a medical device 108 having a curved section 310 and a distal end 348, at least one imaging device 110, and at least one sensor 112. In step S100, the method receives input data from a user input, the imaging device 110, the sensor 112, or a combination thereof. In step S110, the method determines a bending plane and a bending angle of the distal end of the medical device. In step S120, the method predicts a movement and a position of the curved section based on the input data. In step S130, the method displays an image view based on the input data and displays the predicted movement and position of the curved section in the image view.

[0044] The method may also detect a collision of the medical device based on the input data, provide a collision warning of the medical device based on the input data, or limit the bending angle of the distal end of the medical device to avoid a potential future collision with the airway wall.

[0045] The method can represent a constant arc by assuming an insertion distance so that the tip of the distal curved section can be projected onto the bending plane. The predicted movement and position of the bendable section is based on the insertion distance and length of the rigid tip. The constant arc can be represented by intersecting lines on the image views.

[0046] The method provides collision warning and better guides the user to operate the medical device, and is further configured to predict the motion of the medical device based on user input, imaging device 110, sensor 112, or a combination thereof, and previous predictions made by the medical device 100. The user can adjust the attitude of the tip sections and predict the tip misalignment and a combination of stage motion and section articulation. In other words, the medical device 100 can simulate any kind of user adjustment from the current point and visualize how it will affect the tip of the medical device.

[0047] Once the user finds a satisfactory input or series of inputs, they do not have to repeat them manually, but simply transmit them automatically to the control device, either in a stepwise or continuous manner.

[0048] When a bendable medical device or catheter has bendable sections of different lengths, the position of the end effector of the trailing section may not match the position of the tip end effector at the same location. This can be problematic, and the medical device 100 can take this into account when calculating the expected motion.

[0049] The medical devices and catheters of Figures 1-4B can avoid collisions of the medical devices during medical procedures described below with reference to Figures 6 and 7A-7C, which illustrate a steerable catheter with an endoscopic camera at its distal end, according to some embodiments.

[0050] FIG. 6 shows the endoscope illumination and image capture range 610, endoscope lens 611, rigid tip (current) 612, distal bendable section (current) 613, rigid tip (predicted) 614, distal bendable section (predicted) 615, airway 616, endoscope camera view 617, airway wall 618, open space 619, predicted tip of distal bendable section 620, and bending plane (intersecting as a line) 621.

[0051] To provide better guidance when the user manipulates the catheter 300 around bifurcations in the airway, the future position of the tip of the distal bendable section (or one or more selected points on the catheter body) can be predicted based on the current bend angle and plane by assuming a certain insertion / travel distance and rigid tip length. The predicted position can be projected onto the current endoscopic camera view or virtual bronchoscope view. In conjunction with visualizing the lumen opening against the airway wall in the current view, the user can be informed whether the current bend command via input (such as a joystick input) is sufficient for the catheter body to pass through the open space or will cause a collision against the airway wall. If the latter occurs, the user can adjust the bend command (particularly correct the bend angle) until the predicted position of the tip of the bendable section is within the lumen open space of the current endoscopic camera view or virtual bronchoscope view.

[0052] The steerable catheter 300 of FIG. 4A and FIG. 4B includes proximal, middle and distal bendable sections 310A, 310B, 310C and a rigid tip with an endoscopic camera at its distal end 348 (FIG. 6 and FIG. 7). An operator can use a joystick to issue commands to the robot subsystem to steer the distal bendable section 310C in three dimensions to insert / remove the steerable catheter 300. When the bendable sections bend, the rigid tip at the distal end 348 of the distal bendable section 310C does not bend and maintains its straight shape (FIG. 6). The robotic catheter 300 can display real-time endoscopic views from the endoscopic camera and robot status information, i.e., current bending angle and bending plane, current insertion amount, etc. In particular, an operator can check the endoscopic views and robot status information to determine the bending angle, orientation, insertion and removal of the catheter 300 to navigate through the narrow space of the anatomical structure (i.e., the lung airway).

[0053] The robotic function of the follow-the-leader motion allows the proximal and middle bendable sections 310A, 310B to follow the distal bendable section 310C in a continuous arc trajectory (FIG. 7), thus realizing a snake-like insertion along a curved path. In the follow-the-leader motion, the operator can bend the distal bendable section 310C having the rigid tip 348 by using a joystick (FIG. 7A). The operator can then see a real-time endoscopic view and control the direction and amount of bending. For example, in FIG. 7, if the operator wants to follow the curved path shown by the dotted line in FIG. 6, the operator wants to match the bending angle of the distal bendable section 310C (FIG. 7A) and then insert the steerable catheter 300. When the middle bendable section 310B reaches the previous position of the distal bendable section 310C, the control device automatically bends the middle section 310B by the same amount as the distal bendable section 310C at its previous position (FIG. 7B). During this insertion, the operator can also continue to adjust the bend of the distal bendable section 310C to match the dotted line. By continuing this insertion, the distal bendable section 310C can also be automatically bent by the same angle as the middle bendable section 310B at its previous position (FIG. 7B). These steps allow the steerable catheter 300 to follow a curved path while minimizing the risk of hitting the wall of the anatomical structure. In these steps, the operator can only control the distal bendable section 310C, and the control device automatically covers the control of the other bendable sections 310A, 310B.

[0054] Figure 8 is a top view of the rigid tip distal section 310C and the anatomical structure, showing the relationship between the centerline of the airway and the trajectory that the steerable catheter will follow with a follow-the-leader motion. Specifically, Figure 8 also shows, at the bottom of Figure 8, the center of the endoscopic view at the distal end of the rigid tip 310C and a real-time display of the endoscopic view on the display.

[0055] In FIG. 8, the operator can adjust the center of the endoscopic view on the display to the center of the anatomical feature, thereby adjusting the center of the endoscopic view to the center of the airway. In this situation, the endoscopic view is located at the distal end of the stiff tip 310C instead of the distal end of the distal curved section, so the bend angle of the distal section 310C does not match the centerline of the anatomical structure. As the operator continues the insertion, the follow-the-leader motion causes the steerable catheter 300 to follow the dotted line and collide with the airway wall, even if the operator tries to center the endoscopic view to the center of the anatomical structure. This makes the follow-the-leader control counter-intuitive to the operator.

[0056] According to some embodiments, the controller 200 can calculate a dotted line using the robot kinematics and distal bend angle / orientation as a prediction of the follow-the-leader trajectory, and determine the location of the distal end of the distal section (circle in FIG. 8) as a prediction of the location of the tip of the curved section along the dotted line. When determining the circle, the operator can select a vertical distance from the view center, and then the controller 200 will actually determine one white point and display that white point on the real-time endoscopic view. When the white point is displayed on the real-time endoscopic view, the controller 200 can apply a distance conversion ratio between the distance on the endoscopic view and the calculated Euclidean distance. The operator can use this white point as a reference to determine the bend angle to match the centerline of the airway.

[0057] Figure 9 shows a similar top view of the distal section with rigid tip 310C and anatomical structure, but with the follow-the-leader trajectory prediction aligned with the airway centerline. In this situation, the real-time endoscopic view (bottom view of Figure 9) has the view center offset from the airway, but the curved section tip placement prediction (white circle) is over the center of the airway. Thus, using the white dot in the real-time endoscopic view, the operator can adjust the angle of the curved section to align with the airway centerline.

[0058] When calculating the predicted follow-the-leader trajectory and the predicted tip placement in the curved section, the controller uses three coordinate systems (Figure 10). The endoscopic view is in endoscopic coordinates (O t ) on the real-time image. To display the view center (black circle) on the real-time image, the control device calculates the coordinates O t We can expect a black circle at the origin of the curved section. e ) is at the tip of the distal curved section. Since the rigid tip always maintains a straight shape, the coordinate O t and O e The transformation between and can be calculated.

[0059] The orientation of the bending plane and the bending angle can be measured in the following manner. This information is stored in the distal section coordinates (O b1 To calculate the follow-the-leader trajectory prediction, two assumptions are made:

[0060] The shape of the curved section has a constant curvature.

[0061] The prediction of the follow-the-leader trajectory is an extrapolation of the constant curve shape of the curved section.

[0062] By the first assumption, the bending plane and bending angle measured at the known section length are used to determine the bending angle. e and O b1 Also, with this assumption, we can calculate the transformation between the coordinates of O e It is ascertained that the Z axis of O is a tangent to the extrapolation of the constant curve shape of the curved section. t A prediction of the follow-the-leader trajectory based on the coordinates can be calculated and a prediction of the tip placement of the curved section along the determined trajectory can be determined.

[0063] As previously described, the medical device 100 provides a solution to a problem commonly encountered during medical procedures with visual presentations that guide medical device navigation. The medical device 100 provides collision warnings and better guides the user operating the medical device.

[0064] The medical device 100 includes a medical instrument (such as a catheter 300) having bendable sections 310A, 310B, 310C and a distal end, at least one imaging device disposed at the distal end, at least one sensor, and a controller 200 capable of receiving input data from the imaging device and the sensor. The controller 200 can determine a bending plane and bending angle of the distal end of the catheter 300, predict a movement and position of the bendable sections based on the input data, display image views based on the input data, and display the predicted movement and position of the bendable sections in the image views.

[0065] According to some embodiments, the medical device 100 provides collision warnings, better guides the user to operate the medical device, and is further configured to predict the behavior of the medical device based on input data from user input, imaging device 110, sensor 112, or a combination thereof, and previous predictions made by the medical device 100. In other words, the medical device 100 can simulate any kind of user adjustment from the current point and visualize how it will affect the tip of the medical device.

[0066] Once the user finds a satisfactory input or series of inputs, they do not have to repeat them manually, but simply transmit them automatically to the control device, either in a stepwise or continuous manner.

[0067] When a bendable medical device or catheter has bendable sections of different lengths, the position of the end effector of the trailing section may not match the position of the tip end effector at the same location. This can be problematic, and the medical device 100 can take this into account when calculating the expected motion.

[0068] According to some embodiments, there is provided a medical apparatus comprising a medical device having a bendable section and a distal end, at least one imaging device located at the distal end, at least one sensor, and at least one processor, the at least one processor performing the steps of receiving input data from the imaging device and the sensor, determining a bending plane and a bending angle of the distal end of the medical device, predicting a movement and a position of the bendable section based on the input data, displaying an image view based on the input data, and displaying the predicted movement and position of the bendable section in the image view.

[0069] According to some embodiments, there is provided a method for a medical apparatus comprising a medical device having a bendable section and a distal end, at least one imaging device, and at least one sensor, the method including receiving input data from the imaging device and the sensor, determining a bending plane and a bending angle of the distal end of the medical device, predicting a movement and a position of the bendable section based on the input data, displaying an image view based on the input data, and displaying the predicted movement and position of the bendable section in the image view.

[0070] The method may also detect a collision of the medical device based on the input data, provide a collision warning of the medical device based on the input data, or limit the bending angle of the distal end of the medical device to avoid a potential future collision with the airway wall.

[0071] The medical device may be a steerable catheter and the rigid tip may be located at a distal end of the catheter. The rigid tip may be aligned with a normal vector of the distal bendable section. The predicted movement and position of the bendable section may be based on an insertion distance and length of the rigid tip.

[0072] The method can represent a constant arc by assuming an insertion distance so that the tip of the distal bendable section can be projected onto the bending plane. The predicted movement and position of the bendable section is based on the insertion distance and length of the rigid tip. The constant arc can be represented by intersecting lines on the image views.

[0073] Projecting the bendable section placement onto the current virtual bronchoscope viewpoint can be, for example, a bird's-eye view, an over-the-shoulder view, or other type of view, and can take into account the length of the rigid tip. Once the bend angle and bend plane are input, the tip of the current distal bendable section is projected backwards based on the length of the rigid tip. Then, assuming the current bend angle continues, a constant arc is projected forward by assuming an insertion distance. This allows the tip of the bendable section to be projected onto the bend plane and represented by an intersection line on the endoscopic camera view. The axis of the rigid tip can be aligned with the normal vector of the distal curved section, and the bend angle can be limited to avoid potential collisions with the airway wall.

[0074] According to some embodiments, there is provided a storage medium having stored thereon a program configured to cause a computer to execute a method for a medical apparatus comprising a medical instrument having a bendable section and a distal end, at least one imaging device, and at least one sensor.

[0075] Also, an advantage of an additional feature or aspect of the present disclosure is that one or more AI (Artificial Intelligence) or machine learning algorithms, processes, techniques, etc. can be implemented to avoid collision of medical equipment during a medical procedure. Such AI techniques can be trained based on a set of data using neural networks, random forest algorithms, cognitive computing systems, rule-based engines, etc. to assess the type of data and generate an output. For example, the training algorithm can be configured to avoid collision of medical equipment during a medical procedure. The model can be configured as software that receives an image as input and returns a prediction of a given image as output. The model can be an instance of a model architecture (a set of parameter values) obtained by training and selecting a model using machine learning and / or optimization algorithms / processes. The model can generally include, for example: an architecture defined by source code (e.g., a convolutional neural network including layers of parameterized convolution kernels and activation functions) and configuration values ​​(parameters, weights, features, etc.) that are initially set to random values ​​and then iteratively optimized during training given data examples; an objective function (loss function); an optimization algorithm (optimizer), etc.

[0076] At least a portion of the medical images of the detailed configuration of the patient's anatomy relative to the catheter position can be used as input data and provided to the training algorithm. The initial images, output values, and detailed configuration of the catheter position relative to the patient's anatomy can be stored in a database to facilitate accurate real-time correction of local tissue deformations during the endoscopy procedure to obtain new data. Machine learning can find parameters for the AI ​​process through visualization-guided instrument-to-image registrations generated using input mapping to a model or expert studies. The training algorithm is configured to learn physical relationships in the input data to best describe such relationships or correlations. The data set includes information based on many factors, such as the number of images acquired, the angle of the images, the location of the images, the detailed configuration of the medical device relative to the bifurcation model, etc. The data is evaluated using weighted evaluations, where the weights are learned through a training process, subject matter specifications, etc. Deep learning mechanisms can augment the AI ​​process to identify indicators in the image data, which can include, for example, new data images, output values, configuration of the catheter position relative to the patient's anatomy, etc.

[0077] Features of the present disclosure avoid collisions of medical equipment during medical procedures.

[0078] Other embodiments The embodiments of the present disclosure may be implemented by a computerized configuration of a system or device that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may more fully be referred to as a "non-transitory computer-readable storage medium") to perform one or more functions of the aforementioned embodiments and / or includes one or more circuits (e.g., application specific integrated circuits (ASICs)) for performing one or more functions of the aforementioned embodiments, or by a method performed by a computerized configuration of a system or device that reads and executes computer-executable instructions from a storage medium to perform one or more functions of the aforementioned embodiments and / or controls one or more circuits to perform one or more functions of the aforementioned embodiments. The computerized configuration may include one or more processors, one or more memories, circuitry, or a combination thereof (e.g., a central processing unit (CPU), a microprocessing unit (MPU), etc.), and may include a separate computer or a network of separate processors for reading and executing the computer-executable instructions. The computer-executable instructions may be provided to the computerized configuration, for example, from a network or a storage medium. Examples of storage media include one or more of a hard disk, a random access memory (RAM), a read-only memory (ROM), storage of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray Disc (BD) (trademark)), a flash memory device, a memory card, and the like.

[0079] Although the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments, and the scope of the following claims should be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. a medical device (108) having at least one bendable section (310) and a distal end (348); at least one imaging device (110) located at the distal end (348); At least one sensor (203); A control device (200); A medical device (100) comprising: The control device (200) receiving input data from a user input, the imaging device (110), the sensor (203), or a combination thereof; determining a bend plane and a bend angle of the distal end (348) of the medical device (108); predicting a movement and a position of the at least one bendable section (310) based on the input data; displaying an image view based on the input data; displaying the predicted movement and position of the at least one bendable section (310) in the image view; Execute Medical device (100).

2. The control device (200) further performs collision detection of the medical device (108) based on the input data. The medical device (100) of claim 1.

3. The control device (200) is further configured to provide a collision warning for the medical device (108) based on the input data. The medical device (100) of claim 1.

4. The control device (200) further performs limiting the bending angle of the distal end of the medical device to avoid possible future collisions with the airway wall. The medical device (100) of claim 1.

5. the medical device (108) being a steerable catheter; The medical device (100) of claim 1.

6. a rigid tip is provided at the distal end of the medical device (108); The medical device (100) of claim 1.

7. the rigid tip is aligned with a normal vector of the at least one bendable section (310); The medical device (100) of claim 6.

8. the predicted movement and position of the at least one bendable section (310) is based on an insertion distance and length of the stiff tip; The medical device (100) of claim 6.

9. representing a constant arc by assuming an insertion distance so that a tip of the at least one bendable section (310) can be projected onto the bending plane; the predicted movement and position of the at least one bendable section (310) is based on an insertion distance and length of the stiff tip; The medical device (100) of claim 6.

10. the constant arc is represented by intersecting lines on the image view; The medical device (100) of claim 9.

11. A method for a medical device (100) comprising a medical instrument (108) having at least one bendable section (310) and a distal end (348), at least one imaging device (110), and at least one sensor (112), comprising: receiving input data from a user input, the imaging device (110), the sensor (112), or a combination thereof; determining a bend plane and a bend angle of the distal end of the medical device (108); predicting a movement and a position of the at least one bendable section (310) based on the input data; displaying an image view based on the input data; displaying the predicted movement and position of the at least one bendable section (310) in the image view; The method includes:

12. detecting a collision of the medical device (108) based on the input data; The method of claim 11 further comprising:

13. providing a collision warning for the medical device (108) based on the input data; The method of claim 11 further comprising:

14. limiting the bend angle of the distal end (348) of the medical device (108) to avoid potential collisions with the airway wall; The method of claim 11 further comprising:

15. the medical device (108) being a steerable catheter; The method of claim 11.

16. the distal end (348) of the medical device is provided with a hard tip; The method of claim 11.

17. the rigid tip is aligned with a normal vector of the at least one distal bendable section (310); 17. The method of claim 16.

18. the predicted movement and position of the at least one bendable section (310) is based on an insertion distance and length of the stiff tip; The method of claim 11.

19. Representing a constant arc by assuming an insertion distance so that a tip (348) of said at least one bendable section (310) can be projected onto said bending plane; Further comprising: the predicted movement and position of the at least one bendable section is based on an insertion distance and length of the stiff tip; The method of claim 11.

20. the constant arc is represented by intersecting lines on the image view; 20. The method of claim 19.

21. A storage medium having a program stored thereon for causing a computer to execute a method for a medical device (100) comprising a medical instrument (108) having at least one bendable section (310) and a distal end (348), at least one imaging device (110), and at least one sensor (112), the method comprising: The method comprises: receiving input data from a user input, the imaging device (110), the sensor (112), or a combination thereof; determining a bend plane and a bend angle of the distal end (348) of the medical device (108); predicting a movement and a position of the at least one bendable section (310) based on the input data; displaying an image view based on the input data; displaying the predicted movement and position of the at least one bendable section (310) in the image view; Including, storage medium.

Images