Guiding providing system and method for removing stone

Abstract

According to one embodiment, the boot providing system comprises a slave device, a plurality of bootstrapping devices, a plurality of bootstrapping devices and a plurality of bootstrapping devices, the present invention relates to a urinary calculus removal device comprising a guide tube insertable inside a ureter to remove calculus, a guidewire movable inside the guide tube along a longitudinal axis of the guide tube, a drive part for moving the guidewire, a medical tool connected to a distal end of the guidewire and exposed outside the guide tube, and a catheter for removing calculus from the guide tube, the camera is used for taking medical images at the far end of the guide tube; and a master device including a processor that determines a size of a guide image representation indicative of a size of a stone that can be grasped through a basket based on an estimated distance between the medical tool and a camera at a distal end of the guide tube, the size of the guide image representation indicating a size of the stone that can be grasped through the basket, and the size of the guide image representation being determined based on an estimated distance between the medical tool and the camera at the distal end of the guide tube. The master device superimposes and displays the guidance graphic representation having the determined size on the medical image captured by the camera received from the slave device.

Description

Technical Field

[0001] The following involves a guided technique for stone removal using a basket and laser fiber. Background Art

[0002] Urinary stones are stones that form in the urethra (the passage that makes and excretes urine). When the concentration of dissolved minerals in urine (such as calcium, oxalate, phosphoric acid, uric acid, etc.) increases, these minerals can no longer dissolve in the urine and become crystals, which act as nuclei and aggregate with other crystals to form stones. In most cases, urinary stones form in the kidneys and appear in the renal pelvis, ureters, bladder, and urethra. The degree of pain caused by urinary stones depends on the shape, size, and location of the stone. All types of urinary stones can cause hematuria, that is, blood mixed in the urine, among which kidney stones may have no symptoms or only cause mild indigestion. In terms of pain, when ureteral stones are stuck in the upper ureter, the pain extends from the side to the back; when the stone is stuck in the lower ureter, the pain extends from the lower abdomen to the groin, thigh, and perineum. Bladder stones can cause discomfort in the lower abdomen and frequent urination, accompanied by discomfort.

[0003] Urinary stones can be diagnosed by urine chemistry analysis, X-ray, CT, urography, or ultrasound. In addition, urinary stones can be removed by extracorporeal shock wave lithotripsy (ESWL), retrograde intrarenal surgery (RIRS), percutaneous nephrolithotomy (PCNL), and open or laparoscopic nephrolithotomy.

[0004] The above description is information obtained during the process of conceiving the present disclosure, or information already possessed at the time, and is not necessarily known technology before filing this application. Summary of the invention

[0005] Technical issues to be solved

[0006] According to an embodiment, a guidance providing system and method for stone removal may provide a guidance graphic representation displayed by overlaying on a medical image.

[0007] A guidance providing system and method for stone removal according to an embodiment may provide a guidance graphic representation when a basket and a laser fiber are used in a medical tool.

[0008] The guide providing system and method for stone removal according to an embodiment may provide guide graphic representations in different sizes according to varying estimated distances.

[0009] According to an embodiment, a guide providing system and method for stone removal can use both image-based distance estimation and encoder-based distance estimation to estimate the distance between the medical tool and the camera at the distal end of the guide tube.

[0010] However, the technical problems are not limited to the above-mentioned technical problems, and there may be other technical problems.

[0011] Technical methods to solve problems

[0012] According to one embodiment, a guidance providing system for removing stones may include: a slave device, which includes a guide tube that can be inserted into a ureter to remove stones, a guide wire that can be moved inside the guide tube along the longitudinal axis of the guide tube, a drive unit for moving the guide wire, a medical tool connected to the distal endportion of the guide wire and exposed outside the guide tube and including a basket or a laser, and a camera for taking medical images at the distal end of the guide tube; and a master device, which includes a processor, which determines the size of a guidance graphic representation indicating the size of a stone that can be removed by a basket based on an estimated distance between the medical tool and the camera at the distal end of the guide tube, and the master device superimposes and displays the guidance graphic representation with the determined size on the medical image taken by the camera and received from the slave device.

[0013] The processor can be configured to: infer an image-based distance between the medical tool and the camera at the distal end of the guide tube from the captured medical image; estimate an encoder-based distance between the medical tool and the camera at the distal end of the guide tube from encoder information of the drive unit; and determine the estimated distance based on the image-based distance and the encoder-based distance.

[0014] The processor can be configured to: when the medical tool is a basket, estimate the geometric information of the axis from the axis area of ​​the basket based on the captured medical image using an object detection or instance segmentation algorithm, etc.; and infer the distance based on the image from the estimated geometric information based on a machine learning model.

[0015] The geometric information of the axis may include at least one of the following: the area occupied by the axis region of the basket in the medical image, the diameter of the estimated circle corresponding to the distal end of the axis, the radius of the estimated circle corresponding to the distal end of the axis, the length from the boundary of the medical image to the distal end of the axis, the length from the center of the medical image to the distal end of the axis, or the inclination of the axis.

[0016] The processor may infer distances based on the medical images by inputting the medical images into another machine learning model.

[0017] The processor may determine the estimated distance by applying a deviation compensation value determined based on the image and the distance based on the encoder to the distance based on the encoder.

[0018] The deviation compensation value may be a value that converges with the passage of time after the calculation of the estimated distance is started.

[0019] The processor may be configured to: decrease the size of the guide graphical representation when the estimated distance increases; and increase the size of the guide graphical representation when the estimated distance decreases.

[0020] The processor is configured to: when the medical tool is a laser fiber, detect the moment when the distal end of the laser fiber contacts the stone; capture and store the target image at the moment when the distal end of the laser fiber contacts the stone; output the medical image to the display of the main device and output the target image at the same time.

[0021] In response to receiving user input selecting a point in the target image, the processor may output the guide graphical representation onto the target image based on the selected point.

[0022] The processor may detect a stone region in the target image and output the guide graphical representation on the target image based on reference points in the detected stone region.

[0023] The guiding graphic representation may be one of the following shapes: a circle, an ellipse, a quadrilateral, a polygon or a closed curve graphic.

[0024] The size of the guide graphic representation may be determined according to the size of a sheath of a basket in the medical tool.

[0025] The border of the guide graphical representation has a thickness, and the processor may reduce the thickness of the border of the guide graphical representation as the reliability of the estimated distance increases over time.

[0026] According to one embodiment, a method for providing guidance by a processor may include the following steps: obtaining a medical image that includes a medical tool and is photographed at the distal end of a guide tube inserted into the inside of a ureter; inferring an image-based distance between the medical tool and a camera at the distal end of the guide tube from the photographed medical image; obtaining information of an encoder of a drive unit, wherein the drive unit moves a guide wire inside the guide tube; estimating an encoder-based distance between the medical tool and the camera at the distal end of the guide tube from the information of the encoder; determining an estimated distance between the medical tool and the camera at the distal end of the guide tube based on the image-based distance and the encoder-based distance; and superimposing and displaying a guidance graphic representation of a size determined based on the estimated distance on the medical image.

[0027] The step of inferring the image-based distance may include the following steps: when the medical tool is a basket mesh, estimating the geometric information of the axis from the axis region of the medical tool according to the photographed medical image; and inferring the image-based distance from the estimated geometric information based on a machine learning model, such as an object detection or instance segmentation algorithm. When the medical tool is a laser fiber, the geometric information of the laser fiber may use the pixel distance between the distal end of the laser fiber in the image and the center of the image, the length from the boundary of the medical image to the distal end of the laser fiber, the area occupied by the laser fiber region, and the length of the distal end of the laser fiber, and the remaining steps are the same as the basket mesh.

[0028] The step of determining the estimated distance may include the following steps: determining the estimated distance by applying a deviation compensation value determined based on the image distance and the encoder distance to the encoder distance, wherein the deviation compensation value is a value that converges over time after the calculation of the estimated distance begins.

[0029] The step of superimposing and displaying on the medical image also includes the following steps: adjusting the size of the guide graphic representation based on the change of the estimated distance between the medical tool and the camera at the distal end of the guide tube, wherein when the estimated distance increases, the size of the guide graphic representation can be reduced; when the estimated distance decreases, the size of the guide graphic representation can be increased.

[0030] After the step of superimposing and displaying on the medical image, the following steps may also be included: sensing an external input of a user; and changing a position of the guide on the medical image according to the external input of the user.

[0031] Effects of the Invention

[0032] According to an embodiment, a guidance providing system and method for removing stones can remove stones without damaging internal organs of the human body by providing a guidance graphic representation superimposed on a medical image, and there is no need to determine the size of the stone which requires complex calculations.

[0033] According to one embodiment, a guidance providing system and method for stone removal provides a graphical representation of guidance when a basket and a laser fiber are used in a medical tool, thereby enabling a user to determine whether a stone is of a size that can be removed using a basket, and whether a laser fiber needs to be used to break up the stone.

[0034] According to an embodiment, a guide providing system and method for stone removal provides specific actual sizes represented by guide graphics in circles of different sizes according to varying estimated distances, and can accurately determine whether the stone can be removed regardless of the estimated distance, thereby performing a safe operation.

[0035] According to one embodiment, a guidance providing system and method for stone removal can provide a user with a highly reliable guidance graphic representation by simultaneously using image-based distance estimation and encoder-based distance estimation to estimate the distance between a medical tool and a camera at the distal end of a guide tube with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A guide providing system for stone removal according to an embodiment is shown.

[0037] Figure 2 A driving mechanism of a slave device according to an embodiment is shown.

[0038] Figure 3 Shown is a guide tube, camera, and medical tools inserted into a ureter according to one embodiment.

[0039] Figure 4 The figure exemplarily shows a basket that has grasped a stone through a guide providing system for removing the stone according to an embodiment.

[0040] Figure 5 A basket or a laser fiber that has grasped a stone to infer an image-based distance in a guide providing system for stone removal according to an embodiment is exemplarily shown.

[0041] Figure 6 A graph exemplarily shows image-based distances corresponding to geometric information used in a guidance providing system for stone removal according to an embodiment.

[0042] Figure 7 The following exemplarily shows the markings configured in the medical tool used in the guide providing system for removing stones according to one embodiment.

[0043] Figure 8 A graph exemplarily illustrates the relationship between an image-based distance, an encoder-based distance, and an estimated distance in a guidance providing system for stone removal according to an embodiment.

[0044] Fig. 9 The following exemplarily illustrates a fitting method for deriving a function for graphically representing a radius used in a guide providing system for stone removal according to an embodiment.

[0045] Fig.10 A graph between radii corresponding to estimated distances used in a guide providing system for removing stones according to an embodiment is exemplarily shown.

[0046] Fig.11 The following exemplarily illustrates changes in guide graphic representations expressed in different ways according to changes in estimated distances in a guide providing system for removing stones according to an embodiment.

[0047] Fig.12 A screen of a display outputting a guide graphic representation in a guide providing system for stone removal according to an embodiment is exemplarily shown.

[0048] Fig.13 The following exemplarily shows the types of guide graphic representations in a guide providing system for removing stones according to an embodiment.

[0049] Fig.14 The thickness of the border of the guide graphic representation in the guide providing system for removing stones according to an embodiment is exemplarily shown.

[0050] Fig.15 A graph exemplarily illustrates a validation result of a polynomial regression function for inferring an image-based distance in a guidance providing system for stone removal according to an embodiment.

[0051] Fig.16 A graph exemplarily shows a deviation compensation value for calculating an estimated distance based on stone size and time in a guidance providing system for removing stones according to an embodiment, and a graph showing comparison results between the estimated distance and the actual distance using each image estimation method.

[0052] Fig.17 A flow chart showing a guide providing method for stone removal according to an embodiment. DETAILED DESCRIPTION

[0053] The specific structure or function description of the disclosed embodiment is only for illustrative purposes, and various changes can be made to the embodiment. Therefore, the embodiment is not limited or restricted by the specific disclosed form, and all variations, equivalents or substitutes of the embodiment are included in the scope of rights.

[0054] In addition, in the description of various components, the terms such as first or second may be used, which are only used to distinguish the constituent elements from one another. For example, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

[0055] When one constituent element is described as being “connected” to another constituent element, it should be understood that the constituent element may be directly connected or attached to the other constituent element, or that the other constituent element is “connected” between the constituent elements.

[0056] In the absence of special instructions in the content, singular expressions include plural meanings. In this specification, the terms "including" or "having" are used to express the existence of features, numbers, steps, operations, constituent elements, accessories or combinations thereof recorded in the specification, and do not exclude the existence of one or more other features, numbers, steps, operations, constituent elements, accessories or combinations thereof, or additional functions.

[0057] In the absence of other definitions, all terms used herein, including technical or scientific terms, have the common meanings understood by those of ordinary skill in the art. Commonly used terms that are the same as dictionary definitions should be understood to have meanings consistent with the common content of the relevant technology and should not be overly idealized or interpreted as formal meanings unless explicitly mentioned in this application.

[0058] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and repeated descriptions thereof are omitted.

[0059] Among the methods of stone removal, extracorporeal shock wave lithotripsy (ESWL) is suitable for stones with a diameter of less than 1 cm and is only suitable for a small number of stones stuck in the ureter, which has certain limitations. In addition, among the methods of stone removal, percutaneous nephrolithotomy (PCNL) has the problem that it cannot be used when the patient has a tendency to bleed. In addition, the use of flexible ureteroscopic lithotripsy (RIRS) is not too restricted and can be performed on all pregnant women or patients with a tendency to bleed, and it can solve hard stones at one time. In addition, the surgical success rate of flexible ureteroscopic lithotripsy (RIRS) is as high as 94%, and the lowest complication rate is reported to be 0%, and even if complications do occur, their severity is very low. With the development of flexible ureteroscopic technology, the application scope of flexible ureteroscopic lithotripsy (RIRS) has also been continuously expanded, and many studies are currently underway.

[0060] In addition, flexible ureteroscopic lithotripsy (RIRS) can be performed by directly approaching the kidney through an endoscope, directly crushing the stone with a laser fiber, and then grabbing the stone with a basket and removing it from the body. In this flexible ureteroscopic lithotripsy (RIRS), the size of the stone may be one of the important factors of the operation. This is because if a too large stone is removed using a basket, the stone may get stuck in the sheath, thereby damaging the ureter. In addition, when a large stone is crushed using a laser fiber to make it smaller, excessive crushing causes an increase in the number of stones to be removed, which leads to the problem that it takes a long time to remove the stones one by one through the basket. On the contrary, even if the laser fiber is crushed properly, there will be a case where a large stone remains, and there is still a risk of damaging the ureter when the basket grabs the stone afterwards. If the stone is judged to be large after the basket is removed, the medical tool must be replaced from the basket back to the laser fiber to crush the stone, which is troublesome. In addition, since stones can be confirmed by displaying images taken by an endoscopic camera on a monitor, it is difficult to determine the size of the stone from the image displayed on the monitor, and it is also difficult to determine whether the above-mentioned problem will occur.

[0061] To solve the above problems, the present specification proposes a system and method for providing guidance so as to determine whether the stone can be removed.

[0062] Figure 1 A guide providing system for stone removal according to an embodiment is shown.

[0063] According to an embodiment, a guidance providing system for removing stones can display guidance on a medical image. The guidance providing system can provide medical images to a user 1900 (e.g., a surgical staff or a medical staff) during a kidney stone removal operation. The guidance providing system can control a guide tube 1113, a camera 1114 (e.g., an endoscopic camera), and a medical tool 1107 for removing stones based on input from the user 1900 detected by an operating unit (e.g., a controller such as a touch screen or a main handle). For reference, in this specification, a medical image may be an endoscopic image taken by an endoscopic camera.

[0064] The guide providing system may include a slave device 1100 and a master device 1200. The master device 1200 may be connected to the slave device 1100 by wire and / or wireless means and may exchange data. For example, the master device 1200 may transmit a control command to the slave device 1100 based on an input of the user 1900. The slave device 1100 may control the movement and / or operation of at least one of the guide tube 1113, the camera 1114, or the medical tool 1107 in response to the control command. The slave device 1100 may transmit an operation status and a medical image to the master device 1200.

[0065] The slave device 1100 is a slave robot that is equipped with a medical tool and can operate the medical tool 1107 inserted into the body of the patient P under the control of the user 1900 during surgery. The slave device 1100 may include a support body 1101, a driving body 1102, a guide rail 1103, and a driving part 1104. The slave device 1100 can control the movement and / or operation of at least one of the guide tube 1113, the camera 1114, or the medical tool 1107 by operating the driving part 1104 according to the control of the master device 1200. For example, the slave device 1100 can apply at least one of movement (for example, forward or backward), rotation, or bending to at least one of the guide tube 1113, the camera 1114, or the medical tool 1107. This will be described below. Figure 2 and Figure 3 A more specific structure of the slave device 1100 is described in FIG.

[0066] The main device 1200 is a master console, which may include a display, a controller, and a processor.

[0067] The display of the main device 1200 can provide medical information to the user 1900. For example, the display can intuitively output medical images taken by the camera of the slave device 1100. In addition, the display can not only output the above-mentioned real-time medical images, but also output the operation status information, function information and operation information of the slave device 1100 as additional information to assist in the removal and end of the operation. The operation status information is information indicating the state related to the operation of the slave device 1100, and may include, for example, the length of the sheath, the bending angle of the guide tube, and the rolling angle of the guide tube. The function information is information about the functions provided by the main device 1200, and may include a recording function and a snapshot function. The operation information is, for example, information related to the stone removal operation, and may include information (e.g., an image) indicating the shape and position of the guide tube inserted into the urethra.

[0068] The controller may receive input from the user 1900. The master device 1200 may control the master device 1200 and / or the slave device 1100 by interpreting the input of the user 1900 received through the controller. For example, the controller may include at least one of a touch screen, a touch pad, or a master handle. The touch screen is a device integrated with the above-mentioned display, and although this specification mainly describes an example in which the display is a touch screen, it is not limited thereto. The master handle may receive manipulation of the operation of the slave device 1100 from the user 1900 (e.g., movement of the guide tube 1113, the camera 1114, or the medical tool 1107).

[0069] The processor may perform processing and interpretation of the medical image and control of the slave device 1100. For example, the processor may generate a control command for controlling the slave device 1100 based on the interpretation of the operation of the user 1900 detected through the controller (e.g., the master handle). The processor may transmit the generated control command to the slave device 1100.

[0070] The processor may receive a medical image from the slave device 1100 and process or analyze the received medical image. For example, the processor may determine a guidance graphic representation to be displayed on the medical image. The processor may determine the size of the guidance graphic representation based on an estimated distance between the medical tool and the camera at the distal end of the guide tube. The size of the guidance graphic representation may indicate the size of the stone that can be extracted by the basket. The size of the guidance graphic representation displayed on the display will be described later. Fig.10 The master device 1200 may display the guide graphic representation having a determined size by superimposing it on the medical image photographed by the camera 1114 received from the slave device 1100 .

[0071] exist Figure 12 shows a structure in which the display and the controller in the main device 1200 are integrated into one, but the present invention is not limited thereto, and the display and the controller may also be configured as separate modules.

[0072] Figure 2 A driving mechanism of a slave device according to an embodiment is shown.

[0073] The driving structure of the slave device 1100 may include a guide tube holder 1105, a guide wire 1106, a medical tool 1107 (e.g., a basket or laser fiber), an endoscope 1110, and an access sheath 1120. The endoscope 1110 may include a body 1111, a handle 1112, a guide tube 1113, and a camera 1114. The access sheath 1120 may include a sheath holder 1121 and a sheath 1122.

[0074] The slave device 1100 can be configured to place a driver 1102 on the upper part of a support 1101 that is movable relative to the ground, and the driver 1102 is equipped with a guide rail 1103, a drive unit 1104, a guide tube bracket 1105, a guide wire 1106, a medical tool 1107, a camera 1114, an endoscope 1110, and a guide sheath 1120. The driver 1102 can move relative to the support 1101 to perform surgery. For example, the driver 1102 can receive a signal from the master device 1200 and translate in the horizontal and vertical directions relative to the support 1101. The driver 1102 can yaw relative to the support 1101.

[0075] A guide rail 1103 is formed on the driving body 1102, whereby a driving unit 1104 for driving the endoscope can slide in the longitudinal direction of the driving body 1102. The driving unit 1104 supports the endoscope 1110 and can control the endoscope 1110 by receiving a signal from the main device 1200. For example, the driving unit 1104 can move the handle 1112 of the endoscope 1110, thereby controlling the direction in which the guide tube 1113 of the endoscope 1110 is bent. When the endoscope 1110 is supported by the driving unit 1104, the relative movement of the endoscope 1110 relative to the driving unit 1104 can be limited. Thus, the slave device 1100 can reduce the error caused by the movement of the endoscope 1110, thereby improving the stability and accuracy of the operation. In addition, the driving unit 1104 includes an encoder (or a motor including an encoder), etc., and the processor of the main device 1200 can use the encoder information of the driving unit 1104.

[0076] The endoscope 1110 may include a body 1111, a handle 1112, and a guide tube 1113. The endoscope 1110 may control the bending of the guide tube 1113 by relative movement of the handle 1112 relative to the body 1111 of the endoscope 1110. Thus, the guide tube 1113 may be inserted into the interior of the ureter to remove stones. This may mean that the distal end portion of the guide tube 1113 moves toward the location of the stone by bending.

[0077] The guide tube support 1105 can support the guide tube 1113 of the endoscope 1110. The guide tube support 1105 can prevent the guide tube 1113 of the endoscope 1110 from buckling when the driving part 1104 slides or rotates relative to the driving body 1102. For example, the guide tube support 1105 is provided in a plurality along the longitudinal direction of the driving body 1102 and can be spaced apart from each other.

[0078] The guide sheath 1120 may include a sheath support 1121 and a sheath 1122 .

[0079] The sheath support 1121 can mount the sheath 1122 to the distal end of the driver 1102. The sheath 1122 can be mounted to the inner side of the ureter U to guide the guide tube 1113. Therefore, the guide tube 1113 can move relative to the guide sheath 1122 to reach the surgical site of the patient P through the inside of the sheath 1122.

[0080] Figure 3 Shown is a guide tube, camera, and medical tools inserted into a ureter according to one embodiment.

[0081] In addition, the guide tube 1113 can guide the guide wire 1106 and the medical tool 1107 (eg, a basket) placed therein to the location where the stone S is located.

[0082] The guide wire 1106 can move inside the guide tube 1113 along the longitudinal axis (or longitudinal direction) of the guide tube 1113. The guide wire 1106 is formed by covering a core with a flexible tube having high elasticity and flexibility, such as a spring steel wire spirally wound with a steel wire or a Teflon tube, and another structure is bonded to the distal end of the steel wire as needed (for example, bonding a basket and a steel wire of a semicircular ball having high elasticity and flexibility).

[0083] The guide wire 1106 moves smoothly because the entire exterior is coated with Teflon, thereby preventing breakage or excessive bending in the blood vessel during surgery and preventing blood vessel damage caused thereby. The guide wire 1106 can be moved by the drive unit 1104 of the mobile endoscope 1110, or can be moved by another drive unit configuration.

[0084] The medical tool 1107 is detachable, and only one of the basket and the laser fiber can be configured so that they can be installed alternately, but it is not limited to this, and can also be configured in a form in which both are installed at the same time. An example of the medical tool 1107 is a basket, which is connected to the distal end of the guide wire 1106 and exposed to the outside of the guide tube 1113, and can be switched between an expanded state and a contracted state to grasp the stone S. The basket can be configured as a material that returns to its original shape when no external force is applied, and for example, can be a shape memory alloy.

[0085] The medical tool 1107, such as a basket and a laser fiber, may be configured to be mounted on a tool drive so that it can move back and forth.

[0086] As part of the configuration of the endoscope 1110, a camera 1114 can capture medical images at the distal end of the guide tube 1113. Therefore, the camera 1114 can capture the state of the medical tool 1107 approaching a stone inside an organ (e.g., a kidney) to grasp the stone or not grasp the stone, or to break the stone or not break the stone. The camera 1114 can be fixed to the distal end of the guide tube 1113. Since the medical tool 1107 can move relative to the guide tube 1113 (e.g., move forward or backward), the distance between the medical tool 1107 and the camera 1114 can change according to the operation of the user 1900. In the present specification, the distance used as a standard for determining the size of the guide graphic representation is the distance from the camera 1114 to the medical tool 1107, and can be the distance between the camera 1114 and the medical tool 1107.

[0087] Figure 4 The figure exemplarily shows a basket that has grasped a stone through a guide providing system for removing the stone according to an embodiment.

[0088] For example, when the medical tool 1107 is a basket 1107, in order to determine the size represented by the guide image, the processor must estimate the distance d from the distal end of the guide tube 1113 (e.g., the position captured by the camera 1114) to the medical tool 1107 (e.g., the basket) that has the stone S captured. The distance d can also represent the length of the guide wire 1106 exposed from the distal end of the guide tube 1113. When the medical tool is a laser fiber, the above content can also be applied in the same way. Therefore, the processor can infer the image-based distance between the medical tool and the distal end of the guide tube 1113 from the captured medical image, and can estimate the encoder-based distance between the medical tool and the distal end of the guide tube 1113 from the encoder information of the drive unit 1104. For example, the encoder-based distance can be estimated as a value corresponding to a predetermined number of rotations of the encoder (e.g., having a linear form). Then, the processor can determine the estimated distance based on the image-based distance and the encoder-based distance.

[0089] Figure 5 A basket or laser fiber that has grasped a stone to infer an image-based distance in a guidance providing system for removing stones according to an embodiment is exemplarily shown. In addition, in order to infer the image-based distance in a guidance providing system for removing stones according to an embodiment, a laser fiber close to the stone is exemplarily shown, and specifically, the image-based distance can be inferred based on the length of the laser fiber bounding box area in the X or Y or diagonal direction, and the length from the boundary of the medical image to the distal end of the laser fiber. In addition, as a specific example, the image-based distance can be inferred based on the pixel distance between the distal end of the laser fiber in the image and the center of the screen, and the area of ​​the laser fiber area.

[0090] Figure 6 A graph exemplarily shows image-based distances corresponding to geometric information used in a guidance providing system for stone removal according to an embodiment.

[0091] When the medical tool is a basket, the processor can estimate the geometric information of the axis from the axis area of ​​the basket according to the captured medical image. The geometric information can be estimated based on the average length of the detected bounding box. In addition, the processor can infer the image-based distance from the estimated geometric information based on a machine learning model (e.g., an object detection or instance segmentation algorithm). The machine learning model is a model designed and trained to estimate geometric information from a medical image, and may include, for example, a neural network. Without limitation, the processor can infer the image-based distance by inputting the medical image into another machine learning model or fitting function. For example, the image-based distance can be inferred by a function (e.g., a polynomial regression function) or a trained model that is fitted based on the geometric information corresponding to each collected image-based distance. Another machine learning model is a model designed and trained to estimate the image-based distance directly from the medical image, and may include, for example, a neural network.

[0092] In addition, the geometric information is a component value that is insensitive or robust to changes in factors other than the distance between the medical tool and the distal end of the guidewire in the medical image. The geometric information of the axis may include at least one of the following: the area occupied by the axis region of the basket in the medical image, the diameter of the estimated circle corresponding to the distal end of the axis, the radius of the estimated circle corresponding to the distal end of the axis, the length from the boundary of the medical image to the distal end of the axis, the length from the center of the medical image to the distal end of the axis, or the inclination of the axis. For example, the axis region of the basket may be an area in the medical image corresponding to a portion of the guidewire connected to the basket. The geometric information of the axis may, for example, use the diameter of the estimated circle corresponding to the distal end of the axis or the radius 510 of the estimated circle corresponding to the distal end of the axis, but is not limited thereto.

[0093] Figure 7 The following exemplarily shows the markings configured in the medical tool used in the guide providing system for removing stones according to one embodiment.

[0094] In addition, the medical tool (e.g., basket) may be attached or equipped with a mark or pattern to facilitate the extraction of geometric information from the shaft region of the aforementioned basket. The mark 710 may be formed as a circle on a portion of the shaft region, the mark 720 may be surrounded by a band around the shaft region, or the mark 730 may be formed as a pattern shape on the shaft region. This enables the processor to more quickly identify the shaft region of the basket, thereby more quickly extracting geometric information based on image distance.

[0095] Figure 8 A graph exemplarily illustrates the relationship between an image-based distance, an encoder-based distance, and an estimated distance in a guidance providing system for stone removal according to an embodiment.

[0096] The encoder-based distance may have errors in the low frequency band, while the image-based distance may have errors in the high frequency band. Since the error components of each value are different, they can be used complementary. The processor can determine the estimated distance by applying the deviation compensation value determined by the image-based distance and the encoder-based distance to the encoder-based distance. The processor can use mathematical formula 1 to determine the estimated distance.

[0097] [Mathematical formula 1]

[0098]

[0099] D r Denotes the estimated distance, D e Denotes the encoder-based distance, D i Represents image-based distance. The value obtained by cumulatively adding the expected value of the difference between the encoder-based distance and the image-based distance and dividing it by the number of accumulations may be represented. This may represent an offset value, that is, an error value (e.g., a deviation compensation value). The deviation compensation value may have a value that converges (e.g., converges to a specific constant value) over time after the calculation of the estimated distance is started.

[0100] Fig. 9 The following exemplarily illustrates a fitting method for deriving a function for graphically representing a radius used in a guide providing system for stone removal according to an embodiment.

[0101] Fig.10 A graph between radii corresponding to estimated distances used in a guide providing system for removing stones according to an embodiment is exemplarily shown.

[0102] When the estimated distance is input to an nth order polynomial function, the size (e.g., radius) of the corresponding guide graphic representation can be determined. The size of the guide graphic representation can be a criterion for safely removing the stone at the corresponding estimated distance. Therefore, if the stone exists within the size range of the guide graphic representation, the stone can be safely removed. This also means that the size of the guide graphic representation can be defined based on the size of the sheath 1122 guiding the guide tube 1113. The method of fitting an Nth order polynomial function can first obtain an image based on the tool distance (e.g., in millimeters) of the grasped stone.

[0103] Here, Fig. 9 The process shown can be performed on a phantom model that simulates the human kidney and urinary tract, and the stone placed in the phantom model can have a smaller size than the sheath physically. Fig. 9 The stones captured in the medical images in can be of a size that can be protected by the sheath if captured by the basket.

[0104] For example, method 910 is shown by marking point 911 at the outer edge of the stone in each image, while in methods 920, 930, the radius value of each tool distance obtained by fitting it to a circle can be fitted to an n-order polynomial function. Therefore, when radius 921 has a value greater than radius 931, the estimated distance can be set closer than radius 931. This also means that the size of the guide can also change as the estimated distance increases or decreases as the basket moves.

[0105] Fig.11 The following exemplarily illustrates changes in guide graphic representations expressed in different ways according to changes in estimated distances in a guide providing system for removing stones according to an embodiment.

[0106] When the estimated distance increases D1 based on the current basket position 111, the processor can reduce the size of the guide image representation 112. This means that even if the basket does not necessarily grasp the stone, it can be determined whether the stone is of the right size by only approaching the stone (e.g., the estimated distance increases) to grasp the stone. This can be more useful in the case of laser fiber. This can determine the size that needs to be crushed before contacting the stone, so that the operation can be performed faster.

[0107] When the estimated distance decreases D2 based on the current position of the medical tool 111, the processor can increase the size of the guide graphic representation 113. This allows the guide graphic representation to be extracted even when, for example, the basket is moved to remove the stone (e.g., the estimated distance decreases) or the laser is moved to pulverize the stone, or clearly indicates a specific size that needs to be pulverized, which allows for a more accurate determination of whether it can be removed or pulverized at a certain distance.

[0108] When the medical tool 1107 is a laser fiber, the processor may detect the moment when the distal end of the laser fiber contacts the stone, but is not limited thereto. For example, when the user 1900 determines that the laser fiber contacts the stone, the user 1900 may manually input the contact to the main device 1200. When the contact of the laser fiber is detected, the processor may capture and store a target image at the moment when the distal end of the laser fiber contacts the stone. The processor may output the target image together with the medical image to the display of the main device 1200. Thus, the laser fiber may determine whether pulverization is required based on the guide graphic representation on the medical image, so that an appropriate amount of pulverization may be performed, so that the stone can be quickly removed using the basket.

[0109] In addition, the processor may output a guide graphic representation on the target image based on the selected point in response to receiving an input from the user 1900 selecting a point in the target image. This may mean, for example, that when the guide graphic representation is output at a position that is not aligned with the target image, the user 1900 may select a desired point using a touch input to move the position of the guide graphic representation. In another approach, the processor may detect a stone region in the target image and output a guide graphic representation on the target image based on a reference point in the detected stone region. Thus, the user 1900 may easily determine the size of the stone without the inconvenience of a touch input.

[0110] Fig.12 A screen of a display outputting a guide graphic representation in a guide providing system for stone removal according to an embodiment is exemplarily shown.

[0111] 1210 is an area indicating surgical information, for example, it can be an area that outputs an X-ray image of the area where the stone is located that has been taken in advance, or an image that displays in real time the distance and position of the medical tool 1107 entering the body. 1220 is an area that can control the on / off of the guide graphic representation, which can not only turn on / off the guide graphic representation when necessary, but also manually start the operation of the guidance providing system to provide the guide graphic representation. This is because it is not necessary to provide the guide graphic representation at a location other than the area where the stone is located, so the processor in the main device 1200 can be used effectively. 1230 is an area that outputs the medical image taken by the camera 1114, and can display the guide graphic representation 1231 and the medical tools 1107, 1232.

[0112] Fig.13 The following exemplarily shows the types of guide graphic representations in a guide providing system for removing stones according to an embodiment.

[0113] The guide graphic representation may be represented as one of a circle, an ellipse, a quadrilateral, a polygon or a closed curve image, but is not limited thereto. For example, for a stone of a general shape, a guide graphic representation of a circle 131 is sufficient, but for a stone of an abnormal shape (e.g., a stone having a long shape in one direction), a guide graphic representation of a quadrilateral 132 or an ellipse 133 may be more appropriate.

[0114] Fig.14 The thickness of the border of the guide graphic representation in the guide providing system for removing stones according to an embodiment is exemplarily shown.

[0115] The border of the guide graphic representation can have a predetermined thickness. Therefore, as the reliability of the estimated distance increases over time, the processor can reduce the thickness of the border of the guide graphic representation. This may indicate that a border thickness 142 of the guide graphic representation that is thinner than the border thickness 141 of the guide graphic representation initially displayed is a more accurate guide graphic representation. This representation may also be the opposite (for example, the border thickness of the guide graphic representation may become thicker over time). In addition, the reliability of the estimated distance can be determined based on the deviation compensation value in the above mathematical formula 1.

[0116] Fig.15 A graph exemplarily illustrates a validation result of a polynomial regression function for inferring an image-based distance in a guidance providing system for stone removal according to an embodiment.

[0117] Fig.16 A graph exemplarily shows a deviation compensation value for calculating an estimated distance according to stone size and time in a guidance providing system for removing stones according to an embodiment, and a graph showing comparison results between an estimated distance and an actual distance using various image estimation methods.

[0118] Since the polynomial regression function is adopted, the inferred image-based distances are mostly distributed within the normal range 151, so it can be seen that except for the errors in the low-frequency band, accurate image-based distances can be inferred.

[0119] In addition, since the smaller the variance (e.g., error) of the deviation compensation value, the more accurate the determination of the estimated distance based on the image-based distance and the encoder-based distance, and since the variance of the deviation compensation value becomes smaller as the number of cumulative values ​​increases over time, the estimated distance determined over time can be more accurate. 161 represents the cumulative average of the image-based distance error for small stones, and 162 represents the cumulative average of the image-based distance error for large stones. For large stones, the corresponding cumulative average tends to converge to 0 quickly, but after a certain period of time, small stones will also converge to 0. This may mean that the determined estimated distance depends more on time than on the size of the stone.

[0120] In addition, 163 represents the estimated distance determined by each method for a small stone, and 164 represents the estimated distance determined by each method for a large stone. e The actual distance D r The error can be determined as the highest value overall, while the image-based distance D i The actual distance D r The error of may be determined to be a lower value than this. In addition, the estimated distance D determined by the guidance providing system according to an embodiment is f can be determined as the distance D based on the encoder e and the image-based distance D i Compared to the most similar actual distance D r This may mean that the user 1900 can more accurately determine whether the stone can be removed through the guidance graphic representation generated based on the accurate distance. In addition, with respect to the estimated distance D f The estimated distance D at the later time t2 is f The actual distance D r The difference r1 is less than the estimated distance D at the initial time t1 f The actual distance D r The difference between r1 and r2 can therefore determine a more accurate estimate of the distance over time.

[0121] Fig.17 A flow chart showing a guide providing method for stone removal according to an embodiment.

[0122] In step 171, the processor may acquire a medical image that includes a medical tool and is photographed in the distal end of the guide tube inserted into the inside of the ureter. The medical image may be acquired automatically, for example, when the medical tool is operated (e.g., to grab a basket), when a user inputs an online input for outputting a guide graphical representation, or automatically acquired according to a predetermined extension length of the guide tube, but is not limited thereto.

[0123] In step 172, the processor may infer an image-based distance between the medical tool and the camera at the distal end of the guide tube from the captured medical image. For example, when the medical tool is a basket, the processor may estimate geometric information of the axis from the axis region of the basket based on the captured medical image. Then, the processor may infer the image-based distance from the estimated geometric information based on the machine learning model.

[0124] In step 173 , the processor may acquire information of an encoder of a driving unit, wherein the driving unit moves the guide wire inside the guide tube.

[0125] In step 174, the processor may estimate, from the encoder information, an encoder-based distance between the medical tool and the camera at the distal end of the guide tube.

[0126] In step 175, the processor may determine an estimated distance between the medical tool and the camera at the distal end of the guide tube based on the image-based distance and the encoder-based distance. The processor may determine the estimated distance by applying a deviation compensation value determined from the image-based distance and the encoder-based distance to the encoder-based distance, where the deviation compensation value may have a value that converges over time after the calculation of the estimated distance begins.

[0127] In step 176, the processor may overlay and display the guide graphic representation of a size determined based on the estimated distance on the medical image. The processor may adjust the size of the guide graphic representation based on the change in the estimated distance between the medical tool and the camera at the distal end of the guide tube. When the estimated distance increases, the size of the guide graphic representation may be reduced; when the estimated distance decreases, the size of the guide graphic representation may be increased.

[0128] After step 176 , the processor may sense an external input of the user and change the position of the guide on the medical image according to the external input of the user.

[0129] Additional Embodiments

[0130] Embodiment 1: A guide providing device inserted into a subject, comprising:

[0131] ●Guide tube that can be inserted into the subject

[0132] ●Guide wire that can move inside the guide tube

[0133] ●Drive unit for moving the guide wire

[0134] ●A processor for determining the size of objects inside a subject

[0135] Embodiment 2: In the guide providing device corresponding to any of the foregoing embodiments, the guide wire can move along the longitudinal axis of the guide tube.

[0136] Embodiment 3: In the guide providing device corresponding to any of the foregoing embodiments, the drive unit is coupled to the distal end of the guide wire.

[0137] Embodiment 4: In the guide providing device corresponding to any of the foregoing embodiments, it also includes a medical tool exposed outside the guide tube and a camera for capturing images at the distal end of the guide tube.

[0138] Embodiment 5: In a guide providing device corresponding to any of the foregoing embodiments, the processor determines the size of the object based on an estimated distance between the medical tool and the distal end of the guide tube.

[0139] Embodiment 6: In the guide providing device corresponding to any of the foregoing embodiments, the guide tube has a predetermined size.

[0140] Embodiment 7: In the guidance providing device corresponding to any of the foregoing embodiments, a main device is additionally provided, and the graphic representation is superimposed and displayed on the medical image taken by the camera.

[0141] Embodiment 8: In a guidance providing device corresponding to any of the foregoing embodiments, the processor estimates the estimated distance between the medical tool and the distal end of the guide tube from the encoder information of the drive unit, and determines the estimated distance based on the image-based distance and the encoder-based distance.

[0142] Embodiment 9: In a guidance providing device corresponding to any of the foregoing embodiments, the medical tool is a basket including an axial region and an axis, and the processor estimates geometric information of the axis of the basket from the captured medical image, and infers an image-based distance from the estimated geometric information based on a machine learning model.

[0143] Example 10. In a guidance providing device corresponding to any of the aforementioned embodiments, the geometric information of the axis includes at least one of the following: the area occupied by the axis region of the basket axis in the medical image, the diameter of the estimated circle corresponding to the distal end of the axis, the radius of the estimated circle from the distal end of the axis to the boundary of the medical image, the length from the distal end of the axis to the center of the medical image, and the inclination of the axis.

[0144] Embodiment 11, in the guidance providing device corresponding to any one of the foregoing embodiments, the processor infers the image-based distance by inputting the medical image into another machine learning model.

[0145] Embodiment 12: In a guidance providing device corresponding to any of the foregoing embodiments, the processor applies a deviation compensation value to calculate the estimated distance based on the image-based distance and the encoder-based distance, wherein the deviation compensation value is determined according to the encoder-based distance.

[0146] Embodiment 13: In the guidance providing device corresponding to any one of the aforementioned embodiments, the deviation compensation value has a value that converges with the passage of time after the calculation of the estimated distance starts.

[0147] Embodiment 14: In the guidance providing apparatus corresponding to any one of the aforementioned embodiments, the processor reduces the size of the guidance graphic representation as the estimated distance increases, and increases the size of the guidance graphic representation as the estimated distance decreases.

[0148] Embodiment 15: In the guidance providing device corresponding to any of the foregoing embodiments, the medical tool is a laser optical fiber.

[0149] Embodiment 16: In the guidance providing device corresponding to any of the foregoing embodiments, when the medical tool is a laser fiber, the processor detects the moment when the laser fiber contacts the object.

[0150] Embodiment 17: In the guidance providing device corresponding to any of the foregoing embodiments, when the medical tool is a laser fiber, the processor captures and stores a target image at the moment when the distal end of the laser fiber contacts the object.

[0151] Embodiment 18: In the guidance providing device corresponding to any of the foregoing embodiments, the medical tool is a laser optical fiber, which outputs the target image while outputting the medical image to the display of the main device.

[0152] Embodiment 19: In the guidance providing apparatus corresponding to any of the foregoing embodiments, when receiving input from a user selecting a point in a target image, the processor outputs a guidance graphic representation onto the target image based on the selected point.

[0153] Embodiment 20: In a guidance providing device corresponding to any of the foregoing embodiments, a missing area in a target image is detected, and a guidance graphic representation is output to the target image based on a reference point of the detected missing area.

[0154] Embodiment 21: In the guidance providing device corresponding to any of the foregoing embodiments, the guidance graphic is represented as a shape consisting of a circle, an ellipse, a square, a polygon or a closed curve.

[0155] Embodiment 22: In the guide providing device corresponding to any of the foregoing embodiments, the size of the guide graphic representation is determined based on the coverage size of the basket of the medical instrument.

[0156] Embodiment 23: In the guidance providing device corresponding to any of the foregoing embodiments, the boundary of the guidance graphic representation is drawn as a thick line, and as the reliability of the estimated distance increases over time, the thickness of the boundary of the guidance graphic representation gradually decreases.

[0157] Embodiment 24: In the guidance providing device corresponding to any of the above embodiments, the processor performs the following tasks:

[0158] • With the guide catheter inserted into the ureter, a medical image captured from the distal end of the guide catheter having the medical tool is acquired.

[0159] ●Inferring the image-based distance between the medical tool and the distal end of the guide tube from the captured medical images.

[0160] ●Acquire the encoder information of the drive unit used to move the guide wire within the guide tube.

[0161] - Estimate the encoder-based distance between the medical tool and the distal end of the guide tube from the encoder information.

[0162] ●Determine the estimated distance between the medical tool and the distal end of the guide tube based on the image-based distance and the encoder-based distance.

[0163] ● Overlaying a guided graphical representation of the size determined based on the estimated distance on the medical image.

[0164] Embodiment 25: In the guidance providing device corresponding to any one of the aforementioned embodiments, the medical device has a function of detecting a user input after an image is superimposed and displayed thereon.

[0165] Embodiment 26: In a guidance providing device corresponding to any of the foregoing embodiments, the position of the guidance on the medical image changes in response to user input.

[0166] Embodiment 27: Use a guidance providing system corresponding to any of the preceding embodiments during surgery.

[0167] Embodiment 28: A guidance providing system, comprising:

[0168] ●Main device providing signal

[0169] - The guide providing device determines the size of an object within the subject in response to a signal from a guide providing device host device inserted into the subject.

[0170] Embodiment 29: In the guidance providing system of embodiment 28, the guidance providing device comprises:

[0171] ●Guide tube that can be inserted into the subject

[0172] ●Guide wire that can move inside the guide tube

[0173] ●Drive unit for moving the guide wire

[0174] ●A processor for determining the size of objects inside a subject

[0175] Example 30: In the guide providing system of Example 28 or 29, the guide wire can move inside the guide tube along the longitudinal axis of the guide tube.

[0176] Example 31. In a guide providing system corresponding to any one of Examples 28-30, the drive unit is coupled to the distal end of the guide wire.

[0177] Example 32: In the guide providing system corresponding to any one of Examples 28-31, the guide providing device includes a medical tool exposed outside the guide tube and a camera that captures images at the distal end of the guide tube.

[0178] Example 33. In a guide providing system corresponding to any one of Examples 28-32, the processor determines the size of the object based on an estimated distance between the medical tool and the distal end of the guide tube.

[0179] Example 34: In the guide providing system corresponding to any one of Examples 28-33, the guide tube has a predefined size.

[0180] Embodiment 35: In a guidance providing system corresponding to any one of embodiments 28-34, it includes a main device that superimposes and displays a graphical representation on a medical image captured by a camera.

[0181] Example 36. In a guidance providing system corresponding to any one of Examples 28-35, the processor estimates an encoder-based distance between the medical tool and the distal end of the guide tube from encoder information of the guidewire drive unit, and determines the estimated distance based on the image-based distance and the encoder-based distance.

[0182] Example 37. In a guidance providing system corresponding to any one of Examples 28-36, the medical tool is a basket made of a mesh basket, the processor estimates geometric information about the axis of the basket from the captured medical image, and infers the image-based distance from the estimated geometric information based on a machine learning model.

[0183] Example 38. In a guidance providing system corresponding to any one of Examples 28-37, geometric information about the axis of the basket includes one or more of the following: the area where the axial region of the basket is loaded, the diameter of an estimated circle corresponding to the distal end, the radius of an estimated circle corresponding to the distal end, the length from the distal end to the boundary, the length from the center of the medical image to the distal end of the axis, and the inclination of the axis.

[0184] Example 39: In a guidance providing system corresponding to any one of Examples 28-38, the processor infers image-based distances by inputting the medical image into another machine learning model.

[0185] Embodiment 40: In a guidance providing system corresponding to any one of embodiments 28-39, the processor determines the estimated distance by applying a deviation compensation value, wherein the deviation compensation value is determined based on the image-based distance and the encoder-based distance.

[0186] Embodiment 41: In the guidance providing system corresponding to any one of Embodiments 28 to 40, the deviation compensation value has a value that converges with the passage of time after the calculation of the estimated distance starts.

[0187] Embodiment 42: In the guidance providing system corresponding to any one of Embodiments 28-41, the processor reduces the size of the guidance graphic representation as the estimated distance increases, and increases the size of the guidance graphic representation as the estimated distance decreases.

[0188] Example 43. In the guidance providing system corresponding to any one of Examples 28-42, the medical tool is a laser fiber.

[0189] Example 44: In the guidance providing system corresponding to any one of Examples 28-43, when the medical tool is a laser fiber, the processor detects the moment when the laser fiber contacts the object.

[0190] Example 45: In the guidance providing system corresponding to any one of Examples 28-44, when the medical tool is a laser fiber, the processor captures and stores a target image at the moment when the distal end of the laser fiber contacts the object.

[0191] Example 46. In the guidance providing system corresponding to any one of Examples 28-45, when the medical tool is a laser fiber, the processor outputs the medical image to the display of the main device and also outputs the target image.

[0192] Embodiment 47: In the guidance providing system corresponding to any one of Embodiments 28-46, when receiving input from a user selecting a point in a target image, the processor outputs a guidance graphic representation onto the target image based on the selected point.

[0193] Embodiment 48: In a guidance providing system corresponding to any one of Embodiments 28-47, a missing area in a target image is detected, and a guidance graphic representation is output to the target image based on a reference point of the detected missing area.

[0194] Example 49: In the guidance providing system corresponding to any one of Examples 28-48, the guidance graphic is represented as a shape consisting of a circle, an ellipse, a square, a polygon or a closed curve.

[0195] Embodiment 50: In the guide providing system corresponding to any one of embodiments 28-49, the size of the guide graphic representation is determined based on the coverage size of the basket of the medical instrument.

[0196] Embodiment 51: In the guidance providing system corresponding to any one of Embodiments 28-50, the boundary of the guidance graphic representation is drawn as a thick line, and as the reliability of the estimated distance increases over time, the thickness of the boundary of the guidance graphic representation gradually decreases.

[0197] Embodiment 52: In the guidance providing system corresponding to any one of embodiments 28-51, the processor performs the following operations:

[0198] • With the guide catheter inserted into the ureter, a medical image captured from the distal end of the guide catheter having the medical tool is acquired.

[0199] ●Inferring the image-based distance between the medical tool and the distal end of the guide tube from the captured medical images.

[0200] ●Acquire the encoder information of the drive unit used to move the guide wire within the guide tube.

[0201] - Estimate the encoder-based distance between the medical tool and the distal end of the guide tube from the encoder information.

[0202] ●Determine the estimated distance between the medical tool and the distal end of the guide tube based on the image-based distance and the encoder-based distance.

[0203] ● Overlaying a guided graphical representation of the size determined based on the estimated distance on the medical image.

[0204] Embodiment 53: In the guidance providing system corresponding to any one of Embodiments 28-52, the main device includes an input device for manually controlling the guidance display superimposed on the image.

[0205] Embodiment 54: In the guidance providing system corresponding to any one of embodiments 28-53, the user can use an input device to adjust the position of the guidance display on the image.

[0206] Example 55: In a guidance providing system corresponding to any one of Examples 28-54, it is used to provide guidance during surgery.

[0207] These embodiments describe the implementation of guidance providing devices and systems in the medical field, which can help improve the accuracy and efficiency of medical procedures. In addition, this can be used to assist in manipulating objects, improve safety, and assist the work of medical professionals.

[0208] Example 56: In a surgical guidance system, the guidance device includes a tactile feedback function based on the position and movement of the surgical tool. This allows the user to perform the surgery while receiving tactile feedback.

[0209] Example 57: In an endoscopic guidance system, tracking sensors are attached to an endoscope to monitor its position and orientation within the body. The system provides real-time visual feedback to help the surgeon navigate the endoscope and perform precise maneuvers to target specific areas.

[0210] Example 58: A robotic surgery system includes a guidance module that uses preoperative image data to virtually create an anatomical model of the patient. This allows the surgeon to make precise plans and manipulate surgical tools in a virtual environment before surgery.

[0211] Example 59: Autonomous surgical robotic systems use artificial intelligence algorithms and sensors to analyze the patient’s vital signs and the movements of surgical tools in real time. This enables the robot to perform safe and accurate surgery, and the surgeon can also receive automatic assistance from the system.

[0212] Example 60: Innovative medical image guidance systems extract anatomical information by processing images such as CT scans or magnetic resonance imaging (MRI) in real time. Based on this information, the system is able to provide surgeons with precise positioning guidance and provide real-time updated images during surgery.

[0213] Example 61: Ultrasound guidance systems use ultrasound sensors and image processing technology to generate real-time images of internal organs. This enables surgeons to accurately identify and manipulate the exact location during surgery and perform safe and precise surgery.

[0214] Example 62: A surgical guidance system using virtual reality (VR) helps surgeons simulate and practice surgery by virtually reproducing the patient's anatomical model. This enables surgeons to make accurate plans before the actual surgery and improves the safety and efficiency of the surgery.

[0215] These different approaches can be used to provide guidance during surgery and support the surgeon's work. Each system has its own unique features and advantages and can be developed and applied according to different situations and needs in the medical field.

[0216] Example 63: Real-time surgical guidance systems use cameras and image processing technology to help surgeons view and manipulate images in real time during surgery. This enables surgeons to more accurately understand the surgical process and quickly perform necessary operations.

[0217] Example 64: Robotic-assisted surgery systems use robotic arms and tools to assist surgeons in their operations. The robotic arms perform precise and steady movements that more accurately communicate the surgeon’s actions. This can improve the precision and safety of the surgery.

[0218] Example 65: Microscope systems use small precision microscopes to assist surgeons in observing and operating procedures. This enables surgeons to perform precise and safe operations even in delicate procedures.

[0219] Example 66: Surgical guidance systems using neural networks use neural network algorithms to predict and guide the movement of surgical tools. This enables surgeons to predict the next movement of surgical tools and optimize operations.

[0220] Example 67: Real-time position tracking systems use sensors and position tracking technology to track and guide the position of surgical tools in real time. This enables surgeons to obtain precise position information during surgery and perform safe and precise operations.

[0221] Example 68: Automated surgical guidance systems combine artificial intelligence algorithms and robotics to automatically guide surgical procedures. By automating the operation of surgical tools, surgeons can achieve accurate and consistent surgical results and reduce surgical time and costs.

[0222] Example 69: Surgical robotic systems use robotic arms and sensors to support surgeons in performing surgeries remotely. This helps overcome geographical limitations and provide professional medical services.

[0223] Example 70: With the advancement of modern medical technology, surgical guidance systems are also constantly improving and developing. It is expected that more accurate and faster guidance systems will be developed in the future.

[0224] Example 71: The application of surgical guidance systems not only improves the accuracy and safety of surgery, but also improves the work efficiency of medical staff. This has a positive effect on improving the effectiveness and comfort of patient care.

[0225] Example 72: The surgical guidance system promotes collaboration and knowledge sharing in the medical field. Through the system, medical staff can share the latest technology and information to jointly develop and apply more effective treatment methods.

[0226] Example 73: Surgical guidance systems help provide patient-centered healthcare and drive innovation in medical technology. This enables patients to undergo more accurate and safer surgeries and minimize recovery time.

[0227] Example 74: Surgical guidance systems are widely recognized as one of the technologies that represent the future of the medical field. Through continued research and development, it is expected that more advanced systems will be developed, bringing greater innovation and progress to the medical field.

[0228] Example 75: The application of innovative surgical guidance systems means that the boundaries of medical technology are being continuously broken. This will enable medical staff and patients to obtain better medical services and become an important driving force for medical innovation.

[0229] Example 76: Surgical guidance systems offer exciting prospects for the future of medical technology. It is expected that the medical field will develop more advanced guidance systems through the integration of technologies such as real-time image processing, artificial intelligence and robotics.

[0230] Example 77: The various surgical guidance systems and methods described to date are one of the main areas of focus for research and development in the medical field. As medical technology advances, more diverse and innovative surgical guidance systems are expected to emerge.

[0231] Example 78: The application of surgical guidance systems plays an important role in the medical field where patient safety and health are of paramount importance. With the advancement of technology and the efforts of medical staff, surgical guidance systems will continue to advance and develop.

[0232] Example 79: With the development of medical technology, surgical guidance systems are recognized as an important tool to reduce medical costs and improve efficiency. This helps patients receive more affordable and effective medical services.

[0233] Example 80: The surgical guidance system is one of the technologies that represents digitalization and innovation in the medical field. With the development of medical technology, it is expected that more accurate and efficient surgical guidance systems will be developed, which will bring great help to the medical field.

[0234] Example 81: Surgical guidance systems play an important role in improving the quality of medical care. Medical staff can use these systems to perform precise surgeries and protect the health and safety of patients.

[0235] Example 82: Surgical guidance systems encourage and support research and innovation in the medical field. These systems contribute to the development of medical technology and can provide innovative medical services to medical staff and patients.

[0236] Example 83: The application of surgical guidance systems is attracting attention as part of the next generation of medical services in the medical field. Through the integration of innovative technology and medicine, more accurate and personalized medical services can be provided, and patient treatment effects and satisfaction can be improved.

[0237] Various surgical guidance systems and methods help to support and improve surgical procedures in the medical field more accurately and safely. As medical technology advances and research continues, more advanced systems are expected to be developed.

[0238] The embodiments described above can be implemented by hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments can be implemented by, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (digital signal processors), microcomputers, field programmable arrays (FPAs), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions, and can be implemented by more than one general-purpose computer or special-purpose computer. The processing device can execute an operating system (OS) and one or more application software executed in the operating system. In addition, the processing device responds to the execution of the software to access, store, operate, process and generate data. For ease of understanding, it is described as a method with only one processing device, but a person of ordinary skill in the art should understand that the processing device can include multiple processing elements and / or multiple types of processing elements. For example, the processing device can include multiple processors or a processor and a controller. Furthermore, other processing configurations such as parallel processors can also be included.

[0239] Software can include a computer program, code, instructions, or a combination of more than one of them, which can cause a processing device to operate in a desired manner, or, individually or collectively, command a processing device. For interpretation by a processing device or to provide commands or data to a processing device, the software and / or data can be permanently or temporarily embody any type of equipment, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave. The software is distributed on computer systems connected by a network and can be stored or executed in a distributed manner. The software and data can be stored in one or more computer read-write storage media.

[0240] The method according to the embodiment is embodied in the form of a program command that can be executed by a variety of computer means and recorded in a computer read-write medium. The computer read-write medium can include program commands, data files, data structures, etc. in a single or combined form. The program instructions recorded in the medium can be instructions specially designed and constructed for implementing the embodiment, or instructions that ordinary technicians in the field of computer software can use based on known knowledge. The computer read-write recording medium can include magnetic media such as hard disks, floppy disks, and tapes; optical media similar to CD-ROMs, DVDs, etc.; magneto-optical media similar to floptical disks, and hardware devices specially constructed for storing and executing program commands similar to read-only memories (ROMs), random access memories (RAMs), flash memories, etc. Examples of program instructions include not only machine language codes generated by compilers, but also high-level language codes that can be executed by computers by using interpreters, etc.

[0241] The aforementioned hardware devices may be configured to act as one or more software modules in order to perform the operations of the aforementioned embodiments, and vice versa.

[0242] In summary, the embodiments are described through limited drawings, and a person skilled in the art can make various changes and modifications based on the description. For example, the described techniques are performed in a different order from the described method, and / or the described systems, structures, devices, circuits and other components are combined or combined in a different form from the described method, or replaced or substituted by other components or equivalents, and appropriate results can also be obtained.

[0243] Therefore, other embodiments, other examples and equivalents of the claims are all within the scope of the claims of the present invention.

Claims

1. A guide providing system for removing stones, characterized in that: include: A slave device comprising a guide tube insertable into the interior of a ureter to remove stones, a guide wire movable within the guide tube along a longitudinal axis of the guide tube, a drive for moving the guide wire, a medical tool connected to a distal end of the guide wire and exposed to the exterior of the guide tube, and a camera for capturing medical images at the distal end of the guide tube; and A master device comprising a processor which determines the size of a guide graphic representation indicating the size of a stone that can be removed by a basket or laser fiber or needs to be broken up based on an estimated distance between the medical tool and a camera at the distal end of the guide tube, and the master device superimposes and displays the guide graphic representation with the determined size on the medical image captured by the camera and received from the slave device.

2. The guidance providing system according to claim 1, characterized in that: The processor is configured to: inferring an image-based distance between the medical tool and a camera at the distal end of the guide tube from the captured medical image, estimating an encoder-based distance between the medical tool and a camera at the distal end of the guide tube from encoder information of the drive unit, The estimated distance is determined based on the image-based distance and the encoder-based distance.

3. The guidance providing system according to claim 2, characterized in that: The processor is configured to: When the medical tool is a basket, the geometric information of the axis is estimated from the axis region of the basket according to the captured medical image. Based on a machine learning model, distances based on the image are inferred from the estimated geometric information.

4. The guidance providing system according to claim 3, characterized in that: The geometric information of the axis includes at least one of the following: The area occupied by the axis region of the basket in the medical image, the diameter of the estimated circle corresponding to the distal end of the axis, the radius of the estimated circle corresponding to the distal end of the axis, the length from the boundary of the medical image to the distal end of the axis, the length from the center of the medical image to the distal end of the axis, or the inclination of the axis.

5. The guidance providing system according to claim 2, characterized in that: The processor is configured to: Distances based on the medical images are inferred by inputting the medical images into another machine learning model.

6. The guidance providing system according to claim 2, characterized in that: The processor is configured to: The estimated distance is determined by applying a deviation compensation value determined based on the image and the encoder to the encoder-based distance.

7. The guidance providing system according to claim 6, characterized in that: The deviation compensation value is a value that converges with the passage of time after the calculation of the estimated distance is started.

8. The guidance providing system according to claim 1, characterized in that: The processor is configured to: decreasing the size of the guide graphical representation as the estimated distance increases, As the estimated distance decreases, the size of the guide graphical representation is increased.

9. The guidance providing system according to claim 1, characterized in that: The processor is configured to: When the medical tool is a laser fiber, the moment when the distal end of the laser fiber contacts the stone is detected. capturing and storing a target image at the moment when the distal end of the laser optical fiber contacts the stone, The medical image is output to the display of the main device and the target image is output simultaneously.

10. The guidance providing system according to claim 9, characterized in that: The processor is configured to: In response to receiving a user input selecting a point in the target image, the guide graphic representation is output onto the target image based on the selected point.

11. The guidance providing system according to claim 9, characterized in that: The processor is configured to: A stone region in the target image is detected, and the guidance graphical representation is output on the target image based on reference points in the detected stone region.

12. The guidance providing system according to claim 1, characterized in that: The guide graphic is represented as one of the following shapes: Circle, ellipse, quadrilateral, polygon or closed curve figure.

13. The guidance providing system according to claim 1, characterized in that: The size of the guide graphic representation is determined according to the size of a sheath of a basket in the medical tool.

14. The guidance providing system according to claim 1, characterized in that: The border of the guide graphic representation has a thickness, The processor reduces a thickness of a border of the guide graphical representation as reliability of the estimated distance increases over time.

15. A method for providing guidance by a processor, It is characterized in that The following steps are involved: acquiring a medical image captured by including a medical tool in a distal end of a guide tube inserted into the inside of a ureter; inferring an image-based distance between the medical tool and a camera at the distal end of the guide tube from the captured medical image; Acquiring information of an encoder of a driving unit, wherein the driving unit moves the guide wire inside the guide tube; estimating an encoder-based distance between the medical tool and a camera at the distal end of the guide tube from information from the encoder; determining an estimated distance between the medical tool and a camera at the distal end of the guide tube based on the image-based distance and the encoder-based distance; and A guidance graphical representation of a size determined based on the estimated distance is superimposed and displayed on the medical image.

16. The method for providing guidance according to claim 15, characterized in that: The step of inferring the image-based distance comprises the following steps: When the medical tool is a basket or a laser fiber, estimating the geometric information of the axis from the axis region of the medical tool according to the captured medical image; and Based on a machine learning model, distances based on the image are inferred from the estimated geometric information.

17. The method for providing guidance according to claim 15, characterized in that: The step of determining the estimated distance comprises the following steps: determining the estimated distance by applying a deviation compensation value determined based on the image and the encoder to the encoder-based distance, The deviation compensation value is a value that converges with the passage of time after the calculation of the estimated distance starts.

18. The method for providing guidance according to claim 15, characterized in that: The step of superimposing and displaying on the medical image further includes the following steps: adjusting a size of the guide graphical representation based on a change in an estimated distance between the medical tool and a camera at a distal end of the guide tube, Wherein, when the estimated distance increases, the size of the guide graphic representation is reduced; when the estimated distance decreases, the size of the guide graphic representation is increased.

19. The method for providing guidance according to claim 15, characterized in that: The following steps are also included: A medical tool is included in the distal end of a guide tube inserted into the ureter, and a position of the guide on the medical image is changed according to an external input of the user.

20. A method of using the guidance providing system according to claim 1, characterized in that: The following steps are involved: A guide circle is provided only based on the distance of the medical tool, i.e. the basket or laser fiber, and allows the user to compare the size of the stone as it is successively touched.

Images