System and method for processing an electronic medical image to determine an enhanced electronic medical image

Abstract

A system and method for processing an electronic image originating from a medical device includes receiving an image frame originating from the medical device and determining a first color channel and a second color channel in the image frame. A location of an electromagnetic beam halo can be identified by comparing the first color channel and the second color channel. An edge of an electromagnetic beam can be determined based on the electromagnetic beam halo, and a size metric of the electromagnetic beam can be determined based on the edge of the electromagnetic beam. A visual indicator can be displayed on the image frame based on the size metric of the electromagnetic beam.

Description

Technical Field

[0001] Various aspects of the present invention are generally directed to systems and methods useful in planning and / or performing medical procedures. Background Art

[0002] Substantial progress has been made in improving medical outcomes while reducing trauma and risk to patients. Many procedures that once required open surgery can now be performed with less invasive techniques, offering patients less recovery time and risk of infection. Certain procedures requiring biopsy, electrical stimulation, tissue ablation, or removal of natural products or foreign bodies can be performed with minimally invasive procedures.

[0003] For example, in the field of urology, kidney stones or renal calculi can accumulate in the urinary tract and become lodged in the kidneys. Kidney stones are deposits of substances originating from urine, typically minerals and acid salts. While smaller stones may pass naturally from the body, larger stones may require surgical intervention to remove. While open surgery was once the standard treatment for removing stones, other less invasive techniques, such as ureteroscopy and percutaneous nephrolithotomy / nephrolithotomy (PCNL), have emerged as safer and more effective alternatives. Additionally, advances in imaging technology have improved the ability of medical professionals to identify and locate stones before and during surgery. Despite this, medical professionals still have to analyze images to determine the location and size of the stones and whether any stones are present. In addition, images are often obscured, blurred, and / or difficult to assess, making it challenging for medical professionals to discern the presence or size of any stones.

[0004] The systems, devices, and methods of the present invention may remedy some of the deficiencies discussed above and / or address other aspects of the prior art. Summary of the Invention

[0005] Examples of the invention relate, among other things, to medical systems and methods.Each of the examples disclosed herein may include one or more of the features described in conjunction with any of the other disclosed examples.

[0006] In one example, the present invention includes a method for processing an electronic image originating from a medical device, comprising receiving an image frame originating from the medical device and determining a first color channel and a second color channel in the image frame. A location of an electromagnetic beam halo may be identified by comparing the first color channel and the second color channel. An edge of the electromagnetic beam may be determined based on the electromagnetic beam halo, and a size metric of the electromagnetic beam may be determined based on the edge of the electromagnetic beam. A visual indicator may be displayed on the image frame based on the size metric of the electromagnetic beam.

[0007] In another example, the present invention includes a system for processing an electronic image originating from a medical device, the system comprising at least one data storage device storing instructions for processing the electronic image; and at least one processor configured to execute the instructions to perform operations for processing the electronic image. The operations may include processing the electronic image originating from the medical device, comprising receiving an image frame originating from the medical device and determining a first color channel and a second color channel in the image frame. A location of an electromagnetic beam halo may be identified by comparing the first color channel and the second color channel. An edge of the electromagnetic beam may be determined based on the electromagnetic beam halo, and a size metric of the electromagnetic beam may be determined based on the edge of the electromagnetic beam. A visual indicator may be displayed on the image frame based on the size metric of the electromagnetic beam.

[0008] In another example, the present invention includes a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform operations for processing an electronic image originating from a medical device. The operations may include processing an electronic image originating from a medical device, comprising receiving an image frame originating from the medical device and determining a first color channel and a second color channel in the image frame. A location of an electromagnetic beam halo may be identified by comparing the first color channel and the second color channel. An edge of the electromagnetic beam may be determined based on the electromagnetic beam halo, and a size metric of the electromagnetic beam may be determined based on the edge of the electromagnetic beam. A visual indicator may be displayed on the image frame based on the size metric of the electromagnetic beam. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the present invention.

[0010] Figure 1 A medical system according to aspects of the present invention is shown;

[0011] Figure 2 is a flowchart of an exemplary method for processing a medical image according to aspects of the present invention;

[0012] Figure 3 is a flow chart of an exemplary method for determining the size of an object in a medical image according to aspects of the present invention;

[0013] Figure 4 is a flow chart of an exemplary method for medical image enhancement according to aspects of the present invention;

[0014] Figure 5 It shows that according to various aspects of the present invention, Figures 1 to 4 An exemplary system using the techniques discussed in . DETAILED DESCRIPTION

[0015] Examples of the present invention include systems and methods that facilitate and improve the efficiency and safety of minimally invasive surgery. For example, aspects of the present invention can provide users (e.g., physicians, medical technicians, or other healthcare providers) with the ability to more easily identify, size, and therefore remove kidney stones or other substances from a patient's kidneys or other organs. In some embodiments, for example, the present invention can be used to plan and / or perform flexible ureteroscopic surgery with or without the use of laser lithotripsy. The technology discussed herein can also be applied to other medical technologies, such as any medical technology utilizing an endoscope.

[0016] Reference will now be made in detail to the examples of the present invention as described above and illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0017] The terms "proximal" and "distal" are used herein to refer to the relative positions of components of an exemplary medical device or insertion device. As used herein, "proximal" refers to a position relatively closer to the outside of the body or closer to an operator using the medical device or insertion device. Conversely, "distal" refers to a position relatively farther from an operator using the medical device or insertion device, or closer to the inside of the body.

[0018] The foregoing general description and the following detailed description are merely exemplary and illustrative of the claimed features, rather than restrictive descriptions. As used herein, the terms "comprise," "contain," or other variations thereof are intended to encompass non-exclusive content, so that a process, method, article, or apparatus comprising a list of elements includes not only those elements, but may include other elements that are not explicitly listed or that are not inherent to such process, method, article, or apparatus. Additionally, the term "exemplary" is used herein in the sense of "example" rather than "exemplary." As used herein, the terms "approximately," "substantially," and "about" indicate a range of values ​​within + / - 5% of the stated value.

[0019] Figure 1 A medical system 100 is shown that includes a medical device, such as an endoscope or other medical imaging device / medical device 105, a network 110, a user device 115 that can include a display 120 that can be viewed by a user / practitioner / physician / patient 125, and a server that can include a frame processor 135 that can perform the techniques discussed herein. The endoscope 105, the user device 115, and / or the server 130 can be wired (as shown), wirelessly connected, or otherwise communicatively coupled. Alternatively, the functions of the server 130 can be performed on the endoscope 105, the user device 115, etc. The server 130, the endoscope 105, and / or the user device 115 can also comprise a single electronic device.

[0020] like Figure 1 As shown, the endoscope 105 may be an insertion device such as, for example, a ureteroscope (e.g., Boston Scientific's LithoVue TM Digital Flexible Ureteroscope). With the endoscope 105 positioned within the patient's body, for example, through the patient's urethra to reach the patient's kidney, a retrieval device (not shown) can be inserted to retrieve and remove material, such as, for example, kidney stones, with or without the use of laser lithotripsy. The endoscope 105 can record and / or transmit image and / or video data when inserted into the patient's body, and can have a light or other imaging source that can be used to display images of the interior of the patient's blood vessels, organs, etc. A fiber optic cable or other light source can illuminate the interior of the patient. The endoscope 105 can be equipped with or receive a laser that can be projected at a lower power setting so that it acts as an aiming beam. The aiming beam can be used to inform the endoscope user where the laser is aimed without illuminating at a high intensity sufficient to damage tissue or kidney stones. The laser can also emit higher intensity electromagnetic waves based on a signal from the user to perform laser lithotripsy, which can be used to remove, break up, or otherwise destroy one or more organ obstructions, such as kidney stones.

[0021] The displays 120 can be a single display, or at least a dual display having multiple screens or multiple displays on one screen. In one example, one of the displays can display one or more images currently or previously acquired by the endoscope 105. The other display can display images or videos acquired from one or more additional imaging devices 145, such as by x-ray, magnetic resonance imaging, computed tomography, rotational angiography, ultrasound, or another suitable internal imaging device. Alternatively, one of the displays 120 can display an image modified using one or more image enhancement techniques discussed herein, while the other can display an unenhanced image. Alternatively, one of the displays 120 can display an image modified using one or more enhancement techniques discussed herein, while the other of the displays 120 can display an image modified using one or more different enhancement techniques discussed herein.

[0022] The software or application can manipulate, process and interpret the images received from the imaging device 145 to identify the location, size and characteristics of the aiming beam, kidney stones or other objects. As will be discussed further herein, the frame processor 135 can process and enhance the images received from the endoscope 105.

[0023] When performing a medical procedure such as lithotripsy to remove kidney stones, a physician may insert an endoscope 105 into a patient's body. For example, when illuminated by the light on the endoscope 105, the display 120 may be partially or completely obscured by fragments of the kidney stone or other floating particulate matter. Fragments of the kidney stone may need to be removed via the exit channel (access sheath, ureter, etc.) used by the endoscope 105. However, it may be difficult for the physician 125 to determine whether the kidney stone is too large to be discharged from the exit channel and whether it should be further broken up in order to be discharged from the exit channel. The physician may try to remove the kidney stone in question via the exit channel, but if the stone is too large, the sharpness of the stone may cause tissue damage, which may cause harm to the patient, increase recovery time, etc. A technology is needed to more effectively identify whether a kidney stone will be discharged from the exit channel.

[0024] Figure 2 1 is a flow chart of an exemplary method for processing medical images according to aspects of the present invention. A source of video or image frames 205, which can be any medical device such as endoscope 105 or imaging device 145, can provide the frames to signal input 210. The frames can be provided to frame processing program 215, which can store multiple frames. One or more frames 220 can be provided to frame processor 135, which can generate one or more processed frames 230 via the techniques discussed herein. Processed frames 230 can be provided to signal output 235, which can be displayed on display 120.

[0025] Signal input 210 can be a software handler that can transmit a new frame that has been received. Frame handler 215 can send frames directly to display 120 via signal output 235, or it can send one or more frames to frame processor 135. As discussed elsewhere herein, frame processor 135 can perform object size determination techniques. Frame handler 215 can also send the original frame to display 120 and also send a copy of the frame to frame processor 135. Processed frame 230 can be received and also forwarded to display 120. This can allow the original frame to be displayed on display 120 along with the processed frame 230. Alternatively, frame handler 215 can send source frame 220 to frame processor 135, and the frame processor can return processed frame 230 that includes a dual display of the original frame and the enhanced frame. Thus, processed frame 230 can be larger than the source frame. Frame processor 135 can further add buttons or other user interface elements to processed frame 230.

[0026] Although the techniques discussed herein are discussed as occurring on a frame processor 135, which may be depicted as being located on a single device, any of the functionality of the frame processor may be distributed across any number of devices, such as any of the devices depicted in system 100. Furthermore, one or more of signal input 210, frame handler 215, and / or signal output 235 may be hosted on one or more servers 130 or any of the other devices depicted on system 100.

[0027] Figure 3 1 is a flow chart of an exemplary method for processing medical images to determine the size of an object according to aspects of the present invention. A plurality of frames 304 can be received from a frame source. The frame source can include an endoscope 105 or other medical imaging device 145. One or more frames 304 can be accumulated in a frame buffer. The frames can include images of a kidney stone 308 or other object. While the kidney stone 308 can be illuminated by a light source on the tip of the endoscope or other medical device 105, such as an optical fiber emitting white light, a laser beam 316 or other electromagnetic beam can also be present and allow for aiming the tip of the endoscope for purposes such as removing the kidney stone or performing lithotripsy. When illuminated, the light beam 316 can cause a halo 312 to be visible around the beam, which may be due to the intensity of the laser. Steps can then be taken to determine the size of the light beam 316.

[0028] It is possible to distinguish halo 312 from beam 316. Because laser light typically has a specific color, such as red, there may be no trace or a faint trace of beam 316 in the blue or green "unmatched" channels 322, and no trace of halo 312. Conversely, in the "matched" red channel 326, the laser light and halo 312 may appear together as a single, bright, and indistinguishable entity. Similarly, a green laser light and its halo may appear as a single, bright, indistinguishable entity in the green channel, while at least the lower-intensity halo does not appear in the red or blue channels, etc. Therefore, to separate the color channels and perform the techniques discussed herein, the color of the laser light can be initially determined. Furthermore, endoscopes may have white light from a fiber optic cable to aid user navigation. This white light may create reflections that could be confused by the algorithm for an aiming beam. This problem can be avoided by comparing different color channels to find the aiming beam. White light will appear equally in all color channels. Colored laser light, or at least the halo, may appear primarily in the corresponding color channel.

[0029] Even though the aiming beam may be a laser of a specific color, it may be intense in all color channels. This can make it difficult to distinguish the aiming beam from other light reflections, such as those originating from an LED on an endoscope. However, the aiming beam may have an associated halo around it, which can allow it to be distinguished from other light sources or reflections. Furthermore, the halo may only appear in specific color channels, such as the laser color. Therefore, haloes can be reliably identified by comparing different color channels. Once the beam of the matching channel 326 (e.g., the red channel for a red laser) and the smaller beam 322 of the non-matching channel (the green and / or blue channels for a red laser) 322 are determined, the two channels can be combined to form a masked area to more accurately determine the halo. This can be accomplished by subtracting the non-matching channel 322 from the matching channel 326, resulting in an image at 329 with the halo of beam 316 removed. Alternatively, one of the channels can be inverted at 327 to form an inverted channel 328. The inverted channel 328 can be added to, for example, the non-matching channel to form halo 329. A bounding box or other border may be placed around the halo / mask region 330 for further image analysis.

[0030] After determining bounding box 330, aiming beam 339 can be determined within the center of the halo and the bounding box. Image artifacts 340 may also be present. For example, artifacts 340 may occur if the laser reflects off an object near the aiming beam itself. An algorithm comparing various candidate aiming beams can deem the largest object to be the true aiming beam 339 and discard or ignore artifacts 340. The edges of true aiming beam 339 can be determined using algorithms such as Canny edge detection, Hough transform, etc. The algorithm can then approximate the true shape and size of aiming beam 339 by placing a circle or ellipse around the aiming beam to form aiming beam ellipse 342. This can be done dynamically, or the shape of the aiming beam can be previously known. The placement of this ellipse can be performed based on the determined edges of the aiming beam. Multiple candidate ellipses can be fitted to the detected edges, and the best-fitting ellipse, such as aiming beam ellipse 342, can be determined. Metrics, such as major and minor axis measurements, can be determined for aiming beam ellipse 342.

[0031] The aiming beam can have a standard predetermined size and shape. Because the aiming beam 316 is a laser or other highly directional or unidirectional electromagnetic light source, it does not increase significantly with distance. Therefore, any object projected by the aiming beam can be measured using the aiming beam and / or its halo as a standard metric. Therefore, kidney stones or other objects can be measured by evaluating the aiming beam and / or halo. However, difficulties may arise that complicate accurate measurement. Kidney stones are often jagged and contain gaps or other irregularities that distort the apparent shape of the aiming beam. As a result, the aiming beam may appear abnormally small or abnormally large, which may lead to an incorrect estimate of the size of any object measured based on the aiming beam.

[0032] To alleviate this problem, an image of the surface at which the aiming beam is aimed can be evaluated via image analysis within a bounding box or otherwise within a predetermined distance of the aiming beam 316 and / or halo 312 and extracted at 334. Image features can include image lighting, texture, entropy, artifact detection, etc. The extracted image features plus the major and minor axes of the ellipse 342 can be provided to a trained machine learning system, which can apply weights to alter the estimated size of the aiming beam 316 and / or halo 312.

[0033] The machine learning system can be trained. The size of the ground truth indicator (e.g., in pixels) can be determined. The ground truth indicator can be a circle or other shape physically drawn on the kidney stone image that represents the true size of the aiming beam. During the training process, the size of the physically drawn circle is extracted from the image. Correlations between the size of the ground truth indicator and the aiming beam characteristics (major and minor axes of the ellipse 342, image lighting, texture, entropy, artifact detection, etc.) can be determined. In the inference step of the production version, the machine learning model may calculate the size of the aiming beam. This can be performed accurately despite the distortion of the surface projected by the aiming beam.

[0034] As discussed above, the dimensions of the aiming beam may be known. For example, the aiming beam may be a laser that, when shone on a surface, creates a 5mm wide "spot." Because lasers are directional beams, the spot will likely be 5mm wide regardless of the distance from the object it illuminates. Therefore, based on the final estimated shape and / or dimensions of the aiming beam, measurements of the surface it is projected on can be determined. This can be accomplished by evaluating the object's size relative to the known dimensions of the laser spot. For example, if the laser spot is known to be 5mm wide, and the kidney stone it illuminates is approximately twice the width of the laser spot, the kidney stone can be determined to be 10mm wide.

[0035] Using the same technique, the size of the exit passage can be determined and displayed to user 125. Thus, using this scaling technique, the size of an object can be determined, and the exact size of the projected virtual object can also be determined. A visual indicator 350 representing the size of the exit passage relative to the surface on which the aiming beam is projected can be displayed on the display. User 125 can thus determine whether a kidney stone or other object illuminated by the aiming beam will fit into the exit passage visually and with minimal cognitive load. Alternatively or additionally, one or more rulers can be displayed, such as rulers 354 along the X-axis or Y-axis. A visual indicator of the bounding box 330 and / or the final aiming beam ellipse 342 can also be displayed to user 125 on the display.

[0036] Figure 4 is a flow chart of an exemplary method for determining medical image enhancement according to the techniques discussed herein. In step 405, an image frame can be received from a medical device, and in step 410, a first color channel and a second color channel in the image frame can be determined. In step 415, a position of an electromagnetic beam halo can be identified by comparing the first color channel and the second color channel. In step 420, an edge of the electromagnetic beam can be determined based on the electromagnetic beam halo, and in step 425, a size metric of the electromagnetic beam can be determined based on the edge of the electromagnetic beam. In step 430, a visual indicator or other projection can be displayed on the image frame based on the size metric of the electromagnetic beam. As discussed elsewhere herein, the visual indicator can include an exit channel, a ruler, a bounding box, an electromagnetic beam halo, and / or a visual representation of the electromagnetic beam.

[0037] Figure 5 It shows that according to various aspects of the present invention, Figures 1 to 4 An exemplary system using the techniques discussed in . Figure 51 is a simplified functional block diagram of a computer that can be configured as a server 130, an endoscope 105, an imaging device 145, and / or a user device 115, according to an exemplary embodiment of the present invention. Specifically, in one embodiment, any of the user devices, servers, etc. discussed herein can be an assembly of hardware 500, including, for example, a data communication interface 520 for packet data communication. The platform can also include a central processing unit ("CPU") 502 in the form of one or more processors for executing program instructions. The platform can include an internal communication bus 508 and a storage unit 506 (such as a ROM, HDD, SDD, etc.) that can store data on a computer-readable medium 522, although the system 500 can also receive programming and data via network communications. The system 500 can also have a memory 504 (such as RAM) that stores instructions 524 for performing the techniques presented herein, although the instructions 524 can also be temporarily or permanently stored within other modules of the system 500 (e.g., the processor 502 and / or the computer-readable medium 522). The system 500 may also include input and output ports 512 and / or a display 510 for connecting to input and output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Various system functions may be implemented in a distributed manner on multiple similar platforms to distribute the processing load. Alternatively, the system may be implemented by appropriate programming of a single computer hardware platform.

[0038] The disclosed technology can help enable efficient and effective surgery to break down and / or remove matter from a patient's organs. In particular, a user can easily view processed frames to assist, for example, in removing a kidney stone within a patient's kidney.

[0039] Furthermore, while the examples discussed herein are generally directed to ureteroscopic renal stone removal with or without lithotripsy, it is also contemplated that the systems and procedures discussed herein may be equally applicable to other material removal procedures. For example, the systems and methods discussed above may be used during percutaneous nephrolithotomy / nephrolithotripsy (PCNL) to plan and locate any missed renal stones during surgery. The systems and methods discussed above may also be used to plan or perform surgery to remove ureteral stones, gallstones, bile duct stones, and the like.

[0040] Although the principles of the present invention have been described herein with reference to illustrative examples for specific applications, it should be understood that the present invention is not limited thereto. Personnel with ordinary skill in the art and access to the teachings provided herein will recognize that additional modifications, applications, embodiments, and equivalents are within the scope of the features described herein. Therefore, the claimed features should not be construed as being limited to the foregoing description.

Claims

1. A system for processing an electronic image originating from a medical device, comprising: at least one data storage device storing instructions for processing electronic images; as well as at least one processor configured to execute the instructions to perform operations for processing an electronic image, the operations comprising: receiving an image frame from the medical device; determining a first color channel and a second color channel in the image frame; identifying a location of an electromagnetic beam halo by comparing the first color channel and the second color channel; determining an edge of the electromagnetic beam based on the electromagnetic beam halo; determining a dimensional metric of the electromagnetic beam based on the edge of the electromagnetic beam; and A visual indicator is displayed on the image frame based on the size metric of the electromagnetic beam.

2. The system of claim 1 , wherein the operations further comprise: extracting features from the image frame within a predetermined distance of the electromagnetic beam and / or electromagnetic beam halo; as well as A size of the visual indicator on the image frame is determined based on the extracted features. 3 . The system of claim 2 , wherein the size of the visual indicator on the image is determined based on a weight of each of the extracted features.

4. The system of claim 1 , wherein comparing the first color channel and the second color channel further comprises: determining that the first color channel depicts the electromagnetic beam and the electromagnetic beam halo; determining that the second color channel depicts the electromagnetic beam; as well as The second color channel is subtracted from the first color channel to identify the position of the electromagnetic beam halo.

5. The system of claim 1 , wherein comparing the first color channel and the second color channel further comprises: determining that the first color channel depicts the electromagnetic beam and the electromagnetic beam halo; determining that the second color channel depicts the electromagnetic beam; as well as The second color channel and the inverse of the first color channel are added to identify the position of the electromagnetic beam halo.

6. The system of claim 1 , wherein determining the edge of the electromagnetic beam comprises: determining an approximate electromagnetic beam based on the electromagnetic beam halo; as well as The approximate electromagnetic beam is distinguished from the image artifact based on a size of the approximate electromagnetic beam relative to the image artifact.

7. The system of claim 1 , wherein determining the edge of the electromagnetic beam comprises: determining an approximate electromagnetic beam based on the electromagnetic beam halo; distinguishing the approximate electromagnetic beam from the image artifact based on a size of the approximate electromagnetic beam relative to the image artifact; as well as A circle or ellipse is fitted to the approximate electromagnetic beam.

8. The system of claim 1, wherein the first color channel is determined to match a predetermined color of the electromagnetic beam, and wherein the second color channel is determined not to match the predetermined color of the electromagnetic beam.

9. The system of claim 1, wherein the electromagnetic beam is a laser.

10. The system of claim 1, wherein the size metric of the electromagnetic beam corresponds to a diameter or radius of the electromagnetic beam.

11. The system of claim 1 , wherein the visual indicator on the image frame corresponds to a size of an exit passage through which the object passed.

12. The system of claim 1, wherein the image frames are acquired using the medical device while planning or performing a procedure to remove an object from a patient via an exit passage, and wherein the visual indicator corresponds to a size of the exit passage.

Images