Systems and methods for reducing artifacts in medical imaging - Patents.com

Abstract

An exemplary method for imaging tissue of a subject using a rolling shutter imager to provide a video stream includes sequentially resetting a number of pixel rows of the rolling shutter imager from a first row to a last row; transitioning a liquid crystal shutter from a closed state to an open state; illuminating the tissue of the subject with illumination light for an illumination period to accumulate charge in the number of pixel rows after the liquid crystal shutter has been transitioned to the open state and after resetting the last row; sequentially reading the accumulated charge in the pixel rows from the first row to the last row after completion of the illumination period; generating an image frame from the accumulated charge in the sequentially read number of pixel rows; and appending the image frame to the video stream.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to medical imaging, and more particularly to techniques for reducing or eliminating artifacts caused by light events in medical imaging. [Background technology]

[0002] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 63 / 132,444, filed December 30, 2020, the entire contents of which are incorporated herein by reference.

[0003] Medical systems, instruments or tools are utilized before, during or after surgery for various purposes. Some of these medical tools may be used in what are commonly referred to as endoscopic procedures. For example, endoscopy in the medical field allows the internal features of a patient's body to be viewed without the use of traditional fully invasive surgery. Endoscopic imaging systems incorporate endoscopes to allow the surgeon to view the surgical site, and endoscopic tools allow for non-invasive surgery at that site. Endoscopes may be used with camera systems to process images received by the endoscope. Endoscopic camera systems typically include a camera head connected to a camera control unit (CCU), which processes input image data received from the camera's imaging sensor via a cable and outputs image data for display. The CCU may control an illuminator that generates illumination light provided to the imaging scene.

[0004] A variety of imager sensors may be used in endoscopic imaging systems, including charge-coupled device (CCD) sensors and complementary metal-oxide semiconductor (CMOS) sensors. The structure of a CCD is generally more complex than that of a CMOS sensor, and CMOS sensors may be manufactured in high-volume wafer fabrication facilities used for related technologies such as microprocessors and chipsets. As a result, CMOS sensors often cost less than CCDs for comparable performance. In addition to the low cost, the common manufacturing processes used to create CMOS imagers allow CMOS pixel arrays to be integrated on a single circuit with other electronic devices such as clock drivers, digital logic, analog-to-digital converters, and other suitable electronics. The compact structure possible with CMOS imagers can also reduce space requirements and lower power consumption. CMOS imagers also have higher sensitivity and can provide higher video frame rates.

[0005] CMOS-based imagers may utilize an electronic rolling shutter to expose pixels in a sensor array. In an electronic rolling shutter, rows of pixels are cleared (or reset), exposed, and read out in sequence. During integration, the rows of pixels are exposed to light energy and each pixel forms an electrical charge corresponding to the amount and wavelength of light that strikes the pixel. Because the columns are activated and read out in sequence, there is a time lapse between the first row being integrated and the last row being integrated. Because there is a time lapse between the first row starting to integrate and subsequent rows starting to integrate, CMOS imagers with electronic rolling shutters may capture video images that have blurring and other rolling shutter effects. CMOS-based imagers may also utilize a global shutter to expose pixels in a sensor array. In a global shutter, all rows of pixels are exposed at the same time (i.e., exposure begins and ends at the same time), but readout may be (usually) sequential.

[0006] During imaging, unintended and / or undesired light events may occur, causing artifacts in the image. For example, a holmium laser, a laser surgery tool that can be used to remove stones in urological procedures, may produce a brief, intense emission of light (e.g., at visible or IR wavelengths) when interacting with tissue. The light events may produce artifacts in the resulting endoscopic image, such as FIG. 4B described herein. Other tissue ablation devices, such as green light lasers and RF probes, may cause similar artifacts. The duration, repetition rate, and amplitude of the light events may depend on the energy device.

[0007] Unintended / undesired light events can affect imaging in both sensors with a global shutter (e.g., CCD sensors) and sensors with a rolling shutter (e.g., CMOS sensors), but may be more pronounced when using a rolling shutter, as described herein. Additionally, unintended / undesired light events can affect various types of frames, including white light frames and fluorescent image frames. Summary of the Invention

[0008] Disclosed herein are exemplary devices, apparatus, systems, methods, and non-transitory storage media for medical imaging. More generally, exemplary devices, systems, and methods for reducing or eliminating artifacts caused by unintended / undesirable light events are disclosed. The systems, devices, and methods may be used to image tissue of a target, such as in an endoscopic imaging procedure. The imaging may be performed pre-operatively, intra-operatively, post-operatively, and during diagnostic imaging sessions and procedures. The imaging method itself may exclude the insertion of an endoscopic imaging device into a lumen in the body. The endoscopic imaging device may be inserted into a lumen prior to the imaging method. The imaging method itself may exclude any invasive surgical steps.

[0009] Although some of the present techniques are disclosed with respect to particular types of imagers (e.g., rolling shutter imagers, global shutter imagers), it should be understood that the techniques may be applied to any type of imager. Additionally, the present techniques may be applied to non-surgical or non-medical uses.

[0010] An exemplary system may have a "synchronous frame reset" function. In a rolling shutter imager, with the synchronous frame reset function, all rows of pixels in the imaging sensor may be reset simultaneously or within a short time (e.g., less than the rolling shutter's line-by-line offset period). The rows of pixels are then illuminated for an illumination period less than the time between the synchronous frame reset and the first row read, allowing charge to accumulate in the rows of pixels simultaneously for the same amount of time, achieving a global shutter effect. The accumulated charge is then read out sequentially from the first row to the last row to generate an image frame. The image frame may then be added to the video stream. The synchronous frame reset function reduces the period during which light accumulates in the sensor to less than the frame period, thus greatly reducing the effect of light events. Unintended / unwanted light events that occur before the synchronous frame reset do not affect the image frame. In some examples, the synchronous frame reset step eliminates impulsive light events by about 46%.

[0011] Exemplary imaging systems may include various techniques for detecting artifacts in image frames. These techniques may include detecting horizontal or vertical lines in an image frame depending on the mounting orientation of the sensor, detecting a rate of increase in the average value in successive image frames, detecting an increase in saturated pixels in successive image frames, detecting a discrepancy between color channels, detecting an increase in the amount of light in a field stop region, or any combination thereof. In some examples, machine learning models may be used to detect artifacts in image frames, including machine learning models configured to detect the characteristics described above.

[0012] An exemplary system may use a buffer of size n. A sequence of frames captured by an imaging system may be accumulated in the buffer. Each image accumulated in the buffer may be associated with a respective score. The score indicates how likely the image is to be corrupted by a light event. Based on a comparison of the scores, an image frame may be dropped and replaced in the image sequence. The replacement image may be another image in the buffer or a repaired version of the dropped image frame. Advantageously, an N-frame buffer may reduce false positives and maintain a low drop count. In some examples, the image frames may be repaired using one or more machine learning models. For example, a trained image transformation model may be configured to receive a corrupted image frame and generate an output image in which artifacts are reduced or removed. Additionally or alternatively, the trained machine learning model may be configured to repair the image frame by correcting artifact regions based on information from other image frames in the buffer.

[0013] Exemplary systems may include shutters (e.g., liquid crystal or LC shutters, mechanical shutters, DLP mirrors, electro-mechanical shutters). Shutters may be used with rolling shutter imagers to block light after the illumination period (and thus any unintended / undesired light events) to eliminate the effects of light events occurring during sequential readout of rows. Shutters with pulse width control may be used to split the exposure time into multiple shorter periods to reduce the impact of light events on the frame. Shutters may also be operated as standalone devices without communication with the camera and block light from the imaging path when a light event is detected.

[0014] Any of the techniques described herein may be activated, deactivated, or adjusted in terms of their sensitivity / aggression. In some examples, a technique may be activated or its sensitivity may be increased in response to detection of a light event and / or detection of activation of a surgical energy device. A light event may be detected in a number of ways as described herein. It will be understood that any of the variations, aspects, features, and options described in terms of a system apply equally to the method, and vice versa. It will also be apparent that any one or more of the above variations, aspects, features, and options may be combined.

[0015] According to one aspect, a method is provided for imaging tissue of a subject using a rolling shutter imager to provide a video stream, the method including: resetting a number of pixel rows of the rolling shutter imager within a period that is shorter than a line-by-line offset period of the rolling shutter imager; illuminating the tissue of the subject with illumination light for an illumination period after resetting the number of pixel rows to accumulate charge in the number of pixel rows; sequentially reading out the accumulated charge in the number of pixel rows from a first row to a last row; generating an image frame from the accumulated charge in the sequentially read number of pixel rows; and adding the image frame to the video stream.

[0016] Optionally, resetting the plurality of pixel rows of the rolling shutter imager includes triggering a synchronous frame reset function of the rolling shutter imager, the synchronous frame reset function may include a configurable constant parameter indicating a minimum amount of time the plurality of pixel rows are exposed after a trigger, the illumination period being configured to be shorter than the constant parameter, the constant parameter being dynamically adjustable for different image frames.

[0017] Optionally, the method further includes determining whether the image frame satisfies one or more criteria, adding the image frame to the video stream in accordance with a determination that the image frame does not satisfy the one or more criteria, and discarding the image frame in accordance with a determination that the image frame satisfies the one or more criteria. Discarding the image frame may include removing the image frame from the video stream and adding a replacement image frame to the video stream.

[0018] Optionally, the method further includes processing the image frame with a first configuration of an automatic gain control (AGC) algorithm in response to a determination that the image frame does not satisfy the one or more criteria, and processing the image frame with a second configuration of the AGC algorithm or processing the image frame with the AGC algorithm in response to a determination that the image frame satisfies the one or more criteria. Determining whether the image frame satisfies one or more criteria may include identifying one or more artifacts in the image frame. The one or more artifacts may be identified in real time. Identifying one or more artifacts in the image frame may include identifying a line in the image frame.

[0019] Optionally, the method further comprises applying a Sobel filter to the image frames.

[0020] Optionally, identifying one or more artifacts in the image frame comprises calculating a rate of increase from an average value of one or more previous image frames to an average value of the image frame, the rate of increase being calculated with respect to a region of interest in the image frame.

[0021] Optionally, identifying one or more artifacts in the image frame includes calculating an increase from a number of saturated pixels in a previous image frame to a number of saturated pixels in the image frame, the increase being calculated for a region of interest in the image frame.

[0022] Optionally, identifying one or more artifacts in the image frame includes evaluating a difference between at least two of a red channel, a blue channel and a green channel of the image frame.

[0023] Optionally, identifying one or more artifacts in the image frames includes processing the image frames with a trained machine learning algorithm, which may be a neural network.

[0024] Optionally, identifying one or more artifacts in the image frame includes detecting an increase in the amount of light within a field stop region of the image frame.

[0025] Optionally, the method further includes placing the image frames in a buffer of a predefined size, comparing all frames placed in the buffer, and excluding one or more frames in the buffer from the video stream based on the comparison. The predefined size may be three image frames. Comparing all frames placed in the buffer may include assigning a score to each of the frames placed in the buffer and comparing the scores of all frames placed in the buffer.

[0026] Optionally, the method further includes automatically adjusting one or more parameters of the method, the one or more parameters may include one or more thresholds for identifying artifacts in the image frames, a size of an image buffer, a maximum number of image frames that can be dropped from the image buffer, a maximum number of consecutive image frames that can be dropped from the image buffer, or any combination thereof.

[0027] Optionally, the method further includes automatically adjusting the one or more parameters in response to detecting a light event or detecting an actuation of a surgical energy device. Detecting actuation of the surgical energy device can include receiving a signal from the surgical energy device. The surgical energy device is a laser unit. The surgical energy device is an RF probe. Detecting actuation of the surgical energy device can include detecting an increase in power consumption of the surgical energy device. Detecting the light event can include receiving a signal from a photodetector attached to the rolling shutter imager. Detecting actuation of the surgical energy device can include receiving an acoustic signal from the surgical energy device.

[0028] Optionally, the method further comprises automatically adjusting the one or more parameters based on detected movement of the imaging device.

[0029] Optionally, the imaging device comprises a shutter component, the shutter being configured to close at the end of the illumination period.

[0030] Optionally, the shutter component comprises a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electro-mechanical shutter.

[0031] Optionally, the illumination light is generated by at least one LED.

[0032] Optionally, the rolling shutter imaging device is part of an endoscopic imaging device.

[0033] According to one aspect, a computer-implemented method for providing a video stream is provided, the method including: accumulating an image sequence in a buffer of a predefined size, each image of the sequence having an individual score associated therewith; comparing the scores of the image sequences in the buffer; identifying an image of the image sequence based on the comparison; removing the identified image from the image sequence to obtain an updated sequence; and adding the updated image sequence to the video stream.

[0034] Optionally, the sequence of images is acquired by a rolling shutter imager.

[0035] Optionally, the sequence of images is acquired by a global shutter imager configured to read multiple rows of pixels simultaneously.

[0036] Optionally, the image sequence is acquired by an endoscopic imaging device.

[0037] Optionally, the method further comprises replacing the identified image with a replacement image in the sequence of images.

[0038] Optionally, the method further comprises, for each image in the image sequence, identifying one or more artifacts in the individual image and assigning the individual score to the image based on the identification. Identifying one or more artifacts in the image may comprise identifying a line in the image. Identifying one or more artifacts in the image may comprise calculating a rate of increase from an average value of one or more previous images to an average value of the image. The rate of increase may be calculated with respect to a region of interest in the image frame.

[0039] Optionally, identifying one or more artifacts in the image includes calculating an increase from a number of saturated pixels in a previous image to a number of saturated pixels in the image, the increase being calculated for a region of interest in the image frame.

[0040] Optionally, identifying one or more artifacts in the image comprises assessing a difference between at least two of a red channel, a blue channel and a green channel of the image frame.

[0041] Optionally, identifying one or more artifacts in the image comprises processing the image with a trained machine learning algorithm, which may be a neural network.

[0042] Optionally, identifying one or more artifacts in the image frame includes detecting an increase in the amount of light within a field stop of the image frame.

[0043] Optionally, the method further comprises automatically adjusting the predefined size of the buffer.

[0044] Optionally, the method further includes automatically increasing the predefined size of the buffer in response to detecting a light event or detecting an actuation of a surgical energy device.

[0045] According to one aspect, there is provided a method of generating an image using an endoscopic imaging device, the method including: accumulating charge in a pixel array of the endoscopic imaging device during a frame period, deactivating a shutter component n times for less than or equal to one nth of a predefined exposure time to allow light to pass through the shutter, and generating the image from a reading of the charge accumulated in the pixel array after deactivating the shutter component the nth time. The method may exclude insertion of the endoscopic imaging device into a lumen in a body. The endoscopic imaging device may be inserted into a lumen prior to the method.

[0046] Optionally, the shutter component comprises a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electro-mechanical shutter.

[0047] Optionally, the endoscopic imaging device is a rolling shutter imaging device.

[0048] Optionally, the endoscopic imaging device is a global shutter imaging device.

[0049] Optionally, the stops of the shutter are separated by at least the predefined exposure period.

[0050] Optionally, the method further comprises automatically adjusting the value of n. The value of n may be 1 or greater.

[0051] According to one aspect, a method of shielding an endoscopic imaging device from a light event is provided, the method including detecting the light event and, in response to detecting the light event, actuating a shutter to shield a sensor of the endoscopic imaging device from the light event.

[0052] Optionally, the shutter comprises a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electro-mechanical shutter.

[0053] Optionally, detecting the light event includes detecting the light event via a photodiode detector.

[0054] Optionally, the light event includes infrared light.

[0055] Optionally, the light event is generated by a laser, the laser being a Holmium laser.

[0056] Optionally, the method further comprises automatically stopping the shutter after a predefined period of time, the predefined period of time being between about 500us and 1ms, the predefined period of time being dynamically adjustable.

[0057] Optionally, the method further includes automatically deactivating the shutter component in response to detecting the absence of the light event.

[0058] Optionally, the endoscopic imaging device is a rolling shutter imaging device.

[0059] Optionally, the endoscopic imaging device is a global shutter imaging device.

[0060] According to one aspect, a system for imaging tissue of a subject to provide a video stream is provided, the system comprising: an illumination source; and an imaging device having an electronic rolling shutter imager, the imaging device configured to reset a plurality of pixel rows of the rolling shutter imager within a period shorter than a line offset period of the rolling shutter imager, illuminate the tissue of the subject with illumination light for an illumination period to accumulate charge in the plurality of pixel rows after resetting the plurality of pixel rows, sequentially read out the accumulated charge in the plurality of pixel rows from a first row to a last row, generate an image frame from the accumulated charge in the sequentially read plurality of pixel rows, and add the image frame to the video stream.

[0061] Optionally, resetting the plurality of pixel rows of the rolling shutter imager includes triggering a synchronous frame reset function of the rolling shutter imager, the synchronous frame reset function may include a configurable constant parameter indicating a minimum amount of time the plurality of pixel rows are exposed after a trigger, the illumination period being configured to be shorter than the constant parameter, the constant parameter being dynamically adjustable for different image frames.

[0062] Optionally, the imaging device is further configured to determine whether the image frame satisfies one or more criteria, add the image frame to the video stream in accordance with a determination that the image frame does not satisfy the one or more criteria, and discard the image frame in accordance with a determination that the image frame satisfies the one or more criteria. Discarding the image frame may include removing the image frame from the video stream and adding a replacement image frame to the video stream.

[0063] Optionally, the imaging device is further configured to process the image frame using a first configuration of an automatic gain control (AGC) algorithm in response to a determination that the image frame does not satisfy the one or more criteria, and to process the image frame using a second configuration of the AGC algorithm or to process the image frame using the AGC algorithm in response to a determination that the image frame satisfies the one or more criteria. Determining whether the image frame satisfies one or more criteria may include identifying one or more artifacts in the image frame. The one or more artifacts may be identified in real time.

[0064] Optionally, identifying one or more artifacts in the image frame includes identifying a line in the image frame.

[0065] Optionally, the imaging device is further configured to apply a Sobel filter to the image frames.

[0066] Optionally, identifying one or more artifacts in the image frame comprises calculating a rate of increase from an average value of one or more previous image frames to an average value of the image frame, the rate of increase being calculated with respect to a region of interest in the image frame.

[0067] Optionally, identifying one or more artifacts in the image frame includes calculating an increase from a number of saturated pixels in a previous image frame to a number of saturated pixels in the image frame, the increase being calculated for a region of interest in the image frame.

[0068] Optionally, identifying one or more artifacts in the image frame includes evaluating a difference between at least two of a red channel, a blue channel and a green channel of the image frame.

[0069] Optionally, identifying one or more artifacts in the image frames includes processing the image frames with a trained machine learning algorithm, which may be a neural network.

[0070] Optionally, identifying one or more artifacts in the image frame includes detecting an increase in the amount of light within a field stop region of the image frame.

[0071] Optionally, the imaging device is further configured to place the image frames in a buffer of a predefined size, compare all frames placed in the buffer, and exclude one or more frames in the buffer from the video stream based on the comparison. The predefined size may be three image frames. Comparing all frames placed in the buffer may include assigning a score to each of the frames placed in the buffer and comparing the scores of all frames placed in the buffer.

[0072] Optionally, the imaging device may be further configured to automatically adjust one or more parameters of the method, including one or more thresholds for identifying artifacts in the image frames, a size of an image buffer, a maximum number of image frames that can be dropped from the image buffer, a maximum number of consecutive image frames that can be dropped from the image buffer, or any combination thereof.

[0073] Optionally, the imaging device is further configured to automatically adjust the one or more parameters in response to detecting a light event or detecting an actuation of a surgical energy device. Detecting the actuation of the surgical energy device can include receiving a signal from the surgical energy device. The surgical energy device is a laser unit. The surgical energy device is an RF probe. Detecting the actuation of the surgical energy device can include detecting an increase in power consumption of the surgical energy device. Detecting the light event can include receiving a signal from a photodetector attached to the rolling shutter imaging device. Detecting the actuation of the surgical energy device can include receiving an acoustic signal from the surgical energy device.

[0074] Optionally, the imaging device is further configured to automatically adjust the one or more parameters based on detected movement of the imaging device.

[0075] Optionally, the imager comprises a shutter component, the shutter configured to close at the end of the illumination period. The shutter component may include a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electro-mechanical shutter.

[0076] Optionally, the illumination light is generated by at least one LED.

[0077] Optionally, the rolling shutter imaging device is part of an endoscopic imaging device.

[0078] According to one aspect, there is provided a system for providing a video stream having one or more processors, a memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for accumulating an image sequence in a buffer of a predefined size, each image of the sequence having an individual score associated therewith; comparing the scores of the image sequences in the buffer; identifying images of the image sequence based on the comparison; removing the identified images from the image sequence to obtain an updated sequence; and adding the updated image sequence to the video stream.

[0079] Optionally, the sequence of images is acquired by a rolling shutter imager.

[0080] Optionally, the sequence of images is acquired by a global shutter imager configured to read multiple rows of pixels simultaneously.

[0081] Optionally, the image sequence is acquired by an endoscopic imaging device.

[0082] Optionally, the one or more programs further include instructions for replacing the identified image with a replacement image in the sequence of images.

[0083] Optionally, the one or more programs further include instructions for, for each image in the image sequence, identifying one or more artifacts in the individual image and assigning the individual score to the image based on the identification.

[0084] Optionally, identifying one or more artifacts in the image includes identifying lines in the image.

[0085] Optionally, identifying one or more artifacts in the image comprises calculating a rate of increase from an average value of one or more previous images to an average value of the image, the rate of increase being calculated for a region of interest in the image frame.

[0086] Optionally, identifying one or more artifacts in the image includes calculating an increase from a number of saturated pixels in a previous image to a number of saturated pixels in the image, the increase being calculated for a region of interest in the image frame.

[0087] Optionally, identifying one or more artifacts in the image comprises assessing a difference between at least two of a red channel, a blue channel and a green channel of the image frame.

[0088] Optionally, identifying one or more artifacts in the image comprises processing the image with a trained machine learning algorithm, which may be a neural network.

[0089] Optionally, identifying one or more artifacts in the image frame includes detecting an increase in the amount of light within a field stop of the image frame.

[0090] Optionally, the one or more programs further include instructions for automatically adjusting the predefined size of the buffer.

[0091] Optionally, the one or more programs further include instructions for automatically increasing the predefined size of the buffer in response to detecting a light event or detecting activation of a surgical energy device.

[0092] According to one aspect, a system for generating an image using an endoscopic imaging device is provided, the system comprising: a shutter component; and an imaging device configured to accumulate charge in a pixel array of the endoscopic imaging device during a frame period, stop the shutter component n times for less than or equal to 1 / n of a predefined exposure time to allow light to pass through the shutter, and generate the image from a reading of the charge accumulated in the pixel array after the nth stop of the shutter component.

[0093] Optionally, the shutter component comprises a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electro-mechanical shutter.

[0094] Optionally, the endoscopic imaging device is a rolling shutter imaging device.

[0095] Optionally, the endoscopic imaging device is a global shutter imaging device.

[0096] Optionally, the stops of the shutter are separated by at least the predefined exposure period.

[0097] Optionally, the imaging device is configured to automatically adjust the value of n, the value of n being 1 or greater.

[0098] According to one aspect, a system for shielding an endoscopic imaging device from a light event is provided, the system comprising: a shutter; and an imaging device configured to detect the light event and, in response to detecting the light event, actuate the shutter to shield a sensor of the endoscopic imaging device from the light event.

[0099] Optionally, the shutter comprises a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electro-mechanical shutter.

[0100] Optionally, detecting the light event includes detecting the light event via a photodiode detector.

[0101] Optionally, the light event includes infrared light.

[0102] Optionally, the light event is generated by a laser, the laser being a Holmium laser.

[0103] Optionally, the image capture device is configured to automatically stop the shutter after a predefined period of time, which may be between about 500us and 1ms, and which may be dynamically adjustable.

[0104] Optionally, the imaging device is configured to automatically deactivate the shutter component in response to detecting the absence of the light event.

[0105] Optionally, the endoscopic imaging device is a rolling shutter imaging device.

[0106] Optionally, the endoscopic imaging device is a global shutter imaging device.

[0107] According to one aspect, a method is provided for imaging tissue of a subject using a rolling shutter imager to provide a video stream, the method including: sequentially resetting a plurality of pixel rows of the rolling shutter imager from a first row to a last row; transitioning a liquid crystal shutter from a closed state to an open state; illuminating the tissue of the subject with illumination light for an illumination period to accumulate charge in the plurality of pixel rows after the liquid crystal shutter has been transitioned to the open state and after resetting the last row; sequentially reading the accumulated charge in the pixel rows from the first row to the last row after completion of the illumination period; generating an image frame from the accumulated charge in the sequentially read plurality of pixel rows; and appending the image frame to the video stream.

[0108] Optionally, the illumination period is at least a portion of the period between when the last row is reset and when the first row is read.

[0109] Optionally, the illumination period begins when the last row is reset.

[0110] Optionally, the multiple pixel rows are exposed for the same period of time to produce the image.

[0111] Optionally, the method further includes initiating a transition of the liquid crystal shutters from the open state to the closed state after the illumination period ends.

[0112] Optionally, the method further includes initiating a transition of the liquid crystal shutters from the open state to the closed state at the end of the illumination period.

[0113] Optionally, the method further includes initiating a transition of the liquid crystal shutters from the open state to the closed state before an end of the illumination period.

[0114] Optionally, the liquid crystal shutters are opened and closed using a timer device based on the vertical sync pulse.

[0115] Optionally, the liquid crystal shutters are opened or closed based on one or more characteristics of the imaged scene, optionally including brightness and / or modality of the imaged scene.

[0116] Optionally, the illumination light is generated by at least one LED.

[0117] Optionally, the rolling shutter imaging device is part of an endoscopic imaging device.

[0118] Optionally, the rolling shutter imager is part of a flexible scope and / or a tip-on-tip scope.

[0119] According to one aspect, a system for imaging tissue of a subject to provide a video stream is provided, the system including an illumination source and an imaging device having a rolling shutter imager, the imaging device being configured to: sequentially reset a plurality of pixel rows of the rolling shutter imager from a first row to a last row, transition a liquid crystal shutter from a closed state to an open state, illuminate the tissue of the subject with the illumination source for an illumination period to accumulate charge in the plurality of pixel rows after the liquid crystal shutter has been transitioned to the open state and after the reset of the last row, sequentially read the accumulated charge in the pixel rows from the first row to the last row after completion of the illumination period, generate an image frame from the accumulated charge in the sequentially read plurality of pixel rows, and add the image frame to the video stream.

[0120] Optionally, the illumination period is at least a portion of the period between when the last row is reset and when the first row is read.

[0121] Optionally, the illumination period begins when the last row is reset.

[0122] Optionally, the multiple rows of pixels are exposed for the same period of time to produce the image.

[0123] Optionally, the imaging device is further configured to initiate a transition of the liquid crystal shutters from the open state to the closed state after the illumination period has ended.

[0124] Optionally, the imaging device is further configured to initiate a transition of the liquid crystal shutters from the open state to the closed state at the end of the illumination period.

[0125] Optionally, the imaging device is further configured to initiate a transition of the liquid crystal shutters from the open state to the closed state prior to an end of the illumination period.

[0126] Optionally, the liquid crystal shutters are opened and closed using a timer device based on the vertical sync pulse.

[0127] Optionally, the liquid crystal shutters are opened or closed based on one or more characteristics of the imaged scene, optionally including brightness and / or modality of the imaged scene.

[0128] Optionally, the illumination source comprises at least one LED.

[0129] Optionally, the rolling shutter imaging device is part of an endoscopic imaging device.

[0130] Optionally, the rolling shutter imager is part of a flexible scope and / or a tip-on-tip scope. [Brief description of the drawings]

[0131] The invention will now be described, by way of example only, with reference to the accompanying drawings in which: [Figure 1] FIG. 1 is a diagram of an endoscopic camera system according to some examples. [Diagram 2] FIG. 2 is a diagram of a portion of the endoscopic camera system of FIG. 1 and a target object for imaging, according to some examples. [Diagram 3] FIG. 3 is a block diagram of an imaging system according to some examples. [Figure 4A] FIG. 4A is a diagram illustrating how unintended / undesirable light events can create artifacts in a rolling shutter image sensor with a “global shutter” period, according to some examples. [Figure 4B] FIG. 4B shows example artifacts in two image frames, according to some examples. [Diagram 5] FIG. 5 provides an exemplary method for imaging tissue of a subject using a rolling shutter imager to provide a video stream, according to some examples. [Figure 6A] FIG. 6A illustrates how a rolling shutter image sensor with a "synchronous frame reset" feature can be used to reduce or eliminate artifacts, according to some examples. [Figure 6B] FIG. 6B illustrates an example artifact in an image frame, according to some examples. [Figure 7] FIG. 7 illustrates an exemplary N frame buffer, where N is equal to 3, in accordance with some examples. [Figure 8A] FIG. 8A illustrates how a light event can affect exposure duration, according to some examples. [Figure 8B] FIG. 8B illustrates the use of a shutter to mitigate the effect of a light event on an image frame, according to some examples. [Figure 9]FIG. 9 illustrates an example shutter for blocking unintended / undesirable light events in an imaging path of an imaging device, according to some examples. [Figure 10] FIG. 10 illustrates another exemplary operation of a rolling shutter image sensor to reduce artifacts in an image frame, according to some examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0132] Implementations and examples of various aspects and variations of the systems and methods described herein will now be described in detail. Although some exemplary variations of the systems and methods are described herein, other variations of the systems and methods may include aspects of the systems and methods described herein combined in any suitable manner, including combinations of all or some of the described aspects. Examples are described more fully below with reference to the accompanying drawings, which may be embodied in different forms and should not be construed as being limited to the embodiments described herein. Rather, these examples are provided so that this disclosure will be thorough and complete, and will fully convey exemplary implementations to those skilled in the art.

[0133] Disclosed herein are exemplary devices, apparatus, systems, methods, and non-transitory storage media for medical imaging. More generally, exemplary devices, systems, and methods for reducing or eliminating artifacts caused by unintended / undesirable light events are disclosed. The systems, devices, and methods may be used to image tissue of a target, such as in an endoscopic imaging procedure. The imaging may be performed pre-operatively, intra-operatively, post-operatively, and during diagnostic imaging sessions and procedures. The imaging method itself may exclude the insertion of an endoscopic imaging device into a lumen in the body. The endoscopic imaging device may be inserted into a lumen prior to the imaging method. The imaging method itself may exclude any invasive surgical steps.

[0134] Although some of the present techniques are disclosed with respect to particular types of imagers (e.g., rolling shutter imagers, global shutter imagers), it should be understood that the techniques may be applied to any type of imager. Additionally, the present techniques may be applied to non-surgical or non-medical uses.

[0135] An exemplary system may have a "synchronous frame reset" function. In a rolling shutter imager having this function, with the synchronous frame reset function, all rows of pixels in the imaging sensor may be reset simultaneously or within a short time (e.g., shorter than the rolling shutter's line-by-line offset period). The rows of pixels are then illuminated for an illumination period shorter than the time between the synchronous frame reset and the first row read, and charge accumulates in the rows of pixels simultaneously for the same amount of time, achieving a global shutter effect. The accumulated charge is then read out sequentially from the first row to the last row to generate an image frame. The image frame can then be added to the video stream. The synchronous frame reset function can significantly reduce the effect of light events because it reduces the period during which light accumulates in the sensor to less than the frame period. Unintended / unwanted light events that occur before the synchronous frame reset do not affect the image frame. In some examples, the synchronous frame reset step eliminates impulsive light events by about 46%.

[0136] Exemplary imaging systems may include various techniques for detecting artifacts in image frames. These techniques may include detecting horizontal lines in image frames, detecting a rate of increase in the average value in successive image frames, detecting an increase in saturated pixels in successive image frames, detecting inconsistencies between color channels, using machine learning models, detecting an increase in the amount of light in a field stop region, or any combination thereof. In some examples, machine learning models may be used to detect artifacts in image frames, including machine learning models configured to detect the characteristics described above.

[0137] An exemplary system may use a buffer of size n. A sequence of frames captured by an imaging system may be stored in the buffer. Each image stored in the buffer may be associated with a respective score. The score indicates how likely the image is to be corrupted by a light event. Based on a comparison of the scores, an image frame may be dropped and replaced in the image sequence. Advantageously, an N-frame buffer may reduce false positives and maintain a low drop count.

[0138] An exemplary system may include a shutter (e.g., liquid crystal shutter, mechanical shutter, DLP mirror, electromechanical shutter). The shutter may be used with a rolling shutter imager to block light after the illumination period (and thus any unintended or undesired light events) to eliminate the effects of light events occurring during sequential readout of the rows. A shutter with pulse width control may be used to split the exposure time into multiple shorter periods to reduce the impact of light events on the frame. The shutter may also be operated as a standalone device without communication with the camera and block light from the imaging path when a light event is detected.

[0139] Any of the techniques described herein may be activated, deactivated, or adjusted in terms of their sensitivity / aggression. In some examples, a technique may be activated or its sensitivity may be increased in response to detection of a light event and / or detection of activation of a surgical energy device. Detection may be performed in a number of ways as described herein.

[0140] In various examples such as those described above, the illumination light may be modulated using pulse width modulation to provide the appropriate amount of illumination to the scene. The imaging system may control the amount of light so that one or more imaging sensors are optimally exposed, and may do so based on the intensity at the sensor during one or more previous frames.

[0141] In the following description, it should also be understood that the singular forms "a," "an," and "the" used in the following description are intended to include the plural forms, unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein means and encompasses one or more and all combinations of the associated listed items. It should also be understood that the terms "includes," "including," "comprises," and / or "comprising," as used herein, specify the presence of the referenced features, integers, steps, operations, elements, components, and / or units, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, units, and / or groups thereof.

[0142] Certain aspects of the present disclosure include the process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present disclosure can be implemented in software, firmware, or hardware, and when implemented in software, can be present on and downloaded to be operated from different platforms used by various operating systems. As will become apparent from the following description, unless otherwise indicated, descriptions using words such as "processing," "computing," "calculating," "determining," "displaying," "generating," and the like throughout this specification are understood to refer to the operation and processing of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in computer system memory or registers, or other such information storage, transmission, or display devices.

[0143] The present disclosure in some examples also relates to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the required purposes or may include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a non-transitory computer-readable recording medium, such as, but not limited to, a floppy disk, a USB flash drive, an external hard drive, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, any type of disk including an application specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be structured employing a multiple processor design to increase computing power.

[0144] The methods, apparatus, and systems described herein are not inherently related to any particular computer or other apparatus. In addition, various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the present invention as described herein.

[0145] 1 shows an example of an endoscopic imaging system 10 including a scope assembly 11 that may be utilized in an endoscopic procedure. The scope assembly 11 incorporates an endoscope or scope 12 coupled to a camera head 16 by a coupler 13 disposed at the distal end of the camera head 16. Light is provided to the scope by a light source 14 via a light guide 26, such as a fiber optic cable. The camera head 16 is coupled to a camera control unit (CCU) 18 by an electrical cable 15. The CCU 18 is connected to and in communication with the light source 14. The operation of the camera 16 is controlled in part by the CCU 18. The cable 15 may carry motion and / or still image data from the camera head 16 to the CCU 18 and may carry various control signals bidirectionally between the camera head 16 and the CCU 18.

[0146] A control or switch arrangement 17 may be provided on the camera head 16 to allow the user to manually control various functions of the system 10, which may include switching from one imaging mode to another, as further described below. Voice commands may be entered into a microphone 25 attached to a headset 27 worn by the practitioner and coupled to the voice control unit 23. A handheld control device 29, such as a tablet or PDA having a touch screen user interface, may be connected to the voice control unit 23 as a further control interface. In the illustrated example, a recorder 31 and a printer 33 are also connected to the CCU 18. Additional devices, such as image capture and archiving devices, may be included in the system 10 and connected to the CCU 18. Video image data acquired by the camera head 16 and processed by the CCU 18 is converted to images that may be displayed on the monitor 20, recorded by the recorder 31, and / or used to generate still images, hard copies of which may be generated by the printer 33.

[0147] 2 shows an example of a portion of an endoscopic system 10 that is used to illuminate and receive light from an object 1, such as a surgical site on a patient. The object 1 may include a fluorescent marker 2, for example as a result of the patient being administered a fluorescent contrast agent. The fluorescent marker 2 may comprise, for example, indocyanine green (ICG).

[0148] The light source 14 can generate visible illumination light (such as any combination of red, green, and blue light) for generating a visible (e.g., white light) image of the object 1, and can also generate fluorescent excitation illumination light for exciting fluorescent markers 2 in the object to generate a fluorescent image. The illumination light is transmitted to and passes through an optical lens system 22 that focuses the light onto a light pipe 24. The light pipe 24 can generate uniform light, which is transmitted to a fiber optic light guide 26. The light guide 26 can include a number of optical fibers and can be connected to an optical post 28 that is part of the endoscope 12. The endoscope 12 includes an illumination path 12' and an optical channel path 12''.

[0149] The endoscope 12 may include a notch filter 131 that passes a part or all (preferably at least 80%) of the fluorescent emission light (e.g., a wavelength range of 830 nm to 870 nm) emitted by the fluorescent marker 2 in the object 1, passes a part or all (preferably at least 80%) of the visible light (e.g., a wavelength range of 400 nm to 700 nm) such as the visible illumination light reflected by the object 1, while blocking substantially all of the fluorescent excitation light (e.g., infrared light having a wavelength of 808 nm) used to excite the fluorescent emission from the fluorescent marker 2 in the object 1. The notch filter 131 may have an optical density of OD5 or more. In some examples, the notch filter 131 may be disposed in the coupler 13.

[0150] FIG. 3 illustrates an exemplary imaging system 300 that employs an electronic imager 302 to generate images (e.g., still images and / or video) of an object such as a target tissue of a patient, according to some examples. The imager 302 may be a rolling shutter imager (e.g., a CMOS sensor) or a global shutter imager (e.g., a CCD sensor). The system 300 may be used, for example, in the endoscopic imaging system 10 of FIG. 1. The imager 302 includes a CMOS sensor 304 having an array of pixels 305 arranged in pixel rows 308 and pixel columns 310. The imager 302 may include a control component 306 that controls signals generated by the CMOS sensor 304. Examples of the control component include a gain circuit for generating a multi-bit signal indicative of the light incident on each pixel of the sensor 304, one or more analog-to-digital converters, one or more line drivers for acting as buffers and providing driving power to the sensor 304, row circuits, and timing circuits. The timing circuits may include components such as bias circuits, clock / timing generation circuits, and / or oscillators. The row circuitry may enable one or more processing and / or operational tasks, such as addressing rows 308 of pixels, addressing columns 310 of pixels, resetting charge on rows 308 of pixels, enabling exposure of pixels 305, signal decoding, signal amplification, analog to digital signal conversion, application of timing signals, readout signals and reset signals, and other suitable processes or tasks. The imager 302 may also include a mechanical shutter 312 that may be used, for example, to control exposure of the image sensor 304 and / or to control the amount of light received by the image sensor 304.

[0151] The one or more control components may be integrated into the same integrated circuit in which the sensor 304 is integrated, or may be separate components. The imaging device 302 may be incorporated into an imaging head, such as camera head 16 of the system 10.

[0152] One or more control components 306, such as row and timing circuits, may be electrically connected to an imaging controller 320, such as the camera control unit 18 of the system 10. The imaging controller 320 may include one or more processors 322 and memory 324. The imaging controller 320 may receive the imager row readouts and control readout timing and other imager operations, including mechanical shutter operation. The imaging controller 320 may generate image frames, such as video frames, from the row and / or column readouts from the imager 302. The generated frames may be provided to a display 350 for display to a user, such as a surgeon.

[0153] The system 300 of this example includes a light source 330 for illuminating a target scene. The light source 330 is controlled by an imaging controller 320. The imaging controller 320 may determine the type of illumination provided by the light source 330 (e.g., white light, fluorescent excitation light, or both), the intensity of the illumination provided by the light source 330, and / or the on / off time of the illumination synchronized with a rolling shutter operation. The light source 330 may include a first light generator 332 for generating light at a first wavelength and a second light generator 334 for generating light at a second wavelength. For example, in some examples, the first light generator 332 may be a white light generator comprised of multiple separate light generating components (e.g., multiple LEDs of different colors), and the second light generator 334 is a fluorescent excitation light generator such as a laser diode.

[0154] The light source 330 includes a controller 336 for controlling the light output of the light generator. The controller 336 may be configured to provide pulse width modulation of the light generator to modulate the intensity of the light provided by the light source 330, which can be used to manage over-exposure and under-exposure. In some examples, the nominal current and / or voltage of each generator remains constant, and the light intensity is modulated by switching the light generator (e.g., LED) on and off according to a pulse width control signal. In some examples, a PWM control signal is provided by the imaging controller 336. This control signal can be a waveform corresponding to the desired pulse width modulation operation of the light generator.

[0155] The imaging controller 320 may be configured to determine the illumination intensity required from the light source 330 and generate a PWM signal that is communicated to the light source 330. In some examples, depending on the amount of light received at the sensor 304 and the integration time, the light source may be pulsed at different rates to change the illumination intensity at the target scene. The imaging controller 320 may determine the illumination light intensity required for a subsequent frame based on the amount of light received at the sensor 304 in the current frame and / or one or more previous frames. In some examples, the imaging controller 320 may control pixel intensity via PWM of the light source 330 (to increase / decrease the amount of light at the pixel), via operation of the mechanical shutter 312 (to increase / decrease the amount of light at the pixel), and / or via a change in gain (to increase / decrease the amount of light at the pixel). In some examples, the imaging controller 320 uses PWM of the illumination source to control pixel intensity while keeping the shutter open (or at least not operating the shutter) and maintaining a gain level. The controller 320 may operate the shutter 312 and / or change the gain when the light intensity is at a maximum or minimum and further adjustment is needed.

[0156] (Unwanted light events affecting global and rolling shutters) During imaging, unintended and / or undesired light events may occur, causing artifacts in the image. For example, a holmium laser, a laser surgery tool that can be used to remove stones in urological procedures, may produce a brief, intense emission of light (e.g., at visible or IR wavelengths) when interacting with tissue. The light events may produce artifacts in the resulting endoscopic image, such as FIG. 4B described herein. Other tissue ablation devices, such as green light lasers and RF probes, may cause similar artifacts. The duration, repetition rate, and amplitude of the light events may depend on the energy device.

[0157] Although unintended / undesired light events can affect imaging with both sensors with global shutters (e.g., CCD sensors) and sensors with rolling shutters (e.g., CMOS sensors), artifacts can be more noticeable when using a rolling shutter. In a global shutter imaging sensor, all pixels of the sensor array are exposed simultaneously, and the sensor array is only sensitive to light during the exposure window. Thus, an unintended / undesired light event will only cause artifacts if the light event coincides with the exposure window. For example, at a frame rate of 60 Hertz, a frame is captured in a 16.6 millisecond period. If the sensor array is exposed for only 1 millisecond of the 16.6 millisecond period, an unintended / undesired light event will cause artifacts in the frame only if it occurs during the 1 millisecond exposure window. Thus, artifacts may occur less frequently in a global shutter imaging sensor than in a rolling shutter imaging sensor.

[0158] Furthermore, global shutter image sensors may provide interlaced scanning, which can further reduce artifacts caused by unintended / undesired light events. During interlaced scanning, half of the horizontal pixel rows (e.g., even-numbered rows) are captured in one cycle and the other half (e.g., odd-numbered rows) are captured in the next cycle, so two complete scans are required to capture one frame. Thus, if an unintended / undesired light event occurs during one cycle, it only affects half of that frame. For this additional reason, artifacts may be less noticeable with global shutter image sensors.

[0159] However, in rolling shutter imaging sensors, artifacts may be more frequent and more noticeable. Figure 4A illustrates how unintended / undesired light events may generate artifacts in an exemplary rolling shutter imaging sensor with a "global shutter" period. In the illustrated system, illumination of a target scene (e.g., a target tissue of a patient) is controlled to create a global shutter type effect with a long integration time for the pixels of the imaging sensor(s), as described below. Although Figure 4A illustrates how unintended / undesired light events affect the operation of a rolling shutter imaging sensor with a global shutter period, light events affect the operation of any rolling shutter imaging sensor as well as global shutter imaging sensors.

[0160] FIG. 4A illustrates an exemplary operation of a rolling shutter image sensor on a time scale showing the relative timing of resetting and reading out of pixel rows. As shown, the image sensor is driven according to a nominal frame rate of 120 fps (i.e., 8.3 ms per frame). However, rather than reading out a row of pixels of the sensor every possible frame period, a row is read out every other frame period, allowing the sensor pixels to integrate over a longer period (i.e., effectively two frame periods of the nominal frame rate). The longer integration period, referred to as the "global shutter" window, is indicated by the shaded parallelogram 402. The sensor array is sensitive over the entire global shutter window of 16.6 ms. The shaded parallelogram 402 therefore represents the exposure of one single frame. During the global shutter period, illumination is provided during the window 404 so that all rows of pixels are illuminated simultaneously, and the resulting image frame is substantially free of the rolling shutter effect. Further details of rolling shutter image sensors can be found in U.S. patent application Ser. No. 16 / 745,154, entitled “SYSTEMS AND METHODS FOR MEDICAL IMAGING USING A ROLLING SHUTTER IMAGER,” filed Jan. 16, 2020, the contents of which are incorporated by reference in their entirety.

[0161] Referring to FIG. 4A, when an unintended / unwanted light event occurs, there is a 100% probability that an artifact will occur in one or more frames. For example, any light event that occurs during time period 406 (i.e., from the reset of the last row to the readout of the first row), such as light event 2, will undesirably illuminate all rows of the current frame, thus causing an artifact (e.g., blowout) in the current frame. As another example, any light event that occurs within parallelogram 402 outside time period 406, such as light event 1, will undesirably illuminate some rows of the current frame and some rows of the adjacent previous frame, thus causing an artifact (e.g., horizontal lines) in two frames, as shown in FIG. 4B. In other words, regardless of when the unintended / unwanted light event occurs, there is a 100% probability that an artifact will appear in one or more frames. This is a significantly higher probability than a global shutter imaging sensor, where artifacts appear only when the light event lines up with the exposure window (e.g., 1 ms out of 16.6 ms).

[0162] 4A illustrates how an unintended / undesired light event affects the operation of a rolling shutter image sensor having a global shutter period, however, light events affect the operation of any rolling shutter image sensor as well as global shutter image sensors as described above. Additionally, unintended / undesired light events may affect various types of frames, including white light frames and fluorescent image frames.

[0163] (How to reduce artifacts in image frames) FIG. 5 provides an exemplary method for imaging tissue of a subject using a rolling shutter imager to provide a video stream. By the method 500, image frames are generated with reduced or eliminated artifacts. The method 500 may be performed by an imaging system, such as the imaging system 300 of FIG. 3, having a rolling shutter imager, such as the rolling shutter imager 302 of the system 300, and a light source, such as the light source 330 of the system 300. In the process 500, some blocks are optionally combined, some blocks are optionally reordered, and some blocks are optionally omitted. In some examples, additional steps may be performed in combination with the process 500. In some examples, a subset of the blocks are performed in separate processes, e.g., blocks 502-508 and blocks 510-518 are shown together in FIG. 5 but may belong to two separate processes. Thus, the operations illustrated (and described in more detail below) are exemplary in nature and should not be considered as limiting as such.

[0164] In step 502, the multiple pixel rows of the rolling shutter imager are reset. In some examples, the reset is performed via a "synchronous frame reset" feature of the rolling shutter imager. The synchronous frame reset feature clears the charge accumulated in the multiple pixel rows simultaneously or within a short time. In some examples, the multiple pixel rows are reset within a period shorter than the rolling shutter's line-by-line offset period (e.g., the period between the start of line 1 and the start of line 2 in FIG. 4A). However, unlike a global shutter imager, the readout of the multiple pixel rows is performed sequentially row-by-row with a time offset, similar to a rolling shutter imager. In some systems, the synchronous frame reset feature is intended to be used in a photo camera with a flashlight to generate a single frame photographic image. However, in method 500, the synchronous frame reset feature is used in a novel configuration to reduce or eliminate artifacts in an image frame in a video stream, as described below.

[0165] FIG. 6A illustrates a rolling shutter imaging sensor implementing method 500, according to some examples. FIG. 6A illustrates an example operation of a rolling shutter imager, with a time scale showing the relative timing of resetting and reading out pixel rows. As in the example of FIG. 4A, the imaging sensor is driven according to a nominal frame rate of 120 fps (i.e., 8.3 ms per frame), but instead of reading out pixel rows of the sensor every possible frame period, rows are read out every other frame period. Thus, only one row readout occurs during two frame periods (i.e., a 16.6 ms period). Referring to FIG. 6A, at time 0, all of rows 1-N of a frame are reset via a synchronous frame reset function at 607. Thus, at time 0, all of rows 1-N start exposing together and accumulating charge. In some examples, the reset occurs simultaneously or substantially simultaneously across all rows (e.g., less than the offset period per line of the rolling shutter imager).

[0166] Returning to Figure 5, after resetting the plurality of pixel rows in step 504, the subject's tissue is illuminated with illumination light for an illumination period. Referring to Figure 6A, after rows 1-N of a frame are reset via a synchronous frame reset function at time 0, the field of view is illuminated and rows 1-N accumulate charge from light received from the field of view during illumination period 604. In some examples, an illuminator is controlled to provide illumination light for the illumination period, e.g., a controller can control the imaging sensor and the illuminator such that they are synchronized.

[0167] In step 506, the charges stored in the multiple pixel rows are read out sequentially from the first row to the last row. Referring to Figure 6A, the charges stored in rows 1 to N are read out sequentially row by row from the first row to the last row, as shown at 608.

[0168] In some examples, the synchronous frame reset feature provides a configurable constant parameter (also called “G_EXP”) that defines the time between the trigger of the synchronous frame reset and the readout of the first row. Thus, the illumination period needs to be set shorter than (or at most equal to) the parameter so that the readout of the first row does not occur during the illumination period and all pixels of the frame are exposed for the entire illumination period. Referring to FIG. 6A, the illumination period is set shorter than G_EXP and therefore the readout of the first row is set to occur immediately after the illumination period. Thus, when the first row is read out at 608, it has been exposed for the entire illumination period 604. The remaining rows remain exposed until they are read out sequentially, as shown at 608. While the rows are being read out, they are not illuminated (except by an unintended / undesirable light event that happens by chance). In a conventional rolling shutter, the last row gets more light than the first row due to its slower readout time, but in FIG. 6A this is mitigated because all rows stop receiving light at the same time.

[0169] In some examples, after the synchronous frame reset function is triggered, a separate signal is needed to trigger the readout of the first row, at 608. This separate signal needs to be issued at or after the end of the illumination period so that all pixels of the frame are exposed for the entire illumination period before the first row is read out.

[0170] Returning to Figure 5, an image frame is generated from the readout of accumulated charge in multiple rows of pixels at step 508. Referring to Figure 6A, an image frame is generated from accumulated charge read out sequentially in rows 1-N, as shown at 608. Optionally, rows 1-N are then reset.

[0171] In some examples of implementing method 500, the two resets occur at two different times before each image frame is generated. For example, with respect to the second frame of FIG. 6A, a first reset of rows 1-N may occur at a first time 608 after the first frame readout, a second reset (i.e., a synchronous frame reset) of rows 1-N occurs at a second time 616, and the accumulated charge of rows 1-N is read out at a third time 618. No readout of rows 1-N occurs between the first time 608 and the second time 616. In some examples, the reset at 608 is optional.

[0172] The method 500 of novel use of the synchronous frame reset feature can significantly reduce the impact of undesired / unintended light events on image frames. First, any light event occurring before the synchronous frame reset time does not affect the frame. For example, consider light event 2 in FIG. 6A. Light event 2 does not affect any frame because it occurs after the readout of the first frame and before the synchronous frame reset of the second frame. In fact, any light event occurring in window 606 does not affect any image frame.

[0173] Furthermore, even if a light event occurs when a frame is being exposed, the light event only affects one frame. For example, consider light event 1 in FIG. 6A. Because light event 1 occurs before the synchronous frame reset of the second frame, it only affects the first frame and not the second frame, as shown in FIG. 6B. Thus, light event 1 and light event 2 affect one frame in total. In contrast, in the example shown in FIG. 4A without the synchronous frame reset function, light event 1 and light event 2 affect three frames in total. In some examples, the synchronous frame reset feature as shown in FIG. 6A eliminates impulsive light events by about 46%.

[0174] According to some examples, the imager may be configured for any suitable frame rate. Exemplary frame rates include at least 25 fps, at least 30 fps, at least 50 fps, at least 60 fps, at least 100 fps, at least 120 fps, at least 200 fps, and at least 240 fps. The global shutter window time and / or extended vertical blanking time are generally related to the frame rate capability of the imager, with faster frame rates being associated with shorter global shutter window times and / or shorter extended vertical blanking times.

[0175] Returning to Figure 5, in step 510, the system determines whether an image frame satisfies one or more criteria. As discussed above, the synchronous frame reset function may not eliminate the effects of all light events (e.g., light event 1 in Figure 6A). Thus, step 510 may be performed to identify defective image frames and exclude the defective image frames from the video stream.

[0176] In some examples, step 510 includes identifying one or more artifacts in the image frame at step 512. Exemplary techniques for identifying artifacts in an image frame are provided below.

[0177] In some examples, identifying one or more artifacts in the image frame includes identifying horizontal or vertical lines in the image frame. As shown in FIG. 6B, the artifact may include a horizontal line depicting a color change from a darker top to a lighter bottom. The color change is caused by a light event (e.g., light event 1 in FIG. 6A) that occurred during sequential readout of the rows. The horizontal line may appear anywhere in the image. In alternative examples where the sensor is rotated, the artifact line may have a different angle (e.g., vertical). In some examples, the calculation may be performed with respect to the region of interest.

[0178] In some examples, a Sobel filter is used to detect lines in the image frame. The Sobel filter can be modified to focus more on horizontal lines and / or dark to light transitions. An exemplary Sobel convolution can be [1 2 3 2 1; 0 0 0 0 0; -1 -2 -3 -2 -1]. The convolution can be applied to the luminance channel of the image frame. After the Sobel convolution is applied, the Sobel image can be processed to remove all values ​​below a threshold (e.g., 20 for an 8-bit y channel) so that only positive transitions are considered and the noise floor is removed. The Sobel image can be binarized using the mean value and the number of active pixels in the line can be counted. The system can then differentiate the active pixel count along the y-axis. In some examples, a filter (e.g., a Gaussian kernel [0.25 0.5 0.25]) can be applied and the minimum and maximum values ​​can be determined. If the difference between the minimum and maximum values ​​(i.e., the max / min gradient) is above a threshold, then a horizontal line is detected. It should be understood that other techniques for detecting lines in an image frame (e.g., using other types of filters) can be used.

[0179] In some examples, identifying one or more artifacts in an image frame includes calculating a rate of increase of the image frame's mean value from the mean value of one or more previous image frames. Although not all light events cause pixel saturation, many events can still be detected by analyzing the gradient of the increase in the mean value. In some examples, the mean value of the image frame is the average value of all pixels in the image frame. The deviation of the mean value of the image frame (denoted as ΔMean) can then be calculated as the difference between the mean value of the previous image frame and the image frame. The deviation of ΔMean (denoted as ΔΔMean) can then be calculated as the difference between the ΔMean of the last good frame and the ΔMean of the current image frame divided by the number of frames in between. If ΔΔMean is greater than a predefined threshold, the image frame can be determined to have an artifact. In some examples, the calculation can be performed over a region of interest. In some examples, rather than using the mean value, the system can perform the calculation using a median or a modified median value (e.g., a median value that excludes pixels with darkness above a threshold from the calculation).

[0180] Identifying one or more artifacts in an image frame may include calculating an increase in the number of saturated pixels in the image frame from a number of saturated pixels in a previous image frame. A large increase in saturated pixels from one frame to another (pre-gain) may indicate a light event. In some examples, the number of newly saturated pixels (e.g., all pixels above a certain threshold) may be detected in a frame and a difference from the last good frame determined. If the difference is greater than a predefined threshold, the image frame may be determined to have an artifact. In some examples, the calculation may be performed over a region of interest.

[0181] Identifying one or more artifacts in an image frame can be based on color information. For example, the identification can include evaluating a difference between at least two of a red channel, a blue channel, and a green channel of the image frame. If there is a difference between the channels (e.g., between red and one or more of the other channels) that is greater than a predefined threshold, the image frame can be determined to have an artifact. In some examples, the calculation can be performed with respect to a region of interest.

[0182] Identifying one or more artifacts in the image frames may include processing the image frames using a trained machine learning algorithm, such as a trained neural network, where the algorithm may be trained to receive the image frames and classify the image frames as having artifacts or not having artifacts.

[0183] Identifying one or more artifacts in an image frame can be based on pixels in a field stop region of the image frame that would not normally receive light because light rays are blocked by the field stop of the endoscope. During a light event, the blowout can be so significant that stray light impinging on the imaging sensor (e.g., due to stray light reflections in the endoscope or camera) extends into the field stop region, causing an increase in the amount of light in the field stop region. As shown in FIG. 6B, the lower portion of the field stop region in an affected image frame appears brighter than the field stop region in an unaffected image frame, which is completely dark. Thus, if the system detects an increase in the amount of light in the field stop region that exceeds a predefined threshold, a defective image can be identified.

[0184] In some examples, a filter (e.g., a Sobel filter) may be configured to identify lines within the field stop region to identify defective images. For example, the filter may be configured to identify longer lines in the field stop region of a frame near the top or bottom of the frame and shorter lines in the field stop region near the center of the frame.

[0185] The above-described techniques for detecting artifacts may be combined, ordered, and omitted in any manner in step 512 of method 500. For example, detecting horizontal or vertical lines may be applied only to image frames captured using a rolling shutter imager, while other techniques may be applied to image frames captured using either a rolling shutter imager or a global shutter imager. In some examples, additional steps may be performed in combination with the techniques. Thus, the techniques described above are exemplary in nature and should not be considered as limiting as such.

[0186] Returning to FIG. 5, in some examples, step 510 includes placing the frames in a buffer of a predefined size in step 514. A sequence of frames captured by an imaging device may be accumulated in the buffer. Each image accumulated in the buffer may be associated with a respective score. The score indicates how likely the image is to be corrupted by a light event. The score may be calculated based on artifacts (or lack thereof) detected in the image frame, and in some examples may be calculated using any combination of techniques described herein (e.g., horizontal lines, pixel averages, etc.).

[0187] FIG. 7 illustrates an exemplary N-frame buffer, where N is equal to 3. As illustrated, a sequence of three image frames is accumulated in the buffer. The three image frames are associated with scores of 60, 90, and 10, respectively, with higher scores indicating a higher likelihood that the image is defective. The scores of the image frames in the buffer may be compared, and based on the comparison, image frames are identified as images to be excluded from the video stream. In the illustrated example, the system is configured to drop the worst images from the sequence of image frames. Thus, frame 2 has the worst score and is therefore dropped from the sequence of image frames.

[0188] Advantageously, an N-frame buffer can reduce false positives and maintain a low drop count. In the example shown in FIG. 7, if there was no buffer and the system was configured to drop all images with a score above a threshold of 50, the system would have dropped frame 1. However, frame 1 could be a false positive that should not have been dropped - for example, frame 1 could have a high average value of pixels compared to the previous frame due to the instrument moving and detecting bright pixels when there is no light event. Furthermore, if the system is configured not to drop consecutive images, the system would drop frame 1 while keeping frame 2, even though frame 2 has a much higher score and is indeed caused by a real light event. With the buffer, the system would correctly drop frame 2 and keep frame 1.

[0189] In the illustrated example, the system is configured to drop a maximum of one image frame from the buffer. In some examples, the system may drop more image frames. The maximum number of image frames that can be dropped may depend on the frame rate of the camera, the frame rate of the video stream, the aggressiveness of the algorithm (described below), or a combination thereof.

[0190] In some examples, after an image frame is dropped from the sequence of image frames, the system identifies another image frame to replace the dropped image frame in the sequence. In some examples, a replacement image frame is selected from the sequence of image frames based on a score. In the example shown in Figure 7, frame 3 is selected because it has the best score. Thus, the updated sequence of image frames (i.e., frame 1, frame 3, frame 3) can be added to the video stream.

[0191] Buffers may introduce delays into the system. Instead of adding each image frame to the video stream as soon as it is generated, multiple frames are accumulated in a buffer and evaluated together before the first frame is added to the video stream. The size of the buffer determines the amount of delay that occurs - the larger the buffer, the longer the latency. In some instances, if the camera frame rate is higher than the video stream frame rate, the delay may be imperceptible. In other words, the system has a higher delay tolerance. Thus, the buffer size may be determined based on the delay tolerance of the system. In some instances, the buffer size is between 3 and 6 frames.

[0192] In some examples, the buffer has a size of 1 so that only one image frame is stored in memory to replace the next frame if the next frame is defective.

[0193] Returning to FIG. 5, in step 516, an image frame is added to the video stream according to a determination that the image frame does not meet one or more criteria, and in step 518, an image frame is discarded according to a determination that the image frame meets one or more criteria. If an image frame is discarded, it may be replaced with another image frame (e.g., the latest good frame, the best frame in the buffer, a repaired frame). In some examples, the number of frames that may be consecutively discarded is limited to a predefined threshold or based on the aggressiveness of the algorithm, as described below. In some examples, the discarded image frames do not affect the automatic gain exposure ("AGC") algorithm to prevent negative flash; alternatively, the discarded images cause the AGC algorithm to be adjusted by a smaller weighting factor.

[0194] Although the steps of method 500 are described with respect to a rolling shutter imager, the techniques in method 500 can be used with either a rolling shutter imager or a global shutter imager. For example, steps 510-518 can be used to evaluate image frames captured by a global shutter imager to detect defective image frames and remove them from the video stream.

[0195] The aggressiveness or sensitivity of method 500 can be adjusted automatically or manually. In particular, any of the steps and parameters of method 500 can be adjusted automatically, such as one or more thresholds for identifying artifacts in an image frame, the size of the image buffer, the maximum number of image frames that can be dropped from the image buffer, the maximum number of consecutive image frames that can be dropped from the image buffer, or any combination thereof. In some examples, the aggressiveness of the method can be dynamically adjusted upon detection of activation of a surgical energy device (e.g., laser unit, RF probe) and / or upon detection of a light event (e.g., via a photodiode). In these cases, the system can automatically activate any combination of steps of method 500 or increase their aggressiveness or sensitivity to capture defective image frames. On the other hand, in the absence of activation of a surgical energy device or detection of a light event, the system can automatically deactivate any combination of steps of method 500 or decrease their aggressiveness or sensitivity.

[0196] For example, one way to adjust the sensitivity of step 514 is to automatically increase the size of the buffer in response to detection of a light event. In other words, the system accepts higher delays but improves its ability to identify defective images. In the absence of a light event, the system can automatically reduce the size of the buffer or bypass it entirely (i.e., set the size to 0). As another example, the system can automatically activate or increase the sensitivity of any of the techniques of step 512 in response to detection of a light event, and vice versa in the absence of a light event.

[0197] Detecting activation of the surgical energy device can include receiving a signal from the surgical energy device (e.g., laser unit, RF probe). For example, the signal can be a signal from a control unit of the energy device that a Holmium laser has been, is being, or will be output by that unit. In some examples, the energy device is commanded by a foot pedal. Thus, a hardware component (e.g., a pressure sensor) can be coupled to the foot pedal and configured to transmit a signal whenever the foot pedal is pressed.

[0198] Detecting the actuation of the surgical energy device may include detecting an increase in power consumption of the surgical energy device. For example, the detection may be performed by a remote clamp attached to the power cable to determine the increase in power consumption.

[0199] Detecting the light event may include receiving a signal from a photodetector attached to the imaging device. The photodetector may be located within the camera head (e.g., attached to a prism) or clamped to the camera body (e.g., on or near an entrance window of the camera body) to capture stray or reflected light within the camera body. In some examples, the unintended / undesired light event may include infrared light that is not present in the desired image spectrum of the camera. Thus, a filter (e.g., an infrared filter) may be placed in front of the photodetector to ensure that the photodetector detects only unintended / undesired light events. In some examples, the system is configured to treat any light detected by the photodetector that does not correspond to the light source of the imaging device as an unintended / undesired light event. In some examples, whether an image frame is defective may be determined directly by comparing the timing of the detected light event relative to the reset and readout times of the frame, as shown in FIG. 6A and demonstrated in FIG. 6B. If a light event is detected, the system may automatically identify the affected frames and flag those frames (e.g., by adjusting the score associated with the frames).

[0200] Detecting the actuation of the surgical energy device may include receiving an acoustic signal from the surgical energy device. For example, an acoustic microphone may be used to detect an acoustic signal indicative of an optical event, such as a safety tone or distinct chatter produced by a laser during firing. Machine learning algorithms may be trained to recognize the acoustic signal.

[0201] In some examples, motion of the imaging device may be detected and used to adjust the sensitivity of the steps of method 500. Motion of the imaging device may be detected via an inertial measurement unit of the imaging device or via image tracking. If the imaging device is moving, the sensitivity may be reduced to avoid false positive detections.

[0202] Some of the examples include rolling shutter sensors having horizontal lines that are read from the top row to the bottom row, but it should be understood that the techniques described herein are applicable to image sensors having different orientations, layouts, and readout orders without departing from the spirit of the invention.

[0203] (Using a shutter to reduce artifacts) In some examples, shutters can be used to reduce or eliminate artifacts caused by unintended / undesired light events. The shutters can be: liquid crystal shutters, mechanical shutters, DLP mirrors, or electro-mechanical shutters. When actuated, the shutters can block a substantial portion of the light from the imaging path. When deactivated, the shutters allow a substantial portion of the light to pass.

[0204] 6A, a shutter (e.g., an LC shutter) may be configured to actuate at the end of the illumination period 604 to block a substantial portion of the light from the imaging path. Thus, any light events that occur during the sequential readout at 608, such as light event 1, do not affect the image frame. In some examples, the shutter may be configured to transition from a stopped state (i.e., allowing a substantial portion of the light to pass) to an actuated state (i.e., blocking a substantial portion of the light) in a short period of time. Thus, the pixel rows are simultaneously or substantially simultaneously reset via a synchronous frame reset at 607 to begin accumulating charge, and simultaneously or substantially simultaneously stop receiving light via the shutter at the end of the illumination period.

[0205] The shutters are stopped before the synchronous frame reset 607. In some instances, the LC shutters take time to transition from an active state to a stopped state, so the stopping can be triggered to occur early enough so that the LC shutters are completely stopped at 607.

[0206] A shutter with pulse width control can also be used to reduce the effects of unintended / undesired light events, as shown in Figures 8A-B, where a global shutter imager is driven according to a frame rate of 60 fps (i.e., 16.6 ms per frame) and has an exposure window of 1 ms per frame period.

[0207] As shown in FIG. 8A, under standard operation, if a light event happens to coincide with a 1 ms exposure window, the sensor may be exposed to the light event for a large portion of the 1 ms exposure window, causing noticeable artifacts in the image frame.

[0208] In FIG. 8B, instead of having consecutive 1 ms exposure windows during the 16.6 ms frame period, the sensor is configured to be exposed for the entire 16.6 ms frame period to accumulate charge. The shutter with pulse width control is configured to pause n times (n=5 in FIG. 8B) that is less than n times the actual desired 1 ms exposure time (200 us in FIG. 8B) to allow light to pass through the shutter. An image can be generated based on the accumulated charge after the nth pause. In other words, the shutter controls the light to only allow light to reach the sensor for less than the desired exposure time in total, even though the sensor is exposed for the entire 16.6 ms frame period. As shown in FIG. 8B, if a light event occurs within the frame period, the impact is less severe because it can only affect a portion of the total 1 ms exposure time.

[0209] The n stops of the LC shutter can be spaced apart by at least the desired exposure time (e.g., 1 ms in the illustrated example). In some examples, the value of n can be adjusted. For example, n can be automatically increased to reduce the effect of a light event if the system determines, using the techniques described herein, that a light event has occurred, is continuing to occur, or may occur.

[0210] 8B illustrates the use of a shutter in a global shutter imager, it should be understood that the shutter may be used in other types of imagers as well, such as a rolling shutter imager. Additionally, in some examples implementations may be accomplished using density modulation of the shutter.

[0211] Instead of activating and deactivating a shutter, multiple charge accumulation periods for an image frame can be achieved using a global shutter sensor that allows charge accumulation to be pulsed. For example, the sensor allows charge accumulation to be activated or deactivated based on an external control signal. Charge accumulation can be deactivated using, for example, a bypass circuit.

[0212] The shutter can also be configured to operate as a stand-alone device without communication with the imaging device. The shutter can be located in the imaging path. For example, the shutter can be a replacement coupler or drop-in accessory for the eyepiece of the endoscope. The shutter can be powered by its own power source (e.g., internal battery, solar cell).

[0213] In particular, the shutter device can have a photodiode for detecting light events. The photodiode can be located on the outer surface of the imager prism as shown in FIG. 9. For example, the photodiode can be sensitive to IR to detect unintended / unwanted light events (e.g., laser events can include IR), but not sensitive to the imager light source or other desired light for imaging (e.g., emitted fluorescent light). When a light event is detected, the LC shutter closes within a very short period (e.g., 500us to 1ms), protecting the image sensor from most contamination. As soon as the detected light event ends, the LC shutter is configured to open as fast as it closed. Laser light events are typically shorter than 1ms. Thus, if the camera exposure is moderately larger than that, no significant adverse effects are perceived. When the exposure is short, the frame may appear dark rather than blown out.

[0214] Although the photodiode is shown mounted on the LC shutter in Figure 9, the photodiode can be mounted elsewhere in the system. In some examples, the stray light photodiode can be placed on the LC shutter on the image / sensor side. It can be mounted facing the sensor on the LC shutter (or mating component) outside of the incident image beam clear aperture (i.e., beam / image height at this position in Z). The sensor / prism assembly does not capture all incident light, some of that light is reflected back from the prism assembly. The photodiode can capture this stray light exiting the prism / sensor assembly.

[0215] In some examples, a stray light photodiode can be mounted on the LC shutter on the object / endoscope side. It can be mounted facing the endoscope on the LC shutter (or mating component) outside of the incident image beam clear aperture (i.e., beam / image height at this position in Z). The prism assembly does not capture all incident light, and the LC shutter does not transmit all incident light. Reflections at these interfaces result in reflected stray light exiting the system and reflecting off upstream optics (e.g., the endoscope exit window) back to the sensor, but outside of the original imaging beam. The photodiode can capture some of this secondary reflected light from that upstream optic.

[0216] In some examples, a stray light photodiode can be mounted on the prism entrance face. The photodiode can be mounted on the prism glass entrance face outside of the incoming image beam clear aperture (i.e., beam / image height at this position in Z) and pointed (a) toward the endoscope or (b) toward the sensor. The photodiode facing the endoscope can capture secondary reflections from the LC shutter or other upstream optical elements (the primary reflection is out of the system from the sensor / prism assembly) or capture stray light from the incoming beam originating from interfaces of upstream optical elements. The photodiode facing the sensor can capture the primary reflection from the sensor / prism assembly exiting the prism assembly.

[0217] In some examples, a photodiode can be mounted on the prism input face with a pick-off optic or diffractive optical element (DOE). Such a configuration is similar to the endoscope-facing photodiode described above, but with a pick-off optic or DOE to enable input beam sampling (without relying on stray light / system internal reflections).

[0218] In some examples, the photodiode is mounted with a dichroic mirror (beam splitter) and the photodiode is mounted in the free space (i.e., the remaining void in the camera housing not occupied by these components) that exists around the coupler optics, the shutter, the sensor / prism assembly, and the image beam. The dichroic mirror is positioned in the optical path of the imaging beam to reflect at least a portion of the IR content of the light event to the photodiode while passing white light to the sensor assembly.

[0219] In some examples, the photodiode may be mounted in the above-mentioned free space facing the entrance window.

[0220] In some examples, the photodiode may be mounted in the free space described above adjacent an entrance window coupled with a pickoff optic.

[0221] In some examples, upon detection of a light event, the system can compare the timing of the detected light event against the reset and readout times of the frame, as shown in Figure 6A and demonstrated in Figure 6B. In this manner, the system can automatically identify affected frames, which can then be dropped and replaced. For example, the affected frames can be flagged for further processing (e.g., by adjusting a score associated with the frame).

[0222] It should be understood by those skilled in the art that although some examples are described with respect to LC shutters, other types of shutters may be used in the examples of this disclosure, such as mechanical shutters, DLP mirrors, electro-mechanical shutters, optical shutters, etc.

[0223] 10 illustrates another exemplary operation of a rolling shutter image sensor to reduce artifacts in an image frame, according to some examples, in a time scale illustrating the relative timing of resting and readout of pixel rows.

[0224] The example of Fig. 10 is similar to the example of Fig. 4A in that the rows are read out every other frame period, rather than every period. In this way, the system allows the sensor pixels to integrate for a longer period (i.e., substantially two frame periods of the nominal frame rate). The shaded parallelogram 1002 represents the exposure of one single frame when the imaging sensor is driven according to the nominal frame rate of 120 fps (i.e., 8.3 ms per frame). The sensor array is sensitive throughout the 16.6 ms window indicated by the shaded parallelogram 1002. In particular, during the time period 1006 between the reset of the last row N and the readout of the first row, all or substantially all rows of the imaging device can be exposed simultaneously, effectively creating a global shutter window that can provide illumination light such that rolling shutter effects are prevented.

[0225] In the example shown in Figure 10, the illumination period 1004 begins at the beginning of the window 1006. This is in contrast to the example of Figure 4A, where the illumination period 404 begins during, rather than at, the beginning of the window 406. Additionally, the system opens the liquid crystal shutters to expose the pixels in the sensor array to illumination during the illumination period. As shown, the liquid crystal shutters begin to transition from a closed state to an open state slightly before the beginning of the illumination period 1004. In this manner, the liquid crystal shutters can be fully transitioned to an open state at the beginning of the illumination period 1004, thereby preventing chromatic effects that may occur while the shutters transition from a closed state to an open state.

[0226] In some examples, the liquid crystal shutters begin transitioning from the open state to the closed state shortly after the end of the illumination period 1004. In this way, the liquid crystal shutters transition does not affect any portion of the illumination period, again preventing chromatic effects that may occur while the shutters transition between states. In another example, the liquid crystal shutters may close before or at the end of the illumination period, since the closing speed of the liquid crystal shutters is much faster than the opening speed, making chromatic effects less noticeable.

[0227] In one exemplary implementation, the liquid crystal shutter takes approximately 1.3 ms to fully open and 50 us to fully close. Additionally, the shutter is determined to be chromatically stable after approximately 800 us on the open side. Thus, the shutter can be configured to begin transitioning to the open state 800 us prior to the illumination period. The transition to the closed state can begin simultaneously with the end of the illumination period, since there is no effect when the light source is turned off. Another reason to transition to the closed state after the end of the illumination period is so that the system can operate the shutter without knowledge of the desired light pulse and have the automatic gain control in the camera make up the difference. It should be understood that the asymmetric opening and closing times of the liquid crystal shutter are based on how it is configured, and the shutter can be configured in the opposite manner to have a faster opening time and a slower closing time, and the timing of opening and closing the shutter relative to the illumination period can be adjusted accordingly.

[0228] In some examples, rather than starting the illumination period at the beginning of the window 1006, the illumination period can start at another time during the window 1006. The entire illumination period falls within the window 1006. Additionally, the liquid crystal shutters are configured to open and close based on the illumination period as described above. Because the camera and light source vary the exposure time based on the length of 1006, the system can either have a fixed start time for the illumination period and a variable end time, or a fixed end time for the illumination period and a variable start time.

[0229] In some examples, a timer device can be used to set the liquid crystal shutter open and close times based on a vertical synchronization pulse (either the frame acquisition vsync or the light source vsync). In some examples, the system can adjust the liquid crystal shutter open and / or close times based on the imaging scene (e.g., brightness, modality). In some examples, the system can generate a relatively high voltage (e.g., 24V) in the camera head from a lower operating voltage using a temperature converter to drive the liquid crystal shutters at a relatively high voltage.

[0230] Although some examples are described with reference to endoscopes, it should be understood by those skilled in the art that the techniques described herein may be used in any imaging system, including flexible and / or tip-on-tip scopes in which the imaging sensor is located at the distal end of the scope, such as a flexible digital ureteroscope.

[0231] The foregoing disclosure has been disclosed with reference to specific examples for illustrative purposes. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Although features have been described herein as part of the same or separate examples for clarity and conciseness, it will be understood that the scope of the disclosure includes examples having all or any combination of the described features. Many modifications and variations are possible in light of the above teachings. The examples have been selected and disclosed in order to best explain the principles of the technology and their practical application. This will enable those skilled in the art to best utilize the technology and the various examples with various modifications suited to the particular use envisaged.

[0232] Although the present disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications should be understood as being included within the scope of the disclosure and examples as defined by the claims. Finally, the entire disclosures of the patents and publications referenced in this application are incorporated herein by reference.

Claims

1. 1. A method for imaging tissue of a subject using a rolling shutter imager to provide a video stream, the method comprising: sequentially resetting a plurality of pixel rows of the rolling shutter imager from a first row to a last row; transitioning the liquid crystal shutter from a closed state to an open state; illuminating the tissue of the subject with illumination light for an illumination period to accumulate charge in the plurality of rows of pixels after the liquid crystal shutters are transitioned to the open state and after the resetting of the last row; After the illumination period ends, reading the accumulated charges in the pixel rows sequentially from the first row to the last row; generating an image frame from accumulated charges in the sequentially read rows of pixels; and adding the image frame to the video stream.

2. 2. The method of claim 1, wherein the illumination period is at least a portion of the period from when the last row is reset to when the first row is read.

3. The method of claim 2 , wherein the illumination period begins when the last row is reset.

4. 4. The method of claim 1, wherein the rows of pixels are exposed for the same period of time to generate the image.

5. 5. The method of claim 1, further comprising initiating a transition of the liquid crystal shutters from the open state to the closed state after the illumination period ends.

6. 5. The method of claim 1, further comprising initiating a transition of the liquid crystal shutters from the open state to the closed state at the end of the illumination period.

7. 5. The method of claim 1, further comprising initiating a transition of the liquid crystal shutters from the open state to the closed state before an end of the illumination period.

8. 8. The method of claim 1, wherein the liquid crystal shutters are opened or closed using a timer device based on a vertical synchronization pulse.

9. 9. The method of any of claims 1 to 8, wherein the liquid crystal shutters are opened or closed based on one or more characteristics of the imaged scene.

10. 10. The method of claim 9, wherein the one or more characteristics of the imaged scene include brightness and / or modality of the imaged scene.

11. 11. The method according to any of the preceding claims, wherein the illumination light is generated by at least one LED.

12. 12. The method of any preceding claim, wherein the rolling shutter imaging device is part of an endoscopic imaging device.

13. 12. The method according to any of the preceding claims, wherein the rolling shutter imager is part of a flexible scope and / or a tip-on-tip scope.

14. 1. A system for imaging tissue of a subject to provide a video stream, the system comprising: An illumination source; and an imaging device having a rolling shutter imaging device, resetting a plurality of pixel rows of the rolling shutter imager sequentially from a first row to a last row; Transitioning the liquid crystal shutter from a closed state to an open state, illuminating the tissue of the subject with the illumination source for an illumination period to accumulate charge in the plurality of rows of pixels after the liquid crystal shutters are transitioned to the open state and after the reset of the last row; After the illumination period ends, reading the accumulated charges in the pixel rows sequentially from the first row to the last row; generating an image frame from the accumulated charges in the sequentially read rows of pixels; The system is configured to add the image frame to the video stream.

15. 15. The system of claim 14, wherein the illumination period is at least a portion of the period from when the last row is reset to when the first row is read.

16. 16. The system of claim 15, wherein the illumination period begins when the last row is reset.

17. 17. A system according to any of claims 14 to 16, wherein the multiple rows of pixels are exposed for the same period of time to produce the image.

18. 18. The system of any of claims 14-17, wherein the image capture device is further configured to initiate a transition of the liquid crystal shutters from the open state to the closed state after the illumination period has ended.

19. 18. The system of any one of claims 14 to 17, wherein the image capture device is further configured to initiate a transition of the liquid crystal shutters from the open state to the closed state at the end of the illumination period.

20. 18. The system of any of claims 14-17, wherein the image capture device is further configured to initiate a transition of the liquid crystal shutters from the open state to the closed state prior to an end of the illumination period.

21. 21. The system of any of claims 14 to 20, wherein the liquid crystal shutters are opened or closed using a timer device based on a vertical synchronization pulse.

22. 22. A system according to any of claims 14 to 21, wherein the liquid crystal shutters are opened or closed based on one or more characteristics of the imaged scene.

23. 23. The system of claim 22, wherein the one or more characteristics of the imaged scene include brightness and / or modality of the imaged scene.

24. 24. A system according to any one of claims 14 to 23, wherein the illumination source comprises at least one LED.

25. 25. The system of any of claims 14 to 24, wherein the rolling shutter imaging device is part of an endoscopic imaging device.

26. 25. A system according to any of claims 14 to 24, wherein the rolling shutter imager is part of a flexible scope and / or a tip-on-tip scope.

Images