System and method for reducing artifacts in medical imaging

Abstract

An exemplary method for imaging tissue of a subject using a rolling shutter imager to provide a video stream includes sequentially resetting a number of pixel rows of the rolling shutter imager from a first row to a last row; transitioning a liquid crystal shutter from a closed state to an open state; illuminating the tissue of the subject with illumination light for an illumination period to accumulate charge in the number of pixel rows after the liquid crystal shutter has been transitioned to the open state and after resetting the last row; sequentially reading the accumulated charge in the pixel rows from the first row to the last row after completion of the illumination period; generating an image frame from the accumulated charge in the sequentially read number of pixel rows; and appending the image frame to the video stream.

Description

Technical Field

[0001] The present disclosure generally relates to medical imaging, and more particularly, to techniques for reducing or removing artifacts caused by light events in medical imaging.

Background Art

[0002] (Cross-reference to related applications) This application claims the benefit of U.S. Provisional Application No. 63 / 132,444, filed Dec. 30, 2020, the entire content of which is incorporated herein by reference.

[0003] Medical systems, instruments or tools are utilized before, during or after surgery for various purposes. Some of these medical tools can be used in what is generally referred to as an endoscopic procedure. For example, endoscopy in the medical field can view the internal characteristics of a patient's body without using conventional invasive surgery. An endoscopic imaging system incorporates an endoscope so that a surgeon can view the surgical site, and endoscopic tools enable non-invasive surgery at that site. An endoscope can be used with a camera system to process an image received by the endoscope. An endoscopic camera system typically includes a camera head connected to a camera control unit (CCU) that processes input image data received from the imaging sensor of the camera via a cable and outputs image data for display. The CCU can control a illuminator that generates illumination light provided to the imaging scene.

[0004] Endoscopic imaging systems can utilize a variety of imaging device sensors, including charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors. CCD structures are generally more complex than CMOS sensors, which can be manufactured using mass wafer manufacturing facilities used for related technologies such as microprocessors and chipsets. As a result, CMOS sensors are often less expensive than CCDs for comparable performance. In addition to lower cost, common manufacturing processes used to create CMOS imaging devices allow for the integration of CMOS pixel arrays with other electronic devices such as clock drivers, digital logic, analog-to-digital converters, and other appropriate electronic components onto a single circuit. The compact structure possible with CMOS imaging devices reduces space requirements and lowers power consumption. Furthermore, CMOS imaging devices can offer higher sensitivity and higher video frame rates.

[0005] CMOS-based imaging devices can utilize an electronic rolling shutter to expose pixels in a sensor array. In an electronic rolling shutter, pixel rows are cleared (or reset), exposed, and read out sequentially. During integration, the pixel row is exposed to light energy, and each pixel forms a charge corresponding to the amount and wavelength of light hitting the pixel. Because the rows are activated and read out sequentially, there is an elapsed time between the integration of the first row and the integration of the last row. Because there is an elapsed time between the start of integration of the first row and the start of integration of subsequent rows, a CMOS imaging device with an electronic rolling shutter can capture video images with blur and other rolling shutter effects. CMOS-based imaging devices can also utilize a global shutter to expose pixels in a sensor array. In a global shutter, all rows of pixels are exposed simultaneously (i.e., the start and end of exposure are the same), but readout can (usually) be sequential.

[0006] During imaging, unintended and / or undesirable optical events may occur, resulting in artifacts in the image. For example, holmium lasers, which are laser surgical tools that can be used to remove stones in urological procedures, can produce short-lived, intense light emission (e.g., at visible or IR wavelengths) when interacting with tissue. Optical events can generate artifacts in the resulting endoscopic image, as shown in Figure 4B described herein. Other tissue resection devices, such as green light lasers and RF probes, can cause similar artifacts. The duration, repetition rate, and amplitude of optical events may depend on the energy device.

[0007] Unintended / undesirable optical events can affect imaging with both sensors having a global shutter (e.g., CCD sensors) and sensors having a rolling shutter (e.g., CMOS sensors), but can be more pronounced when using a rolling shutter, as described herein. Furthermore, unintended / undesirable optical events can affect various types of frames, including white light frames and fluorescence image frames. [Overview of the Initiative]

[0008] This specification discloses exemplary devices, apparatus, systems, methods, and non-temporary storage media for medical imaging. More generally, exemplary devices, systems, and methods for reducing or eliminating artifacts caused by unintended / undesirable optical events are disclosed. Systems, devices, and methods may be used to image target tissue in endoscopic imaging procedures, etc. Imaging may be performed preoperatively, intraoperatively, postoperatively, and during imaging sessions and procedures. The imaging method itself may exclude the insertion of an endoscopic imaging device into a lumen in the body. The endoscopic imaging device may be inserted into the lumen before the imaging method. The imaging method itself may exclude any invasive surgical steps.

[0009] While some aspects of this technology are disclosed in relation to specific types of imaging devices (e.g., rolling shutter imaging devices, global shutter imaging devices), it should be understood that the technology can be applied to any type of imaging device. Furthermore, this technology can be applied to non-surgical or non-medical uses.

[0010] An exemplary system may have a "synchronous frame reset" function. In a rolling shutter imaging device, the synchronous frame reset function allows all rows of pixels in the imaging sensor to be reset simultaneously or within a short time (e.g., shorter than the line-by-line offset period of the rolling shutter). Subsequently, for an illumination period shorter than the time between the synchronous frame reset and the reading of the first row, the row of pixels is illuminated, and charge accumulates in the row of pixels for the same amount of time simultaneously, achieving a global shutter effect. The accumulated charge is then read sequentially from the first row to the last row to generate an image frame. The image frame can then be added to a video stream. The synchronous frame reset function can significantly reduce the impact of light events because it shortens the period during which light accumulates in the sensor to less than the frame period. Unintended / undesirable light events occurring before the synchronous frame reset do not affect the image frame. In some examples, the synchronous frame reset step eliminates approximately 46% of shocking light events.

[0011] An exemplary imaging system may include various techniques for detecting artifacts within an image frame. These techniques may include detecting horizontal or vertical lines within the image frame depending on the sensor's mounting orientation, detecting the rate of increase in the average value within consecutive image frames, detecting an increase in saturated pixels within consecutive image frames, detecting mismatches between color channels, detecting an increase in light intensity within the field-of-view aperture region, or any combination thereof. In some examples, machine learning models may be used to detect artifacts within an image frame, including machine learning models configured to detect the aforementioned characteristics.

[0012] An exemplary system may use an n-sized buffer. A sequence of frames captured by the imaging system can be stored in the buffer. Each image stored in the buffer can be associated with its own score. The score indicates the likelihood that the image is corrupted by a light event. Based on the score comparison, image frames in the image sequence may be dropped and replaced. The replacement image can be another image in the buffer or a restored version of the dropped image frame. Advantageously, an N-frame buffer can reduce false positives and maintain a low drop count. In some examples, image frames may be restored using one or more machine learning models. For example, a trained image transformation model may be configured to receive a corrupted image frame and produce an output image with reduced or removed artifacts. In addition, or alternatively, a trained machine learning model may be configured to restore the image frame by correcting artifact regions based on information from other image frames in the buffer.

[0013] Exemplary systems may include shutters (e.g., liquid crystal or LC shutters, mechanical shutters, DLP mirrors, electromechanical shutters). Shutters can be used with rolling shutter imaging devices to eliminate the effects of light events occurring during sequential row readout, blocking light after the illumination period (and thus any unintended / undesirable light events). Shutters with pulse width control can be used to divide the exposure time into multiple shorter periods, reducing the impact of light events on the frame. Shutters can also operate as standalone devices without communication with the camera, blocking light from the imaging path when a light event is detected.

[0014] Any of the techniques described herein may be activated, deactivated, or modified in terms of their sensitivity / aggressiveness. In some examples, a technique may be activated or its sensitivity may be increased in response to the detection of a light event and / or the activation of a surgical energy device. Light events can be detected in numerous ways as described herein. It will be understood that any of the modifications, embodiments, features, and options described in terms of a system are equally applicable to a method, and vice versa. It will also be apparent that one or more of the above modifications, embodiments, features, and options can be combined.

[0015] According to one embodiment, a method is provided for imaging the tissue of a subject using a rolling shutter imaging device to provide a video stream, the method comprising: resetting a plurality of pixel rows of the rolling shutter imaging device within a period shorter than the line-by-line offset period of the rolling shutter imaging device; illuminating the tissue of the subject with illumination light for an illumination period to accumulate charge in the plurality of pixel rows after the reset of the plurality of pixel rows; reading the accumulated charge in the plurality of pixel rows sequentially from the first row to the last row; generating an image frame from the sequentially read accumulated charge in the plurality of pixel rows; and adding the image frame to the video stream.

[0016] Optionally, resetting the plurality of pixel rows of the rolling shutter imaging device includes triggering a synchronous frame reset function of the rolling shutter imaging device. The synchronous frame reset function may include a configurable constant parameter indicating the minimum amount of time the plurality of pixel rows are exposed after the trigger, and the illumination period is configured to be shorter than the constant parameter. The constant parameter is dynamically adjustable for different image frames.

[0017] Optionally, the method further includes determining whether the image frame satisfies one or more criteria; adding the image frame to the video stream in accordance with the determination that the image frame does not satisfy one or more criteria; and discarding the image frame in accordance with the determination that the image frame satisfies one or more criteria. Discarding the image frame may include removing the image frame from the video stream and adding a replacement image frame to the video stream.

[0018] Optionally, the method further includes processing the image frame using a first configuration of an automatic gain control (AGC) algorithm in accordance with the determination that the image frame does not satisfy one or more of the criteria, and processing the image frame using a second configuration of the AGC algorithm or processing the image frame using the AGC algorithm in accordance with the determination that the image frame satisfies one or more of the criteria. Determining whether the image frame satisfies one or more of the criteria may include identifying one or more artifacts in the image frame. The one or more artifacts may be identified in real time. Identifying one or more artifacts in the image frame may include identifying lines in the image frame.

[0019] Optionally, the method further includes applying a Sobel filter to the image frame.

[0020] Optionally, identifying one or more artifacts within the image frame includes calculating the rate of increase from the average value of one or more previous image frames to the average value of the image frame. The rate of increase may be calculated with respect to the region of interest within the image frame.

[0021] Optionally, identifying one or more artifacts within the image frame includes calculating the increase from the number of saturated pixels in the previous image frame to the number of saturated pixels in the current image frame. The increase may be calculated with respect to the region of interest within the image frame.

[0022] Optionally, identifying one or more artifacts within the image frame includes evaluating the differences between at least two of the red, blue, and green channels of the image frame.

[0023] Optionally, identifying one or more artifacts within the image frame includes processing the image frame using a trained machine learning algorithm. The trained machine learning algorithm may be a neural network.

[0024] Optionally, identifying one or more artifacts within the image frame includes detecting an increase in light intensity within the field-of-view aperture region of the image frame.

[0025] Optionally, the method further includes placing the image frames into a buffer of a predefined size, comparing all the frames placed in the buffer, and, based on the comparison, removing one or more frames from the buffer from the video stream. The predefined size may be three image frames. Comparing all the frames placed in the buffer may include assigning a score to each of the frames placed in the buffer and comparing the scores of all the frames placed in the buffer.

[0026] Optionally, the method further includes automatically adjusting one or more parameters of the method. The one or more parameters can include one or more thresholds for identifying artifacts in the image frame, the size of the image buffer, the maximum number of image frames droppable from the image buffer, the maximum number of consecutive image frames droppable from the image buffer, or any combination thereof.

[0027] Optionally, the method further includes automatically adjusting the one or more parameters in response to detection of a light event or detection of activation of a surgical energy device. Detecting activation of the surgical energy device can include receiving a signal from the surgical energy device. The surgical energy device can be a laser unit. The surgical energy device can be an RF probe. Detecting activation of the surgical energy device can include detecting an increase in power consumption of the surgical energy device. Detecting the light event can include receiving a signal from a photodetector attached to the rolling shutter imaging device. Detecting activation of the surgical energy device can include receiving an acoustic signal from the surgical energy device.

[0028] Optionally, the method further includes automatically adjusting the one or more parameters based on detected movement of the imaging device.

[0029] Optionally, the imaging device has a shutter component, and the shutter is configured to close at the end of the illumination period.

[0030] Optionally, the shutter component includes a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electro-mechanical shutter.

[0031] Optionally, the illumination light is generated by at least one LED.

[0032] As an option, the rolling shutter imaging device is part of the endoscopic imaging device.

[0033] According to one embodiment, a computer implementation method for providing a video stream is provided, comprising: accumulating a sequence of images in a buffer of a predefined size, wherein each image in the sequence is associated with a separate score; comparing the scores of the image sequence in the buffer; identifying an image in the image sequence based on the comparison; removing the identified image from the image sequence to obtain an updated sequence; and adding the updated image sequence to the video stream.

[0034] Optionally, the image sequence is acquired by a rolling shutter imaging device.

[0035] Optionally, the image sequence is acquired by a global shutter imaging device configured to read multiple pixel rows simultaneously.

[0036] Optionally, the image sequence is acquired by an endoscopic imaging device.

[0037] Optionally, the method further includes replacing the identified image with a replacement image in the image sequence.

[0038] Optionally, the method further includes, for each image in the image sequence, identifying one or more artifacts within the individual image, and assigning the individual score to the image based on the identification. Identifying one or more artifacts in the image may include identifying lines within the image. Identifying one or more artifacts in the image may include calculating the rate of increase from the mean of one or more previous images to the mean of the image. The rate of increase may be calculated with respect to the region of interest within the image frame.

[0039] Optionally, identifying one or more artifacts in the image includes calculating the increase in the number of saturated pixels from the previous image to the number of saturated pixels in the current image. The increase may be calculated with respect to the region of interest within the image frame.

[0040] Optionally, identifying one or more artifacts in the image includes evaluating the differences between at least two of the red, blue, and green channels of the image frame.

[0041] Optionally, identifying one or more artifacts in the image includes processing the image using a trained machine learning algorithm. The trained machine learning algorithm may be a neural network.

[0042] Optionally, identifying one or more artifacts within the image frame includes detecting an increase in the amount of light within the field aperture of the image frame.

[0043] Optionally, the method further includes automatically adjusting the predefined size of the buffer.

[0044] Optionally, the method further includes automatically increasing the predefined size of the buffer in response to the detection of an optical event or the detection of the operation of a surgical energy device.

[0045] According to one embodiment, a method is provided for generating an image using an endoscopic imaging device, the method comprising: during a frame period, accumulating charge in the pixel array of the endoscopic imaging device during the frame period; stopping a shutter component n times for no more than 1 / n of a predetermined exposure time to allow light to pass through the shutter; and generating the image from reading the charge accumulated in the pixel array after the nth stopping of the shutter component. The method may exclude the insertion of the endoscopic imaging device into a lumen in the body. The endoscopic imaging device may be inserted into the lumen before the method.

[0046] Optionally, the shutter components include a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electromechanical shutter.

[0047] As an option, the endoscopic imaging device is a rolling shutter imaging device.

[0048] As an option, the endoscopic imaging device is a global shutter imaging device.

[0049] Optionally, the stopping times of the shutters are separated by at least the predetermined exposure period.

[0050] Optionally, the method further includes automatically adjusting the value of n. The value of n may be 1 or greater.

[0051] According to one embodiment, a method is provided for shielding an endoscope imaging device from a light event, the method comprising detecting the light event and, in response to the detection of the light event, activating a shutter to shield the sensor of the endoscope imaging device from the light event.

[0052] Optionally, the shutter may include a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electromechanical shutter.

[0053] Optionally, detecting the light event includes detecting the light event via a photodiode detector.

[0054] Optionally, the optical event includes infrared light.

[0055] Optionally, the optical event is generated by a laser, which is a holmium laser.

[0056] Optionally, the method further includes automatically stopping the shutter after a predetermined period. The predetermined period may be approximately 500us to 1ms. The predetermined period is dynamically adjustable.

[0057] Optionally, the method further includes automatically stopping the shutter component in response to the detection of the absence of the light event.

[0058] As an option, the endoscopic imaging device is a rolling shutter imaging device.

[0059] As an option, the endoscopic imaging device is a global shutter imaging device.

[0060] According to one embodiment, a system is provided for imaging the tissue of a subject to provide a video stream, the system comprising an illumination light source and an imaging device having an electronic rolling shutter imaging device, wherein the imaging device is configured to reset a plurality of pixel rows of the rolling shutter imaging device within a period shorter than the line-by-line offset period of the rolling shutter imaging device, illuminate the tissue of the subject with illumination light for an illumination period to accumulate charge in the plurality of pixel rows after the reset of the plurality of pixel rows, read the accumulated charge in the plurality of pixel rows sequentially from the first row to the last row, generate an image frame from the sequentially read charge in the plurality of pixel rows, and add the image frame to the video stream.

[0061] Optionally, resetting the plurality of pixel rows of the rolling shutter imaging device includes triggering a synchronous frame reset function of the rolling shutter imaging device. The synchronous frame reset function may include a configurable constant parameter that indicates the minimum amount of time the plurality of pixel rows are exposed after the trigger, and the illumination period is configured to be shorter than the constant parameter. The constant parameter is dynamically adjustable for different image frames.

[0062] Optionally, the imaging device is configured to further determine whether the image frame satisfies one or more criteria, and to add the image frame to the video stream if it is determined that the image frame does not satisfy one or more criteria, and to discard the image frame if it is determined that the image frame satisfies one or more criteria. Discarding the image frame may include removing the image frame from the video stream and adding a replacement image frame to the video stream.

[0063] Optionally, the imaging device is configured to process the image frame using a first configuration of an automatic gain control (AGC) algorithm if it determines that the image frame does not meet the one or more criteria, and to process the image frame using a second configuration of the AGC algorithm or to process the image frame using the AGC algorithm if it determines that the image frame meets the one or more criteria. Determining whether or not the image frame meets the one or more criteria may include identifying one or more artifacts within the image frame. The one or more artifacts may be identified in real time.

[0064] Optionally, identifying one or more artifacts within the image frame includes identifying lines within the image frame.

[0065] Optionally, the imaging device is further configured to apply a Sobel filter to the image frame.

[0066] Optionally, identifying one or more artifacts within the image frame includes calculating the rate of increase from the average value of one or more previous image frames to the average value of the image frame. The rate of increase may be calculated with respect to the region of interest within the image frame.

[0067] Optionally, identifying one or more artifacts within the image frame includes calculating the increase from the number of saturated pixels in the previous image frame to the number of saturated pixels in the current image frame. The increase may be calculated with respect to the region of interest within the image frame.

[0068] Optionally, identifying one or more artifacts within the image frame includes evaluating the differences between at least two of the red, blue, and green channels of the image frame.

[0069] Optionally, identifying one or more artifacts within the image frame includes processing the image frame using a trained machine learning algorithm. The trained machine learning algorithm may be a neural network.

[0070] Optionally, identifying one or more artifacts within the image frame includes detecting an increase in light intensity within the field-of-view aperture region of the image frame.

[0071] Optionally, the imaging device is configured to further place the image frames into a buffer of a predetermined size, compare all the frames in the buffer, and, based on the comparison, remove one or more frames from the buffer from the video stream. The predetermined size may be three image frames. Comparing all the frames in the buffer may include assigning a score to each of the frames in the buffer and comparing the scores of all the frames in the buffer.

[0072] Optionally, the imaging device is configured to automatically adjust one or more parameters of the method. The one or more parameters may include one or more thresholds for identifying artifacts in the image frame, the size of the image buffer, the maximum number of image frames that can be dropped from the image buffer, the maximum number of consecutive image frames that can be dropped from the image buffer, or any combination thereof.

[0073] Optionally, the imaging device may be configured to automatically adjust the one or more parameters in response to the detection of an optical event or the detection of operation of a surgical energy device. Detecting the operation of the surgical energy device may include receiving a signal from the surgical energy device. The surgical energy device is a laser unit. The surgical energy device is an RF probe. Detecting the operation of the surgical energy device may also include detecting an increase in the power consumption of the surgical energy device. Detecting an optical event may include receiving a signal from a photodetector attached to the rolling shutter imaging device. Detecting the operation of the surgical energy device may also include receiving an acoustic signal from the surgical energy device.

[0074] Optionally, the imaging device is further configured to automatically adjust the one or more parameters based on the detected movement of the imaging device.

[0075] Optionally, the imaging device has a shutter component, which is configured to close at the end of the illumination period. The shutter component may include a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electromechanical shutter.

[0076] Optionally, the illumination light is generated by at least one LED.

[0077] As an option, the rolling shutter imaging device is part of the endoscopic imaging device.

[0078] According to one embodiment, a system for providing a video stream is provided, comprising one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for accumulating an image sequence in a buffer of a predetermined size, wherein each image in the sequence is associated with a separate score, comparing the scores of the image sequence in the buffer, identifying an image in the image sequence based on the comparison, removing the identified image from the image sequence to obtain an updated sequence, and adding the updated image sequence to the video stream.

[0079] Optionally, the image sequence is acquired by a rolling shutter imaging device.

[0080] Optionally, the image sequence is acquired by a global shutter imaging device configured to read multiple pixel rows simultaneously.

[0081] Optionally, the image sequence is acquired by an endoscopic imaging device.

[0082] Optionally, the one or more programs further include instructions for replacing the identified image with a replacement image in the image sequence.

[0083] Optionally, the one or more programs further include instructions for identifying one or more artifacts within each of the images in the image sequence, and for assigning the individual score to the image based on the identification.

[0084] Optionally, identifying one or more artifacts in the image includes identifying lines in the image.

[0085] Optionally, identifying one or more artifacts in the image includes calculating the rate of increase from the average value of one or more previous images to the average value of the image. The rate of increase may be calculated with respect to the region of interest within the image frame.

[0086] Optionally, identifying one or more artifacts in the image includes calculating the increase in the number of saturated pixels from the previous image to the number of saturated pixels in the current image. The increase may be calculated with respect to the region of interest within the image frame.

[0087] Optionally, identifying one or more artifacts in the image includes evaluating the differences between at least two of the red, blue, and green channels of the image frame.

[0088] Optionally, identifying one or more artifacts in the image includes processing the image using a trained machine learning algorithm. The trained machine learning algorithm may be a neural network.

[0089] Optionally, identifying one or more artifacts within the image frame includes detecting an increase in the amount of light within the field aperture of the image frame.

[0090] Optionally, the one or more programs further include instructions for automatically adjusting the predefined size of the buffer.

[0091] Optionally, the one or more programs further include instructions for automatically increasing the predefined size of the buffer in response to the detection of an optical event or the activation of a surgical energy device.

[0092] According to one embodiment, a system for generating an image using an endoscope imaging device is provided, comprising: a shutter component; and an imaging device configured to accumulate charge in the pixel array of the endoscope imaging device during a frame period, to stop the shutter component n times for less than 1 / n of a predetermined exposure time to allow light to pass through the shutter, and to generate the image from reading the charge accumulated in the pixel array after the nth stop of the shutter component.

[0093] Optionally, the shutter components include a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electromechanical shutter.

[0094] As an option, the endoscopic imaging device is a rolling shutter imaging device.

[0095] As an option, the endoscopic imaging device is a global shutter imaging device.

[0096] Optionally, the stopping times of the shutters are separated by at least the predetermined exposure period.

[0097] Optionally, the imaging device is configured to automatically adjust the value of n. The value of n is 1 or greater.

[0098] According to one embodiment, a system is provided for shielding an endoscope imaging device from a light event, comprising: a shutter; and an imaging device configured to detect the light event and, in response to the detection of the light event, activate the shutter to shield the sensor of the endoscope imaging device from the light event.

[0099] Optionally, the shutter may include a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electromechanical shutter.

[0100] Optionally, detecting the light event includes detecting the light event via a photodiode detector.

[0101] Optionally, the optical event includes infrared light.

[0102] Optionally, the optical event is generated by a laser, which is a holmium laser.

[0103] Optionally, the imaging device is configured to automatically stop the shutter after a predetermined period of time. The predetermined period may be approximately 500us to 1ms. The predetermined period is dynamically adjustable.

[0104] Optionally, the imaging device is configured to automatically stop the shutter component in response to the detection of the absence of the light event.

[0105] As an option, the endoscopic imaging device is a rolling shutter imaging device.

[0106] As an option, the endoscopic imaging device is a global shutter imaging device.

[0107] According to one embodiment, a method is provided for imaging the tissue of a subject using a rolling shutter imaging device to provide a video stream, the method comprising: sequentially resetting a plurality of pixel rows of the rolling shutter imaging device from the first row to the last row; transitioning a liquid crystal shutter from a closed state to an open state; illuminating the tissue of the subject with illumination light for an illumination period in order to accumulate charge in the plurality of pixel rows after the liquid crystal shutter has transitioned to the open state and after the last row has been reset; after the illumination period has ended, sequentially reading the accumulated charge in the pixel rows from the first row to the last row; generating an image frame from the charges accumulated in the plurality of pixel rows that have been sequentially read; and adding the image frame to the video stream.

[0108] Optionally, the illumination period is at least a portion of the period from when the last row is reset until when the first row is read.

[0109] Optionally, the illumination period begins when the last row is reset.

[0110] Optionally, the multiple pixel rows are exposed for the same amount of time to generate the image.

[0111] Optionally, the method further includes initiating the transition of the liquid crystal shutter from the open state to the closed state after the illumination period has ended.

[0112] Optionally, the method further includes initiating a transition of the liquid crystal shutter from the open state to the closed state at the end of the illumination period.

[0113] Optionally, the method further includes initiating the transition of the liquid crystal shutter from the open state to the closed state before the end of the illumination period.

[0114] Optionally, the liquid crystal shutter can be opened or closed using a timer device based on vertical synchronization pulses.

[0115] Optionally, the liquid crystal shutter may be opened or closed based on one or more characteristics of the captured scene. Optionally, the one or more characteristics of the captured scene may include the brightness and / or modality of the captured scene.

[0116] Optionally, the illumination light is generated by at least one LED.

[0117] As an option, the rolling shutter imaging device is part of the endoscopic imaging device.

[0118] Optionally, the rolling shutter imaging device is part of a flexible scope and / or a tip-on-tip scope.

[0119] According to one embodiment, a system is provided for imaging the tissue of a subject to provide a video stream, the system comprising an illumination light source and an imaging device having a rolling shutter imaging device, wherein the imaging device is configured to sequentially reset a plurality of pixel rows of the rolling shutter imaging device from the first row to the last row, transition the liquid crystal shutter from a closed state to an open state, illuminate the tissue of the subject with the illumination light source for an illumination period in order to accumulate charge in the plurality of pixel rows after the liquid crystal shutter has transitioned to the open state and after the last row has been reset, after the illumination period has ended, sequentially read the accumulated charge in the pixel rows from the first row to the last row, generate an image frame from the sequentially read accumulated charge in the plurality of pixel rows, and add the image frame to the video stream.

[0120] Optionally, the illumination period is at least a portion of the period from when the last row is reset until when the first row is read.

[0121] Optionally, the illumination period begins when the last row is reset.

[0122] Optionally, the multiple pixel rows are exposed for the same amount of time to generate the image.

[0123] Optionally, the imaging device is further configured to initiate the transition of the liquid crystal shutter from the open state to the closed state after the illumination period has ended.

[0124] Optionally, the imaging device is further configured to initiate a transition of the liquid crystal shutter from the open state to the closed state at the end of the illumination period.

[0125] Optionally, the imaging device is further configured to initiate the transition of the liquid crystal shutter from the open state to the closed state before the end of the illumination period.

[0126] Optionally, the liquid crystal shutter can be opened or closed using a timer device based on vertical synchronization pulses.

[0127] Optionally, the liquid crystal shutter may be opened or closed based on one or more characteristics of the captured scene. Optionally, the one or more characteristics of the captured scene may include the brightness and / or modality of the captured scene.

[0128] Optionally, the illumination light source has at least one LED.

[0129] As an option, the rolling shutter imaging device is part of the endoscopic imaging device.

[0130] Optionally, the rolling shutter imaging device is part of a flexible scope and / or a tip-on-tip scope. [Brief explanation of the drawing]

[0131] The present invention will be described below with reference to the attached drawings. [Figure 1] Figure 1 shows diagrams of endoscopic camera systems, including several examples. [Figure 2] Figure 2 shows a portion of the endoscopic camera system shown in Figure 1 and a target object for imaging, relating to several examples. [Figure 3] Figure 3 is a block diagram of an imaging system, relating to several examples. [Figure 4A] Figure 4A illustrates how unintended / undesirable light events can generate artifacts in rolling shutter imaging sensors with a "global shutter" period, as shown in several examples. [Figure 4B] Figure 4B shows exemplary artifacts in two image frames related to several examples. [Figure 5] Figure 5 provides an exemplary method for imaging subject tissue using a rolling shutter imaging device to provide a video stream, relating to several examples. [Figure 6A] Figure 6A illustrates how artifacts can be reduced or eliminated using rolling shutter imaging sensors with a "synchronous frame reset" function, as shown in several examples. [Figure 6B] Figure 6B shows exemplary artifacts in image frames related to several examples. [Figure 7] Figure 7 shows an exemplary N frame buffer where N is equal to 3, relating to several examples. [Figure 8A] Figure 8A illustrates how light events can affect the exposure period in several examples. [Figure 8B] Figure 8B shows the use of shutters to mitigate the effects of light events on image frames, in several examples. [Figure 9]Figure 9 shows exemplary shutters for blocking unintended / undesirable light events in the imaging path of an imaging device, relating to several examples. [Figure 10] Figure 10 shows another exemplary operation of a rolling shutter imaging sensor to reduce artifacts within an image frame, relating to several examples. [Modes for carrying out the invention]

[0132] Herein, various embodiments and modifications of the systems and methods described herein will be described in detail, along with examples. While several exemplary variations of the systems and methods are described herein, other variations of the systems and methods may include embodiments of the systems and methods described herein, combined in any suitable manner, having all or some of the described embodiments. Examples will be described more fully below with reference to the accompanying drawings, but these may be implemented in different forms and should not be construed as limiting to the embodiments described herein. Rather, these examples are provided to ensure that this disclosure is thorough and complete and to fully convey exemplary implementations to those skilled in the art.

[0133] This specification discloses exemplary devices, apparatus, systems, methods, and non-temporary storage media for medical imaging. More generally, exemplary devices, systems, and methods for reducing or eliminating artifacts caused by unintended / undesirable optical events are disclosed. Systems, devices, and methods may be used to image target tissue in endoscopic imaging procedures, etc. Imaging may be performed preoperatively, intraoperatively, postoperatively, and during imaging sessions and procedures. The imaging method itself may exclude the insertion of an endoscopic imaging device into a lumen in the body. The endoscopic imaging device may be inserted into the lumen before the imaging method. The imaging method itself may exclude any invasive surgical steps.

[0134] While some aspects of this technology are disclosed in relation to specific types of imaging devices (e.g., rolling shutter imaging devices, global shutter imaging devices), it should be understood that the technology can be applied to any type of imaging device. Furthermore, this technology can be applied to non-surgical or non-medical uses.

[0135] An exemplary system may have a "synchronous frame reset" function. In a rolling shutter imaging device with this function, all rows of pixels in the imaging sensor can be reset simultaneously or within a short time (e.g., shorter than the line-by-line offset period of the rolling shutter) using the synchronous frame reset function. Subsequently, for an illumination period shorter than the time between the synchronous frame reset and the reading of the first row, the row of pixels is illuminated, and charge accumulates in the row of pixels for the same amount of time simultaneously, achieving a global shutter effect. The accumulated charge is then read sequentially from the first row to the last row to generate an image frame. The image frame can then be added to the video stream. The synchronous frame reset function can significantly reduce the impact of light events because it shortens the period during which light accumulates in the sensor to less than the frame period. Unintended / undesirable light events occurring before the synchronous frame reset do not affect the image frame. In some examples, the synchronous frame reset step eliminates approximately 46% of shocking light events.

[0136] An exemplary imaging system may include various techniques for detecting artifacts within an image frame. These techniques may include detecting horizontal lines within an image frame, detecting the rate of increase of the mean value in consecutive image frames, detecting an increase in saturated pixels in consecutive image frames, detecting mismatches between color channels, using machine learning models, detecting an increase in light intensity within a field-of-view aperture region, or any combination thereof. In some examples, machine learning models may be used to detect artifacts within an image frame, including machine learning models configured to detect the characteristics described above.

[0137] An exemplary system can use an n-size buffer. Frame sequences captured by the imaging system can be stored in the buffer. Each image stored in the buffer can be associated with a score. The score indicates the likelihood that the image is corrupted by a light event. Based on the score comparison, image frames in the image sequence can be dropped and replaced. Advantageously, an N-frame buffer can reduce false positives and maintain a low drop count.

[0138] Exemplary systems may include shutters (e.g., liquid crystal shutters, mechanical shutters, DLP mirrors, electromechanical shutters). Shutters are used in conjunction with rolling shutter imaging devices to eliminate the effects of light events occurring during sequential row readout, thereby blocking light after the illumination period (and thus any unintended or unwanted light events). Using shutters with pulse width control, the exposure time can be divided into multiple shorter periods, reducing the impact of light events on the frame. Shutters can also operate as standalone devices without communication with the camera, blocking light from the imaging path when a light event is detected.

[0139] Any of the techniques described herein may be activated, deactivated, or modified in terms of their sensitivity / aggressiveness. In some examples, a technique may be activated or its sensitivity may be increased in response to the detection of a light event and / or the activation of a surgical energy device. Detection may be carried out in numerous ways as described herein.

[0140] In the various examples described above, the illumination light may be modulated using pulse width modulation to provide an appropriate amount of illumination for the scene. The imaging system may control the amount of light so that one or more imaging sensors are optimally exposed, and may do so based on the intensity at the sensors in one or more previous frames.

[0141] It should be understood that in the following descriptions, the singular forms "a," "an," and "the" are intended to include the plural form unless the context clearly indicates otherwise. It should also be understood that the phrase "and / or" as used herein means and encompasses one or more and all combinations of the related enumerated items. Furthermore, it should be understood that the phrases "includes," "including," "comprises," and / or "comprising," as used herein, identify the presence of the features, integers, processes, operations, elements, components, and / or units mentioned, but do not exclude the presence or addition of one or more other features, integers, processes, operations, elements, components, units, and / or groups thereof.

[0142] Certain aspects of this disclosure include the processing steps and instructions described herein in the form of algorithms. It should be noted that the processing steps and instructions of this disclosure may be implemented in software, firmware, or hardware, and if implemented in software, they may reside on different platforms used by various operating systems and may be downloaded to operate from such different platforms. As will be apparent from the following description, unless otherwise specified, throughout this specification, any use of terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” and “generating” is understood to refer to the operation and processing of a computer system or similar electronic processing unit that manipulates and transforms data represented as physical (electronic) quantities in computer system memory or registers, or in other such information recording devices, transmitting devices, or display devices.

[0143] In some examples, the present disclosure also relates to apparatus for performing the operations described herein. Such apparatus may be specifically configured for a particular purpose, or may include a general-purpose computer that is selectively started or reconfigured by a computer program stored in the computer. Such computer programs may be stored on non-temporary computer-readable recording media, such as (but not limited to) floppy disks, USB flash drives, external hard drives, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROMs, EEPROMs, magnetic or optical cards, disks of any type including application-specific integrated circuits (ASICs), or any type of medium suitable for storing electronic instructions, each of which may be coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor, or they may be structures employing a multi-processor design to increase computing power.

[0144] The methods, apparatus, and systems described herein are not inherently related to any particular computer or other device. Furthermore, various general-purpose systems may be used for programs following the teachings herein, and it may prove convenient to construct more specialized devices to perform the necessary method steps. The structures required for these various systems will become apparent from the following description. Moreover, the present invention is not described with reference to any particular programming language. It will be understood that various programming languages ​​may be used to carry out the teachings of the present invention as described herein.

[0145] Figure 1 shows an example of an endoscopic imaging system 10 including a scope assembly 11 that can be used for endoscopic procedures. The scope assembly 11 contains an endoscope or scope 12 coupled to a camera head 16 by a coupler 13 located at the distal end of the camera head 16. Light is supplied to the scope by a light source 14 via an optical guide 26, such as an optical fiber cable. The camera head 16 is coupled to a camera control unit (CCU) 18 by an electrical cable 15. The CCU 18 is connected to and communicates with the light source 14. The operation of the camera 16 is partially controlled by the CCU 18. The cable 15 can transmit moving and / or still image data from the camera head 16 to the CCU 18 and can also transmit various control signals bidirectionally between the camera head 16 and the CCU 18.

[0146] A control or switch array 17 may be provided on the camera head 16 to allow the user to manually control various functions of the system 10, which may include switches from one imaging mode to another, as further described later. Voice commands may be input to a microphone 25 attached to a headset 27 worn by the operator and coupled to the voice control unit 23. A handheld device 29, such as a tablet or PDA with a touchscreen user interface, may be connected to the voice control unit 23 as a further control interface. In the illustrated example, a recorder 31 and a printer 33 are also connected to the CCU 18. Additional devices, such as image capture and archiving devices, may be included in the system 10 and connected to the CCU 18. Video image data acquired by the camera head 16 and processed by the CCU 18 is converted into images that can be displayed on the monitor 20, recorded by the recorder 31, and / or used to generate still images, and a hard copy thereof can be produced by the printer 33.

[0147] Figure 2 shows an example of a part of an endoscopic system 10 used to illuminate an object 1, such as a patient's surgical site, and receive light from it. Object 1 may include a fluorescent marker 2, for example, as a result of the patient being administered a fluorescent contrast agent. The fluorescent marker 2 may consist of, for example, indocyanine green (ICG).

[0148] The light source 14 can generate visible illumination light (any combination of red, green, and blue light, etc.) to produce a visible (e.g., white light) image of the object 1, and can also generate fluorescence excitation illumination light to excite a fluorescent marker 2 in the object to produce a fluorescence image. The illumination light is transmitted to and passes through an optical lens system 22 that focuses the light onto an optical pipe 24. The optical pipe 24 can generate uniform light, which is transmitted to an optical fiber guide 26. The optical guide 26 includes multiple optical fibers and may be connected to an optical post 28, which is part of the endoscope 12. The endoscope 12 includes an illumination path 12' and an optical channel path 12''.

[0149] The endoscope 12 may include a notch filter 131 that allows some or all (preferably at least 80%) of the fluorescent light emitted by the fluorescent marker 2 in the object 1 (e.g., in the wavelength range of 830 nm to 870 nm) to pass through, and some or all (preferably at least 80%) of the visible light (e.g., in the wavelength range of 400 nm to 700 nm) such as the visible illumination light reflected by the object 1 to pass through, while substantially blocking all of the fluorescent excitation light (e.g., infrared light with a wavelength of 808 nm) used to excite the fluorescent emission from the fluorescent marker 2 in the object 1. The notch filter 131 may have an optical density of OD5 or higher. In some examples, the notch filter 131 may be located inside the coupler 13.

[0150] Figure 3 schematically shows an exemplary imaging system 300 employing an electronic imaging device 302 to generate images (e.g., still images and / or videos) of an object such as a patient's target tissue, relating to several examples. The imaging device 302 may be a rolling shutter imaging device (e.g., a CMOS sensor) or a global shutter imaging device (e.g., a CCD sensor). System 300 can be used, for example, in the endoscopic imaging system 10 of Figure 1. The imaging device 302 includes a CMOS sensor 304 having an array of pixels 305 arranged in rows 308 of pixels and columns 310 of pixels. The imaging device 302 may include control components 306 for controlling the signals generated by the CMOS sensor 304. Examples of control components include a gain circuit for generating a multibit signal indicating the light incident on each pixel of the sensor 304, one or more analog-to-digital converters, one or more line drivers for supplying driving power to the sensor 304 by acting as buffers, row circuits, and timing circuits. The timing circuits may include components such as bias circuits, clock / timing generation circuits, and / or oscillators. The row circuit may enable one or more processing and / or operational tasks, such as addressing row 308 of pixels, addressing column 310 of pixels, resetting the charge on row 308 of pixels, enabling exposure of pixel 305, decoding signals, amplifying signals, analog / digital signal conversion, applying timing signals, readout signals and reset signals, and other appropriate processes or tasks. The imaging device 302 may also include a mechanical shutter 312, which can be used, for example, to control the exposure of the imaging sensor 304 and / or to control the amount of light received by the imaging sensor 304.

[0151] One or more control components may be integrated into the same integrated circuit into which the sensor 304 is integrated, or they may be separate components. The imaging device 302 may be incorporated into an imaging head such as the camera head 16 of the system 10.

[0152] One or more control components 306, such as row circuits and timing circuits, may be electrically connected to an imaging controller 320, such as a camera control unit 18 of the system 10. The imaging controller 320 may include one or more processors 322 and memory 324. The imaging controller 320 can receive row readouts from the imaging device and control readout timing and other imaging device operations, including mechanical shutter operation. The imaging controller 320 may generate image frames, such as video frames, from row and / or column readouts from the imaging device 302. The generated frames may be provided to a display 350 for display to a user, such as a surgeon.

[0153] The system 300 in this example includes a light source 330 for illuminating the target scene. The light source 330 is controlled by an imaging controller 320. The imaging controller 320 may determine the type of illumination provided by the light source 330 (e.g., white light, fluorescence-excited light, or both), the intensity of the illumination provided by the light source 330, and / or the on / off time of the illumination synchronized with the rolling shutter operation. The light source 330 may include a first photogenerator 332 for generating light of a first wavelength and a second photogenerator 334 for generating light of a second wavelength. For example, in some examples, the first photogenerator 332 may be a white light generator composed of a plurality of separate photogenerating components (e.g., a plurality of LEDs of different colors), and the second photogenerator 334 may be a fluorescence-excited light generator such as a laser diode.

[0154] The light source 330 includes a controller 336 for controlling the light output of the photogenerator. The controller 336 may be configured to provide pulse width modulation of the photogenerator to modulate the intensity of the light provided by the light source 330, which can be used to manage overexposure and underexposure. In some examples, the nominal current and / or voltage of each generator remains constant, and the light intensity is modulated by switching the photogenerator (e.g., LED) on and off according to the pulse width control signal. In some examples, the PWM control signal is provided by the imaging controller 336. This control signal can be a waveform corresponding to the desired pulse width modulation operation of the photogenerator.

[0155] The imaging controller 320 is configured to determine the required illumination intensity from the light source 330 and can generate a PWM signal to be communicated to the light source 330. In some examples, depending on the amount of light received by the sensor 304 and the integration time, the light source may be pulsed at different rates to change the illumination intensity in the target scene. The imaging controller 320 may determine the required illumination intensity for subsequent frames based on the amount of light received by the sensor 304 in the current frame and / or one or more previous frames. In some examples, the imaging controller 320 can control pixel intensity via the PWM of the light source 330 (to increase / decrease the amount of light in a pixel), via the operation of the mechanical shutter 312 (to increase / decrease the amount of light in a pixel), and / or via a change in gain (to increase / decrease the amount of light in a pixel). In some examples, the imaging controller 320 uses the PWM of the illumination source to control pixel intensity while maintaining a gain level with the shutter open (or at least without operating the shutter). The controller 320 may operate the shutter 312 and / or change the gain when the light intensity is at its maximum or minimum and further adjustment is needed.

[0156] (Undesirable optical events affecting global shutter and rolling shutter) During imaging, unintended and / or undesirable optical events may occur, resulting in artifacts in the image. For example, holmium lasers, which are laser surgical tools that can be used to remove stones in urological procedures, can produce short-lived, intense light emission (e.g., at visible or IR wavelengths) when interacting with tissue. Optical events can generate artifacts in the resulting endoscopic image, as shown in Figure 4B described herein. Other tissue resection devices, such as green light lasers and RF probes, can cause similar artifacts. The duration, repetition rate, and amplitude of optical events may depend on the energy device.

[0157] Unintended / undesirable light events can affect imaging with both global shutter sensors (e.g., CCD sensors) and rolling shutter sensors (e.g., CMOS sensors), but artifacts can be more pronounced when using a rolling shutter. In a global shutter imaging sensor, all pixels in the sensor array are exposed simultaneously, and the sensor array is only sensitive to light during the exposure window. Therefore, unintended / undesirable light events will only produce artifacts if the event coincides with the exposure window. For example, at a frame rate of 60 Hz, one frame is captured with a period of 16.6 milliseconds. If the sensor array is exposed for only 1 millisecond within the 16.6 millisecond period, an unintended / undesirable light event will only produce an artifact in the frame if it occurs within the 1 millisecond exposure window. For this reason, artifacts may occur less frequently in global shutter imaging sensors than in rolling shutter imaging sensors.

[0158] Furthermore, global shutter image sensors may offer interlaced scanning, which can further mitigate artifacts caused by unintended or unwanted light events. During interlaced scanning, half of a horizontal pixel row (e.g., even-numbered rows) is captured in one cycle, and the other half (e.g., odd-numbered rows) is captured in the next cycle, requiring two full scans to capture one frame. Therefore, if an unintended or unwanted light event occurs during one cycle, it will only affect half of that frame. For this additional reason, artifacts may be less noticeable in global shutter image sensors.

[0159] However, artifacts can be more frequent and pronounced in rolling shutter imaging sensors. Figure 4A illustrates how unintended / undesirable light events can generate artifacts in an exemplary rolling shutter imaging sensor with a “global shutter” period. In the illustrated system, illumination of the target scene (e.g., target tissue of a patient) is controlled to produce a global shutter type effect with a long integration time for pixels of (one or more) imaging sensors, as described below. Figure 4A shows how unintended / undesirable light events affect the operation of a rolling shutter imaging sensor with a global shutter period, but light events affect the operation of any rolling shutter and global shutter imaging sensor.

[0160] Figure 4A illustrates the exemplary operation of a rolling shutter image sensor on a timescale showing the relative timing of pixel row resets and readouts. As shown, the image sensor is driven according to a nominal frame rate of 120 fps (i.e., 8.3 milliseconds per frame). However, instead of reading out rows of pixels from the sensor every possible frame period, rows are read out every other frame period, allowing the sensor pixels to integrate over a longer period (i.e., effectively two frames of the nominal frame rate). This longer integration period, called the "global shutter" window, is indicated by the shaded parallelogram 402. The sensor array is sensitive over the entire 16.6 millisecond global shutter window. Thus, the shaded parallelogram 402 represents the exposure of a single frame. During the global shutter period, illumination is provided between windows 404 so that all rows of pixels are illuminated simultaneously, and the resulting image frame is virtually free of the rolling shutter effect. Further details of the rolling shutter image sensor can be found in U.S. Patent Application No. 16 / 745154, filed on January 16, 2020, entitled "SYSTEMS AND METHODS FOR MEDICAL IMAGING USING A ROLLING SHUTTER IMAGER," the contents of which are incorporated in their entirety by reference.

[0161] Referring to Figure 4A, if an unintended / undesirable light event occurs, there is a 100% probability that artifacts will appear in one or more frames. For example, any light event occurring during time period 406 (i.e., from the reset of the last row to the readout of the first row), such as light event 2, undesirably illuminates all rows in the current frame, thus causing an artifact (e.g., a blowout) in the current frame. As another example, any light event occurring within parallelogram 402 outside of time period 406, such as light event 1, undesirably illuminates several rows in the current frame and several rows in the adjacent previous frame, causing artifacts (e.g., a horizontal line) in two frames, as shown in Figure 4B. In other words, regardless of when the unintended / undesirable light event occurs, there is a 100% probability that artifacts will appear in one or more frames. This is significantly higher than the probability for global shutter image sensors where artifacts only appear when light events align within an exposure window (e.g., 1 millisecond out of 16.6 milliseconds).

[0162] Figure 4A illustrates how unintended / undesirable optical events affect the operation of a rolling shutter image sensor with a global shutter duration, but optical events affect the operation of any rolling shutter and global shutter image sensors as described above. Furthermore, unintended / undesirable optical events can affect various types of frames, including white light frames and fluorescence image frames.

[0163] (Methods for reducing artifacts within an image frame) Figure 5 provides an exemplary method for imaging subject tissue using a rolling shutter imaging device to provide a video stream. Method 500 generates image frames with artifacts reduced or removed. Method 500 may be performed by an imaging system such as the imaging system 300 in Figure 3, which has a rolling shutter imaging device such as the rolling shutter imaging device 302 of system 300 and a light source such as the light source 330 of system 300. In process 500, some blocks are combined at will, the order of some blocks is changed at will, and some blocks are omitted at will. In some examples, additional steps may be performed in combination with process 500. In some examples, subsets of blocks are performed in separate processes; for example, blocks 502-508 and blocks 510-518 are shown together in Figure 5 but may belong to two separate processes. Thus, the operations illustrated (and described in more detail below) are essentially exemplary and should not be considered limiting as such.

[0164] In step 502, multiple pixel rows of the rolling shutter imaging device are reset. In some examples, the reset is performed via a “synchronous frame reset” function of the rolling shutter imaging device. The synchronous frame reset function clears the charge accumulated in multiple pixel rows simultaneously or within a short time. In some examples, multiple pixel rows are reset within a period shorter than the line-by-line offset period of the rolling shutter (e.g., the period between the start of line 1 and the start of line 2 in Figure 4A). However, unlike a global shutter imaging device, the readout of multiple pixel rows is performed sequentially, row by row, with a time offset, similar to a rolling shutter imaging device. In some systems, the synchronous frame reset function is intended to be used in a photo camera with a flashlight to produce a single-frame photographic image. However, in method 500, the synchronous frame reset feature is used in a novel configuration to reduce or remove artifacts in image frames within a video stream, as described below.

[0165] Figure 6A shows a rolling shutter imaging sensor implementing Method 500 in several examples. Figure 6A shows exemplary operation of the rolling shutter imaging device on a time scale showing the relative timing of pixel row resets and readouts. Similar to the example in Figure 4A, the imaging sensor is driven according to a nominal frame rate of 120 fps (i.e., 8.3 milliseconds per frame), but instead of reading a pixel row of the sensor every possible frame period, a row is read out every other frame period. Thus, only one row readout occurs between two frame periods (i.e., a 16.6 millisecond period). Referring to Figure 6A, at time 0, all rows 1-N of the frame are reset via the synchronous frame reset function of 607. Thus, at time 0, all rows 1-N begin exposure together and accumulate charge. In some examples, the reset occurs simultaneously or substantially simultaneously across all rows (e.g., shorter than the line-by-line offset period of the rolling shutter imaging device).

[0166] Returning to Figure 5, in step 504, after resetting multiple pixel rows, the subject's tissue is illuminated with illumination light over the illumination period. Referring to Figure 6A, at time 0, rows 1-N of the frame are reset via a synchronous frame reset function, after which the field of view is illuminated, and rows 1-N accumulate charge from the light received from the field of view during the illumination period 604. In some examples, the illuminator is controlled to provide illumination light over the illumination period, and for example, a controller can control the imaging sensor and the illuminator so that they are synchronized.

[0167] In step 506, the charges accumulated in multiple pixel rows are read out sequentially from the first row to the last row. Referring to Figure 6A, as shown in 608, the charges accumulated in rows 1 to N are read out sequentially row by row from the first row to the last row.

[0168] In some examples, the synchronous frame reset function provides a configurable constant parameter (also called "G_EXP") that defines the time between the trigger of the synchronous frame reset and the reading of the first row. Therefore, the illumination period must be set shorter than (or at best equal to) the parameter so that the reading of the first row does not occur during the illumination period, and all pixels in the frame are exposed for the entire duration of the illumination period. Referring to Figure 6A, the illumination period is set shorter than G_EXP, and therefore the reading of the first row is set to occur immediately after the illumination period. Thus, when the first row is read at 608, it has been exposed for the entire illumination period 604. The remaining rows remain exposed until they are read sequentially, as shown at 608. While rows are being read, they are not illuminated (except in the case of accidental unintended / undesirable light events). In conventional rolling shutters, the last row receives more light than the first row due to its slower read time, but this is mitigated in Figure 6A because all rows stop receiving light simultaneously.

[0169] In some examples, after the synchronous frame reset function is triggered, a separate signal is required in 608 to trigger the reading of the first row. This separate signal needs to be issued at or after the end of the illumination period so that all pixels of the frame are exposed throughout the entire illumination period before the first row is read.

[0170] Returning to Figure 5, in step 508, an image frame is generated from the readout of accumulated charge in multiple pixel rows. Referring to Figure 6A, as shown in 608, an image frame is generated from the accumulated charge read out sequentially in rows 1 to N. Optionally, rows 1 to N are then reset.

[0171] In some examples of implementing Method 500, the two resets occur at two different times before each image frame is generated. For example, with respect to the second frame in Figure 6A, the first reset of rows 1-N may occur at a first time 608 after the first frame has been read out, the second reset of rows 1-N (i.e., a synchronous frame reset) may occur at a second time 616, and the accumulated charge of rows 1-N is read out at a third time 618. No readout of rows 1-N occurs between the first time 608 and the second time 616. In some examples, the reset at 608 is optional.

[0172] The novel method 500 of using the synchronous frame reset function can significantly reduce the impact of undesirable / unintended light events on image frames. First, any light event occurring before the synchronous frame reset time does not affect the frame. For example, consider light event 2 in Figure 6A. Since light event 2 occurs after the readout of the first frame and before the synchronous frame reset of the second frame, it does not affect either frame. In fact, any light event occurring in window 606 does not affect any image frame.

[0173] Furthermore, even if a light event occurs while a frame is being exposed, the light event will only affect one frame. For example, consider light event 1 in Figure 6A. Since light event 1 occurs before the synchronous frame reset of the second frame, it affects only the first frame and not the second, as shown in Figure 6B. Thus, light event 1 and light event 2 affect a total of one frame. In contrast, in the example shown in Figure 4A without the synchronous frame reset feature, light event 1 and light event 2 affect a total of three frames. In some examples, the synchronous frame reset feature, as shown in Figure 6A, eliminates approximately 46% of disruptive light events.

[0174] According to several examples, the imaging device can be configured for any suitable frame rate. Illustrative frame rates include at least 25 fps, at least 30 fps, at least 50 fps, at least 60 fps, at least 100 fps, at least 120 fps, at least 200 fps, and at least 240 fps. The global shutter window time and / or extended vertical blanking time are generally related to the frame rate capability of the imaging device, with faster frame rates associated with shorter global shutter window times and / or shorter extended vertical blanking times.

[0175] Returning to Figure 5, in step 510, the system determines whether the image frame meets one or more criteria. As mentioned above, the synchronous frame reset function may not eliminate the effects of all optical events (e.g., optical event 1 in Figure 6A). Therefore, step 510 may be performed to identify defective image frames and remove them from the video stream.

[0176] In some examples, step 510 includes identifying one or more artifacts in the image frame in step 512. Exemplary techniques for identifying artifacts in an image frame are provided below.

[0177] In some examples, identifying one or more artifacts within an image frame involves identifying horizontal or vertical lines within the image frame. As shown in Figure 6B, an artifact may include a horizontal line depicting a color change from a darker top to a brighter bottom. The color change is caused by a light event that occurred during sequential reading of the rows (e.g., light event 1 in Figure 6A). The horizontal line can appear anywhere in the image. In alternative examples where the sensor is rotated, the artifact line may have a different angle (e.g., vertical). In some examples, the calculation may be performed with respect to the region of interest.

[0178] In some examples, a Sobel filter is used to detect lines within an image frame. The Sobel filter can be modified to further focus on horizontal lines and / or dark-to-light transitions. An exemplary Sobel convolution may be [1 2 3 2 1; 0 0 0 0 0; -1 -2 -3 -2 -1]. The convolution can be applied to the luminance channel of the image frame. After the Sobel convolution is applied, the Sobel image may be processed to remove all values ​​below a threshold (e.g., 20 in an 8-bit y channel) so that only positive transitions are considered and the noise floor is removed. The Sobel image can be binarized using the mean value, and the number of active pixels in a line can be counted. The system can then differentiate the active pixel count along the y axis. In some examples, a filter (e.g., a Gaussian kernel [0.25 0.5 0.25]) can be applied, and minimum and maximum values ​​can be determined. A horizontal line is detected if the difference between the minimum and maximum values ​​(i.e., the maximum / minimum gradient) exceeds a threshold. It should be understood that other techniques for detecting lines within an image frame (e.g., using other types of filters) are available.

[0179] In some examples, identifying one or more artifacts within an image frame involves calculating the rate of increase from the mean of one or more previous image frames to the mean of the current image frame. Not all light events cause pixel saturation, but many events can still be detected by analyzing the gradient of the increase in the mean. In some examples, the mean of an image frame is the mean of all pixels in the image frame. Then, the deviation of the mean of the image frame (denoted as ΔMean) can be calculated as the difference between the mean of the previous image frame and the mean of the current image frame. The deviation of ΔMean (denoted as ΔΔMean) can then be calculated as the difference between the ΔMean of the last good frame and the ΔMean of the current image frame, divided by the number of frames in between. If ΔΔMean is greater than a predefined threshold, the image frame can be determined to have an artifact. In some examples, the calculation may be performed with respect to the region of interest. In some examples, instead of using the mean, the system may perform the calculation using the median or a modified median (e.g., a median that excludes pixels with darkness above a threshold from the calculation).

[0180] Identifying one or more artifacts within an image frame may involve calculating the increase in the number of saturated pixels in the current image frame from the number of saturated pixels in the previous image frame. A significant increase in saturated pixels from one frame to another (pre-gain) may indicate a light event. In some examples, the number of newly saturated pixels (e.g., all pixels above a certain threshold) may be detected in the frame, and the difference from the last good frame is determined. If the difference is greater than a predefined threshold, the image frame can be determined to have an artifact. In some examples, the calculation may be performed with respect to the region of interest.

[0181] Identifying one or more artifacts within an image frame can be based on color information. For example, identification may involve evaluating the difference between at least two of the image frame's red, blue, and green channels. If the difference between channels (e.g., between red and one or more of the other channels) exceeds a predefined threshold, the image frame can be determined to have an artifact. In some examples, the calculation can be performed with respect to the region of interest.

[0182] Identifying one or more artifacts within an image frame may involve processing the image frame using a trained machine learning algorithm, such as a trained neural network. The algorithm can be trained to receive image frames and classify them as either having artifacts or not having artifacts.

[0183] Identifying one or more artifacts within an image frame can be based on pixels within the aperture region of the image frame that would normally not receive light because the light rays are blocked by the endoscope's aperture. During a light event, blowouts can have a significant impact, causing stray light (e.g., from stray light reflections within the endoscope or camera) hitting the imaging sensor to extend into the aperture region, resulting in an increase in the amount of light within that region. As shown in Figure 6B, the lower part of the aperture region in the affected image frame appears brighter than the completely dark aperture region in the unaffected image frame. Therefore, if the system detects an increase in light intensity in the aperture region that exceeds a predefined threshold, a defective image can be identified.

[0184] In some cases, a filter (e.g., a Sobel filter) may be configured to identify lines within the field of view aperture region in order to identify defective images. For example, the filter may be configured to identify longer lines within the field of view aperture region of the frame near the top or bottom of the frame, and shorter lines within the field of view aperture region near the center of the frame.

[0185] The techniques described above for detecting artifacts can be arbitrarily combined, ordered, and omitted in step 512 of method 500. For example, detecting horizontal or vertical lines may be applied only to image frames captured using a rolling shutter imaging device, while other techniques may be applied to image frames captured using either a rolling shutter imaging device or a global shutter imaging device. In some examples, additional steps may be performed in combination with the techniques. Thus, the techniques described above are essentially illustrative and should not be considered limiting in that sense.

[0186] Returning to Figure 5, in some examples, step 510 includes placing the frames in a buffer of a predefined size in step 514. The frame sequence captured by the imaging device can be stored in the buffer. Each image stored in the buffer can be associated with its own score. The score indicates the degree to which the image is likely to be degraded by a light event. The score can be calculated based on artifacts (or their absence) detected within the image frame, and in some examples, it can be calculated using any combination of the techniques described herein (e.g., horizontal lines, average pixel values, etc.).

[0187] Figure 7 shows an exemplary N-frame buffer where N equals 3. As shown, a sequence of three image frames is stored in the buffer. The three image frames are associated with scores of 60, 90, and 10, respectively, with higher scores indicating a greater likelihood of image defects. The scores of the image frames in the buffer can be compared, and based on this comparison, image frames are identified as images that should be removed from the video stream. In the illustrated example, the system is configured to drop the worst image from the sequence of image frames. Therefore, frame 2 has the worst score and is dropped from the sequence of image frames.

[0188] Advantageously, an N-frame buffer can reduce false positives and maintain a low drop count. In the example shown in Figure 7, if there were no buffer and the system were configured to drop all images with scores above a threshold of 50, the system would drop frame 1. However, frame 1 could be a false positive that should not be dropped—for example, frame 1 might have a higher average pixel count compared to the previous frame because the instrument moved and detected a bright pixel when there was no light event. Furthermore, if the system were configured not to drop consecutive images, the system would keep frame 2 while dropping frame 1, even if frame 2 had a much higher score and was indeed caused by a real light event. With a buffer, the system correctly drops frame 2 and keeps frame 1.

[0189] In the illustrated example, the system is configured to drop a maximum of one image frame from the buffer. In some examples, the system may be able to drop more image frames. The maximum number of image frames that can be dropped may depend on the camera's frame rate, the video stream's frame rate, the algorithm's assertiveness (described below), or a combination of these.

[0190] In some examples, after an image frame is dropped from a sequence of image frames, the system identifies another image frame to replace the dropped image frame in the sequence. In some examples, the replacement image frame is selected from the sequence of image frames based on a score. In the example shown in Figure 7, frame 3 is selected because it has the best score. Thus, the updated sequence of image frames (i.e., frame 1, frame 3, frame 3) can be added to the video stream.

[0191] Buffers can introduce delay into a system. Instead of adding each image frame to the video stream as soon as it is generated, multiple frames are stored in a buffer and evaluated together before the first frame is added to the video stream. The size of the buffer determines the amount of delay that occurs—the larger the buffer, the longer the latency. In some examples, if the camera's frame rate is higher than the video stream's frame rate, the delay may be imperceptible. In other words, the system has a higher delay tolerance. Therefore, the buffer size can be determined based on the system's delay tolerance. In some examples, the buffer size is between 3 and 6 frames.

[0192] In some examples, the buffer has a size of 1 so that only one image frame is stored in memory, in order to replace the next frame if the next frame is defective.

[0193] Returning to Figure 5, in step 516, an image frame is added to the video stream according to the determination that the image frame does not meet one or more criteria, and in step 518, an image frame is discarded according to the determination that the image frame does not meet one or more criteria. If an image frame is discarded, it can be replaced with another image frame (e.g., the most recent good frame, the best frame in the buffer, a repaired frame). In some examples, the number of frames that can be consecutively discarded is limited to a predefined threshold or based on the aggressiveness of the algorithm as described below. In some examples, discarded image frames do not affect the automatic gain exposure ("AGC") algorithm for preventing negative flash; or, discarded images cause the AGC algorithm to adjust with a smaller weighting coefficient.

[0194] Although the steps of Method 500 are described in relation to a rolling shutter imaging device, the techniques in Method 500 can be used with either a rolling shutter imaging device or a global shutter imaging device. For example, steps 510-518 can be used to evaluate image frames captured by a global shutter imaging device to detect defective image frames and remove them from the video stream.

[0195] The aggressiveness or sensitivity of Method 500 can be adjusted automatically or manually. Specifically, any of the steps and parameters of Method 500, such as one or more thresholds for identifying artifacts in image frames, the size of the image buffer, the maximum number of image frames that can be dropped from the image buffer, the maximum number of consecutive image frames that can be dropped from the image buffer, or any combination thereof, can be adjusted automatically. In some examples, the aggressiveness of the Method can be dynamically adjusted when detecting the activation of a surgical energy device (e.g., a laser unit, an RF probe) and / or when detecting an optical event (e.g., via a photodiode). In these cases, the system can automatically activate any combination of the steps of Method 500 or increase their aggressiveness or sensitivity to capture defective image frames. On the other hand, when there is no activation of a surgical energy device or detection of an optical event, the system can automatically deactivate any combination of the steps of Method 500 or decrease their aggressiveness or sensitivity.

[0196] For example, one way to adjust the sensitivity of step 514 is to automatically increase the buffer size in response to the detection of a light event. In other words, the system accepts a higher delay but improves its ability to identify defective images. If there is no light event, the system can automatically decrease the buffer size or bypass it completely (i.e., set the size to 0). As another example, the system can automatically activate or increase the sensitivity of any of the techniques in step 512 in response to the detection of a light event, and the opposite can be done if there is no light event.

[0197] Detecting the activation of a surgical energy device may include receiving a signal from the surgical energy device (e.g., a laser unit, an RF probe). For example, the signal may be from the control unit of the energy device, indicating that the holmium laser has been, is being, or will be emitted by that unit. In some examples, the energy device is commanded by a foot pedal. Thus, a hardware component (e.g., a pressure sensor) may be coupled to the foot pedal and configured to transmit a signal whenever the foot pedal is pressed.

[0198] Detecting the operation of a surgical energy device may include detecting an increase in the power consumption of the surgical energy device. For example, detection may be performed by a remote clamp attached to a power cable to determine the increase in power consumption.

[0199] Detecting a light event may involve receiving a signal from a photodetector attached to the imaging device. The photodetector may be located in the camera head (e.g., mounted on a prism) or clamped to the camera body (e.g., on or near the camera body's entry window) to capture stray or reflected light within the camera body. In some examples, unintended / undesirable light events may include infrared light that is not present in the desired image spectrum of the camera. Therefore, a filter (e.g., an infrared filter) can be placed in front of the photodetector to ensure that the photodetector detects only unintended / undesirable light events. In some examples, the system is configured to treat any light detected by the photodetector that does not correspond to the light source of the imaging device as an unintended / undesirable light event. In some examples, whether an image frame is defective may be determined directly by comparing the timing of the detected light event with the frame's reset time and readout time, as shown in Figure 6A and demonstrated in Figure 6B. If a light event is detected, the system can automatically identify the affected frames and flag them (e.g., by adjusting the score associated with the frames).

[0200] Detecting the operation of a surgical energy device may involve receiving acoustic signals from the surgical energy device. For example, an acoustic microphone can be used to detect acoustic signals indicating optical events such as a safety tone or distinct chatter produced by the laser during firing. Machine learning algorithms can be trained to recognize acoustic signals.

[0201] In some cases, the movement of the imaging device may be detected and used to adjust the sensitivity of the steps in method 500. The movement of the imaging device can be detected via the imaging device's inertial measurement unit or via image tracking. If the imaging device is moving, the sensitivity may be reduced to avoid detecting false positives.

[0202] Some examples include rolling shutter sensors having a horizontal line read from the top row to the bottom row, but it should be understood that, without departing from the spirit of the invention, the techniques described herein are applicable to imaging sensors having different orientations, layouts, and readout sequences.

[0203] (Reducing artifacts using shutters) In some cases, a shutter can be used to reduce or eliminate artifacts caused by unintended or unwanted light events. The shutter can be a liquid crystal shutter, a mechanical shutter, a DLP mirror, or an electromechanical shutter. When activated, the shutter can block a significant portion of the light from the imaging path. When deactivated, the shutter allows a significant portion of the light to pass through.

[0204] Referring to Figure 6A, the shutter (e.g., an LC shutter) may be configured to operate at the end of the illumination period 604 to block a substantial portion of the light from the imaging path. Thus, any optical events occurring during sequential readout at 608, such as optical event 1, do not affect the image frame. In some examples, the shutter may be configured to transition quickly from a stopped state (i.e., allowing a substantial portion of the light to pass through) to an operated state (i.e., blocking a substantial portion of the light). Thus, the pixel rows are reset simultaneously or substantially simultaneously via a synchronous frame reset at 607 to begin charge accumulation, and simultaneously or substantially simultaneously via the shutter to stop receiving light at the end of the illumination period.

[0205] The shutter is stopped before the synchronous frame reset 607. In some examples, the LC shutter takes time to transition from the operating state to the stopped state. Therefore, the stop can be triggered to occur well in advance so that the LC shutter is completely stopped at 607.

[0206] Furthermore, as shown in Figures 8A-B, a shutter with pulse width control can be used to reduce the effects of unintended / undesirable light events. In Figures 8A-B, the global shutter imaging system is driven according to a frame rate of 60 fps (i.e., 16.6 milliseconds per frame) and has an exposure window of 1 millisecond per frame period.

[0207] As shown in Figure 8A, under standard operation, if a light event happens to coincide with a 1ms exposure window, the sensor may be exposed to the light event for most of the 1ms exposure window, potentially causing noticeable artifacts within the image frame.

[0208] In Figure 8B, instead of setting consecutive 1ms exposure windows during a 16.6ms frame period, the sensor is configured to be exposed over the entire 16.6ms frame period to accumulate charge. A shutter with pulse width control is configured to stop only n times (n=5 in Figure 8B), which is less than n times the actual desired 1ms exposure time (200us in Figure 8B), allowing light to pass through the shutter. The image can be generated based on the accumulated charge after the nth stop. In other words, the shutter controls the light so that, even though the sensor is exposed over the entire 16.6ms frame period, light only reaches the sensor for a period less than the desired total exposure time. As shown in Figure 8B, if a light event occurs within the frame period, the impact is not very severe because it can only affect a portion of the total 1ms exposure time.

[0209] The n stops of the LC shutter can be spaced at least by a desired exposure time (e.g., 1 ms in the illustrated example). In some examples, the value of n can be adjusted. For example, n can be automatically increased to reduce the impact of a light event if the system determines, is occurring, continuing to occur, or is likely to occur using the techniques described herein.

[0210] Figure 8B illustrates the use of a shutter in a global shutter imaging system, but it should be understood that shutters can be used similarly in other types of imaging systems, such as rolling shutter imaging systems. Furthermore, in some examples, implementation can be achieved using shutter density modulation.

[0211] Instead of activating and stopping the shutter, multiple charge accumulation periods for an image frame can be achieved using a global shutter sensor that allows for pulsed charge accumulation. For example, the sensor can be activated or stopped based on an external control signal. Charge accumulation can be stopped, for example, using a bypass circuit.

[0212] The shutter can also be configured to operate as a standalone device without communicating with the imaging device. The shutter can be installed within the imaging path. For example, the shutter can be a replacement coupler or drop-in accessory for the eyepiece of an endoscope. The shutter can be powered by its own power source (e.g., an internal battery, a solar cell).

[0213] In particular, the shutter device may have a photodiode for detecting light events. The photodiode can be positioned on the outer surface of the imaging device prism, as shown in Figure 9. For example, the photodiode may be sensitive to IR to detect unintended / undesirable light events (e.g., laser events may include IR), but not to the light source of the imaging device or other desired light for imaging (e.g., emitted fluorescence). When a light event is detected, the LC shutter closes within a very short period (e.g., 500us to 1ms) to protect the image sensor from most contamination. As soon as the detected light event has finished, the LC shutter is configured to open at the same speed it closed. Laser light events are typically shorter than 1ms. Therefore, if the camera exposure is moderately longer than that, the adverse effects are not very noticeable. When the exposure is short, the frame may appear dark rather than blow out.

[0214] Although Figure 9 shows the photodiode mounted on the LC shutter, the photodiode can be mounted elsewhere in the system. In some examples, the stray light photodiode can be placed on the LC shutter on the image / sensor side. It can be mounted facing the sensor on the LC shutter (or mating component) outside the incident image beam clearing aperture (i.e., the beam / image height at this position in Z). The sensor / prism assembly does not capture all incident light, and some of that light is reflected back from the prism assembly. The photodiode can capture this stray light emanating from the prism / sensor assembly.

[0215] In some cases, a stray light photodiode can be mounted on the object / endoscope-side LC shutter. It can be mounted facing the endoscope on the LC shutter (or mating component) outside the incident image beam clearing aperture (i.e., beam / image height at this position in Z). The prism assembly does not capture all incident light, and the LC shutter does not transmit all incident light. Reflections at these interfaces cause reflected stray light to exit the system, be reflected back to the sensor by an upstream optical element (e.g., the endoscope exit window), but outside the original imaging beam. The photodiode can capture a portion of this secondary reflected light from the said upstream optical element.

[0216] In some cases, stray light photodiodes can be mounted on the prism incidence surface. The photodiode can be mounted on the prism glass incidence surface, outside the incident image beam clearing aperture (i.e., the beam / image height at this position in Z), oriented either (a) towards the endoscope or (b) towards the sensor. A photodiode facing the endoscope can capture secondary reflections from the LC shutter or other upstream optical elements (primary reflections are from the sensor / prism assembly out of the system) or stray light from the incident beam originating from the interface of the upstream optical elements. A photodiode facing the sensor can capture primary reflections from the sensor / prism assembly exiting the prism assembly.

[0217] In some examples, the photodiode can be mounted on the prism incidence surface together with a pick-off optical element or a diffractive optical element (DOE). Such a configuration is similar to the photodiode facing the endoscope described above, but with the addition of a pick-off optical element or DOE to enable incident beam sampling (without relying on stray light / internal system reflections).

[0218] In some examples, the photodiode is mounted with a dichroic mirror (beam splitter), and the photodiode is mounted in the free space surrounding the coupler optics, shutter, sensor / prism assembly, and image beam (i.e., the remaining void within the camera housing not occupied by these components). The dichroic mirror is positioned in the optical path of the imaging beam to reflect at least a portion of the IR content of the light event to the photodiode while allowing white light to pass through the sensor assembly.

[0219] In some cases, the photodiode can be mounted in the aforementioned free space facing the entrance window.

[0220] In some examples, the photodiode may be mounted in the aforementioned free space adjacent to the incident window coupled with the pick-off optical element.

[0221] In some examples, upon detection of a light event, the system can compare the timing of the detected light event with the frame's reset and readout times, as shown in Figure 6A and demonstrated in Figure 6B. In this way, the system can automatically identify affected frames, which can then be dropped and replaced. For example, affected frames can be flagged for further processing (e.g., by adjusting the score associated with the frame).

[0222] While some examples are described in relation to LC shutters, it should be understood by those skilled in the art that other types of shutters, such as mechanical shutters, DLP mirrors, electromechanical shutters, and optical shutters, can be used in the examples of this disclosure.

[0223] Figure 10 shows another exemplary operation of a rolling shutter imaging sensor to reduce artifacts within an image frame, relating to several examples. Figure 10 shows the operation on a time scale that illustrates the relative timing of pixel row pauses and readouts.

[0224] The example in Figure 10 is similar to the example in Figure 4A, in that rows are read out every frame period, rather than every period. In this way, the system allows the sensor pixels to be integrated over a longer period (i.e., substantially two frames of the nominal frame rate). The shaded parallelogram 1002 represents the exposure of one single frame when the imaging sensor is driven according to a nominal frame rate of 120 fps (i.e., 8.3 milliseconds per frame). The sensor array is sensitive over the entire 16.6 millisecond window indicated by the shaded parallelogram 1002. In particular, during the time period 1006 between the reset of the last row N and the readout of the first row, all or substantially all rows of the imaging device can be exposed simultaneously, effectively creating a global shutter window that can provide illumination light to prevent the rolling shutter effect.

[0225] In the example shown in Figure 10, illumination period 1004 begins at the start of window 1006. This is in contrast to the example in Figure 4A, where illumination period 404 begins during, rather than at, window 406. Furthermore, the system opens the liquid crystal shutter to expose the pixels in the sensor array to illumination during the illumination period. As shown, the liquid crystal shutter begins its transition from closed to open state just before the start of illumination period 1004. This way, the liquid crystal shutter can be fully transitioned to the open state at the start of illumination period 1004, thus preventing chromatic effects that may occur during the shutter's transition from closed to open state.

[0226] In some examples, the liquid crystal shutter begins its transition from open to closed state shortly after the end of the illumination period 1004. This way, the transition of the liquid crystal shutter does not affect any part of the illumination period, and any chromatic effects that might occur while the shutter is transitioning between states can be prevented again. In another example, the liquid crystal shutter can be closed before or at the end of the illumination period, as its closing speed is much faster than its opening speed, and the chromatic effects may become less pronounced.

[0227] In one exemplary implementation, the liquid crystal shutter takes approximately 1.3 ms to fully open and 50 us to fully close. Furthermore, the shutter is judged to be color-stable after approximately 800 us on the aperture side. Therefore, the shutter can be configured to begin transitioning to the open state 800 us before the illumination period. Since there is no effect when the light source is turned off, the transition to the closed state can begin simultaneously with the end of the illumination period. Another reason for transitioning to the closed state after the end of the illumination period is that the system can operate the shutter without knowing the desired light pulses, and the automatic gain control in the camera can compensate for the difference. It should be understood that the asymmetric opening and closing times of the liquid crystal shutter are based on how it is configured, and the shutter can be configured in the reverse manner to make the opening time faster and the closing time slower, and the timing of opening and closing the shutter relative to the illumination period can be adjusted accordingly.

[0228] In some examples, the illumination period can begin at a different time within window 1006, rather than at the beginning of window 1006. The entire illumination period fits within window 1006. Furthermore, the liquid crystal shutter is configured to open and close based on the illumination period, as described above. Since the camera and light source vary the exposure time based on the length of 1006, the system can either fix the start time of the illumination period and vary the end time, or fix the end time of the illumination period and vary the start time.

[0229] In some examples, a timer device can be used to set the open and closed times of the liquid crystal shutter based on vertical synchronization pulses (either frame acquisition vsync or light source vsync). In some examples, the system can adjust the open and / or closed times of the liquid crystal shutter based on the imaging scene (e.g., brightness, modality). In some examples, a thermoconverter can be used to generate a relatively high voltage (e.g., 24V) into the camera head from a lower operating voltage to drive the liquid crystal shutter at a relatively high voltage.

[0230] While some examples are illustrated with reference to endoscopes, it should be understood by those skilled in the art that the techniques described herein can be used in any imaging system, including flexible and / or tip-on-tip scopes, such as flexible digital ureteroscopes, where an imaging sensor is located at the distal end of the scope.

[0231] The foregoing disclosures are disclosed with reference to specific examples for illustrative purposes. However, the above illustrative discussions are not intended to be exhaustive or to limit the invention to the exact forms disclosed. While features have been described in this specification as part of the same or separate examples for clarity and brevity, it will be understood that the scope of the disclosure includes examples having all or some combinations of the described features. In light of the above teachings, many modifications and variations are possible. The examples have been selected and disclosed to best illustrate the principles of the art and their practical applications. This will enable those skilled in the art to best utilize the art and the various examples by making various modifications suitable for specific conceivable applications.

[0232] While the present disclosure and examples have been adequately described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications should be understood to fall within the scope of the disclosure and examples as defined by the claims. Finally, the entire disclosures of the patents and publications referenced in this application are incorporated herein by reference.

Claims

1. A method for operating a system for imaging the tissue of a subject to provide a video stream, wherein the system comprises a control unit, an illumination light source, and an imaging device having a rolling shutter imaging device, and the method involves the control unit controlling the illumination light source and the imaging device, The process involves sequentially resetting multiple pixel rows of the rolling shutter imaging device from the first row to the last row, Transitioning the liquid crystal shutter from the closed state to the open state, After the liquid crystal shutter transitions to the open state and after the last row is reset, light is generated from the illumination light source to illuminate the tissue of the subject over the illumination period in order to accumulate charge in the plurality of pixel rows. The liquid crystal shutter is transitioned from the open state to the closed state, wherein the liquid crystal shutter in the closed state blocks light from light sources other than the illumination light source after the illumination period has ended. After the illumination period has ended and the liquid crystal shutter is in the closed state, the accumulated charge in the pixel row is read sequentially from the first row to the last row. To generate an image frame from the charge accumulated in the plurality of pixel rows that were read sequentially, Adding the aforementioned image frame to the video stream, Methods that include...

2. A method according to claim 1, wherein the illumination period is at least a portion of the period from when the last row is reset to when the first row is read.

3. A method according to claim 2, wherein the illumination period begins when the last row is reset.

4. A method according to any one of claims 1 to 3, wherein the control unit causes the plurality of pixel rows to be exposed for the same amount of time to generate the image frame.

5. A method according to any one of claims 1 to 4, wherein the control unit causes the liquid crystal shutter to transition from the open state to the closed state after the end of the illumination period.

6. A method according to any one of claims 1 to 4, wherein the control unit causes the liquid crystal shutter to transition from the open state to the closed state at the end of the illumination period.

7. A method according to any one of claims 1 to 4, wherein the control unit causes the liquid crystal shutter to transition from the open state to the closed state before the end of the illumination period.

8. A method according to any one of claims 1 to 7, wherein the control unit uses a timer device based on vertical synchronization pulses to open or close the liquid crystal shutter.

9. A method according to any one of claims 1 to 8, wherein the control unit causes the liquid crystal shutter to open or close based on one or more characteristics of the captured scene.

10. A method according to claim 9, wherein the one or more characteristics of the captured scene include the brightness and / or modality of the captured scene.

11. A method according to any one of claims 1 to 10, wherein the illumination light source is at least one LED.

12. A method according to any one of claims 1 to 11, wherein the rolling shutter imaging device is part of an endoscope imaging device.

13. A method according to any one of claims 1 to 11, wherein the rolling shutter imaging device is part of a flexible scope and / or a tip-on-tip scope.

14. A system for imaging the tissue of a subject in order to provide a video stream, wherein the system is Light source and The imaging device has a rolling shutter imaging device, and the imaging device has The multiple pixel rows of the rolling shutter imaging device are sequentially reset from the first row to the last row. The liquid crystal shutter is transitioned from the closed state to the open state. After the liquid crystal shutter has transitioned to the open state and after the last row has been reset, the tissue of the subject is illuminated with the illumination light source over an illumination period in order to accumulate charge in the plurality of pixel rows. The liquid crystal shutter is transitioned from the open state to the closed state, and the liquid crystal shutter in the closed state blocks light from light sources other than the illumination light source after the illumination period has ended. After the illumination period has ended and the liquid crystal shutter is in the closed state, the accumulated charge in the pixel row is read sequentially from the first row to the last row. An image frame is generated from the charge accumulated in the plurality of pixel rows that were read sequentially. A system configured to add the aforementioned image frames to the video stream.

15. The system according to claim 14, wherein the illumination period is at least a portion of the period from when the last row is reset to when the first row is read.

16. The system according to claim 15, wherein the illumination period begins when the last row is reset.

17. A system according to any one of claims 14 to 16, wherein the plurality of pixel rows are exposed for the same amount of time to generate the image frame.

18. A system according to any one of claims 14 to 17, wherein the imaging device is further configured to initiate a transition of the liquid crystal shutter from the open state to the closed state after the end of the illumination period.

19. A system according to any one of claims 14 to 17, wherein the imaging device is further configured to initiate a transition of the liquid crystal shutter from the open state to the closed state at the end of the illumination period.

20. A system according to any one of claims 14 to 17, wherein the imaging device is further configured to initiate a transition of the liquid crystal shutter from the open state to the closed state before the end of the illumination period.

21. A system according to any one of claims 14 to 20, wherein the liquid crystal shutter is opened or closed using a timer device based on vertical synchronization pulses.

22. A system according to any one of claims 14 to 21, wherein the liquid crystal shutter is opened or closed based on one or more characteristics of the captured scene.

23. The system according to claim 22, wherein the one or more characteristics of the captured scene include the brightness and / or modality of the captured scene.

24. A system according to any one of claims 14 to 23, wherein the illumination light source has at least one LED.

25. A system according to any one of claims 14 to 24, wherein the rolling shutter imaging device is part of an endoscope imaging device.

26. A system according to any one of claims 14 to 24, wherein the rolling shutter imaging device is part of a flexible scope and / or a tip-on-tip scope.

Images