Methods and systems for automatically switching the light source on / off of an endoscope during surgical procedures.

Abstract

This patent disclosure discloses a machine learning-based system for automatically switching an endoscopic camera's light source on / off during a surgical procedure. The disclosed system receives a sequence of video images captured by the endoscopic camera when the light source is switched on. The system then analyzes the video image sequence using a machine learning classifier to classify each video image into a first category (image inside the patient's body) or a second category (image outside the patient's body). The system then determines, based on the classified video images, whether the endoscopic camera is inside or outside the patient's body. When it is determined that the endoscopic camera is outside the patient's body, the system generates a control signal to immediately switch off the light source.

Description

Technical Field

[0001] This disclosure relates generally to surgical automation, and more specifically, to systems, apparatus and techniques for automatically switching the light source of the endoscope on / off for automatic white balance when the endoscope is inserted into / removed from the patient's body or pointed at a white object. Background Technology

[0002] During laparoscopic or robotic surgery, it may be necessary to remove an endoscope from the patient's body for various reasons, such as lens cleaning, and then reinsert it to continue its imaging function. Endoscopes are typically equipped with a strong light source, such as a light-emitting diode (LED) or laser, in which light propagates downwards to the distal end of the endoscope inside the patient's body to illuminate the body cavity, allowing the endoscopic camera to "see" and record. However, if this strong light source is not turned off when removing the endoscope from the patient's body, accidentally pointing the endoscope into the eye of any nearby surgical personnel can cause potentially permanent and significant retinal damage. To avoid this potential health risk when the endoscope is outside the patient's body, endoscope operators are trained to immediately keep their palm against the end of the endoscope until the light source is manually turned off. Similarly, before inserting the endoscope back into the patient's body, the operator needs to manually turn the light back on, keeping his / her palm against the endoscope tip to block the light, until the endoscope has entered the cannula or otherwise inserted into the patient's body.

[0003] However, manual control of the endoscope light source remains a risk to surgical personnel if the surgeon removing the endoscope has poor coordination in blocking and turning off the light, or simply forgets to turn it off. Furthermore, in some operating room (OR) settings, the responsibility of the circulating nurse or scrubbing technician lies with the surgeon whether they are pushing the endoscope into the patient or removing it to turn the light source on or off. These settings rely on coordination between the surgical personnel and the surgeon to ensure the safety of people in the OR, and therefore may have a higher associated risk of accidental light exposure compared to the surgeon's control over the endoscope and the light source's on / off state.

[0004] Therefore, an automated process is needed to turn the endoscope light source on and off when inserting / removing the endoscope into / from the patient's body, without the aforementioned problems. Summary of the Invention

[0005] This patent discloses various embodiments of a machine learning-based detection / control technique for automatically switching the endoscope's light source on / off for automatic white balance when the endoscope is inserted into / removed from the patient's body or pointed at a white object. In some embodiments, to enable automatic switching of LED or laser light sources, a statistical classifier for video scene analysis is first constructed, particularly for distinguishing scenes from inside and outside the patient's body. Next, the statistical classifier can be applied to real-time video images captured by the endoscope to determine the endoscope's state, such as whether the endoscope is inside or outside the patient's body. This allows for automatic detection of changes in the endoscope's state.

[0006] More specifically, if the endoscope changes from being inside the patient's body to being outside the patient's body, a control signal can be immediately generated to turn off the endoscope's light source. Conversely, if the endoscope changes from being outside the patient's body to being inside the patient's body, a control signal can be immediately generated to turn on the endoscope's light source. Therefore, the disclosed control technology eliminates the need for manual intervention when turning the endoscope's light source on / off. It should be noted that the disclosed technology essentially detects the transition of the endoscope between these two states; therefore, the detection of the state change is instantaneous with minimal delay. In other words, the disclosed technology can immediately detect the moment the endoscope is outside the patient's body and immediately turn off the light source.

[0007] Another object of this disclosure is to provide a technique for enabling automatic switching of the endoscope's white balance mode for automatic white balancing. More specifically, the proposed automatic white balancing technique is designed to detect and identify the endoscope pointing at a white balance object, such as a white card or a white sponge. The proposed automatic white balancing technique can then trigger white balance operation, including turning on a light source to allow automatic white balancing to be performed. In some embodiments, the proposed automatic white balancing technique can also detect the end of the white balance operation by recognizing that the endoscope is being pointed away from the white balance object. For example, the proposed automatic white balancing technique may include a video analysis tool for detecting scene changes from the image of the white balance object to different objects. The proposed automatic white balancing technique can then turn off the white balance mode, including turning off the light source again. It should be noted that the light source can remain off until the endoscope is reinserted into the patient, whereby the proposed automatic light source control technique will turn the light source back on to restore imaging. Therefore, the proposed automatic white balancing technique completely eliminates the need for manual intervention during white balancing, and thus saves significant OR time and hassle.

[0008] In one aspect, a process for automatically turning an endoscope camera's light source on / off during a surgical procedure is disclosed. The process can begin by receiving a first sequence of video images captured by the endoscope camera when the light source is turned on. The process then analyzes the first video image sequence using a statistical classifier (e.g., a machine learning-based classifier) ​​to classify each video image in the first video image sequence as either a first category image inside the patient's body or a second category image outside the patient's body. The process then determines whether the endoscope camera is inside or outside the patient's body based on the classified first video image sequence. When it is determined that the endoscope camera is outside the patient's body, the process subsequently generates a first control signal for turning off the light source, wherein the first control signal is used to immediately turn off the light source.

[0009] In some implementations, the process continues to monitor the status of the endoscope camera once it is determined that the camera is inside the patient's body.

[0010] In some implementations, the process determines the state of the endoscope camera as being outside the patient's body by identifying a first transition event, based on a classified first video image sequence, that moves the endoscope camera from inside the patient's body to outside the patient's body.

[0011] In some implementations, after the light source has been turned off, the process may further receive a second sequence of video images captured by the endoscopic camera when the light source was disconnected. The process then processes the second video image sequence to determine whether the endoscopic camera remains outside the patient's body. If so, the process continues to monitor the status of the endoscopic camera. Otherwise, the process generates a second control signal for turning the light source back on.

[0012] In some implementations, the process processes a second video image sequence to determine whether the endoscope camera remains outside the patient's body by: classifying each video image in the second video image sequence into a first category image inside the patient's body or a second category image outside the patient's body using a statistical classifier; and determining the state of the endoscope camera as inside or outside the patient's body based on the classified second video image sequence.

[0013] In some implementations, the process determines the state of the endoscope camera as being inside the patient's body by identifying a second transition event, based on a classified second video image sequence, that moves the endoscope camera from outside the patient's body into the patient's body.

[0014] In some implementations, while the light source remains off and the endoscope camera is outside the patient's body, the process further detects that the endoscope camera is pointing at a white balance object and subsequently generates a third control signal to turn on the light source for white balance operation.

[0015] In some implementations, the process detects that the endoscope camera is pointing at a white balance object by: receiving a real-time video image captured by the endoscope camera when the light source is turned off; and processing the video image to identify a predefined white balance object within the video image.

[0016] In some implementations, the process trains the statistical classifier by: receiving multiple surgical videos, each of which contains video images of an endoscope both inside and outside the patient's body; for each of the multiple surgical videos, labeling the corresponding video image as a first-class image or a second-class image; and training the statistical classifier to distinguish the input image into a first-class image or a second-class image based on the labeled video images.

[0017] In some implementations, the process trains a statistical classifier to distinguish input images based on statistics of different sets of image features such as color, texture, and contrast.

[0018] In another aspect, an endoscope system is disclosed. This endoscope system may include: an endoscope camera module; a light source module coupled to the endoscope camera module for providing a light source to the endoscope camera module; and a light source control module coupled to the endoscope camera module and the light source module. More specifically, the light source control module is configured to automatically turn the light source in the light source module on / off during surgical procedures by: receiving a first sequence of video images captured by the endoscope camera module when the light source is turned on, and determining whether the endoscope camera module is inside or outside the patient's body based on classifying each video image in the first video image sequence into a first category of images inside the patient's body or a second category of images outside the patient's body using an image classifier. If it is determined that the endoscope camera module is inside the patient's body, the light source control module is configured to continue monitoring the status of the endoscope camera. However, if it is determined that the endoscope camera module is outside the patient's body, the light source control module is further configured to generate a first control signal for turning off the light source, wherein the control signal is used to immediately turn off the light source.

[0019] In some implementations, the light source control module is further configured to automatically turn the light source in the light source module on / off during surgical procedures by: receiving a second video image sequence captured by the endoscope camera module when the light source is off; and processing the second video image sequence to determine whether the endoscope camera module remains outside the patient's body. If it is determined that the endoscope camera module remains outside the patient's body, the light source control module continues to monitor the status of the endoscope camera module. However, if it is determined that the endoscope camera module is no longer outside the patient's body, the light source control module generates a second control signal for turning the light source on. Attached Figure Description

[0020] The structure and operation of this disclosure will be understood by reviewing the following detailed description and accompanying drawings, in which similar reference numerals refer to similar components, and wherein:

[0021] Figure 1 A block diagram of an exemplary endoscope system including an automatic light source on / off control subsystem according to some embodiments described herein is shown.

[0022] Figure 2 A flowchart illustrating an exemplary process for automatically turning the light source of an endoscopic camera module on / off during surgical procedures, according to some embodiments described herein, is presented.

[0023] Figure 3 A flowchart illustrating an exemplary process for training a disclosed statistical classifier to classify surgical video images as being inside or outside a patient's body, according to some embodiments described herein, is presented.

[0024] Figure 4 The photographic images presented show the conventional method of manually turning the light source on / off the endoscope camera module.

[0025] Figure 5 A computer system is conceptually illustrated that can be used to implement some of the embodiments of the techniques in this subject matter. Detailed Implementation

[0026] The specific embodiments listed below are intended to describe various configurations of the subject matter and are not intended to represent the only configuration in which the subject matter can be practiced. The accompanying drawings are incorporated herein and form part of the specific embodiments. The specific embodiments include particular details intended to provide a thorough understanding of the subject matter. However, the subject matter is not limited to the particular details listed herein and can be practiced without these particular details. In some cases, structures and components are shown in block diagrams to avoid obscuring the concept of the subject matter.

[0027] This patent discloses various embodiments of a machine learning-based detection / control technique for automatically switching the endoscope's light source on / off for automatic white balance when the endoscope is inserted into / removed from the patient's body or pointed at a white object. In some embodiments, to enable automatic switching of LED or laser light sources, a statistical classifier for video scene analysis is first constructed, particularly for distinguishing scenes from inside and outside the patient's body. Next, the statistical classifier can be applied to real-time video images captured by the endoscope to determine the endoscope's state, such as whether the endoscope is inside or outside the patient's body. This allows for automatic detection of changes in the endoscope's state.

[0028] More specifically, if the endoscope changes from being inside the patient's body to being outside the patient's body, a control signal can be immediately generated to turn off the endoscope's light source. Conversely, if the endoscope changes from being outside the patient's body to being inside the patient's body, a control signal can be immediately generated to turn on the endoscope's light source. Therefore, the disclosed control technology eliminates the need for manual intervention when turning the endoscope's light source on / off. It should be noted that the disclosed technology essentially detects the transition of the endoscope between these two states; therefore, the detection of the state change is instantaneous with minimal delay. In other words, the disclosed technology can immediately detect the moment the endoscope is outside the patient's body and immediately turn off the light source.

[0029] It is important to note that automatic white balancing is a mandatory step in preparing the endoscope before initiating any surgical procedure. Furthermore, during surgical procedures, after the endoscope has been temporarily placed outside the patient for cleaning or other reasons (described in more detail below), the endoscope needs to be recalibrated; that is, another white balancing is performed before the endoscope can be reinserted into the patient. Assuming the endoscope light source is turned off before performing white balancing for the described safety reasons, the surgeon will typically need to manually turn the white balancing mode on / off. This involves manually turning the light source on to perform the white balancing operation and then manually turning it off when the white balancing is complete.

[0030] Another object of this disclosure is to provide a technique for enabling automatic switching of the endoscope's white balance mode for automatic white balancing. More specifically, the proposed automatic white balancing technique is designed to detect and identify the endoscope pointing at a white balance object, such as a white card or a white sponge. The proposed automatic white balancing technique can then trigger white balance operation, including turning on a light source to allow automatic white balancing to be performed. In some embodiments, the proposed automatic white balancing technique can also detect the end of the white balance operation by recognizing that the endoscope is being pointed away from the white balance object. For example, the proposed automatic white balancing technique may include a video analysis tool for detecting scene changes from the image of the white balance object to different objects. The proposed automatic white balancing technique can then turn off the white balance mode, including turning off the light source again. It should be noted that the light source can remain off until the endoscope is reinserted into the patient, whereby the proposed automatic light source control technique will turn the light source back on to restore imaging. Therefore, the proposed automatic white balancing technique completely eliminates the need for manual intervention during white balancing, and thus saves significant OR time and hassle.

[0031] During laparoscopic or robotic surgery, for various reasons, such as lens cleaning, it may be necessary to remove the endoscope from the patient's body and then reinsert it to continue its imaging function. The period during which the endoscope is removed from the patient's body and subsequently reinserted during a surgical procedure is referred to as an outside-of-body (OOB) event. It is important to note that each surgical procedure may include multiple OOB events, which occur for various reasons. For example, an OOB event occurs if the endoscope lens must be cleaned, such as when the endoscope lens is covered in blood. Another type of OOB event involves changing the endoscope lens from one field of view size to another for different anatomical structures / fields of view (FOV). Typically, the endoscope camera continues recording during an OOB event. The proposed technique can use video images captured during an OOB event to determine when to turn the light source on / off during the OOB event to ensure the safety of surgical personnel in the OR, and when to turn the light source on / off for automatic white balance.

[0032] Figure 1 A block diagram of an exemplary endoscope system 100 including an automatic light source on / off control subsystem 110 according to some embodiments described herein is shown. Figure 1As can be seen, the endoscope system 100 includes an endoscope camera module 102, a light source unit 104, and a proposed automatic light source on / off control subsystem 110 (or "light source control subsystem 110" hereinafter). The light source control subsystem 110 is coupled to the output of the endoscope camera module 102 and configured to receive real-time surgical procedure video and / or still images (collectively referred to below as "surgical video images 120") during live surgical procedures. The endoscope camera module 102 is typically a rod-shaped or tubular structure containing physical imaging components such as a CCD camera, optical lenses, and optical fibers connecting to sources such as light sources. The light source control subsystem 110 is also coupled to the input of the light source module 104 and configured to generate and output an automatic light source on / off control signal 130 (or "light source control signal 130" hereinafter) during live surgical procedures. Figure 1 As can be seen, the light source module 104 includes a physical light source 106, such as an LED light source or a laser light source, and a power control unit 108. The power control unit 108 can be configured to receive a light source control signal 130 from the light source control subsystem 110, and immediately turn the light source 106 on or off based on the light source control signal 130.

[0033] The proposed light source on / off control subsystem 110 may include one or more video image processing modules, including video image processing modules 112 and 114. Specifically, video image processing module 112 is configured to perform surgical scene analysis to determine whether the endoscope camera module 102 is inside or outside the patient's body. Video image processing module 112 is further configured to detect two transition events: (1) when the endoscope camera module 102 is moved from inside the patient's body to outside the patient's body, hereinafter also referred to as the "first transition event"; and (2) when the endoscope camera module 102 is moved from outside the patient's body to inside the patient's body, hereinafter also referred to as the "second transition event". Video image processing module 112 is configured to generate a "power off" signal 130-1 for turning off the light source 106 in the light source module 104 when the first transition event is detected, and to generate a "power on" signal 130-2 for turning on the light source 106 in the light source module 104 when the second transition event is detected. The light source control subsystem 110 is configured to output control signals 130-1 and 130-2 generated by the video image processing module 112 to the light source module 104. Upon receiving the control signal 130, the power control unit 108 of the light source module 104 is configured to immediately turn off the light source 106 of the endoscope camera module 102 upon receiving the power-off signal 130-1, or immediately turn on the light source 106 of the endoscope camera module 102 upon receiving the power-on signal 130-2.

[0034] The proposed video image processing module 114 is configured to perform surgical scene analysis to detect the start and end of white balance operation when the endoscope camera module 102 is outside the patient's body. The video image processing module 114 is also configured to generate a "power-on" signal 130-3 to turn on the light source 106 in the light source module 104 when the start of white balance operation is detected, or a "power-off" signal 130-4 to turn off the light source 106 in the light source module 104 when the end of white balance operation is detected. The light source control subsystem 110 is configured to output control signals 130-3 and 130-4 generated by the video image processing module 114 to the light source module 104. Upon receiving the control signal 130, the power control unit 108 of the light source module 104 is configured to immediately turn on the light source 106 of the endoscope camera module 102 upon receiving the power-on signal 130-3, or immediately turn off the light source 106 of the endoscope camera module 102 upon receiving the power-off signal 130-4. It should be noted that the video image processing module 114 does not need to engage when the endoscope camera module 102 is inside the patient's body. In some embodiments, the video image processing module 114 engages after the video image processing module 112 detects a first transition event, and disengages after the video image processing module 112 detects a second transition event.

[0035] In some embodiments, the video processing module 112 may include a machine learning-based or computer vision-based statistical classifier trained to distinguish between scenes originating inside and outside the patient's body. The video processing module 112 may apply this machine learning-based or computer vision-based statistical classifier to real-time surgical video images 120 captured by the endoscope camera module 102 to classify the real-time scene depicted in the surgical video images 120 as (1) inside or (2) outside the patient's body. In some embodiments, the statistical classifier is configured to classify / label each received video image 120 as (1) inside or (2) outside the patient's body. It should be noted that a first transition event or a second transition event is a continuous action represented by a consecutive sequence of video images. Therefore, after the sequence of video images 120 has been classified / labeled, a transition event can be detected if the sequence of video images 120 contains a first transition event or a second transition event. While the first and second transition events can be detected based on only one or a few classified / labeled video frames, the sequence generation decision based on labeled video images 120 allows for the identification of correlations between video image sequences, in order to generate a more accurate decision about whether the first or second transition event exists in the video image sequence.

[0036] In some implementations, if the endoscope camera module 102 is initially inside the patient's body (and the light source 106 is on), and the statistical classifier has classified one or more newly received surgical video images 120 as being outside the patient's body, a first transition event can be detected, and the video processing module 112 immediately generates a "power off" control signal 130-1. Alternatively, if the endoscope camera module 102 is initially inside the patient's body (and the light source 106 is on), and the statistical classifier has classified one or more newly received surgical video images 120 as being inside the patient's body, no transition event occurs, and the video processing module 112 continues to monitor the state of the endoscope camera module 102 without generating any control signal 130. Note that by detecting the first transition event, the video processing module 112 can detect the moment when the endoscope camera module 102 is taken outside the patient's body with minimal delay, so as to immediately turn off the light source.

[0037] Similarly, if the endoscope camera module 102 is initially outside the patient's body (and the light source 106 is off), and the statistical classifier has classified one or more newly received surgical video images 120 as being inside the patient's body, a second transition event can be detected, and the video processing module 112 immediately generates a "power-on" control signal 130-3. Alternatively, if the endoscope camera module 102 is initially outside the patient's body (and the light source 106 is off), and the statistical classifier has classified one or more newly received surgical video images 120 as being outside the patient's body, no transition event occurs, and the video processing module 112 continues to monitor the state of the endoscope camera module 102 without generating any control signal 130.

[0038] It should be noted that another important application of the disclosed light source control subsystem 110 is that the video processing module 112 automatically turns off the light source 106 whenever the endoscope camera module 102 is removed from the patient's body. Therefore, even if the endoscope camera module 102 is subsequently pointed at a whiteboard containing sensitive information in the OR or another object in the OR associated with privacy issues, the sensitive information cannot be clearly captured or identified in the dark image captured by the endoscope camera module 102 when the light source 106 is turned off.

[0039] In some embodiments, before classifying the surgical video images 120 using the proposed statistical classifier, the statistical classifier is trained based on a large number of surgical videos containing the first and second transition events described above. More specifically, a large number of surgical videos containing the first and second transition events are first collected. The collected surgical videos may include videos of actual surgical procedures performed by a surgeon. Additionally, the collected surgical videos may include artificially generated procedure videos created to include the first and second transition events. In some embodiments, each training video may contain enough video frames to depict the actions of the endoscope both inside and outside the patient's body, and the actions of transferring the endoscope from inside / outside the patient's body to outside / inside the patient's body (i.e., the first and second transition events).

[0040] Next, for each training surgical video, each video image in the video frame is annotated / labeled as either a first-class image inside the patient's body or a second-class image outside the patient's body. In some embodiments, before annotating the training surgical videos, the training surgical videos may be segmented into a set of video segments by a phase segmentation engine based on a first transition event and a second transition event, and each video segment in the set of video segments may belong to one of two phases, namely, an inner phase when the endoscope is inside the patient's body, or an outer phase when the endoscope is completely outside the patient's body. Those skilled in the art will understand that a surgical video including multiple OOB events may contain multiple video segments of the inner phase and multiple video segments of the outer phase. Next, for each video segment in the set of inner phase video segments, the video image within the video segment may be labeled as a first-class image inside the patient's body; and for each video segment in the set of outer phase video segments, the video image within the video segment may be labeled as a second-class image outside the patient's body.

[0041] Once all training videos have been correctly labeled, a statistical classifier can be trained based on the labeled set of surgical videos, enabling the trained statistical classifier to distinguish a given input video frame into either a first-class image or a second-class image. Furthermore, the statistical classifier can be trained to distinguish transition events within a sequence of video images 120 into first-transition events or second-transition events based on its ability to correctly label each image frame as either a first-class or second-class image. In some embodiments, the proposed statistical classifier may include a feature-based model that can be used to distinguish input images into first-class or second-class images based on statistics of a set of user-specified image features such as color, texture, contrast, and other features. In other embodiments, the proposed statistical classifier may include a deep learning model that does not require manual feature identification by the user.

[0042] In various implementations, the proposed statistical classifier may include machine learning models built on regression models, deep neural network-based models, support vector machines, decision trees, Naive Bayes classifiers, Bayesian networks, or k-nearest neighbors (KNN) models. In some implementations, each of these machine learning models is built on a convolutional neural network (CNN) architecture, a recurrent neural network (RNN) architecture, or another form of deep neural network (DNN) architecture.

[0043] As described above, the video image processing module 114 is configured to generate a "power-on" control signal 130-3 for the light source module 104 when the start of a white balance operation is detected, or to generate a "power-off" control signal 130-4 for the light source module 104 when the end of a white balance operation is detected. In some embodiments, the video processing module 114 may include a machine learning-based or computer vision-based statistical classifier trained to detect white balance objects. Some suitable white balance objects may include a white balance card, a white sponge, a white board, a piece of white paper, a piece of white cloth, and any other suitable white medium. In some embodiments, the statistical classifier is configured to classify / label each received video image 120 as (1) a white balance object or (2) a non-white balance object.

[0044] In some implementations, if the endoscope camera module 102 is outside the patient's body during an OOB event when the light source 106 is off, and the statistical classifier has classified one or more newly received surgical video images 120 as white-balanced objects, the video processing module 114 immediately generates a "power-on" control signal 130-3 to trigger white-balance operation. Alternatively, if the endoscope camera module 102 is initially pointed at a white-balanced object when the light source 106 is on for white-balance, and the statistical classifier has classified one or more newly received surgical video images 120 as non-white-balanced objects, the video processing module 114 immediately generates a "power-off" control signal 130-4 to terminate white-balance operation. In some implementations, the statistical classifier can be trained on different datasets to identify different types of white-balanced and non-white-balanced objects before using the proposed statistical classifier to classify the surgical video images 120 as white-balanced or non-white-balanced objects.

[0045] In some implementations, the disclosed automatic light source control subsystem 110 can be implemented in computer software, electronic hardware, or a combination of both, and can be implemented as a module separate from the light source module 104 (e.g., Figure 1 (As shown). In some other embodiments, the disclosed automatic light source control subsystem 110 may be implemented in computer software, electronic hardware, or a combination of both, and implemented as a component of the light source module 104 (as shown). Figure 1(Not shown in the image).

[0046] Figure 2 A flowchart illustrating an exemplary process 200 for automatically turning on / off the light source of an endoscopic camera module during a surgical procedure, according to some embodiments described herein, is presented. In one or more embodiments, the process may be omitted, repeated, and / or performed in a different order. Figure 2 One or more steps in the process. Therefore, Figure 2 The specific arrangement of the steps shown should not be construed as limiting the scope of this technology.

[0047] Process 200 can begin by receiving a sequence of real-time video images captured by the endoscope camera module when the light source is turned on (step 202). Specifically, the endoscope camera module is initially located inside the patient's body. Next, process 200 analyzes the newly received sequence of video images to determine whether the endoscope camera module is inside or outside the patient's body (step 204). In some embodiments, process 200 uses a statistical classifier to classify each image in the sequence of newly received video images into a first category of images inside the patient's body or a second category of images outside the patient's body. Process 200 can determine the state of the endoscope camera module as inside or outside the patient's body based on the classified image sequence. More specifically, process 200 can determine the state of the endoscope camera module as outside the patient's body by detecting a first transition event within the newly received video image sequence.

[0048] If process 200 determines at step 204 that the endoscope camera module is still inside the patient's body, process 200 returns to step 202 and continues to monitor the status of the endoscope camera module. Otherwise, if process 200 determines that the endoscope camera module is outside the patient's body (i.e., a first transition event is detected), process 200 generates a control signal for shutting off the light source for safety and other reasons (step 206). The light source module uses the control signal to immediately shut off the light source to prevent accidental eye injury.

[0049] Next, when the endoscope camera module is outside the patient's body with the light source off, process 200 analyzes the real-time video image to determine if the endoscope camera module is pointing at a white balance object (step 208). For example, process 200 may analyze the video image to identify a predefined white balance object within the video image. If not, process 200 may return to step 208 and continue searching for a cue indicating the start of white balance operation. However, if a white balance object is detected at step 208, process 200 generates another control signal to turn on the light source for white balance (step 210).

[0050] Next, when the endoscope camera module is outside the patient's body with the light source on, process 200 continues to analyze the real-time video images to determine whether the white balance operation is complete (step 212). If not, process 200 can return to step 212 and continue searching for a cue indicating the end of the white balance operation. Otherwise, the white balance operation is complete, and process 200 generates another control signal to turn off the light source (step 214). Process 200 then analyzes another sequence of newly received video images to determine whether the endoscope camera module remains outside the patient's body (step 216). Similarly, process 200 can use a statistical classifier to classify each image in the sequence of newly received video images into a first category of images inside the patient's body or a second category of images outside the patient's body. More specifically, process 200 can determine the state of the endoscope camera module as being inside the patient's body by detecting a second transition event within the sequence of newly received video images.

[0051] If process 200 determines at step 216 that the endoscope camera module remains outside the patient's body, process 200 returns to step 216 to continue monitoring the state of the endoscope camera module. Otherwise, if process 200 determines at step 216 that the endoscope camera module is inside the patient's body (i.e., a second transition event is detected), process 200 generates another control signal for turning on the light source (step 218). The light source module uses the control signal to immediately turn on the light source to resume endoscopic imaging.

[0052] Figure 3 A flowchart illustrating an exemplary process 300 for training a disclosed statistical classifier to classify surgical video images as inside or outside a patient's body, according to some embodiments described herein, is presented. In one or more embodiments, the process may be omitted, repeated, and / or performed in a different order. Figure 3 One or more steps in the process. Therefore, Figure 3 The specific arrangement of the steps shown should not be construed as limiting the scope of this technology.

[0053] Process 300 begins by collecting a large number of training surgical videos, each containing one or more Out-of-Body (OOB) events (step 302). As described above, each OOB event typically includes a first transition event (i.e., removal of the endoscope from the patient's body) and a second transition event (i.e., placement of the endoscope back into the patient's body). It should be noted that the collected training surgical videos may include videos of actual surgical procedures performed by surgeons and specially created, artificially generated procedure videos to provide training data for training the statistical classifier.

[0054] Step 300 then segments each training surgical video into a set of video segments based on one or more associated OOB events, where each video segment in the set belongs to either the inner phase when the endoscope is inside the patient or the outer phase when the endoscope is outside the patient (step 304). Note that the set of transition events within the surgical video provides natural phase boundaries for segmenting the surgical video into the inner and outer phases. Also note that a surgical video containing multiple OOB events can be segmented into corresponding sets of video segments consisting of multiple inner and multiple outer phases.

[0055] Next, for each video segment of the inner phase from the segmented training surgical videos, process 300 labels the video image within the video segment as a first-class image inside the patient's body; while for each video segment of the outer phase from the segmented surgical videos, process 300 labels the video image within the video segment as a second-class image outside the patient's body (step 306). After all the training surgical videos have been correctly labeled, process 200 proceeds to train a statistical classifier based on the labeled set of training surgical videos, such that the trained statistical classifier can be used to distinguish a given input video frame as either a first-class image or a second-class image (step 308).

[0056] Figure 4 The photographic images presented show the conventional method of manually turning the light source of the endoscope camera module 400 on / off. (Example) Figure 4 As can be seen, the endoscope camera module 400 includes a light source on / off button 402 located at the proximal end of the endoscope camera module 400. After the surgeon removes the endoscope camera module 400 from the patient, the surgeon must hold the endoscope camera module 400 at one end and manually operate the light source on / off button 402 at the other end of the endoscope camera module 400. The disclosed automatic light source control technology eliminates the need to manually turn the endoscope light source on / off during several critical events within the surgical procedure, thereby significantly reducing or eliminating any chance of injury to surgical personnel from a strong endoscopic light source, and saving considerable time and hassle in the surgical procedure.

[0057] Figure 5A computer system conceptually illustrates some embodiments that can be used to implement the techniques of this subject matter. Computer system 500 may be a client, server, computer, smartphone, PDA, laptop, or tablet computer with one or more processors embedded therein or coupled thereto, or any other type of computing device. Such computer systems include various types of computer-readable media and interfaces for various other types of computer-readable media. Computer system 500 includes a bus 502, a processing unit 512, system memory 504, read-only memory (ROM) 510, persistent storage device 508, input device interface 514, output device interface 506, and network interface 516. In some embodiments, computer system 500 is part of a robotic surgical system.

[0058] Bus 502 collectively represents all system buses, peripheral buses, and chipset buses that communicatively connect multiple internal devices of computer system 500. For example, bus 502 communicatively connects processing unit 512 to ROM 510, system memory 504, and permanent storage device 508.

[0059] Processing unit 512 retrieves instructions to be executed and data to be processed from these various memory units in order to perform the various processes described in this patent disclosure, including combining Figures 1-3 The system automatically switches the light source of the endoscopic camera module on / off during surgical procedures and trains a disclosed statistical classifier to classify surgical video images as occurring inside or outside the patient's body. The processing unit 512 may include any type of processor, including but not limited to microprocessors, graphics processing units (GPUs), tensor processing units (TPUs), intelligent processor units (IPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). In different implementations, the processing unit 512 may be a single processor or a multi-core processor.

[0060] ROM 510 stores static data and instructions required by processing unit 512 and other modules of the computer system. On the other hand, permanent storage device 508 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the computer system 500 is off. Some specific embodiments of this subject matter disclosure use mass storage devices (such as disks or optical discs and their corresponding disk drives) as permanent storage device 508.

[0061] Other embodiments use removable storage devices (such as floppy disks, flash drives, and their corresponding disk drives) as permanent storage device 508. Similar to permanent storage device 508, system memory 504 is a read-write memory device. However, unlike storage device 508, system memory 504 is volatile read-write memory, such as random access memory. System memory 504 stores some of the instructions and data required by the processor during operation. In some embodiments, the various processes described in this patent disclosure are stored in system memory 504, permanent storage device 508, and / or ROM 510; these processes include combining... Figures 1-3 The endoscope camera module's light source is automatically switched on / off during surgical procedures, and a publicly disclosed statistical classifier is trained to classify surgical video images as occurring inside or outside the patient's body. The processing unit 512 retrieves instructions to be executed and data to be processed from these various memory units to perform specific implementation procedures.

[0062] Bus 502 is also connected to input device 514 and output device 506. Input device 514 enables a user to transmit information to the computer system and select commands for the computer system. Input device 514 may include, for example, an alphanumeric keypad and a pointing device (also known as a "cursor control device"). Output device 506 enables, for example, the display of images generated by computer system 500. Output device 506 may include, for example, a printer and a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD). Some embodiments include devices that function as both input and output devices, such as a touch screen.

[0063] Finally, as Figure 5 As shown, bus 502 also couples computer system 500 to a network (not shown) via network interface 516. Thus, the computer can be part of a network of computers (such as a local area network (“LAN”), wide area network (“WAN”), intranet) or a network of network groups (such as the Internet). Any or all components of computer system 500 can be used in conjunction with the disclosure of this subject matter.

[0064] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed in this patent disclosure can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

[0065] Hardware for implementing the various exemplary logics, logic blocks, modules, and circuits described in conjunction with the aspects disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic components, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of receiver devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuitry specific to a given function.

[0066] In one or more exemplary aspects, the functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a non-transitory computer-readable storage medium or a non-transitory processor-readable storage medium. The steps of the methods or algorithms disclosed herein may be embodied in processor-executable instructions that may reside on a non-transitory computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable storage medium may be any storage medium accessible by a computer or processor. By way of example, but not limitation, such non-transitory computer-readable or processor-readable storage media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. As used herein, magnetic disks and optical disks include compact discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein magnetic disks typically reproduce data magnetically, while optical discs utilize lasers to reproduce data optically. The combinations described above are also included within the scope of non-transitory computer-readable and processor-readable media. Additionally, the operation of a method or algorithm may reside as one or any combination or set of code and / or instructions on a non-transitory processor-readable storage medium and / or computer-readable storage medium, thereby being incorporated into a computer program product.

[0067] While this patent document contains numerous details, these details should not be construed as limiting the scope of any disclosed technology or content protected by the claims, but rather as descriptions of features that may be specific to particular embodiments of a particular technology. Certain features described in this patent document in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any sub-combination in multiple embodiments. Furthermore, while features may be described above as functioning in certain combinations and even initially so protected by the claims, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may involve sub-combinations or variations thereof.

[0068] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or requiring the performance of all illustrated operations to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0069] Only a few specific implementations and examples are described, but other specific implementations, enhancements and variations can be derived based on the content described and illustrated in this patent document.

Claims

1. A computer program product comprising instructions, which, when executed by a computer, cause the computer to perform a method for automatically switching on / off a light source of an endoscopic camera during a surgical procedure, the method comprising: Receive a first video image sequence captured by the endoscope camera when the light source is turned on; The computer uses a machine learning classifier to analyze the first video image sequence to classify each video image in the first video image sequence into a first type of image inside the patient's body or a second type of image outside the patient's body; The computer determines whether the endoscope camera is inside or outside the patient's body based on a classified first video image sequence, wherein determining the state of the endoscope camera as being outside the patient's body includes identifying a first transition event that moves the endoscope camera from inside the patient's body to outside the patient's body based on the classified first video image sequence; as well as When it is determined that the endoscopic camera is outside the patient's body, a first control signal is generated for shutting off the light source, wherein the first control signal is used to immediately shut off the light source. The machine learning classifier includes a feature-based model used to classify an input image into either the first class or the second class based on statistics of a set of image features. The machine learning classifier is trained based on surgical videos containing a first transition event (transferring the endoscope camera from inside the patient's body to outside the patient's body) and a second transition event (transferring the endoscope camera from outside the patient's body to inside the patient's body), such that the trained machine learning classifier can be used to distinguish a given input image into either the first type of image or the second type of image.

2. The computer program product of claim 1, further comprising instructions that, when executed by a computer, cause the computer to perform the following steps: Once it is determined that the endoscope camera is inside the patient's body, the status of the endoscope camera continues to be monitored.

3. The computer program product of claim 1, further comprising instructions, when executed by a computer, to cause the computer to perform the following steps after the light source has been turned off: Receive a second video image sequence captured by the endoscope camera when the light source is disconnected; Process the second video image sequence to determine whether the endoscopic camera remains outside the patient's body; and If so, continue monitoring the status of the endoscope camera. Otherwise, a second control signal is generated to turn on the light source.

4. The computer program product of claim 3, wherein processing the second video image sequence to determine whether the endoscopic camera remains outside the patient's body comprises: The machine learning classifier is used to classify each video image in the second video image sequence into either a first type of image inside the patient's body or a second type of image outside the patient's body; as well as The state of the endoscopic camera, whether inside or outside the patient's body, is determined based on the classified second video image sequence.

5. The computer program product of claim 4, wherein determining the state of the endoscope camera within the patient's body includes identifying a second transition event that moves the endoscope camera from outside the patient's body into the patient's body based on a classified second video image sequence.

6. The computer program product of claim 1, wherein when the light source remains off and the endoscopic camera is outside the patient's body, the method further comprises: Detects whether the endoscope camera is pointing at a white balance object; as well as A third control signal is generated to turn on the light source for white balance operation.

7. The computer program product of claim 6, wherein detecting that the endoscope camera is pointing at the white balance object comprises: Receive real-time video images captured by the endoscope camera when the light source is turned off; as well as The video image is processed to identify predefined white balance objects within the video image.

8. The computer program product of claim 1, wherein the method further comprises training the machine learning classifier by: Receive multiple surgical videos, wherein each of the multiple surgical videos contains video images of the endoscope both inside and outside the patient's body; For each of the plurality of surgical videos, the corresponding video image is labeled as either the first type of image or the second type of image; and The machine learning classifier is trained to distinguish input images into either the first class or the second class based on the labeled video images.

9. The computer program product of claim 8, wherein training the machine learning classifier to distinguish input images comprises training the machine learning classifier based on a set of different image features, the set of different image features including statistics on color, texture, and contrast.

10. A device for automatically switching the light source of an endoscopic camera on / off during a surgical procedure, the device comprising: One or more processors; and A memory coupled to the one or more processors; and The memory stores a set of instructions that, when executed by the one or more processors, cause the device to: Receive a first video image sequence captured by the endoscope camera when the light source is turned on; A machine learning classifier is used to analyze the first video image sequence to classify each video image in the first video image sequence as either a first type of image outside the patient's body or a second type of image outside the patient's body; Determining whether the endoscope camera is inside or outside the patient's body based on the classified first video image sequence, wherein determining the state of the endoscope camera as being outside the patient's body also includes identifying a first transition event that moves the endoscope camera from inside the patient's body to outside the patient's body; as well as If it is determined that the endoscopic camera is inside the patient's body, then the status of the endoscopic camera continues to be monitored, or If it is determined that the endoscopic camera is outside the patient's body, a first control signal is generated to turn off the light source, wherein the first control signal is used to immediately turn off the light source. The machine learning classifier includes a feature-based model used to classify an input image into either the first class or the second class based on statistics of a set of image features. The machine learning classifier is trained based on surgical videos containing a first transition event (transferring the endoscope camera from inside the patient's body to outside the patient's body) and a second transition event (transferring the endoscope camera from outside the patient's body to inside the patient's body), such that the trained machine learning classifier can be used to distinguish a given input image into either the first type of image or the second type of image.

11. The device of claim 10, wherein the memory further stores a set of instructions that, when executed by the one or more processors, cause the device to: Receive a second video image sequence captured by the endoscope camera when the light source is disconnected; Process the second video image sequence to determine whether the endoscopic camera remains outside the patient's body; and If so, continue monitoring the status of the endoscope camera. Otherwise, a second control signal is generated to turn on the light source.

12. The device of claim 11, wherein processing the second video image sequence to determine whether the endoscopic camera remains outside the patient's body further comprises: The machine learning classifier is used to classify each video image in the second video image sequence into either a first type of image inside the patient's body or a second type of image outside the patient's body; as well as The state of the endoscopic camera, whether inside or outside the patient's body, is determined based on the classified second video image sequence.

13. The device of claim 12, wherein determining the state of the endoscope camera within the patient's body includes recognizing a second transition event that moves the endoscope camera from outside the patient's body into the patient's body.

14. The device of claim 10, wherein when the light source remains off and the endoscopic camera is outside the patient's body, the memory further stores a set of instructions that, when executed by the one or more processors, cause the device to: Detecting that the endoscopic camera is pointing at a white balance object; and A third control signal is generated to turn on the light source for white balance operation.

15. The device of claim 14, wherein detecting that the endoscopic camera is pointing at the white balance object comprises: Receive real-time video images captured by the endoscope camera when the light source is turned off; as well as The video image is processed to identify predefined white balance objects within the video image.

16. An endoscope system comprising: Endoscopic camera module; A light source module, which is coupled to the endoscope camera module to provide a light source for the endoscope camera module; as well as A light source control module, coupled to the endoscope camera module and the light source module, is configured to automatically turn the light source in the light source module on / off during surgical procedures in the following manner: Receive a first video image sequence captured by the endoscope camera module when the light source is turned on; The light source control module classifies each video image in the first video image sequence into a first type of image inside the patient's body or a second type of image outside the patient's body using a machine learning classifier. Based on the classified first video image sequence, it determines whether the endoscope camera module is inside or outside the patient's body. Determining the state of the endoscope camera as outside the patient's body includes identifying a first transition event of moving the endoscope camera from inside the patient's body to outside the patient's body based on the classified first video image sequence. as well as If it is determined that the endoscopic camera module is inside the patient's body, then the status of the endoscopic camera continues to be monitored, or If it is determined that the endoscopic camera module is outside the patient's body, a first control signal is generated to turn off the light source, wherein the control signal is used to immediately turn off the light source. The machine learning classifier includes a feature-based model used to classify an input image into either the first class or the second class based on statistics of a set of image features. The machine learning classifier is trained based on surgical videos containing a first transition event (transferring the endoscope camera from inside the patient's body to outside the patient's body) and a second transition event (transferring the endoscope camera from outside the patient's body to inside the patient's body), such that the trained machine learning classifier can be used to distinguish a given input image into either the first type of image or the second type of image.

17. The endoscope system of claim 16, wherein the light source control module is further configured to automatically turn the light source in the light source module on / off during surgical procedures by: Receive a second video image sequence captured by the endoscope camera module when the light source is turned off; Process the second video image sequence to determine whether the endoscopic camera module remains outside the patient's body; and If so, continue monitoring the status of the endoscope camera module. Otherwise, a second control signal is generated to turn on the light source.

Images