User in / out-of-position recognition during robotic surgery using deep learning

Abstract

The invention name of the present disclosure is "User In / Out-Of-Position Recognition During Robotic Surgery Using Deep Learning". Various user in / out-of-position recognition techniques based on deep learning are disclosed herein. More specifically, the various user in / out-of-position recognition techniques include constructing / training a CNN-based image recognition model including a user in / out-of-position classifier based on training images collected from a user seating area of a surgeon console under various clinically relevant conditions / circumstances. The trained user in / out-of-position classifier can then be used to monitor / track a user of the user seating area of the surgeon console during a teleoperation / surgical procedure and continuously classify real-time video images of the user seating area as a user in position state or a user out of position state. In some embodiments, the disclosed techniques can be used to detect a user switch event at the surgeon console when a second user is detected to have entered the user seating area after a first user is detected to have left the user seating area. If the second user is identified as a new user, the disclosed techniques can trigger a re-calibration procedure for the new user.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to robotic surgical systems, and more particularly, to systems, devices, and techniques for automatically identifying a user / surgeon in or out of position at a surgeon console during robotic surgery. BACKGROUND

[0002] Endoscopic surgery involves viewing inside a patient and performing surgery inside the patient using an endoscope and other surgical tools. For example, laparoscopic surgery can use a laparoscope to access and view the abdominal cavity. Endoscopic surgery can be performed using manual tools and / or robotic surgical systems with robotically assisted tools.

[0003] A surgeon can remotely operate a robotic surgical system to command robotically assisted tools located at a surgical table. This operation of robotically assisted tools by a surgeon remotely can be generally referred to as teleoperation. For example, a surgeon can use a computer console located in an operating room (or it can be located in a different city) to command a robot to manipulate surgical tools mounted on a surgical table. The robotically assisted tools can include an endoscope mounted on a robotic arm. Thus, a remote surgeon can use a surgical robotic system to perform endoscopic surgery.

[0004] A surgeon console of a robotic surgical system typically includes an eye tracking mechanism configured to track a user / surgeon’s line of sight during a surgical procedure. For a particular surgeon using the surgeon console during a new surgical procedure, the eye tracking mechanism needs to be calibrated or otherwise configured for that surgeon before starting the new surgical procedure. During a subsequent surgical procedure, that surgeon can need to leave the console for a variety of reasons, such as to assist in performing the procedure. After that surgeon leaves the console, a new surgeon can enter and take over the procedure. When such a user change event occurs, the robotic surgical system needs to ensure that the eye tracking settings are recalibrated or otherwise reconfigured for the new surgeon, as different users typically have different eye tracking settings. However, existing robotic surgery systems do not include a mechanism to detect such user change events during a procedure. Thus, existing robotic surgical systems will continue to use the eye tracking settings for the surgeon who was previously in charge of the new procedure, which can result in inaccurate eye tracking control signals, further resulting in unsafe / unintended robotic arm or tool movements. SUMMARY

[0005] Disclosed herein are various user in / out-of-position recognition techniques based on image processing and deep learning. More specifically, the various user in / out-of-position recognition techniques include constructing / training a convolutional neural network (CNN) based image recognition model including a user in / out-of-position classifier based on training images collected from a user seating area of a surgeon console within a robotic surgical system under various clinically relevant conditions / circumstances. The trained user in / out-of-position classifier can then be used to monitor / track a user of the user seating area of the surgeon console during a teleoperation / surgical procedure and continuously classify real-time video images of the user seating area as a user in position state or a user out of position state.

[0006] In some embodiments, the disclosed user in / out-of-position recognition techniques can be used to detect a user transition event at the surgeon console when detecting that a second user has entered the user seating area after detecting that a first user has left the user seating area. The detection of the user transition event can enable prompting the second user in the user seating area to identify himself / herself as the same user or a different user as the first user. If the second user is identified as a new user, the disclosed user in / out-of-position recognition techniques can trigger a re-calibration process of user console settings for the new user, including but not limited to: user line-of-sight tracking settings; user UID control settings; user seating settings (e.g., armrest settings).

[0007] In one aspect, a process for detecting user in / out-of-position at a surgeon console in a robotic surgical system is disclosed. The process can begin with receiving a set of raw video images capturing a user seating area of a surgeon console. Next, in the process, the set of raw video images is processed to generate a set of training images. In the process, a convolutional neural network (CNN) model is then trained using the set of training images. More specifically, the CNN model includes an image classifier configured to output a binary classification for each input image, where the binary classification is either a user in position classification or a user out of position classification. Further, in the process, a set of real-time video images capturing the user seating area during a surgical procedure time is received. Then, in the process, the trained CNN model is applied to the set of real-time video images to automatically classify each video image in the set of real-time video images as either the user in position classification or the user out of position classification.

[0008] In some embodiments, in the process, the set of raw video images is processed to generate the set of training images by: for each raw video image in the set of raw video images, generating a plurality of augmented images of the raw video image using a set of image augmentation techniques; and including one or more augmented images in the plurality of augmented images in the set of training images.

[0009] In some embodiments, in the process, the plurality of enhanced images is generated using the set of image enhancement techniques by first placing a bounding box of a first predetermined size within a border of the original video image. Then, in the process, the original image is cropped using the bounding box to generate a first enhanced image of the original image having the first predetermined size.

[0010] In some embodiments, the method further comprises: upscaling the first enhanced image using a random upsampling factor to generate a second enhanced image of the original image having the first predetermined size.

[0011] In some embodiments, the method further comprises: horizontally flipping the second enhanced image to generate a third enhanced image of the original image having the first predetermined size.

[0012] In some embodiments, the first predetermined size comprises a vertical dimension and a horizontal dimension that are each equal to a smaller dimension of the original image.

[0013] In some embodiments, after the set of training images is generated, the method further comprises: automatically annotating each training image in the set of training images with a user-in-seat label indicating that a user in the training image is seated or a user-not-in-seat label indicating that any user in the training image is not seated.

[0014] In some embodiments, in the process, a training image in the set of training images is automatically annotated by first detecting a seat object within the training image using an image recognition technique. In the process, next it is determined whether a portion of the detected seat object is obstructed by another object in the training image. If so, the training image is labeled with the user-in-seat label. Otherwise, in the process, the training image is labeled with the user-not-in-seat label.

[0015] In some embodiments, in the process, a training image in the set of training images is annotated by labeling the training image with the user-in-seat label if the image recognition technique fails to detect a seat object.

[0016] In some embodiments, the CNN model is composed of a stack of feature extraction layers comprising a series of pooling layers and a series of convolutional layers having a first set of parameters. The stack of feature extraction layers is followed by a fully connected layer having a second set of parameters, which is further followed by a final output layer.

[0017] In some embodiments, to train the CNN model using the set of training images, in the process, a set of two-dimensional (2D) feature maps of each training image in the set of training images is extracted using the stack of the feature extraction layers while fixing the first set of parameters. In the process, then, the second set of parameters of the fully connected layer is trained based on at least the plurality of sets of extracted 2D feature maps extracted from the set of training images to obtain a trained fully connected layer. In the process, then, the first set of parameters and the trained second set of parameters in the CNN model are fixed to obtain a trained CNN model.

[0018] In some embodiments, the stack of the feature extraction layers includes a portion of an Inception-v3 image recognition model from the input layer up to the bottleneck layer. Further, the Inception-v3 image recognition model is previously trained to detect and recognize objects that are not related to detecting user in / out of position at the surgeon console.

[0019] In another aspect, an apparatus for detecting user in / out of position at a surgeon console in a robotic surgery system is disclosed. The apparatus includes one or more processors; and a memory coupled to the one or more processors. The memory of the apparatus stores instructions that, when executed by the one or more processors, cause the apparatus to: receive a set of raw video images capturing a user seating area of a surgeon console; process the set of raw video images to generate a set of training images; train a deep learning image classifier using the set of training images, wherein the deep learning image classifier is configured to receive an input image, process the input image, and output a binary classification for the input image, and wherein the binary classification is a user in position classification or a user out of position classification; receive a set of real-time video images capturing the user seating area during a surgery time; and apply the trained deep learning image classifier to the set of real-time video images to automatically classify each video image in the set of real-time video images as the user in position classification or the user out of position classification.

[0020] In another aspect, a robotic surgical system is disclosed. The robotic surgical system can include: a surgeon console including a user seating area; and a computer at the surgeon console configured to detect user presence / absence / absence at the surgeon console by: receiving a set of raw video images capturing the user seating area at the surgeon console; processing the set of raw video images to generate a set of training images; training a CNN model using the set of training images, the CNN model including an image classifier configured to output a binary classification for each input image, and wherein the binary classification is either a user presence classification or a user absence classification; further, receiving a set of real-time video images capturing the user seating area during a surgical procedure time; and applying the trained CNN model to the set of real-time video images to automatically classify each video image in the set of real-time video images as either the user presence classification or the user absence classification. BRIEF DESCRIPTION OF DRAWINGS

[0021] The structure and operation of this disclosure will be understood by reviewing the following detailed description and accompanying drawings, in which like reference characters refer to like parts and in which:

[0022] Figure 1 An exemplary operating room environment is shown, in accordance with some embodiments described herein, with a robotic surgical system for implementing the disclosed user presence / absence recognition techniques.

[0023] Figure 2 Two exemplary video images of a user seating area are shown, in accordance with some embodiments described herein, showing a user absence state and a user presence state of the user seating area during a surgical procedure.

[0024] Figure 3 A block diagram of the disclosed user presence / absence recognition model building system for building a CNN-based user presence / absence classifier based on video images of a user seating area, in accordance with some embodiments described herein, is shown.

[0025] Figure 4 A block diagram of an exemplary implementation of the disclosed data augmentation module in the disclosed user presence / absence recognition model building system, in accordance with some embodiments described herein, is shown.

[0026] Figure 5 An exemplary process of labeling a time series of video images as either a user presence classification or a user absence classification, in accordance with some embodiments described herein, is shown.

[0027] Figure 6An exemplary implementation of the disclosed transfer learning module based on use of an Inception-v3 image recognition model is shown in accordance with some embodiments described herein.

[0028] Figure 7 A flow diagram is presented in accordance with some embodiments described herein showing an exemplary process for constructing a CNN-based user on / off seat classifier based on captured video images of a user seating area.

[0029] Figure 8 An exemplary surgical workflow is shown in accordance with some embodiments described herein that uses a trained user on / off seat image classifier to track a user at a surgeon console during a surgical procedure time.

[0030] Figure 9 A flow diagram is presented in accordance with some embodiments described herein showing an exemplary process for tracking a user at a surgeon console during a surgical procedure time using a trained user on / off seat classifier.

[0031] Figure 10 A computer system that can be used to implement some embodiments of the subject technology is shown conceptually. DETAILED DESCRIPTION

[0032] The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology can be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein and can be practiced without these specific details. In some instances, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology.

[0033] Various user on / off seat recognition techniques based on image processing and deep learning are disclosed herein. More specifically, the various user on / off seat recognition techniques include constructing / training a convolutional neural network (CNN)-based image recognition model including a user on / off seat classifier based on training images collected from a user seating area of a surgeon console within a robotic surgical system under various clinically relevant conditions / circumstances. The trained user on / off seat classifier can then be used to monitor / track a user of the user seating area of the surgeon console during a teleoperation / surgical procedure and continuously classify real-time video images of the user seating area as either a user on status or a user off status.

[0034] In some embodiments, the disclosed user in / out identification techniques can be used to detect a user changeover event at the surgeon console when detecting that a second user has entered the user seating area after detecting that the first user has exited the user seating area. Detection of a user changeover event can enable prompting the second user in the user seating area to identify himself / herself as the same user as the first user or a different user. If the second user is identified as a new user, the disclosed user in / out identification techniques can trigger a recalibration process for the surgeon console settings for the new user, including but not limited to: user line of sight tracking settings; user UID control settings; user seating settings (e.g., armrest settings).

[0035] Figure 1 An exemplary operating room environment is shown in accordance with some embodiments described herein, with a robotic surgical system 100 for implementing the disclosed user in / out identification techniques. As shown, the robotic surgical system 100 includes a user / surgeon console 120, a control tower 130, and one or more surgical robot arms 112 located at a robotic surgical platform 116 (e.g., a table or bed, etc.), with surgical tools having end effectors attached to the distal ends of the robot arms 112 for performing a surgical procedure. The robot arms 112 are shown as a table-mounted system, but in other configurations, the robot arms can be mounted in a cart, ceiling or sidewall, or other suitable support surface. The robotic surgical system 100 can include any currently existing or future developed robotically assisted surgical system for performing robotically assisted surgical procedures. Figure 1

[0036] In general, a user / operator 140 (e.g., a surgeon or other operator) can remotely manipulate the robot arms 112 and / or surgical instruments (e.g., teleoperation) using the surgeon console 120. The surgeon console 120 can be located in the same operating room as the robotic surgical system 100, as shown. In other environments, the surgeon console 120 can be located in an adjacent or nearby room, or teleoperated from a remote location in a different building, city, or country. The surgeon console 120 can include a seat 132, foot-operated controls 134, one or more handheld user interface devices (UIDs) 136, and at least one user display / monitor 138 configured to display, for example, a view of a surgical site within a patient. As shown in the example surgeon console 120, a surgeon located in the seat 132 and viewing the user display / monitor 138 can manipulate the foot-operated controls 134 and / or the UIDs 136 to remotely control the robot arms 112 and / or surgical instruments mounted to the distal ends of the arms. Figure 1

[0037] ​​In some variations, the user can also operate the robotic surgical system 100 in an "over the bed" (OTB) mode, in which the user is at the side of the patient and simultaneously manipulates a robotically-driven tool / end effector attached thereto (e.g., with a handheld user interface device (UID) 136 held in one hand) and a manual laparoscopic tool. For example, the user's left hand can manipulate the handheld UID 136 to control the robotic surgical components, while the user's right hand can manipulate the manual laparoscopic tool. Thus, in these variations, the user can perform both robotically-assisted (minimally invasive surgical) MIS and manual laparoscopic surgery on the patient.

[0038] During an example procedure or surgical operation, the patient is prepared and draped in a sterile manner to achieve anesthesia. Initial access to the surgical site can be performed manually with the robotic surgical system 100 in a stowed or withdrawn configuration to facilitate access to the surgical site. Once access is achieved, initial positioning and / or preparation of the robotic system can be performed. During the procedure, the surgeon in the surgeon's console 120 can manipulate various surgical tools / end effectors and / or imaging systems to perform the surgical operation using foot-operated controls 134 (e.g., one or more foot pedals) and / or the UID 136. Manual assistance can also be provided at the operating table by personnel in sterile garments, who can perform tasks including but not limited to retracting tissue, or perform manual repositioning or tool changes involving one or more of the robotic arms 112. There can also be non-sterile personnel to assist the surgeon at the surgeon's console 120. When the procedure or surgical operation is complete, the robotic surgical system 100 and / or the surgeon's console 120 can be configured or set up in a state that facilitates one or more post-operative procedures, including but not limited to robotic surgical system 100 cleaning and / or sterilization, and / or healthcare record input or printout, whether electronic or hard copy, e.g., via the surgeon's console 120.

[0039] In some aspects, communication between the robotic surgical platform 116 and the surgeon console 120 can be through a control tower 130, which can translate user commands from the surgeon console 120 into robotic control commands and transmit them to the robotic surgical platform 116. The control tower 130 can also transmit status and feedback from the robotic surgical platform 116 back to the surgeon console 120. Connections between the robotic surgical platform 116, the surgeon console 120, and the control tower 130 can be via wired and / or wireless connections, and can be proprietary and / or performed using any of a variety of data communication protocols. Any wired connections can optionally be built into the floor and / or walls or ceiling of the operating room. The robotic surgical system 100 can provide video output to one or more displays, including displays within the operating room as well as remote displays accessed via the internet or other network. The video output or feed can also be encrypted to ensure privacy, and all or part of the video output can be saved to a server or electronic health record system.

[0040] In some implementations, a user / operator 140 can hold and move a UID 136 with one hand to provide input commands to move a robotic arm actuator 117 in a robotic surgical system 100. The UID 136 can be communicatively coupled to the rest of the robotic surgical system 100, e.g., via a console computer system (not shown) of the surgeon’s console 120. The UID 136 can be ungrounded relative to another component of the robotic surgical system 100, while tethered or untethered to the surgeon’s console 120. The term “ungrounded” is intended to refer to implementations in which, for example, two UIDs are neither mechanically nor dynamically constrained relative to the surgeon’s console 120. For example, the user / operator 140 can hold the UID 136 in a hand and move it freely to any possible position and orientation within a space limited only by, e.g., a tracking mechanism of the surgeon’s console. The UID 136 can thereby generate a spatial state signal corresponding to the movement of the UID, e.g., the position and orientation of the handheld housing of the UID, and the spatial state signal can be an input signal that controls the motion of the robotic arm actuator 117. The robotic surgical system 100 can use a control signal derived from the spatial state signal to control the proportional motion of the actuator 117. In some embodiments, the console computer system of the surgeon’s console 120 receives the spatial state signal from the UID 136 and generates a corresponding control signal. Based on these control arm actuator 117 energized to move a segment or link of the corresponding robotic arm 112, the movement of the corresponding surgical tool attached to the distal end of the robotic arm 112 can mimic the movement of the UID 136. Similarly, interactions between the user / operator 140 and the UID 136 can generate, for example, a clamp control signal that causes the jaws of a gripper of the corresponding surgical tool to close and clamp tissue of a patient on the top of the robotic surgical platform 116.

[0041] The robotic surgical system 100 can include multiple UIDs 136, with a respective control signal being generated for each UID 136 that controls an actuator and surgical tool (end effector) attached to a respective arm 112. For example, a user / operator 140 can move a first UID 136 (e.g., a left UID) with one hand (e.g., the left hand) to control the motion of an arm actuator 117 located in a left robotic arm 112, where the actuator responds by moving links, gears, etc. in the left robotic arm 112. Similarly, the user / operator 140 can move a second UID 136 (e.g., a right UID) with the other hand (e.g., the right hand) to control the motion of another arm actuator 117 located in a right robotic arm 112, where the arm actuator in turn moves other links, gears, etc. in the right robotic arm 112. The motion of the respective actuator 117 in the respective robotic arm 112 can be controlled by a spatial state signal generated from the respective UID 136. When the user / operator 140 has finished manipulating the surgical tools with the UIDs 136, the user / operator can dock (i.e., store / place) the UIDs 136 in designated docking areas / docking stations at the console 120. For example, a surgeon console 120 can include a docking station at each of a left arm rest and a right arm rest (both not shown) of a seat 132. To dock the left and right UIDs 136, the user can move the left UID 136 to the left docking station and the right UID 136 to the right docking station, and place each of the UIDs in its respective docking station holder.

[0042] To detect user occupancy or vacancy at the surgeon's console 120 (or "console 120" hereinafter), video images of the user seating area of ​​console 120 can be captured and analyzed. Within console 120, the user seating area disclosed herein generally refers to surgeon seat 132, which may or may not have a user / operator 140. In some embodiments, images of the user seating area can be captured by a camera mounted on or integrated into a user display / monitor 138 (also referred to as "monitor 138" or "monitor") and aimed / pointed at the user seating area and seat 132. For example, camera 150 may be located in the middle of the bottom bezel portion of monitor 138. Those skilled in the art will understand that images of the user seating area captured by camera 150 can be categorized into two types: the first type of image primarily includes the main seat 132 and the background behind seat 132, but without any user / operator on seat 132. Images of the first type are categorized as "user vacancy status" or "user vacancy classification". Alternatively, the second type of image of the user seating area captured by camera 150 will primarily include the user / operator 140 seated in seat 132 (where the seat itself is often obscured) and the background behind seat 132. This second type of image is classified as either "user in position" or "user in position classification." As will be discussed in more detail below, the disclosed user in / out classifier is configured to recognize / distinguish between the two types of seating area images described above, and to classify any given user seating area image as either user out or user in position.

[0043] Figure 2 Two exemplary video images of a user seating area 200 according to some embodiments described herein are shown, illustrating user disengagement and user in-position states in the user seating area during surgical procedures. For example, Figure 2 The left image 202 is a user seating area image associated with a user aberration, as there is no user or anyone in image 202. More specifically, the left image 202 shows a console seat 204 including a seat back (upper seat) and headrest / headrest holder in front of a background 206. Note that for training purposes, image 202 is labeled as user aberration-classified, for example, using the label "A" or the binary number "0". For user identification purposes, the disclosed user positioning / abortion classifier is configured to automatically classify image 202 using user aberration classification. Conversely, Figure 2The right image 208 in FIG. 2B is a user seated zone image associated with a user seated state. More specifically, the right image 208 shows a user 210 sitting in a console seat 204 in front of the background 206, while the console seat 204 in the image 208 is mostly blocked by the user 210. Note that for training purposes, the image 208 is labeled as a user seated classification, e.g., using a label “P” or a binary number “1”. For user identification purposes, the disclosed user seated / off- seated classifier is configured to utilize the user seated classification to automatically classify the image 208.

[0044] Note that the video image 202 is only one example of many possible variations of a user off-seated state, while the video image 208 is only one example of many possible variations of a user seated state. As will be discussed in more detail below, there are also some less obvious in-between scenarios compared to the video images 202 and 208, e.g., during a time period when a user is leaving / exiting or returning / entering a user seated zone. The disclosed user seated / off-seated identification technology is also configured to detect such situations and provide appropriate classifications for these situations.

[0045] The disclosed user seated / off-seated identification technology includes a user seated / off-seated identification model building system for building a CNN-based user seated / off-seated classifier based on training images collected from a user seated zone of a surgeon console under various clinically relevant conditions / situations. The disclosed user seated / off-seated identification technology also includes a user seated / off-seated tracking process based on the trained user seated / off-seated classifier for automatically identifying video images of the user seated zone as either a user seated state or a user off-seated state during a surgical procedure, and triggering appropriate interactions with a current user if a transition event from a user off-seated state to a user seated state (i.e., a potential user handover event) is detected.

[0046] Figure 3 A block diagram of a user seated / off-seated identification model building system 300 for building a CNN-based user seated / off-seated classifier based on video images of a user seated zone is shown in accordance with some embodiments described herein. As shown, the user seated / off-seated identification model building system 300 (hereinafter also referred to as “model building system 300”) can include a data collection module 302, a data augmentation module 304, a data labeling module 306, a transfer learning module 308, which are coupled in the manner shown. Note that the model building system 300 generates a trained user seated / off-seated classifier 310 as its output. Figure 3

[0047] ​In some embodiments, the data collection module 302 in the model construction system 300 obtains different sets of raw / original video images 312 of the user seating area of the surgeon console under various clinically relevant conditions / situations. These various clinically relevant situations can include various backgrounds for the user seating area, such as different background colors (e.g., white, light blue, etc.) and backgrounds with / without a person present. The various clinically relevant situations can also include different lighting conditions for the user seating area. The various clinically relevant situations can also include various screen angles for the monitor 138, including but not limited to a right angle and various tilted angles. The various clinically relevant situations for the user seating area can also include various heights for the monitor 138.

[0048] The various clinically relevant situations for the user seating area can also include different distances between the user 140 and the monitor 138, such as a plurality of distances ranging from 20 cm to 100 cm. The various clinically relevant situations for the user seating area can also include states of the UID 136, where the states can include a stationary UID 136 (i.e., when the UID 136 is docked / laid down) and a mobile UID 136 (i.e., when the UID 136 is in teleoperational use in the user’s hand). The various clinically relevant situations for the user seating area can also include a user changeover event (i.e., an event where a first user exits the user seating area followed by a second user entering the user seating area). The various clinically relevant situations for the user seating area can also include various durations of video clips captured of the user seating area for obtaining the training images 312, such as between 30 seconds to 60 seconds.

[0049] The various clinically relevant situations of the user seated area can include different scenarios related to whether the user 140 is wearing one or more of the following on her / his head: a QR tag, a face mask, and a three-dimensional (3D) headset. The various clinically relevant situations of the user seated area can also include different scenarios related to whether the user 140 is wearing a hospital gown. The various clinically relevant situations of the user seated area can also include the user 140 wearing a hospital gown of different colors, such as white or light blue. The various clinically relevant situations of the user seated area can also include various headrest angles of the seat 132. The various clinically relevant situations of the user seated area can also include various types of head movements of the user 140, such as moving left, moving right, and moving up and down. The various clinically relevant situations of the user seated area can also include a plurality of different types of surgeon console seats. Last but not least, the various clinically relevant situations of the user seated area can include collecting user seated area images from different surgeon consoles of different robotic surgical systems. Note that the data collection module 302 can collect video images 312 of the user seated area, including some or all of the types of clinically relevant scenarios / situations described above, as well as other types of clinically relevant scenarios / situations not listed above. Moreover, because all captured images of the user seated area will be classified as either a user-in-position state or a user-out-of-position state, the initial video images 312 generated by the data collection module 302 should include a sufficient number of training images for each of the two image classifications.

[0050] In some embodiments, the data augmentation module 304 in the model construction system 300 is configured to receive initial video images 312 (also referred to as “raw images 312” below), pre-process the received raw images, and generate pre-processed training images 314. Figure 4 A block diagram illustrating an exemplary implementation 400 of the data augmentation module 304 according to some embodiments described herein is shown. As Figure 4 As shown, the input to the data augmentation module 400 is the initial video images 312 captured by the camera 150 of the user seated area. Figure 4 An exemplary initial video image 420 of the user seated area is shown as an example of the initial video images 312, where the video image 420 captures a user seated in front of a background. In this particular example, the video image 420 is a high-definition image having a 16:9 aspect ratio. However, other examples of the initial video images 312 can have other image formats different from the example image 420.

[0051] In Figure 4In some embodiments, the bounding box placement submodule 402 is configured to place the bounding box with a predetermined size within the bounding box placement area of the initial video image 312. In some embodiments, the predetermined size of the bounding box used by the bounding box placement submodule 402 is determined based on a maximum bounding box that fits within the bounding box placement area of the initial video image 312. Because many CNN image recognition models require square input images, the CNN input images prepared for model training or image detection / recognition require square images to be cropped out of the initially non-square video images. Thus, in some embodiments, the maximum bounding box that fits within the bounding box placement area of the non-square initial video image 312 is also a square image, with a vertical dimension and a horizontal dimension that are each equal to the smaller dimension (in two-dimensional size) of the initial video image 312. For example, Figure 4 It is also shown that a maximum square bounding box 422 is randomly placed within the example video image 420. In this example, because the bounding box 422 has filled the entire vertical extent of the video image 420, the bounding box placement submodule 402 achieves a degree of randomness in placing the bounding box 422 only in the horizontal direction, which is shown to be offset from the center towards the right boundary of the video image 420. However, the randomness in placing the bounding box 422 proposed means that the bounding box 422 can be located anywhere within the video image 420 between the left and right boundaries of the video image 420.

[0052] The bounding box placement submodule 402 in the data augmentation module 400 is followed by a zoom-in submodule 404, which is configured to randomly zoom in on the initial video image 312 based on a random factor. For example, Figure 4 It is also shown that the effect of zooming in on the initial video image 420 by the random factor on both the video image 420 and the bounding box 422. As Figure 4 shown, the zoom-in operation increases the size of the initial video image 420 by the random factor to create a larger zoomed-in video image 424, while the size of the bounding box 422 within the zoomed-in video image 424 remains unchanged. However, it can be clearly observed from the zoomed-in video image 424 that the image portion within the bounding box 422 now corresponds to a smaller portion of the initial video image 420 than the portion placed within the same bounding box 422 within the initial video image 312 before the zoom-in effect was applied by the zoom-in submodule 404.

[0053] The zoom-in submodule 404 in the data augmentation module 400 is further followed by an image cropping submodule 406, which is configured to crop out the portion of the video image output by the zoom-in submodule 404 that is within the placed bounding box and thereby generate a cropped image. For example, Figure 4The effect of the image cropping submodule 406 is shown, which includes cropping out the portion of the zoomed-in video image 424 that is inside the bounding box 422 to obtain a cropped image 426. The image cropping submodule 406 in the data augmentation module 400 is further followed by an image flipping submodule 408, which is configured to horizontally flip the cropped image from the previous stage with a predetermined probability p (e.g., p = 50%) to generate a flipped version of the cropped image or the original cropped image. For example, Figure 4 The effect of horizontally flipping the cropped image 426 to generate a flipped image 428 is shown. Note that because the image flipping submodule 408 performs the image flipping operation based only on a predetermined probability p (p e (0, 1)), the output of the image flipping submodule 408 does not necessarily include the flipping effect, i.e., it can also output the cropped image 426 instead of the flipped image 428.

[0054] Finally, in the data augmentation module 400, the image flipping submodule 408 is followed by an image resizing submodule 410, which is configured to resize the output image of the image flipping submodule 408 to a predetermined size for the downstream CNN image recognition model. For example, if the downstream CNN model is an Inception-v3 model, the predetermined size in the image resizing submodule 410 can be 299 x 299. Finally, the data augmentation module 400 outputs a preprocessed training image 314 of the originally input video image 312 with the predetermined size (such as 299 x 299).

[0055] As mentioned above, each of the bounding box placement submodule 402 and the zooming submodule 404 within the disclosed data augmentation module 400 can be configured to have a degree of randomness. In addition, the image flipping submodule 408 performs the operation based on a predetermined probability p, and thus can have two different outcomes. Therefore, the randomness and the probability combined by the data augmentation module 400 ensure that the data augmentation module 400 will almost always generate different preprocessed training images 314 whenever the same input video image 312 passes through the data augmentation module 400. In some embodiments, for each of a set of originally video images 312 generated by the data collection module 302, the image 312 is passed through the data augmentation module 400 multiple times to generate multiple preprocessed training images 314 based on the same original image 312. In this way, a single original image 312 can be used to generate a set of different augmented training images 314. Note that because the number of originally / originally training images 312 collected by the data collection module 302 is typically limited due to time and resource constraints, the disclosed data augmentation module 400 allows a sufficiently large and diverse training dataset to be generated based on a smaller set of originally video images 312, thereby saving the time and resources required to collect the original training images 312.

[0056] Return to reference Figure 3 Note that in the model building system 300, the data augmentation module 304 is coupled to the data labeling module 306, which is configured to receive preprocessed training images 314 and subsequently label each training image in the preprocessed training images 314 as either a user out-of-place classification (e.g., using the label "A" or the binary number "0") or a user in-place classification (e.g., using the label "P" or the binary number "1"). Note that the data labeling module 306 generates a labeled training image dataset 316 as output.

[0057] Return to reference Figure 2 Note that while exemplary video images 202 and 208 represent more common and obvious cases of user displaced and in-place states, respectively, there are many different scenarios / variations of user in-place states that are not as obvious as in image 208 used for receiving user in-place classification. For example, the user in a given image may be off-center and / or only partially visible. Similarly, there are many different scenarios / variations of user displaced states that are not as obvious as in image 202 used for receiving user displaced classification. For example, a part of the user's body may be visible in a given image, but the user is clearly not in the seat shown in the image. In some embodiments, Figure 3 The data labeling module 306 is configured to label each preprocessed training image 314 based on the following rules: images showing fully visible seats (i.e., no part of the seat is blocked by the user's body) are labeled with the user off-seat classification; while images showing partially visible or invisible seats (i.e., any part of the seat is blocked by the user's body) are labeled with the user in-seat classification.

[0058] Figure 5 Exemplary processes for labeling time-series video images for user in-situ or user out-of-situ classifications according to some embodiments described herein are illustrated. Figure 5 As shown, a series of 15 video images 501 to 515 represent 15 consecutive video frames of a video clip showing the event of a user entering and taking a seat in the user seating area. Note that video images 501 to 503 are labeled "A," indicating user departure. This is because these three video images show fully visible seats, even though part of the user's body is visible in the images. Also note that images 504 to 515 are labeled "P," indicating user seating. This is because these twelve video images show partially visible / partially obstructed seats or completely obstructed seats.

[0059] In some embodiments, the above labeling process can be fully automated by constructing a machine learning based object recognition model trained to detect and identify fully visible / unobstructed seats in a user seating area. For example, the above proposed object recognition model can be individually trained based on training images of various configurations of fully visible / unobstructed seats under various user seating area conditions (e.g., under different backgrounds, lighting conditions, and possible seat angles relative to the monitor / camera). When the object recognition module is implemented as the data labeling module 306 in the model building system 300, the data labeling module 306 can then apply the trained object recognition model to each pre-processed training image 314 and automatically generate the binary decision and corresponding label / classification for the training image 314, i.e., (1) a fully visible seat identified decision and corresponding user out-of-seat label / classification; or (2) a fully visible seat not identified decision and corresponding user in-seat label / classification.

[0060] Referring back to Figure 3 , note that the data labeling module 306 in the model building system 300 is coupled to a transfer learning module 308 configured to receive the labeled training image dataset 316, train a CNN based user in-seat / out-of-seat classifier within the transfer learning module 308 based on the labeled training image dataset 316, and then generate the trained user in-seat / out-of-seat classifier 310 as an output. In some embodiments, the transfer learning module 308 employs a trained CNN based image recognition model to perform transfer learning for user in-seat / out-of-seat identification purposes. More specifically, the transfer learning module 308 can be configured to “freeze” all layers in the trained CNN based image recognition model except the last few layers. Note that the trained CNN based image recognition model can be a well-known or previously otherwise constructed image recognition model for other object recognition applications, such as for facial recognition. In some embodiments, the phrase “freeze all layers in the trained model except the last few layers” means freezing all layers from the first layer of the trained CNN based model to the bottleneck layer of the trained CNN based model except the last two layers: i.e., the fully connected layer and the Softmax layer after the bottleneck layer. Note that the phrase “freeze a given layer” in the trained CNN based image recognition model means fixing the previously constructed parameters in the given layer, including the weights and biases of the given layer in the disclosed transfer learning module 308. In the above example, performing transfer learning within the transfer learning module 308 using the trained CNN based image recognition model means fixing the previously constructed weights and biases in all network layers of the trained CNN based model up to and including the bottleneck layer, but without fixing the trainable parameters in the last fully connected layer and the Softmax layer of the trained CNN based model.

[0061] Figure 6 Exemplary implementations 600 based on a transfer learning module 308 using the Inception-v3 image recognition model are shown, according to some embodiments described herein. Figure 6 As shown, the transfer learning module 600 includes a feature extractor 602, which consists of the entire Inception-v3 image recognition model (or "Inception-v3 model" hereinafter), except for the final fully connected (FC) layer and Softmax layer of the initial Inception-v3 model. More specifically, the feature extractor 602 includes a CNN 604 of the Inception-v3 model, which includes all layers of the Inception-v3 model up to and before the bottleneck layer. Note that the CNN 604 consists of a stack of feature extraction layers, which includes a series of convolutional layers interleaved with a series of pooling layers. For transfer learning purposes, in the transfer learning module 600, the parameters of the weights and biases in all layers, such as CNN 604, are fixed to the trained values ​​of the Inception-v3 model. In the feature extractor 602, the bottleneck layer 606 of the initial Inception-v3 model follows the CNN 604 of the feature extractor 602. In order to perform transfer learning in the transfer learning module 600, the parameters of the weights and biases in the bottleneck layer 606 are also fixed to the trained values ​​of the Inception-v3 model.

[0062] The feature extractor 602 of the transfer learning module 600 receives a labeled training image dataset 316 (reference). Figure 3 The input image (the same items in the labeled training image dataset 316) is used as input, where the size of each input image in the dataset 316 has been resized to 299×299×3 (3 channels) according to the input size requirements of the Inception-v3 model. Using a feature extractor 602 with fixed parameters, the transfer learning module 600 extracts a set of feature maps 608 for each input training image 316 as the output of the bottleneck layer 606. In some implementations, the size of the feature map 608 for each input training image 316 is 8×8×2048.

[0063] Note that because the parameters in the feature extractor 602 are fixed, the feature extractor 602 will always generate the same set of feature maps 608 for the same input training image 316. In some embodiments, the transfer learning module 600 can use the same labeled training image dataset 316 multiple times, referred to as multiple “training epochs,” and during the training epochs, the same labeled training image dataset 316 is used to train the user in-seat / out-of-seat classifier. Because the outputted sets of feature maps 608 do not change across different training epochs based on the same training image dataset 316, the set of feature maps 608 for the labeled training image dataset 316 can be stored as two-dimensional (2D) images in memory or on a hard disk after the first training epoch. Thus, during the second training epoch and subsequent epochs, the set of feature maps 608 can simply be retrieved from memory or disk without having to be recomputed by passing the same training image dataset 316 through the feature extractor 602, thereby saving training time and computational resources during transfer learning.

[0064] Note that the transfer learning module 600 does not include the original last FC layer and the Softmax layer of the original Inception-v3 model. Instead, the transfer learning module 600 uses a custom FC layer 610 and a custom Softmax layer 612 with a set of variables as the decision layer 616 of the disclosed user in-seat / out-of-seat classifier 618. As Figure 6 shown, the user in-seat / out-of-seat classifier 618 includes the feature extractor 602 and the decision layer 616.

[0065] As Figure 6 shown, the custom FC layer 610 receives the set of feature maps 608 of the processed training image dataset 316 as input, which is used to train the set of variables within the custom FC layer 610. In some embodiments, the custom FC layer 610 can include 512 nodes, and each of the 512 nodes is associated with a weight variable and a bias variable. As described above, training the custom FC layer 610 can include performing multiple training epochs (e.g., epochs = 20) using the same set of feature maps 608 extracted from the same training image dataset 316. After performing the multiple training epochs, the custom FC layer 610 will include a set of trained parameters. For the above example of 512 nodes in the FC layer 610, the set of trained parameters will include 512 trained weight values and 512 trained bias values.

[0066] Finally, in the transfer learning module 600, the output from the custom FC layer 610 is coupled to a Softmax layer 612 comprising a binary Softmax function configured to generate a user-in-seat (e.g., having a binary value of 1) or user-out-of-seat (e.g., having a binary value of 0) classification / decision for each input training image 316. Note that after the custom FC layer 610 is sufficiently trained, e.g., after a number of training processes based on the training image dataset 316, the training variables within the custom FC layer 610 can be fixed, and the trained user-in / out-of-seat classifier 618 becomes the trained user-in / out-of-seat classifier 310, which is the output of the user-in / out-of-seat recognition model building system 300.

[0067] Note that although the transfer learning module 308 / 600 has been described based on using the Inception-v3 model as the trained feature extractor, the disclosed transfer learning module within the disclosed user-in / out-of-seat recognition model building system 300 is not meant to be limited to using a particular trained deep learning framework. In general, the trained feature extractor within the disclosed transfer learning module 308 / 600 can be implemented with other known image recognition models, including but not limited to GGNet, ResNet, DenseNet, AlexNet, Dual Path Network, MobileNet, and Inception v1 and Inception v2.

[0068] Figure 7 A flow diagram is shown in accordance with some embodiments described herein, showing an exemplary process 700 for building a CNN-based user-in / out-of-seat classifier based on captured video images of a user seating area. In one or more embodiments, one or more of the steps in Figure 7 may be omitted, repeated, and / or performed in a different order. Accordingly, Figure 7 The particular arrangement of steps shown should not be understood as limiting the scope of the present technology.

[0069] The process 700 can begin with collecting different sets of original video images of a user seating area in various clinically relevant situations (step 702). For example, these various clinically relevant situations can include various backgrounds of the user seating area; different lighting conditions of the user seating area; various screen angles and heights of the monitor, different distances between the user and the monitor; the state of the UID; whether the user wears one or more of the following: a QR tag, a face mask, a 3D headset, and a hospital gown; different colors of the hospital gown; various headrest angles of the seat; various types of head movements of the user, and others.

[0070] Next, for each of the different sets of raw video images of the user seating area, the process 700 generates different sets of augmented / preprocessed training images for that raw video image (step 704). For example, the different sets of augmented / preprocessed training images for that raw video image can include one or more of the following augmented images: (1) one or more cropped images obtained from the raw video image by first randomly placing a largest square bounding box within the raw image, and then cropping the raw image using the bounding box; (2) one or more cropped and scaled images obtained from the raw video image by first randomly placing a largest square bounding box within the raw image, then randomly scaling the raw image based on a random factor, and finally cropping the scaled image using the bounding box; and (3) one or more cropped, scaled, and possibly flipped images obtained from the raw video image by first randomly placing a largest square bounding box within the raw video image, then randomly scaling the raw image based on a random factor, next cropping the scaled image using the bounding box, and finally flipping the cropped image based on a predetermined probability level. In some embodiments, each of the different sets of augmented / preprocessed training images for a given raw video image is resized to a predetermined size required by an input size constraint of the downstream CNN classifier to be trained.

[0071] Next, for each of the different sets of augmented / preprocessed training images generated from the sets of raw video images, the process 700 labels each preprocessed training image with either a user off-seat classification or a user on-seat classification (step 706). In some embodiments, the process 700 labels a preprocessed training image with a user off-seat classification if the preprocessed image shows a seat that is completely visible, and labels the preprocessed training image with a user on-seat classification if the preprocessed image shows a seat that is partially visible or not visible. After labeling the sets of preprocessed training images, the process 700 generates a labeled training image dataset for the downstream user on-seat / off-seat classifier.

[0072] Subsequently, process 700 trains a CNN-based user positioning / disposition classifier using a labeled training image dataset via transfer learning. Specifically, in process 700, multiple sets of feature maps are first extracted from the labeled training image dataset using a trained CNN image recognition model (also referred to as a “feature extractor”) with fixed network parameters (step 708). In some embodiments, the trained CNN image recognition model is an Inception-v3 image recognition model that goes up to the bottleneck layer but not to the initial final FC layer and Softmax layer of the Inception-v3 model. Next, in process 700, a custom FC layer containing a set of variables is trained using the extracted multiple sets of feature maps (step 710). In some embodiments, in process 700, multiple training processes are performed using the same extracted multiple sets of feature maps from the same labeled training image dataset to train the custom FC layer. Finally, in process 700, the fixed-parameter CNN feature extractor, the custom FC layer with that set of training variables, and the Softmax binary classifier are combined to obtain the disclosed user positioning / disposition image classifier (step 712).

[0073] Figure 8 An exemplary surgical workflow 800 according to some embodiments described herein is shown, which uses the disclosed (trained) user placement / displacement classifier to track users at the surgeon's console during surgical procedures. Figure 8 As shown, the surgical workflow 800 is represented by a timeline, where time 802 is on the horizontal axis, and the classifier decision 804 of the trained user positioning / disposition classifier (hereinafter referred to as the "user positioning / disposition classifier") is on the vertical axis. More specifically, the user positioning / disposition classifier, such as the trained user positioning / disposition classifier 310 generated by the user positioning / disposition recognition model building system 300, is used to receive and process real-time video images of the user seating area of ​​the surgeon's console. For each frame of the real-time video image, the user positioning / disposition classifier generates a binary decision 804 of either user positioning (i.e., a value of 1 on the vertical axis) or user disposition (i.e., a value of 0 on the vertical axis).

[0074] Note that the surgical workflow 800 is integrated with a user transition detection mechanism that is designed to detect an event when a new user enters the user seating area after a previous user has left the user seating area. As will be discussed in greater detail below, the disclosed user transition detection mechanism can also interact with the user who has just entered the user seating area to determine whether the user is the same user as the previous user. If the user confirms that he / she is not the same user as the previous user, the disclosed user transition detection mechanism can trigger a recalibration or reconfiguration of the surgeon console settings for the new user, such as user implementation tracking settings, user UID control settings, and user seating settings (e.g., armrest settings). The surgical workflow 800 will now be described in greater detail in conjunction with the user transition detection mechanism.

[0075] Note that the surgical workflow 800 begins with a first time period from to to ti during which the user on / off classifier outputs a user on decision (1) indicating that the user is on seat at the surgeon console. Next, at or immediately after time ti, a first decision change occurs in which the user on / off classifier begins outputting a user off decision (0) indicating that the user has left the surgeon console at or around time ti. From time ti onward, the user on / off classifier continues to output the user off decision (0) until time t2 at which a second decision change occurs in which the user on / off classifier begins outputting the user on decision (1). As noted above, the transition from the user off decision to the user on decision can indicate that the user has returned to the surgeon console. In some embodiments, the user transition detection mechanism detects such a transition in the surgical workflow 800 and subsequently generates a dialog window / frame on the monitor of the surgeon console to prompt the user to confirm whether she / he is the same user or a new user. In some embodiments, to avoid false positives, the disclosed user transition detection mechanism only generates the dialog window / frame if there are a minimum number (e.g., 10) of consecutive user on decisions (1) generated after time t2. In the surgical workflow 800, a dialog window / frame 806 including the question "Are you the same surgeon?" is generated by the user transition detection mechanism at some time after time t2.

[0076] Beginning at time t2 in the surgical workflow 800, the user in / out classifier outputs a user in determination (1) only for a brief time interval between t2 and t3. This brief time interval is followed by a third determination change at time t3, after which the user in / out classifier begins outputting a user out determination (0). Note that this brief time interval (t2, t3) can be the result of the user entering the user seating area but only staying for a very brief moment and then leaving the user seating area. Thus, the user can not have responded to the conversation window / frame 806 during the brief time interval (t2, t3).

[0077] Beginning at time t3, the user in / out classifier continues to output a user out determination (0) until time t4, at which a fourth determination change occurs and the user in / out classifier begins outputting a user in determination (1) again. However, as shown in the surgical workflow 800, this new user in determination period is very short and ends almost immediately after beginning at time t4 at time t5. In this scenario, the disclosed user transition detection mechanism does not generate a new conversation window / frame because the previously generated conversation window / frame 806 has not been responded to and thus is still valid. Note that even if the conversation window / frame 806 did not appear, the user transition detection mechanism can not generate a conversation window / frame because the number of consecutive user in determinations (1) encompassed by the time interval (t4, t5) can be below a minimum number (e.g., 10) required to avoid false positives.

[0078] Beginning at time t5, the user in / out classifier continues to output a user out determination (0) until time t6, at which a fifth determination change occurs and the user in / out classifier begins outputting a user in determination (1) again. Beginning at time t6, the user in / out classifier continues to output a user in determination (1) for an extended period of time, indicating that the user continues to be in at the surgeon console. In addition, the returning user can now answer “yes” in response to the conversation window / frame 806, confirming that he / she is the same user as the previous user who was at the user seating area before time t1. Note that the conversation window / frame 806 generated around time t2 remains valid at time t6 because it has not been responded to, such that the disclosed user transition detection mechanism does not need to generate another conversation window / frame for the returning user around time t6.

[0079] Figure 9 A flow diagram is shown illustrating an exemplary process 900 for tracking a user at a surgeon console during a surgical procedure using a trained user in / out classifier, in accordance with some embodiments described herein. In one or more embodiments, one or more of the steps in Figure 9 may be omitted, repeated, and / or performed in a different order. Accordingly,Figure 9 The particular arrangement of steps shown should not be construed as limiting the scope of the present technology.

[0080] The process 900 can begin with receiving a first series of video images of a user seating area of a surgeon console (step 902). In the process 900, the first series of video images is then processed to detect an event of a first user in the user seating area leaving the user seating area (step 904). To detect the event of the first user leaving the user seating area, in the process 900, a series of user-in / out determinations corresponding to the first series of video images can be generated using the disclosed user-in / out image classifier, e.g., the trained user-in / out classifier 310, and a transition of the determination output from a user-in determination (1) to a user-out determination (0) is detected. In the process 900, a second series of video images of the user seating area is then received (step 906). In the process 900, the second series of video images is then processed to detect an event of a second user in the user seating area entering the user seating area (step 908). To detect the event of the second user entering the user seating area, in the process 900, a series of user-in / out determinations corresponding to the second series of video images can be generated using the disclosed user-in / out image classifier, and a transition of the determination output from a user-out determination (0) to a user-in determination (1) is detected.

[0081] When the second user entering the user seating area is detected, in the process 900, a dialog window is then generated on a monitor of the surgeon console asking the second user to confirm whether he / she is the same user as the first user (step 910). In some embodiments, in the process 900, the dialog window is generated only after a minimum number of consecutive user-in determinations (1) have been generated by the user-in / out classifier following a user-out determination (0), so as to avoid false positives.

[0082] In the process 900, a response from the second user is then received through the dialog window (step 912), and based on the response, it is determined whether the second user is the same user as the first user (step 914). If yes, it is determined in the process 900 that the surgeon console settings do not need to be recalibrated or reconfigured, and the process 900 terminates. However, if the second user is not the same user as the first user, i.e., the second user is a new user, it is determined in the process 900 that the surgeon console settings need to be recalibrated or reconfigured, and a surgeon console settings recalibration procedure is then initiated (step 916). For example, in the process 900, another dialog window can simply be generated prompting the user to input his / her identity, which causes the robotic surgical system 100 to load new surgeon console settings, e.g., user eye gaze tracking settings, user UID control settings, and user seating settings for the second user.

[0083] Figure 10 A computer system that can be used to implement some embodiments of the subject technology is conceptually illustrated. Computer system 1000 can be a client, a server, a computer, a smartphone, a PDA, a laptop, or a tablet computer with one or more processors embedded therein or coupled thereto, or any other type of computing device. Such a computer system includes various types of computer readable media and interfaces for various other types of computer readable media. Computer system 1000 includes a bus 1002, a processing unit 1012, a system memory 1004, a read-only memory (ROM) 1010, a permanent storage device 1008, an input device interface 1014, an output device interface 1006, and a network interface 1016. In some embodiments, computer system 1000 is part of a robotic surgical system.

[0084] Bus 1002 collectively represents all system and peripheral buses connected to the computer system 1000. For instance, bus 1002 connects the processing unit 1012 with ROM 1010, system memory 1004, and permanent storage device 1008.

[0085] From the various memory units, processing unit 1012 retrieves instructions to be executed and data to be processed, in order to perform the various processes described in this patent disclosure, including processes for constructing a CNN-based user in- / out-of-position classifier and for using the in- / out-of-position classifier in conjunction with a surgical robot to control a surgical robot. Figures 3 to 9 The disclosed user in- / out-of-position classifier described is tracking a user at a surgeon’s console during a surgical procedure time using the above-described techniques. Processing unit 1012 can include any type of processor, including but not limited to microprocessors, graphics processing units (GPUs), tensor processing units (TPUs), intelligent processor units (IPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). In different implementations, processing unit 1012 can be a single processor or a multi-core processor.

[0086] ROM 1010 stores static data and instructions that are needed by processing unit 1012 and other modules of the computer system. Permanent storage device 1008, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when computer system 1000 is off. Some implementations of the subject disclosure use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as permanent storage device 1008.

[0087] Other implementations use removable storage devices (for example, floppy disks, flash drive) and their corresponding disk drives, as permanent storage devices 1008. Like permanent storage device 1008, system memory 1004 is a read-and-write memory device. However, unlike storage device 1008, system memory 1004 is a volatile read-and-write memory, such as a random access memory. System memory 1004 stores some of the instructions and data that the processor needs at runtime. In some implementations, the various processes described in this patent disclosure, including the process for using the CNN-based user in-position / out-of-position classifier described in connection with Figures 3 to 9 The disclosed user in-position / out-of-position classifier described is constructed during a surgical procedure time and the above-described techniques for tracking a user at a surgeon’s console, based on a CNN, are stored in system memory 1004, permanent storage device 1008, and / or ROM 1010. Processing unit 1012 retrieves from these various memory units the instructions to be executed and the data to be processed in order to execute the processes of some implementations.

[0088] Bus 1002 also connects to input and output device interfaces 1014 and 1006. Input device interface 1014 enables the user to communicate information and select commands to the computer system. Input devices used with input device interface 1014 can include, for example, alphanumeric keyboards and pointing devices (also referred to as “cursor control devices”). Output device interface 1006 enables, for example, the display of images generated by the computer system 1000. Output devices used with output device interface 1006 can include, for example, printers and display devices, for example, cathode ray tube (CRT) or liquid crystal display (LCD) screens. Some implementations include devices, for example, touchscreens, that function as both input and output devices.

[0089] Finally, as shown in Figure 10 Bus 1002 also couples computer system 1000 to a network (not shown) through network interface 1016. In this manner, the computer can be a part of a network of computers such as a local area network (“LAN”), a wide area network (“WAN”), an intranet, or a network of networks, for example, the Internet. Any or all components of computer system 1000 can be used in conjunction with the subject disclosure.

[0090] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0091] The hardware used to implement the various illustrative logics, logical blocks, modules, and circuits described in connection with the aspects disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field

[0092] In one or more exemplary aspects, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or code on a non-transitory computer-readable storage medium or a non-transitory processor-readable storage medium. The steps of a method or algorithm described herein can be embodied in a processor-executable instruction that can reside on a non-transitory computer- or processor-readable storage medium. Non-transitory computer- or processor-readable storage media can be any storage media that can be accessed by a computer or processor. By way of example but not limitation, such non-transitory computer- or processor-readable storage media can include RAM, ROM, EEPROM, FLASH memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes compact discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy disks and blu-ray discs where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of non-transitory computer- and processor-readable media. Additionally, the operations of a method or algorithm can reside in one or any combination of the above memory hardware, as one or any combination of the code and / or instructions that can reside on a non-transitory processor-readable storage medium and / or computer-readable storage medium, thereby making a computer program product.

[0093] Although the patent document contains many details, these should not be understood as limiting the scope of any disclosed technologies or claimable content to the specific embodiments described, but rather as a description of features that can be particular to the specific implementation of a particular technology. Certain features described in the context of separate embodiments in this patent document can also be implemented in a combined form in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any subcombination in multiple embodiments. Also, while features can be described above as acting in certain combinations and even initially so claimed by the claims, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can then be directed to a subcombination or variation of a subcombination.

[0094] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order, nor that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0095] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A computer-implemented method for detecting user in / out of seat at a surgeon console in a robotic surgical system, the method comprising: receiving a set of raw video images capturing a user seating area of the surgeon console, wherein the user seating area includes a seat object; processing the set of raw video images to generate a set of training images; training a convolutional neural network (CNN) model using the set of training images, wherein the trained CNN model includes a trained CNN classifier configured to output a binary classification for each input image, and wherein the binary classification is a user in seat classification or a user out of seat classification; receiving a set of real-time video images capturing the user seating area during a surgical procedure time; and applying the trained CNN classifier to the set of real-time video images to automatically classify each video image in the set of real-time video images as the user in seat classification or the user out of seat classification, wherein a video image in the set of real-time video images receives the user out of seat classification when the seat object in the video image is fully visible.

2. The computer-implemented method of claim 1, wherein, processing the set of raw video images to generate the set of training images includes, for each raw video image in the set of raw video images: generating a plurality of augmented images of the raw video image using a set of image augmentation techniques; and including one or more augmented images in the plurality of augmented images in the set of training images.

3. The computer-implemented method of claim 2, wherein, generating the plurality of augmented images of the raw video image using the set of image augmentation techniques includes: placing a bounding box of a first predetermined size within a border of the raw video image; and cropping the raw video image using the bounding box to generate a first augmented image of the raw video image having the first predetermined size.

4. The computer-implemented method of claim 3, wherein, the method further comprising: augmenting the first augmented image using a random zoom factor to generate a second augmented image of the raw video image having the first predetermined size.

5. The computer-implemented method of claim 4, wherein, the method further comprising: horizontally flipping the second augmented image to generate a third augmented image of the raw video image having the first predetermined size.

6. The computer-implemented method of claim 3, wherein, the first predetermined size includes a vertical dimension and a horizontal dimension each equal to a smaller dimension of two dimensions of the raw video image.

7. The computer-implemented method of claim 1, wherein, after generating the set of training images, the method further comprising automatically annotating each training image in the set of training images with a user in seat label indicating a user in seat in the training image or a user out of seat label indicating any user out of seat in the training image.

8. The computer-implemented method of claim 7, wherein, the automatically annotating the training images in the set of training images includes: determining whether the seat object is partially obstructed or fully obstructed by another object in the training image; and if yes, labeling the training image with the user in seat label; otherwise, labeling the training image with the user out of seat label.

9. The computer-implemented method of claim 1, wherein, The CNN model includes a stack of feature extraction layers including a series of pooling layers and a series of convolutional layers having a first set of parameters, the stack of feature extraction layers followed by a fully connected layer having a second set of parameters, the fully connected layer further followed by a final output layer, and wherein training the CNN model using the set of training images includes: for each training image in the set of training images, extracting a set of two-dimensional (2D) feature maps for the training image using the stack of feature extraction layers while fixing the first set of parameters; and training the second set of parameters of the fully connected layer based at least on the sets of extracted 2D feature maps extracted from the set of training images to obtain a trained fully connected layer; and fixing both the first set of parameters and the trained second set of parameters in the CNN model to obtain the trained CNN classifier.

10. The computer-implemented method of claim 9, wherein, The stack of feature extraction layers includes a portion of an Inception-v3 image recognition model up to a bottleneck layer, and wherein the Inception-v3 image recognition model was previously trained to detect and recognize objects that are not related to detecting user in / out of seat at a surgeon’s console.

11. A device for detecting user in / out of seat at a surgeon’s console in a robotic surgery system, the device comprising: one or more processors; memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the device to: receive a set of raw video images capturing a user seating area of the surgeon’s console, wherein the user seating area includes a seat object; process the set of raw video images to generate a set of training images; train a convolutional neural network (CNN) model using the set of training images, wherein the trained CNN model includes a trained CNN classifier configured to output a binary classification for each input image, and wherein the binary classification is either a user in seat classification or a user out of seat classification; receive a set of real-time video images of the user seating area captured during a surgical procedure time; and apply the trained CNN classifier to the set of real-time video images to automatically classify each video image in the set of real-time video images as either the user in seat classification or the user out of seat classification, wherein a video image in the set of real-time video images receives the user out of seat classification when the seat object in the video image is fully visible.

12. The apparatus of claim 11, wherein, The memory further stores instructions that, when executed by the one or more processors, cause the device to generate the set of training images by: for each raw video image in the set of raw video images: generate a plurality of augmented images of the raw video image using a set of image augmentation techniques; and include one or more augmented images in the plurality of augmented images in the set of training images. ​ 13. The apparatus of claim 12, wherein, The memory further stores instructions that, when executed by the one or more processors, cause the device to generate the plurality of augmented images of the original video image by: placing a bounding box of a first predetermined size within a frame of the original video image; and cropping the original video image using the bounding box to generate a first augmented image of the original video image having the first predetermined size.

14. The apparatus of claim 11, wherein, The memory further stores instructions that, when executed by the one or more processors, cause the device to automatically annotate each training image in the set of training images with a user-in-seat label indicating a user in seat in the training image or a user-out-of-seat label indicating any user out of seat in the training image.

15. The apparatus of claim 14, wherein, The memory further stores instructions that, when executed by the one or more processors, cause the device to automatically annotate the training image by: determining whether the seat object is partially obstructed or completely obstructed by another object in the training image; and if yes, labeling the training image with the user-in-seat label; otherwise, labeling the training image with the user-out-of-seat label.

16. The apparatus of claim 11, wherein, The CNN model includes a stack of feature extraction layers including a series of pooling layers and a series of convolutional layers having a first set of parameters, the stack of feature extraction layers followed by a fully connected layer having a second set of parameters, the fully connected layer further followed by a final output layer, and wherein training the CNN model using the set of training images includes: for each training image in the set of training images, extracting a set of two-dimensional (2D) feature maps of the training image using the stack of feature extraction layers while fixing the first set of parameters; and training the second set of parameters of the fully connected layer based at least on the sets of extracted 2D feature maps extracted from the set of training images to obtain a trained fully connected layer; and fixing both the first set of parameters and the trained second set of parameters in the CNN model to obtain the trained CNN classifier.

17. The apparatus of claim 16, wherein, The stack of feature extraction layers includes a portion of an Inception-v3 image recognition model from an input layer up to a bottleneck layer, and wherein the Inception-v3 image recognition model is previously trained to detect and recognize objects unrelated to detecting user-in-seat / out-of-seat at a surgeon console.

18. A robotic surgical system comprising: a surgeon console including a user seating area; and a computer at the surgeon console configured to detect user-in-seat / out-of-seat at the surgeon console by: receiving a set of original video images capturing the user seating area of the surgeon console, wherein the user seating area includes a seat object; processing the set of original video images to generate a set of training images; training a convolutional neural network, CNN, model using the set of training images, wherein the trained CNN model comprises a trained CNN classifier configured to output a binary classification for each input image, and wherein the binary classification is either a user-in-seat classification or a user-out-of-seat classification; receiving a set of real-time video images of the user seating area captured during a surgical procedure time; and applying the trained CNN classifier to the set of real-time video images to automatically classify each video image in the set of real-time video images as either the user-in-seat classification or the user-out-of-seat classification, wherein a video image in the set of real-time video images receives the user-out-of-seat classification when the seating object in the video image is completely visible.

19. The robotic surgical system of claim 18, wherein, the CNN model comprises a stack of feature extraction layers comprising a series of pooling layers and a series of convolutional layers having a first set of parameters, the stack of feature extraction layers being followed by a fully connected layer having a second set of parameters, the fully connected layer being further followed by a final output layer, and wherein training the CNN model using the set of training images comprises: for each training image in the set of training images, extracting a set of two-dimensional, 2D, feature maps of the training image using the stack of feature extraction layers while fixing the first set of parameters; and training the second set of parameters of the fully connected layer based at least on the plurality of sets of extracted 2D feature maps extracted from the set of training images to obtain a trained fully connected layer; and fixing both the first set of parameters and the trained second set of parameters in the CNN model to obtain the trained CNN classifier.

Images